Lines Matching refs:precision
1318 * This algorithm depends on the precision in white-scale and that is
1394 * need up to 10 decimal digits to preserve the original precision and avoid
1398 * The approach used here is to preserve the maximum precision within the
2749 * precision.
2753 double fp, unsigned int precision)
2760 if (precision < 1)
2761 precision = DBL_DIG;
2763 /* Enforce the limit of the implementation precision too. */
2764 if (precision > DBL_DIG+1)
2765 precision = DBL_DIG+1;
2768 if (size >= precision+5) /* See the requirements below. */
2857 if (cdigits+czero-clead+1 < (int)precision)
2962 while (cdigits+czero-clead < (int)precision && fp > DBL_MIN);
2964 /* The total output count (max) is now 4+precision */
2987 * 5+precision - see check at the start.
2994 * digit output here so far is at most 1+precision - no
3000 *ascii++ = 69, --size; /* 'E': PLUS 1 TOTAL 2+precision */
3011 *ascii++ = 45, --size; /* '-': PLUS 1 TOTAL 3+precision */
3392 * fixed point arithmetic. This code has sufficient precision for either 8-bit
3396 * precision floating point arithmetic would work fine.
3500 * 16-bit precision in the interpolation:
3534 * 8 bits. Do this with maximum precision.
3542 * precision and that 'lg2' is current scaled by an extra 12 bits, so adjust
4081 /* Notice that the '16 from 1' table should be full precision, however
