Lines Matching full:precision
26 Decimal floating point has finite precision with arbitrarily large bounds.
283 longer than precision). The result is [0,qNaN].
317 on creation and either the precision exceeds the capability of the
335 operation (or sequence of operations) caused a loss of precision.
363 current precision, with the sign of the intermediate result. For
519 # uses a precision 2 greater than normal
520 return +s # Convert result to normal precision
1154 Rounds the number (if more than precision digits)
1421 """Return (self // other, self % other), to context.prec precision.
1690 # maximum length of payload is precision if clamp=0,
1691 # precision-1 if clamp=1.
1699 """Round if it is necessary to keep self within prec precision.
1868 """Round self to the nearest integer, or to a given precision.
2056 'insufficient precision: pow() 3rd '
2058 'precision digits')
2093 digits of precision. Return None if self**other is not
2114 # representable (at *any* precision), xc must be the nth power of a
2200 # exactly with p digits of precision.
2338 where p is the context precision.
2345 modulo with unbounded precision, but is computed more
2461 # try for an exact result with precision +1
2479 # compute correctly rounded result: start with precision +3,
2480 # then increase precision until result is unambiguously roundable
2776 # the desired precision and express self in the form c*100**e
2781 # precision p is n*10**e where n = round_half_even(sqrt(c)),
2787 # place (precision p+1 instead of precision p), rounding down.
2794 # use an extra digit of precision
3106 # indistinguishable from 1 at the given precision, while for
3127 # compute correctly rounded result: increase precision by
3262 # correctly rounded result: repeatedly increase precision by 3
3343 # correctly rounded result: repeatedly increase precision
3620 # if precision == 1 then we don't raise Clamped for a
3805 # special values don't care about the type or precision
3823 precision = spec['precision']
3824 if precision is not None:
3826 self = self._round(precision+1, rounding)
3828 self = self._rescale(-precision, rounding)
3829 elif spec['type'] in 'gG' and len(self._int) > precision:
3830 self = self._round(precision, rounding)
3839 if not self and precision is not None:
3840 dotplace = 1 - precision
3910 prec - precision (for use in rounding, division, square roots..)
5184 in 'precision' digits.
5193 - modulo must be nonzero and have at most 'precision' digits
5197 unbounded precision, but is computed more efficiently. It is
5268 quantize operation would be greater than precision then an Invalid
5329 to precision digits if necessary. The sign of the result, if
5499 """Square root of a non-negative number to context precision.
5594 left-hand-operand, 1E+0 as the right-hand-operand, and the precision
5595 of the operand as the precision setting; Inexact and Rounded flags
5624 left-hand-operand, 1E+0 as the right-hand-operand, and the precision
5625 of the operand as the precision setting, except that no flags will
5691 # Let exp = min(tmp.exp - 1, tmp.adjusted() - precision - 1).
5789 # approximation to z*M. To avoid loss of precision, the y below
5816 # increase precision by 2; compensate for this by dividing
5850 # Increase precision by 2. The precision increase is compensated
5965 precision.
5973 digits of precision, and with an error in d of at most 1. This is
5977 # we'll call iexp with M = 10**(p+2), giving p+3 digits of precision
5980 # compute log(10) with extra precision = adjusted exponent of c*10**e
5993 # reduce remainder back to original precision
6007 of precision, and with an error in c of at most 1. (This is
6185 # [[fill]align][sign][#][0][minimumwidth][,][.precision][type]
6197 (?:\.(?P<precision>0|(?!0)\d+))?
6227 precision: nonnegative integer giving precision, or None
6260 # minimumwidth defaults to 0; precision remains None if not given
6262 if format_dict['precision'] is not None:
6263 format_dict['precision'] = int(format_dict['precision'])
6265 # if format type is 'g' or 'G' then a precision of 0 makes little
6267 if format_dict['precision'] == 0:
6269 format_dict['precision'] = 1
