Lines Matching refs:precision
30 Decimal floating point has finite precision with arbitrarily large bounds.
256 longer than precision). The result is [0,qNaN].
290 on creation and either the precision exceeds the capability of the
308 operation (or sequence of operations) caused a loss of precision.
336 current precision, with the sign of the intermediate result. For
474 # uses a precision 2 greater than normal
475 return +s # Convert result to normal precision
1086 Rounds the number (if more then precision digits)
1353 """Return (self // other, self % other), to context.prec precision.
1632 # maximum length of payload is precision if _clamp=0,
1633 # precision-1 if _clamp=1.
1641 """Round if it is necessary to keep self within prec precision.
1905 'insufficient precision: pow() 3rd '
1907 'precision digits')
1942 digits of precision. Return None if self**other is not
1963 # representable (at *any* precision), xc must be the nth power of a
2049 # exactly with p digits of precision.
2187 where p is the context precision.
2194 modulo with unbounded precision, but is computed more
2310 # try for an exact result with precision +1
2328 # compute correctly rounded result: start with precision +3,
2329 # then increase precision until result is unambiguously roundable
2635 # the desired precision and express self in the form c*100**e
2640 # precision p is n*10**e where n = round_half_even(sqrt(c)),
2646 # place (precision p+1 instead of precision p), rounding down.
2653 # use an extra digit of precision
2965 # indistinguishable from 1 at the given precision, while for
2986 # compute correctly rounded result: increase precision by
3121 # correctly rounded result: repeatedly increase precision by 3
3202 # correctly rounded result: repeatedly increase precision
3479 # if precision == 1 then we don't raise Clamped for a
3664 # special values don't care about the type or precision
3680 precision = spec['precision']
3681 if precision is not None:
3683 self = self._round(precision+1, rounding)
3685 self = self._rescale(-precision, rounding)
3686 elif spec['type'] in 'gG' and len(self._int) > precision:
3687 self = self._round(precision, rounding)
3696 if not self and precision is not None:
3697 dotplace = 1 - precision
3767 prec - precision (for use in rounding, division, square roots..)
4971 in 'precision' digits.
4980 - modulo must be nonzero and have at most 'precision' digits
4984 unbounded precision, but is computed more efficiently. It is
5055 quantize operation would be greater than precision then an Invalid
5116 to precision digits if necessary. The sign of the result, if
5286 """Square root of a non-negative number to context precision.
5361 left-hand-operand, 1E+0 as the right-hand-operand, and the precision
5362 of the operand as the precision setting; Inexact and Rounded flags
5391 left-hand-operand, 1E+0 as the right-hand-operand, and the precision
5392 of the operand as the precision setting, except that no flags will
5458 # Let exp = min(tmp.exp - 1, tmp.adjusted() - precision - 1).
5572 # approximation to z*M. To avoid loss of precision, the y below
5599 # increase precision by 2; compensate for this by dividing
5633 # Increase precision by 2. The precision increase is compensated
5748 precision.
5756 digits of precision, and with an error in d of at most 1. This is
5760 # we'll call iexp with M = 10**(p+2), giving p+3 digits of precision
5763 # compute log(10) with extra precision = adjusted exponent of c*10**e
5776 # reduce remainder back to original precision
5790 of precision, and with an error in c of at most 1. (This is
5930 # [[fill]align][sign][0][minimumwidth][,][.precision][type]
5941 (?:\.(?P<precision>0|(?!0)\d+))?
5971 precision: nonnegative integer giving precision
6005 # minimumwidth defaults to 0; precision remains None if not given
6007 if format_dict['precision'] is not None:
6008 format_dict['precision'] = int(format_dict['precision'])
6010 # if format type is 'g' or 'G' then a precision of 0 makes little
6012 if format_dict['precision'] == 0:
6014 format_dict['precision'] = 1
