Lines Matching full:precision
42 // zero value has precision 0
92 // TODO(gri) test how precision is set for zero value results
397 // (rounding by SetPrec after SetInt64 using default precision
399 // precision)
678 // check precision
721 // check precision when set automatically
1265 // chose base such that we cross the mantissa precision limit
1298 // chose base such that we cross the mantissa precision limit
1481 preci := 200 // precision of integer part
1482 precf := 20 // precision of fractional part
1537 const dprec = 3 // max. precision variation
1549 // vary operand precision (only ok as long as a, b can be represented correctly)
