| /external/python/cpython2/Lib/test/decimaltestdata/ |
| remainder.decTest | 23 precision: 9
88 precision: 6
99 precision: 9
118 precision: 9
315 precision: 6
317 precision: 3
322 precision: 5
342 precision: 9
465 precision: 9
479 precision: 1
[all...] |
| reduce.decTest | 25 precision: 9
152 precision: 3
157 precision: 3
187 precision: 9
214 precision: 40
230 precision: 5
|
| /external/python/cpython3/Lib/test/decimaltestdata/ |
| remainder.decTest | 23 precision: 9
88 precision: 6
99 precision: 9
118 precision: 9
315 precision: 6
317 precision: 3
322 precision: 5
342 precision: 9
465 precision: 9
479 precision: 1
[all...] |
| reduce.decTest | 25 precision: 9
152 precision: 3
157 precision: 3
187 precision: 9
214 precision: 40
230 precision: 5
|
| /prebuilts/gdb/darwin-x86/lib/python2.7/test/decimaltestdata/ |
| remainder.decTest | 23 precision: 9
88 precision: 6
99 precision: 9
118 precision: 9
315 precision: 6
317 precision: 3
322 precision: 5
342 precision: 9
465 precision: 9
479 precision: 1
[all...] |
| /prebuilts/gdb/linux-x86/lib/python2.7/test/decimaltestdata/ |
| remainder.decTest | 23 precision: 9
88 precision: 6
99 precision: 9
118 precision: 9
315 precision: 6
317 precision: 3
322 precision: 5
342 precision: 9
465 precision: 9
479 precision: 1
[all...] |
| /prebuilts/python/darwin-x86/2.7.5/lib/python2.7/test/decimaltestdata/ |
| remainder.decTest | 23 precision: 9
88 precision: 6
99 precision: 9
118 precision: 9
315 precision: 6
317 precision: 3
322 precision: 5
342 precision: 9
465 precision: 9
479 precision: 1
[all...] |
| /prebuilts/python/linux-x86/2.7.5/lib/python2.7/test/decimaltestdata/ |
| remainder.decTest | 23 precision: 9
88 precision: 6
99 precision: 9
118 precision: 9
315 precision: 6
317 precision: 3
322 precision: 5
342 precision: 9
465 precision: 9
479 precision: 1
[all...] |
| /external/eigen/blas/testing/ |
| dblat1.f | 19 *> Test program for the DOUBLE PRECISION Level 1 BLAS.
54 DOUBLE PRECISION SFAC
131 DOUBLE PRECISION SFAC
136 DOUBLE PRECISION SA, SB, SC, SS, D12
139 DOUBLE PRECISION DA1(8), DATRUE(8), DB1(8), DBTRUE(8), DC1(8),
247 DOUBLE PRECISION SFAC
254 DOUBLE PRECISION DTRUE1(5), DTRUE3(5), DTRUE5(8,5,2), DV(8,5,2),
258 DOUBLE PRECISION DASUM, DNRM2
341 DOUBLE PRECISION SFAC
346 DOUBLE PRECISION S
[all...] |
| /toolchain/binutils/binutils-2.27/gas/doc/ |
| c-mips.texi | 144 Enable use of floating-point operations on odd-numbered single-precision
436 Disable or enable double-precision floating-point operations. Note
437 that by default double-precision floating-point operations are always
839 @item 1 - Double-precision
840 This variant indicates that double-precision support is used. For 64-bit
843 required and double-precision operations use pairs of registers.
845 @item 2 - Single-precision
846 This variant indicates that single-precision support is used. Double
847 precision operations will be supported via soft-float routines.
859 @item 5 - Double-precision 32-bit CPU, 32-bit or 64-bit FP
[all...] |
| /device/linaro/bootloader/edk2/StdLib/LibC/Softfloat/bits64/ |
| softfloat.c | 72 Floating-point rounding mode, extended double-precision rounding precision,
217 Returns the fraction bits of the single-precision floating-point value `a'.
229 Returns the exponent bits of the single-precision floating-point value `a'.
241 Returns the sign bit of the single-precision floating-point value `a'.
253 Normalizes the subnormal single-precision floating-point value represented
273 single-precision floating-point value, returning the result. After being
292 and significand `zSig', and returns the proper single-precision floating-
294 value is simply rounded and packed into the single-precision format, with
301 precision floating-point number.
[all...] |
| /external/cblas/testing/ |
| c_zblat2.f | 74 DOUBLE PRECISION RZERO, RHALF, RONE
82 DOUBLE PRECISION EPS, ERR, THRESH
97 DOUBLE PRECISION G( NMAX )
102 DOUBLE PRECISION DDIFF
253 * Compute EPS (the machine precision).
428 9998 FORMAT( ' RELATIVE MACHINE PRECISION IS TAKEN TO BE', 1P, E9.1 )
479 DOUBLE PRECISION RZERO
482 DOUBLE PRECISION EPS, THRESH
493 DOUBLE PRECISION G( NMAX )
497 DOUBLE PRECISION ERR, ERRMA
[all...] |
| c_zblat1.f | 12 DOUBLE PRECISION SFAC
81 DOUBLE PRECISION SFAC
87 DOUBLE PRECISION SA
92 DOUBLE PRECISION STRUE2(5), STRUE4(5)
95 DOUBLE PRECISION DZASUMTEST, DZNRM2TEST
282 DOUBLE PRECISION SFAC
535 DOUBLE PRECISION SFAC
538 DOUBLE PRECISION SCOMP(LEN), SSIZE(LEN), STRUE(LEN)
543 DOUBLE PRECISION SD
546 DOUBLE PRECISION SDIF
[all...] |
| /external/tensorflow/tensorflow/contrib/metrics/python/ops/ |
| metric_ops.py | 409 """Computes the precision of the predictions with respect to the labels.
413 precision. This value is ultimately returned as `precision`, an idempotent
419 `precision`. `update_op` weights each prediction by the corresponding value in
431 metrics_collections: An optional list of collections that `precision` should
438 precision: Scalar float `Tensor` with the value of `true_positives`
442 `precision`.
450 return metrics.precision(
[all...] |
| /external/deqp/android/cts/master/src/ |
| gles3-hw-issues.txt | 2 dEQP-GLES3.functional.shaders.builtin_functions.precision.clamp.highp_fragment.scalar
3 dEQP-GLES3.functional.shaders.builtin_functions.precision.clamp.highp_fragment.vec2
4 dEQP-GLES3.functional.shaders.builtin_functions.precision.clamp.highp_fragment.vec3
5 dEQP-GLES3.functional.shaders.builtin_functions.precision.clamp.highp_fragment.vec4
6 dEQP-GLES3.functional.shaders.builtin_functions.precision.clamp.highp_vertex.scalar
7 dEQP-GLES3.functional.shaders.builtin_functions.precision.clamp.highp_vertex.vec2
8 dEQP-GLES3.functional.shaders.builtin_functions.precision.clamp.highp_vertex.vec3
9 dEQP-GLES3.functional.shaders.builtin_functions.precision.clamp.highp_vertex.vec4
10 dEQP-GLES3.functional.shaders.builtin_functions.precision.max.highp_fragment.scalar
11 dEQP-GLES3.functional.shaders.builtin_functions.precision.max.highp_fragment.vec
[all...] |
| /bionic/libm/upstream-freebsd/lib/msun/bsdsrc/ |
| b_log.c | 62 * the leading term is calculated to extra precision in two
67 * precision for the leading term.
69 * In this case, use a relative precision of 24 bits.
81 * Used for generation of extend precision logarithms.
411 /* u1 + u2 = 2f/(2F+f) to extra precision. */
425 * Extra precision variant, returning struct {double a, b;};
|
| /device/linaro/bootloader/edk2/AppPkg/Applications/Python/Python-2.7.2/Lib/test/decimaltestdata/ |
| reduce.decTest | 25 precision: 9
152 precision: 3
157 precision: 3
187 precision: 9
214 precision: 40
230 precision: 5
|
| /device/linaro/bootloader/edk2/AppPkg/Applications/Python/Python-2.7.2/Lib/test/ |
| test_format.py | 79 # test some ridiculously large precision, expect overflow
82 # check for internal overflow validation on length of precision
149 # next one gets two leading zeroes from precision, and another from the
185 # next one gets one leading zero from precision
189 # but reduce precision, and base marker should add a zero
191 # one leading zero from precision, and another from "0" flag & width
|
| /device/linaro/bootloader/edk2/StdLib/LibC/Stdio/ |
| vsnprintf_ss.c | 129 int prec; /* precision from format (%.3d), or -1 */
359 * ``... diouXx conversions ... if a precision is
368 * explicit precision of zero is no characters.''
433 * required by a decimal [diouxX] precision, then print the
435 * floating precision; finally, if LADJUST, pad with blanks.
471 /* leading zeroes from decimal precision */
|
| /external/compiler-rt/lib/sanitizer_common/ |
| sanitizer_common_interceptors_format.inc | 353 int precisionIdx; // precision argument index, or -1 if not specified (".*n$")
413 // Precision
416 // Actual precision is optional (surprise!)
519 // Dynamic precision
535 // FIXME: properly support starred precision for strings.
538 // Won't read more than "precision" symbols.
|
| /external/deqp/framework/delibs/debase/ |
| deMath.h | 191 /* Convert int to float. If the value cannot be represented exactly in native single precision format, return
196 * \note Value returned is in native single precision format. For example with x86 extended precision, the value
197 * returned might not be representable in IEEE single precision float.
201 /* Convert to float. If the value cannot be represented exactly in IEEE single precision floating point format,
205 /* Convert to float. If the value cannot be represented exactly IEEE single precision floating point format,
|
| /external/eigen/lapack/ |
| dlarft.f | 28 * DOUBLE PRECISION T( LDT, * ), TAU( * ), V( LDV, * )
91 *> V is DOUBLE PRECISION array, dimension
106 *> TAU is DOUBLE PRECISION array, dimension (K)
113 *> T is DOUBLE PRECISION array, dimension (LDT,K)
176 DOUBLE PRECISION T( LDT, * ), TAU( * ), V( LDV, * )
182 DOUBLE PRECISION ONE, ZERO
|
| /external/gemmlowp/doc/ |
| quantization.md | 230 low-precision.md](low-precision.md#efficient-handling-of-offsets) for that. The
285 low-precision.md](low-precision.md#efficient-handling-of-offsets) for
312 [low-precision.md](low-precision.md) and the newer one described in this
|
| /external/libpcap/ |
| savefile.c | 273 pcap_open_offline_with_tstamp_precision(const char *fname, u_int precision,
307 p = pcap_fopen_offline_with_tstamp_precision(fp, precision, errbuf);
323 pcap_t* pcap_hopen_offline_with_tstamp_precision(intptr_t osfd, u_int precision,
343 return pcap_fopen_offline_with_tstamp_precision(file, precision,
365 pcap_fopen_offline_with_tstamp_precision(FILE *fp, u_int precision,
400 p = (*check_headers[i])(magic, fp, precision, errbuf, &err);
|
| /external/valgrind/none/tests/ppc32/ |
| test_isa_2_06_part2.c | 469 * When src is a SNaN, it's converted to a QNaN first before rounding to single-precision,
470 * so we can't just copy the double-precision value to the corresponding slot in the
471 * single-precision array (i.e., in the loop at the end of this function). Instead, we
572 precision_type_t precision; member in struct:vx_fp_test
1088 precision_type_t precision; member in struct:simple_test
1628 precision_type_t precision; member in struct:div_ext_test
[all...] |
