数值计算中的相对误差，绝对误差

相对误差：
Relative tolerance (RTOL) controls local error relative to the size of the solution— RTOL = 10-4 means that errors are controlled to 0.01%

绝对误差：
Absolute tolerances (ATOL) control error when a solution component may be small
— Ex: solution starting at a nonzero value but decaying to noise level, ATOL should be set to noise level

Operator Numerical Check

1. 基本公式

其中：

atol: absolute tolerance
rtol: relative tolerance
NaN and Inf行为：
NaNs are treated as equal if they are in the same place and if equal_nan=True. Infs are treated as equal if they are in the same place and of the same sign in both arrays.

2. 软件行为

NumPy
Numpy对应的接口有两个：

[numpy.allclose(a, b, rtol=1.e-5, atol=1.e-8, equal_nan=False)](https://github.com/numpy/numpy/blob/19989c21b6b7a45da80b21d3abde485542ae4eb1/numpy/core/numeric.py#L2189)

[testing.assert_allclose(actual, desired, rtol=1e-07, atol=0, equal_nan=True)](https://numpy.org/doc/stable/reference/generated/numpy.testing.assert_allclose.html)

PyTorch
PyTorch采用Numpy相同的default tolerance，API为:

torch.allclose(input, other, rtol=1e-05, atol=1e-08, equal_nan=False)。

每个op可使用default也可自设，具体每个op的tolerance可见source code; Caffe2算子的tolerance可见source code。

Examples

from code

TensorFlow
TF使用如下test util判断：

assertAllCloseAccordingToType(a, b, rtol=1e-06, atol, float_rtol, float_atol, half_rtol, half_atol, bfloat16_rtol, bfloat16_atol, msg=None)

或它的简化版本ssertAllClose。
对不同的数据类型，default tolerance如下：

TF default tolerance的选择标准描述如下：

The default atol and rtol is 10 * eps, where eps is the smallest representable positive number such that 1 + eps != 1. This is about 1.2e-6 in 32bit, 2.22e-15 in 64bit, and 0.00977 in 16bit. See numpy.finfo.

具体每个op的tolerance可见相应op的test code或相应kernel的test code。

Examples

code

https://www.cnblogs.com/Matrix_Yao/p/15830144.html