python - pytorch 如何计算简单函数的导数？

Question

当我们谈论 pytorch 中的自微分时，我们通常会看到基于张量公式的图形结构，pytorch 将使用链式规则通过跟踪图形树来计算梯度。但是，我想知道叶子节点会发生什么？pytorch 是否使用其解析导数对整个基本函数列表进行硬编码，还是使用数值方法计算梯度？一个简单的例子：

import torch

def f(x):
    return x ** 2
x = torch.tensor([1.0], requires_grad=True)
y = f(x)
y.backward()
print(x.grad) # 2.0

在这个例子中，pytorch 是通过 $$ (x^2)' = 2x = 2 * 1 = 2 $$ 计算导数，还是 pytorch 以类似于 $$ (1.00001^2 - 1^2) / (1.000001 - 1) ~ 2 $$ ?

谢谢！

Answer 1

See this paper for exact answer, specifically section 2.1 or figure 2.

In short, PyTorch has a list of basic functions and the expression of their derivatives. So, what is done in your case (y =xx), is evaluating $$ y' = 2x $$.

The numerical method you mentioned is called numerical differentiation or finite differences, and it is an approximation of the derivative. But it is not what PyTorch does.