# 向量, 矩阵与张量梯度的理论推导

## 基本思路

In order to simplify a given calculation, it is often useful to write out the explicit formula for a single scalar element of the output in terms of nothing but scalar variables. Once one has an explicit formula for a single scalar element of the output in terms of other scalar values, then one can use the calculus that you used as a beginner, which is much easier than trying to do matrix math, summations, and derivatives all at the same time.

## 例子

${\bf X}: B \times N$矩阵, ${\bf W}: N \times M$矩阵, ${\bf Y}: B\times M$矩阵, 而$\tilde{ {\bf Y} }$表示如下:

$\tilde{ {\bf Y} } = {\bf X} \cdot {\bf W},$

$\frac{\partial L}{ \partial w_ {n, m} } = \frac{\partial L}{\partial \tilde{ {\bf Y} }} \frac{ \partial \tilde{ {\bf Y} } }{ \partial w_ {n, m} },$

### 第一项: $\partial L / \partial \tilde{ {\bf Y} }$

\begin{aligned} \frac{ \partial L }{\partial \tilde{ y }_ {i, j} } &= \frac{ \partial }{ \partial \tilde{ y }_ {i, j} } \left[ \sum_b \sum_m ( \tilde{ y }_ {b, m} - y_ {b, m} )^2 \right] \\ &= 2 ( \tilde{ y }_ {i, j} - y_ {i, j} ), \end{aligned}

$\frac{\partial L}{\partial \tilde{ {\bf Y} }} = 2 ( \tilde{ {\bf Y} } - {\bf Y}), \label{first_term}$

### 第二项: $\partial \tilde{ {\bf Y} } / \partial w_ {n, m}$

\begin{aligned} \frac{ \partial \tilde{ y }_ {i, j} }{ \partial w_ {n, m} } &= \frac{\partial }{ \partial w_ {n, m} } \left[ \sum_ {k=1}^N x_{i, k} w_{k, j} \right] \\ &= \frac{\partial }{ \partial w_ {n, m} } \left[ x_ {i, 1} w_ {1, j} + \ldots + x_ {i, n} w_ {n, j} + \ldots + x_ {i, N} w_ {N, j} \right] \\ &= \begin{cases} 0, & j \ne m \\ x_ {i, n}, & j = m \end{cases} \end{aligned},

$\frac{ \partial \tilde{ {\bf Y} } }{ \partial w_ {n, m} } = \left[ \begin{array}{ccccc} 0 & \ldots & x_{1, n} & \ldots & 0\\ 0 & \ldots & x_{2, n} & \ldots & 0\\ \vdots & \ddots & \vdots & \ddots & \vdots\\ 0 & \ldots & x_{B, n} & \ldots & 0\\ \end{array} \right]_ {B\times M}, \label{second_term}$

### 结合

\begin{aligned} \frac{\partial L}{ \partial w_ {n, m} } &= \sum_{i}\sum_{j} 2(\tilde{ {\bf Y} } - {\bf Y} ) \circ \left[ \begin{array}{ccccc} 0 & \ldots & x_{1, n} & \ldots & 0\\ 0 & \ldots & x_{2, n} & \ldots & 0\\ \vdots & \ddots & \vdots & \ddots & \vdots\\ 0 & \ldots & x_{B, n} & \ldots & 0\\ \end{array} \right]_ {B\times M} \\ &= 2 \sum_{i=1}^B x_{i, n} ( \tilde{y}_ {i, m} - y_{i, m} )\\ &= {\bf X}_ n^\intercal \cdot 2( \tilde{ {\bf Y} } - {\bf Y} )_ m, \end{aligned}

\begin{aligned} \frac{\partial L}{ \partial {\bf W} } &= {\bf X}^\intercal \cdot 2 (\tilde{ {\bf Y} } - {\bf Y}) \\ &= {\bf X}^\intercal \frac{\partial L}{\partial \tilde{ {\bf Y} } }. \end{aligned}

$\frac{\partial f} {\partial {\bf B} } = {\bf A}^\intercal \frac{\partial f}{ \partial {\bf C} }.$

Ph.D candidate of EIC at HUST.