最小二乘法求解线性方程组公式推导

M行N列方程组如下。其中x,y是已知量，k是未知量：

$${\left\{\begin{matrix} k_{1}x_{1,1}+k_{2}x_{1,2}+ \cdots +k_{N}x_{1,N}=y_{1}\\ k_{1}x_{2,1}+k_{2}x_{2,2}+ \cdots +k_{N}x_{2,N}=y_{2}\\ \vdots \\ k_{1}x_{M,1}+k_{2}x_{M,2}+ \cdots +k_{N}x_{M,N}=y_{M} \end{matrix}\right.}$$

 构造误差函数：

$${ e=\sum_{m=1}^{M}\left ( k_{1}x_{m,1}+k_{2}x_{m,2}+ \cdots +k_{N}x_{m,N}-y_{m} \right )^{2} }$$

 对${k_{n}}$求导，并使导数为零得：

$${\left\{\begin{matrix} \frac{\partial e}{\partial k_{1}}=2 \sum_{m=1}^{M} x_{m,1} \left ( k_{1}x_{m,1}+k_{2}x_{m,2}+ \cdots +k_{N}x_{m,N}-y_{m} \right )=0 \\ \frac{\partial e}{\partial k_{2}}=2 \sum_{m=1}^{M} x_{m,2} \left ( k_{1}x_{m,1}+k_{2}x_{m,2}+ \cdots +k_{N}x_{m,N}-y_{m} \right )=0 \\ \vdots \\ \frac{\partial e}{\partial k_{N}}=2 \sum_{m=1}^{M} x_{m,N} \left ( k_{1}x_{m,1}+k_{2}x_{m,2}+ \cdots +k_{N}x_{m,N}-y_{m} \right )=0 \end{matrix}\right.}$$

 整理得：

$${\left\{\begin{matrix} \sum_{m=1}^{M} x_{m,1}x_{m,1}k_{1}+\sum_{m=1}^{M} x_{m,1}x_{m,2}k_{2}+ \cdots +\sum_{m=1}^{M} x_{m,1}x_{m,N}k_{N}=\sum_{m=1}^{M} x_{m,1}y_{m} \\ \sum_{m=1}^{M} x_{m,2}x_{m,1}k_{1}+\sum_{m=1}^{M} x_{m,2}x_{m,2}k_{2}+ \cdots +\sum_{m=1}^{M} x_{m,2}x_{m,N}k_{N}=\sum_{m=1}^{M} x_{m,2}y_{m}\\ \vdots \\ \sum_{m=1}^{M} x_{m,N}x_{m,1}k_{1}+\sum_{m=1}^{M} x_{m,N}x_{m,2}k_{2}+ \cdots +\sum_{m=1}^{M} x_{m,N}x_{m,N}k_{N}=\sum_{m=1}^{M} x_{m,N}y_{m} \end{matrix}\right. }$$

改写成矩阵形式如下：

$${ \begin{pmatrix} \sum_{m=1}^{M} x_{m,1}x_{m,1} & \sum_{m=1}^{M} x_{m,1}x_{m,2} & \cdots & \sum_{m=1}^{M} x_{m,1}x_{m,N} \\ \sum_{m=1}^{M} x_{m,2}x_{m,1} & \sum_{m=1}^{M} x_{m,2}x_{m,2} & \cdots & \sum_{m=1}^{M} x_{m,2}x_{m,N} \\ \vdots & \vdots & \ddots & \vdots \\ \sum_{m=1}^{M} x_{m,N}x_{m,1} & \sum_{m=1}^{M} x_{m,N}x_{m,2} & \cdots & \sum_{m=1}^{M} x_{m,N}x_{m,N} \end{pmatrix} \begin{pmatrix} k_{1}\\ k_{2}\\ \vdots \\ k_{N} \end{pmatrix}= \begin{pmatrix} \sum_{m=1}^{M} x_{m,1}y_{m} \\ \sum_{m=1}^{M} x_{m,2}y_{m} \\ \vdots \\ \sum_{m=1}^{M} x_{m,N}y_{m} \end{pmatrix} }$$

结果就是：

$${ \begin{pmatrix} k_{1}\\ k_{2}\\ \vdots \\ k_{N} \end{pmatrix}= \begin{pmatrix} \sum_{m=1}^{M} x_{m,1}x_{m,1} & \sum_{m=1}^{M} x_{m,1}x_{m,2} & \cdots & \sum_{m=1}^{M} x_{m,1}x_{m,N} \\ \sum_{m=1}^{M} x_{m,2}x_{m,1} & \sum_{m=1}^{M} x_{m,2}x_{m,2} & \cdots & \sum_{m=1}^{M} x_{m,2}x_{m,N} \\ \vdots & \vdots & \ddots & \vdots \\ \sum_{m=1}^{M} x_{m,N}x_{m,1} & \sum_{m=1}^{M} x_{m,N}x_{m,2} & \cdots & \sum_{m=1}^{M} x_{m,N}x_{m,N} \end{pmatrix} ^{-1} \begin{pmatrix} \sum_{m=1}^{M} x_{m,1}y_{m} \\ \sum_{m=1}^{M} x_{m,2}y_{m} \\ \vdots \\ \sum_{m=1}^{M} x_{m,N}y_{m} \end{pmatrix} }$$

 从上式可以看出来，其实它就是公式：

$${\mathbf{K}=\left ( \mathbf{X^{T}}\mathbf{X} \right )^{-1}\mathbf{X^{T}}\mathbf{Y} }$$