Suppose $A\mathbf{x} = \mathbf{b}$ has no solution. We can compute an approximate solution by determining the least-squares solution. Let $A$ be the $m \times n$ matrix and $\mathbf{b}$ a vector in $\mathbb{R}^m$. A least-squares solution of $A\mathbf{x} = \mathbf{b}$ are the solutions to $A^T A \hat{\mathbf{x}} = A^T \mathbf{b}$. Steps:
- Compute $A^T A$ and $A^T \mathbf{b}$.
- Form augmented matrix $[A^T A | A^T \mathbf{b}]$ for equation $A^T A \hat{\mathbf{x}} = A^T \mathbf{b}$.
- Find RREF of $[A^T A | \mathbf{b}’]$ where $\mathbf{b}’ = A^T \mathbf{b}$.
- $\hat{\mathbf{x}}$ least squares solution to $A’\hat{\mathbf{x}} = \mathbf{b}’$ is the best approximation to $A\mathbf{x} = \mathbf{b}$.
- The approximation error is $|\mathbf{b} - A\hat{\mathbf{x}}|$.
#Exercise
Least Squares