From the article "Appreciation of the Identity \det(\exp(\boldsymbol{A})) = \exp(\text{Tr}(\boldsymbol{A}))", we learned that the matrix \exp(\boldsymbol{A}) is always invertible, and its inverse is simply \exp(-\boldsymbol{A}). The issue is that \exp(\boldsymbol{A}) is primarily a theoretical definition; writing it this way has little practical value because it requires calculating every power \boldsymbol{A}^n.
Are there any concrete examples? Yes. In this post, we will construct an explicit matrix that is always invertible.
The idea is actually very simple. Suppose \boldsymbol{x} and \boldsymbol{y} are two k-dimensional column vectors. Then \boldsymbol{x}\boldsymbol{y}^{\top} is a k \times k matrix. Let us consider: \begin{aligned} \exp\left(\boldsymbol{x}\boldsymbol{y}^{\top}\right) &= \sum_{n=0}^{\infty}\frac{\left(\boldsymbol{x}\boldsymbol{y}^{\top}\right)^n}{n!} \\ &= \boldsymbol{I} + \boldsymbol{x}\boldsymbol{y}^{\top} + \frac{\boldsymbol{x}\boldsymbol{y}^{\top}\boldsymbol{x}\boldsymbol{y}^{\top}}{2} + \frac{\boldsymbol{x}\boldsymbol{y}^{\top}\boldsymbol{x}\boldsymbol{y}^{\top}\boldsymbol{x}\boldsymbol{y}^{\top}}{6} + \dots \end{aligned} Notice that: \boldsymbol{y}^{\top}\boldsymbol{x} = \langle \boldsymbol{x}, \boldsymbol{y} \rangle is actually just a scalar. Therefore, we can simplify the expression further: \begin{aligned} \exp\left(\boldsymbol{x}\boldsymbol{y}^{\top}\right) &= \boldsymbol{I} + \boldsymbol{x}\boldsymbol{y}^{\top} \left(1 + \frac{\langle \boldsymbol{x}, \boldsymbol{y} \rangle}{2} + \frac{\langle \boldsymbol{x}, \boldsymbol{y} \rangle^2}{6} + \dots\right) \\ &= \boldsymbol{I} + \boldsymbol{x}\boldsymbol{y}^{\top} \left(\frac{e^{\langle \boldsymbol{x}, \boldsymbol{y} \rangle} - 1}{\langle \boldsymbol{x}, \boldsymbol{y} \rangle}\right) \end{aligned} Now this matrix is very concrete. It can be calculated quite easily because it only involves scalar exponential operations. The term (e^x - 1)/x in the parentheses has a removable discontinuity at x=0; when x=0, its value is 1.
According to the identity \det(\exp(\boldsymbol{A})) = \exp(\text{Tr}(\boldsymbol{A})), the determinant of this matrix is: \det\left(\exp\left(\boldsymbol{x}\boldsymbol{y}^{\top}\right)\right) = e^{\langle \boldsymbol{x}, \boldsymbol{y} \rangle} Its inverse matrix is: \exp\left(-\boldsymbol{x}\boldsymbol{y}^{\top}\right) = \boldsymbol{I} - \boldsymbol{x}\boldsymbol{y}^{\top} \left(\frac{1 - e^{-\langle \boldsymbol{x}, \boldsymbol{y} \rangle}}{\langle \boldsymbol{x}, \boldsymbol{y} \rangle}\right) which is also an explicit result.
Of course, a general matrix has k^2 independent parameters, whereas the matrix constructed here from two vectors has only 2k parameters. In terms of expressive power, this is likely insufficient. To enhance the expressivity, one could consider multiplying several such matrices together: \exp\left(\boldsymbol{x}_1\boldsymbol{y}_1^{\top}\right)\exp\left(\boldsymbol{x}_2\boldsymbol{y}_2^{\top}\right)\exp\left(\boldsymbol{x}_3\boldsymbol{y}_3^{\top}\right)\dots Note that this is generally not equal to: \exp\left(\boldsymbol{x}_1\boldsymbol{y}_1^{\top} + \boldsymbol{x}_2\boldsymbol{y}_2^{\top} + \boldsymbol{x}_3\boldsymbol{y}_3^{\top} + \dots\right)
Well, after all this derivation, what is its actual use?
Uh... I don’t know what it’s useful for yet; just treat it as something to appreciate.
(Actually, my original intention was to use a constructive method to directly build invertible neural networks. Based on this ready-made matrix, constructing an invertible fully connected network is not difficult, but I haven’t figured out how to generalize it to convolutional layers. Once I have settled that generalization, I will come back to discuss the applications of this matrix!)
When reprinting, please include the original article address: https://kexue.fm/archives/6407
For more detailed reprinting matters, please refer to: "Scientific Space FAQ"