Optimization Algorithms from a Dynamics Perspective (VI): Why Doesn't SimSiam Degenerate?

Deep Image Prior

First of all, it was discovered long ago that a randomly initialized CNN model can be used directly to extract visual features, and the effect is not particularly bad. This conclusion can be traced back to the 2009 paper "What is the best multi-stage architecture for object recognition?". This can be understood as CNNs having a natural ability to process images. Later, this characteristic was given a sophisticated name, "Deep Image Prior," from the paper "Deep Image Prior". They conducted experiments showing that starting from a randomly initialized CNN model, tasks like image completion and denoising can be completed without any supervised learning, further confirming the characteristic that CNNs naturally possess image processing capabilities.

According to my understanding, "Deep Image Prior" stems from three points:

1. Image Continuity: This means that the image itself can be directly viewed as a continuous vector, without needing to learn an Embedding layer like in NLP. This means we can do many tasks using simple methods like "original image + K-nearest neighbors."

2. CNN Architecture Prior: This refers to the local perception design of CNNs, which indeed simulates the visual processing of the human eye well. Since the visual classification results we provide are also based on our human eyes, the two are consistent.

3. Good Initialization: This is not hard to understand. Even the best model with all-zero initialization probably won’t work. A previous article "Understanding Model Parameter Initialization Strategies from a Geometric Perspective" also briefly discussed initialization methods. From a geometric sense, mainstream initialization methods are a kind of approximate "orthogonal transformation," which can preserve the information of input features as much as possible.

Dynamics of Non-Degeneration

Again, the deep image prior means that a randomized CNN model is already a "not-so-bad" encoder. So what we need to do next can be summarized into two points: learn in a better direction, and do not degenerate toward a constant.

Learning in a better direction means designing some prior signals to allow the model to better integrate this prior knowledge. SimSiam, BYOL, etc., let the same image undergo two different data augmentations and then make the corresponding feature vectors as similar as possible. This is a good signal guidance, telling the model that simple transformations should not affect our visual understanding. In fact, this is one of the designs used by all contrastive learning methods.

The difference lies in the point of "not degenerating toward a constant." General contrastive learning methods tell the model which image features should not be similar by constructing negative samples, thereby preventing the model from degenerating. However, SimSiam and BYOL are different; they have no negative samples. In fact, they prevent degeneration by decomposing the model’s optimization process into two synchronized but differently paced modules. Taking SimSiam as an example, its optimization objective can be written as: \mathcal{L}(\varphi, \theta)=\mathbb{E}_{x, \mathcal{T}_1,\mathcal{T}_2}\Big[l\left(h_{\varphi}(f_{\theta}(\mathcal{T}_1(x))), f_{\theta}(\mathcal{T}_2(x))\right)\Big] Then using gradient descent to optimize, the corresponding dynamical equations are: \begin{aligned} \frac{d\varphi}{dt} = - \frac{\partial\mathcal{L}}{\partial \varphi} =& -\mathbb{E}_{x, \mathcal{T}_1,\mathcal{T}_2}\bigg[\frac{\partial l}{\partial h}\frac{\partial h}{\partial \varphi}\bigg]\\ \frac{d\theta}{dt} = - \frac{\partial\mathcal{L}}{\partial \theta} =& -\mathbb{E}_{x, \mathcal{T}_1,\mathcal{T}_2}\bigg[\frac{\partial l}{\partial h}\frac{\partial h}{\partial f}\frac{\partial f}{\partial \theta} \color{blue}{\,+\underbrace{\frac{\partial l}{\partial f}\frac{\partial f}{\partial \theta}}_{\begin{aligned}\text{SimSiam}\\\text{removed this}\end{aligned}}}\bigg] \end{aligned} The above equation indicates the difference brought by the presence or absence of the stop_gradient operator. Simply put, if the stop_gradient operator is added, then \frac{d\theta}{dt} lacks the second term. In this case, both \frac{d\varphi}{dt} and \frac{d\theta}{dt} share the factor \frac{\partial l}{\partial h}. Since h_{\varphi} is closer to the output layer and the initialized f_{\theta} is also a decent encoder, at the start of learning, h_{\varphi} will be optimized faster, while parts closer to the input layer optimize slower. That is to say, \frac{d\varphi}{dt} is the fast dynamics part, and \frac{d\theta}{dt} is the slow dynamics part. Relatively speaking, \frac{d\varphi}{dt} will converge to 0 more quickly. This means \frac{\partial l}{\partial h} will quickly become very small. Since \frac{d\theta}{dt} also contains the \frac{\partial l}{\partial h} term, \frac{d\theta}{dt} becomes small along with it. Before it can degenerate, the force driving it to degenerate has become negligible, so it won’t degenerate. Conversely, if there is the second term \frac{\partial l}{\partial f}\frac{\partial f}{\partial \theta} (whether it’s added back or only it is kept), it is equivalent to adding a "fast track," making it a fast term. Even if \frac{\partial l}{\partial h}=0, because the second term exists, it will continue to drive it toward degeneration.

Let’s take a simple specific example. Consider: l = \frac{1}{2}(\varphi\theta - \theta)^2 For simplicity, \varphi, \theta are scalars. The corresponding dynamical equations are: \frac{d\varphi}{dt}=-(\varphi\theta - \theta)\theta, \quad\frac{d\theta}{dt}=-(\varphi\theta - \theta) \varphi \color{blue}{+ \underbrace{(\varphi\theta - \theta)}_{\begin{aligned}\text{SimSiam}\theta\\ \text{removed this}\end{aligned}}} Suppose \varphi(0)=0.6, \theta(0)=0.1 (chosen arbitrarily). The evolution of both is:

As can be seen, after stopping the gradient of the second \theta, the equations for \varphi and \theta are quite consistent; \varphi quickly tends to 1, while \theta stabilizes at a non-zero value (meaning no degeneration). Conversely, if the second term of \frac{d\theta}{dt} is added, or if only the second term is kept, the result is that \theta quickly tends to 0, and \varphi cannot tend to 1. This means the dominance is taken over by \theta.

This example itself isn’t highly persuasive, but it simply reveals the changes in dynamics:

The introduction of the predictor (\varphi) divides the model’s dynamics into two major parts. The introduction of the stop_gradient operator makes the dynamics of the encoder part (\theta) slower and enhances the synchronization between the encoder and the predictor. In this way, the predictor fits the target with "lightning speed," so that the optimization process stops before the encoder has time to degenerate.