Sophon is your intelligent research coscientist for the agent era.
The derivation holds. Carrying the temperature tau through the InfoNCE bound makes the transition into Eq. 5 cleaner, and the batch-size claim in Section 4 still needs a citation.
Researchers at the world’s great institutions use Sophon
Harvard, Stanford, UC Berkeley, Carnegie Mellon, Penn, MIT, Caltech, Waterloo, Michigan, Yale, Brown.
Sophon has already read your library. Margin notes wait inside every paper, written with your experiments in mind.
This paper presents SimCLR: a simple framework for contrastive learning of visual representations. We simplify recently proposed contrastive self-supervised learning algorithms without requiring specialized architectures or a memory bank. In order to understand what enables the contrastive prediction tasks to learn useful representations, we systematically study the major components of our framework. We show that (1) composition of data augmentations plays a critical role in defining effective predictive tasks, (2) introducing a learnable nonlinear transformation between the representation and the contrastive loss substantially improves the quality of the learned representations, and (3) contrastive learning benefits from larger batch sizes and more training steps compared to supervised learning. By combining these findings, we are able to considerably outperform previous methods for self-supervised and semi-supervised learning on ImageNet. A linear classifier trained on self-supervised representations learned by SimCLR achieves 76.5% top-1 accuracy, matching the performance of a supervised ResNet-50.
Learning effective visual representations without human supervision is a long-standing problem. Most mainstream approaches fall into one of two classes: generative or discriminative. Generative approaches learn to model pixels in the input space, which is computationally expensive and may be more than representation learning requires. Discriminative approaches train networks on pretext tasks where both the inputs and the labels are derived from an unlabeled corpus, so the quality of the learned features depends heavily on the choice of task.
In this paper we present a simple framework for contrastive learning that requires neither specialized architectures nor a memory bank. A key ingredient is a learnable nonlinear projection head placed between the representation and the contrastive loss, which improves the learned features substantially. The representation before the head, rather than after it, is what transfers to downstream tasks.
When the pretext task is defined well, a standard ResNet encoder trained with this objective closes most of the gap to its supervised counterpart. We analyze each component of the framework in isolation, holding the encoder, the optimizer, and the training schedule fixed, and report linear evaluation accuracy on ImageNet throughout.
Our ablations point to three findings. Augmentation composition defines the difficulty of the prediction task. The projection head absorbs the invariances that the loss demands. And the number of negatives visible to the loss at each step governs how sharp the resulting embedding space becomes.
Each of these observations holds across encoder depths and widths, and together they let a conceptually minimal method match far more intricate pipelines on standard benchmarks.
The contrastive prediction task is defined on pairs of augmented views. Given a positive pair, the other 2(N-1) augmented examples in the minibatch serve as negatives. The loss, a normalized temperature-scaled cross entropy we denote NT-Xent, applies a temperature tau to cosine similarities before the softmax, and the choice of tau interacts strongly with the similarity normalization.
We train all encoders with the LARS optimizer to stabilize large-batch training, and we evaluate with the standard linear protocol. Under this protocol the encoder is frozen and a linear classifier is trained on top of the average-pooled features, so any gain reflects the representation itself and never the classifier.
Figure 2 summarizes the effect of individual augmentations and their compositions. No single transformation suffices to learn good representations, even though the model can nearly solve the contrastive task under each of them. Random cropping paired with strong color distortion stands out: without color distortion, the network can exploit color histograms alone to identify positive pairs, and the task stops teaching it anything about shape or texture. We also find that unsupervised contrastive learning profits from stronger augmentation than supervised training does, and that the same augmentation policy transfers across encoder depths and widths with no further tuning.
Scale matters as well. Contrastive learning benefits from much larger batch sizes than supervised learning, and training with batches of 4096 supplies thousands of negative examples at every step, sharpening the embedding space early in training. Longer schedules recover part of the gap at smaller batches, though the compute trade is rarely favorable.
Proceedings of the 37th International Conference on Machine Learning, Vienna, Austria, PMLR 119, 2020.
The composition is the finding here. Crop alone or color jitter alone barely beats the baseline in their ablation, so the invariance comes from stacking the two. Worth keeping in mind before we trim the augmentation pipeline.
Note that the linear probe is run on the layer before the projection head, not after. The head throws away color and augmentation information, which is exactly what the loss asks it to do. Evaluating after it costs them about 10 points.
This result quietly depends on a batch of 4096, since every other example in the batch serves as a negative. At our batch sizes a memory bank or a momentum encoder gets the same effective count for far less GPU.
Your whole lab in one document. Live cursors, instant comments, and every edit landing for everyone at once.
Should tau match Eq. 4 here?
Ask how anything connects. Sophon answers with the papers, the evidence, and the graph to prove it.
The flat loss in run 14 is dimensional collapse. Effective rank of the embeddings falls from 118 to 23 at step 12k, and the closest match in your library describes this exact failure mode.
Shows contrastive embeddings collapsing to a low rank subspace while the loss keeps improving, the same signature run 14 reproduces.
The embedding vectors end up spanning a lower-dimensional subspace instead of the entire available embedding space.
Describe what you are hunting for. Sophon searches the literature with your entire project as context.
Five anchors for your survey: the two canonical objectives, their predictive-coding root, the bridge paper that unifies them, and the analysis your reviewers will expect.
A method mentioned in Slack, run in Colab, published on arXiv. Sophon ties every trace into one graph.
sweeping tau with the mask ratio fixed the collapse on run 14
matches what the SimCLR appendix found, adding it to the ablation table
Following SimCLR, we adopt the temperature-scaled InfoNCE objective, where τ sharpens the similarity distribution over negatives. We sweep τ jointly with the mask ratio.
Every source of context, one workspace
Oh yeah, we're also the fastest LaTeX editor on the web
We present a masked-contrastive pretraining objective that couples span masking with an InfoNCE loss over quantized targets. A single temperature-annealed encoder, trained once on unlabeled audio and images, transfers across eleven downstream benchmarks. Our models match wav2vec 2.0 on LibriSpeech with a quarter of the compute and improve linear-probe accuracy on ImageNet by 2.1 points over SimCLR.
Self-supervised pretraining now rivals supervised baselines across speech and vision, yet the two communities optimize different objectives under different masking regimes. We ask whether one contrastive recipe can serve both modalities at once. Our answer couples masked prediction with a temperature-scaled contrastive loss, so the encoder learns representations that survive aggressive span corruption. For a positive pair the loss takes the familiar InfoNCE form
where the temperature τ follows a cosine schedule. Annealing the temperature sharpens the similarity distribution late in training, which we find is the single largest contributor to transfer quality.
Given a batch of N masked views, we learn an encoder by pulling each positive pair together while repelling the remaining 2(N-1) negatives. Spans are drawn from a geometric distribution and masked targets are quantized before the contrastive head, following the wav2vec 2.0 recipe. We pretrain for 400k steps at batch size 4096 with cosine decay.