Contrastive learning unifies t-SNE and UMAP

• Verfasst von: Damrich, Sebastian [VerfasserIn]
• Verfasst von: Böhm, Jan Niklas [VerfasserIn]
• Verfasst von: Hamprecht, Fred [VerfasserIn]
• Verfasst von: Kobak, Dmitry [VerfasserIn]
• Titel: Contrastive learning unifies t-SNE and UMAP
• Verf.angabe: Sebastian Damrich, Jan Niklas Böhm, Fred A. Hamprecht, Dmitry Kobak
• E-Jahr: 2022
• Jahr: 3 Jun 2022
• Umfang: 29 S.
• Fussnoten: Gesehen am 18.10.2022
• Titel Quelle: Enthalten in: De.arxiv.org
• Ort Quelle: [S.l.] : Arxiv.org, 1991
• Jahr Quelle: 2022
• Band/Heft Quelle: (2022), Artikel-ID 2206.01816, Seite 1-29
• DOI: doi:10.48550/arXiv.2206.01816
• Datenträger: Online-Ressource
• Sprache: eng
• Sach-SW: Computer Science - Human-Computer Interaction
• Sach-SW: Computer Science - Machine Learning
• K10plus-PPN: 181907434X

Abstract

Neighbor embedding methods $t$-SNE and UMAP are the de facto standard for visualizing high-dimensional datasets. They appear to use very different loss functions with different motivations, and the exact relationship between them has been unclear. Here we show that UMAP is effectively negative sampling applied to the $t$-SNE loss function. We explain the difference between negative sampling and noise-contrastive estimation (NCE), which has been used to optimize $t$-SNE under the name NCVis. We prove that, unlike NCE, negative sampling learns a scaled data distribution. When applied in the neighbor embedding setting, it yields more compact embeddings with increased attraction, explaining differences in appearance between UMAP and $t$-SNE. Further, we generalize the notion of negative sampling and obtain a spectrum of embeddings, encompassing visualizations similar to $t$-SNE, NCVis, and UMAP. Finally, we explore the connection between representation learning in the SimCLR setting and neighbor embeddings, and show that (i) $t$-SNE can be optimized using the InfoNCE loss and in a parametric setting; (ii) various contrastive losses with only few noise samples can yield competitive performance in the SimCLR setup.