Sampling-free variational inference of Bayesian neural networks by variance backpropagation

• Verfasst von: Haußmann, Manuel [VerfasserIn]
• Verfasst von: Hamprecht, Fred [VerfasserIn]
• Verfasst von: Kandemir, Melih [VerfasserIn]
• Titel: Sampling-free variational inference of Bayesian neural networks by variance backpropagation
• Verf.angabe: Manuel Haußmann, Fred A. Hamprecht, Melih Kandemir
• Jahr: 2018
• Umfang: 15 S.
• Fussnoten: Last revised 12 Jun 2019 ; Gesehen am 13.07.2022
• Titel Quelle: Enthalten in: De.arxiv.org
• Ort Quelle: [S.l.] : Arxiv.org, 1991
• Jahr Quelle: 2018
• Band/Heft Quelle: (2018), Artikel-ID 1805.07654, Seite 1-15
• DOI: doi:10.48550/arXiv.1805.07654
• Datenträger: Online-Ressource
• Sprache: eng
• Sach-SW: Computer Science - Machine Learning
• Sach-SW: Statistics - Machine Learning
• K10plus-PPN: 181009206X

Abstract

We propose a new Bayesian Neural Net formulation that affords variational inference for which the evidence lower bound is analytically tractable subject to a tight approximation. We achieve this tractability by (i) decomposing ReLU nonlinearities into the product of an identity and a Heaviside step function, (ii) introducing a separate path that decomposes the neural net expectation from its variance. We demonstrate formally that introducing separate latent binary variables to the activations allows representing the neural network likelihood as a chain of linear operations. Performing variational inference on this construction enables a sampling-free computation of the evidence lower bound which is a more effective approximation than the widely applied Monte Carlo sampling and CLT related techniques. We evaluate the model on a range of regression and classification tasks against BNN inference alternatives, showing competitive or improved performance over the current state-of-the-art.