Review of Autoencoders (Deep Learning)

500px-Stacked_Combined
An auto-encoder, autoassociator or Diabolo network is an artificial neural network used for learning efficient codings. The aim of an auto-encoder is to learn a compressed, distributed representation (encoding) for a set of data, typically for the purpose of dimensionality reduction. Architecturally, the simplest form of the auto-encoder is a feedforward, non-recurrent neural net that is very similar to the multilayer perceptron (MLP), with an input layer, an output layer and one or more hidden layers connecting them.

An auto-encoder is often trained using one of the many back propagation variants (conjugate gradient method, steepest descent, etc.). Though often reasonably effective, there are fundamental problems with using back propagation to train networks with many hidden layers. Once the errors get back propagated to the first few layers, they are minuscule, and quite ineffectual. This causes the network to almost always learn to reconstruct the average of all the training data.

Piotr Mirowski (of Microsoft Bing London) presented Review of Auto-Encoders to the Computational Intelligence Unconference 2014, with our Deep Learning stream. These are his slides. Slides are a bit old but it does a great job in comparing some of the models relevant today.

Original link here: https://piotrmirowski.files.wordpress.com/2014/08/piotrmirowski_ciunconf_2014_reviewautoencoders.pptx

He also has Matlab-based tutorial on auto-encoders available here:
https://github.com/piotrmirowski/Tutorial_AutoEncoders/

Leave a Reply

Skip to toolbar