Autoencoder

PCA:

• N dimensions, M orthogonal directions with most variance.
• Reconstruct by using mean value over all the data on the N-M directions that are not represented.

USe backprop to implement PCA inefficiently.

• M hidden units as bottleneck

INPUT vector ⇒ Code ⇒ OUTPUT vector

Activities in the hidden units form an efficient code.

If hidden and output layers are linear, it weill learn hidden units that are line function of data and minimize squared reconstruciton error (like PCA). M hidden units will span same space as the first M components of PCA, but weights vectors may not be orthogonal and will have equal variances.

Allows generalization of PCA.

With non-linear layers before and after the code, it should be possible to efficiently represent data that lies on or near a non-linear manifold.

input vector ⇒ encoder ⇒ encode weights ⇒ decoding weights ⇒ output vector

Looked like nice way to do non-linear dimensionality reduction:

• Encoding model compact and fast;
• learning time is linear in the number of training cases.

But very difficult to optimize deep autoencoders using backprop: Small initial weights ⇒ backprop gradient dies.

Optimize with: unsupervised layer-by-layer pre-training. Or initialize weights carefully as in Echo-State nets.

Stack of 4 RBMs, then unroll them. Fine-tune with gentle backprop.

784 → 1000 → 500 → 250 → 30 linear models → 250 → 500 → 1000 → 784. W_1 → W_2 → W_3 → W_4 → 30 linear units → W^T_4 → W^T_3 → W^T_2 → W^T_1

Convert each doc into “bag of words” (ignore stop words) Reduce each query vector using a deep autoencoder.

Input vec 2000 counts ⇒ Output vec 2000 reconstructed counts.

Divide counts in a bag of words vector by N, where N is the total number of non-stop works in the document. Output of autoencoder 2000 softmax.

When training the first RBM in the stack: Treat word counts as probs, but make visible to hidden weights N times bigger than hidden to visible, because we have N obersvations from the prop distr.

Convert doc in memory address. Find similar docs in nearby addresses.

Autoencoder 30 logistic units in code layer During fine-tuning add noise to inputs to the code units.

• Noise forces activities to become bimodal in order to resist the effects of the noise.
• Simply threshold activities of 30 code units to get binary code.

Learn binary features for representation.

Deep-autoencoder as hash function.

Query (supermaket search): Hash, get address, get nearby addresses (semantically similar documents).