Differences
This shows you the differences between two versions of the page.
| Both sides previous revisionPrevious revisionNext revision | Previous revisionLast revisionBoth sides next revision | ||
| data_mining:neural_network:autoencoder [2017/05/04 15:55] – [Semantic hashing] phreazer | data_mining:neural_network:autoencoder [2017/07/30 18:01] – [Autoencoder] phreazer | ||
|---|---|---|---|
| Line 1: | Line 1: | ||
| ====== Autoencoder ====== | ====== Autoencoder ====== | ||
| + | |||
| + | * Unsupervised learning: Feature extraction, Generative models, Compression, | ||
| + | * Loss as evaluation metric | ||
| + | * Difference to RBM: Deterministic approach (not stochastic). | ||
| + | |||
| + | Encoder compresses to few dimensions, Decoder maps back to full dimensionality | ||
| + | ===== Comparison with PCA ===== | ||
| + | |||
| PCA: | PCA: | ||
| Line 69: | Line 77: | ||
| Reconstructing 32x32 color images from 256 bit codes. | Reconstructing 32x32 color images from 256 bit codes. | ||
| + | |||
| + | ===== Shallow autoencoders for pre-training ===== | ||
| + | |||
| + | Just have 1 layer. RBMs can be seen as shallow autoencoders. | ||
| + | |||
| + | Train RBM with one-step constrastive divergence: Makses resconstruction look like data. | ||
| + | |||
| + | |||
| + | ===== Conclusion about pre-training ===== | ||
| + | |||
| + | For data sets without huge number of labeled cases: Pre-training helps subsequent discriminative learning, espescially if unlabeled extra data is available. | ||
| + | |||
| + | For very large, labeled datasets: Not necessary, but if nets get much larger pre-training is necessary again. | ||