Differences
This shows you the differences between two versions of the page.
| Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
| data_mining:neural_network:belief_nets [2017/04/30 09:44] – [Discriminative fine-tuning for DBNs] phreazer | data_mining:neural_network:belief_nets [2017/07/30 16:05] (current) – [Structure] phreazer | ||
|---|---|---|---|
| Line 120: | Line 120: | ||
| * Initial gradients are sensibel, backprop only needs to perform a local search from a sensible start point. | * Initial gradients are sensibel, backprop only needs to perform a local search from a sensible start point. | ||
| + | Fine-tuning only modifies features slightly to get category boundaries right (does not need to discover new features). | ||
| + | |||
| + | Objection: Many features are learned that are useless for a particular discrimination. | ||
| + | |||
| + | Example model (MNIST): Add 10-way softmax at the top and do backprop. | ||
| + | |||
| + | |||
| + | More layers => lower error with pretraining. | ||
| + | |||
| + | Solutions are qualitative different. | ||
| + | |||
| + | ==== Model real-valued data with RBMS ==== | ||
| + | |||
| + | Mean-field logistic units cannot represent precise inetermediate values (e.g. pixel intensity in image). | ||
| + | |||
| + | Model pixels as Gaussian variables. Alternating Gibbs sampling, with lower learning rate. | ||
| + | |||
| + | Parabolic containment function. (keep visible unit close to b_i). | ||
| + | Energy-gradient. | ||
| + | |||
| + | Stepped sigmoid units. Many copies of a stochastic binary unist. All copies have same weiths and bias, b, but they have different fixed offsets to the bias (b-0.5, b-1.5, ...). | ||
| + | |||
| + | ==== Structure ==== | ||
| + | |||
| + | Autoencoder, | ||