This is an old revision of the document!
Hopfield nets
Architecture:
- Recurrent connections
- Binary theshold units
Idea: If connections are symmetric, there is a global energy function.
Each binary configuration has an energy. Binary threshold decision rule causes network to settle to a minimum of this energy function.
Energy function
Global energy:
$E = - \sum_i s_i b_i - \sum_{i<j} s_i s_j w_ij$
- $w_ij$ is connection weight.
- Binary states of two neurons
Local computation for each unit:
$\Delta E_i = E(s_i=0) - E(s_i=1) = b_i \sum_j s_j w_{ij}$
Storing memories
* memories could be energy minima of a neural net. * Binary threshold decision rule can the nbe used to clean up incomplete or corrupted memories.
- Activities of 1 and -1 to store a binary state vector (by incrementing weight betweend any two units by the product of their activities). Bias is permantly on unit.
- $\Delta w_{ij} = s_i s_j$
Storage capacity
N units = 0.15 N memories (At N bits per memory this is only 0.15N^2).
- Net has $N^2$ weights and biases.
- After storing $M$ memories, each connection weights has an integer value in the range of $[-M, M]$.
- Number of bits required to store weights and biases: $N^2 log(2M+1)$.
Spurious minima limit capactiy
When new configuration is memorizes, we hope to create a new energy minimum (if two minima merge, capacity decreases).
Avoiding spurious minima by unlearning
Let net settle from random initial state, then do unlearning.
Better storage rule: Instead of trying to store vectors in one shot, cycle through training set many times. (Pseudo likelihood technique).
Hopfield nets with hidden units
Instead of memories, store interpretations of sensory input.
- Input represented as visible units
- Intepretation represented as hidden units.
- Badness of interpreation rep. as energy.