This is an old revision of the document!
Hopfield nets
Architecture:
- Recurrent connections
- Binary theshold units
Idea: If connections are symmetric, there is a global energy function.
Each binary configuration has an energy. Binary threshold decision rule causes network to settle to a minimum of this energy function.
Energy function
Global energy:
$E = - \sum_i s_i b_i - \sum_{i<j} s_i s_j w_ij$
- $w_ij$ is connection weight.
- Binary states of two neurons
Local computation for each unit:
$\Delta E_i = E(s_i=0) - E(s_i=1) = b_i \sum_j s_j w_{ij}$
Storing memories
* memories could be energy minima of a neural net. * Binary threshold decision rule can the nbe used to clean up incomplete or corrupted memories.
- Activities of 1 and -1 to store a binary state vector (by incrementing weight betweend any two units by the product of their activities). Bias is permantly on unit.
- $\Delta w_{ij} = s_i s_j$
Storage capacity
N units = 0.15 N memories (At N bits per memory this is only 0.15N^2).
- Net has $N^2$ weights and biases.
- After storing $M$ memories, each connection weights has an integer value in the range of $[-M, M]$.
- Number of bits required to store weights and biases: $N^2 log(2M+1)$.
Spurious minima limit capactiy
When new configuration is memorizes, we hope to create a new energy minimum (if two minima merge, capacity decreases).
Avoiding spurious minima by unlearning
Let net settle from random initial state, then do unlearning.
Better storage rule: Instead of trying to store vectors in one shot, cycle through training set many times. (Pseudo likelihood technique).