Hopfield nets

Architecture:

• Recurrent connections
• Binary theshold units

Idea: If connections are symmetric, there is a global energy function.

Each binary configuration has an energy. Binary threshold decision rule causes network to settle to a minimum of this energy function.

Global energy:

$E = - \sum_i s_i b_i - \sum_{i<j} s_i s_j w_ij$

• $w_ij$ is connection weight.
• Binary states of two neurons

Local computation for each unit:

$\Delta E_i = E(s_i=0) - E(s_i=1) = b_i \sum_j s_j w_{ij}$

* memories could be energy minima of a neural net. * Binary threshold decision rule can the nbe used to clean up incomplete or corrupted memories.

• Activities of 1 and -1 to store a binary state vector (by incrementing weight betweend any two units by the product of their activities). Bias is permantly on unit.
• $\Delta w_{ij} = s_i s_j$

N units = 0.15 N memories (At N bits per memory this is only 0.15N^2).

• Net has $N^2$ weights and biases.
• After storing $M$ memories, each connection weights has an integer value in the range of $[-M, M]$.
• Number of bits required to store weights and biases: $N^2 log(2M+1)$.

When new configuration is memorizes, we hope to create a new energy minimum (if two minima merge, capacity decreases).

Let net settle from random initial state, then do unlearning.

Better storage rule: Instead of trying to store vectors in one shot, cycle through training set many times. (Pseudo likelihood technique).

Instead of memories, store interpretations of sensory input.

• Input represented as visible units
• Intepretation represented as hidden units.
• Badness of interpreation rep. as energy.

Issues:

• How to avoid getting trapped in poor local minima of the energy function?
• How to learn the weights on the connections to the hidden units and between the hidden units?