Hopfield nets

Architecture:

• Recurrent connections
• Binary theshold units

Idea: If connections are symmetric, there is a global energy function.

Each binary configuration has an energy. Binary threshold decision rule causes network to settle to a minimum of this energy function.

Global energy:

$E = - \sum_i s_i b_i - \sum_{i<j} s_i s_j w_ij$

• $w_ij$ is connection weight.
• Binary states of two neurons

Local computation for each unit:

$\Delta E_i = E(s_i=0) - E(s_i=1) = b_i \sum_j s_j w_{ij}$

* memories could be energy minima of a neural net. * Binary threshold decision rule can the nbe used to clean up incomplete or corrupted memories.

• Activities of 1 and -1 to store a binary state vector (by incrementing weight betweend any two units by the product of their activities). Bias is permantly on unit.
• $\Delta w_{ij} = s_i s_j$

N units = 0.15 N memories (At N bits per memory this is only 0.15N^2).

• Net has $N^2$ weights and biases.
• After storing $M$ memories, each connection weights has an integer value in the range of $[-M, M]$.
• Number of bits required to store weights and biases: $N^2 log(2M+1)$.