Application: Hopfield Neural Networks

Application: Hopfield Neural Networks#

A Hopfield network is a type of recurrent neural network where all neurons are connected to each other. It is commonly used for associative memory—storing and recalling patterns. The network evolves to a stable state, recalling a stored pattern when given noisy or incomplete input. We can understand its behavior using matrix and vector operations.

Components of a Hopfield Network#

Neurons (States): Each neuron has a binary state, either $1$ (active) or $- 1$ (inactive). The state of all neurons is represented by a vector $s$ . For a network of 3 neurons, the state vector might be:

$\begin{array}{r} s = (\begin{array}{c} 1 \\ - 1 \\ 1 \end{array}) \end{array}$
Weights (Connections): Neurons are connected by weights, represented by a symmetric matrix $W$ . For a 3-neuron network, the weight matrix might look like:

$\begin{array}{r} W = (\begin{array}{c} 0 & w_{12} & w_{13} \\ w_{21} & 0 & w_{23} \\ w_{31} & w_{32} & 0 \end{array}) \end{array}$

The diagonal elements are zero because a neuron does not connect to itself.
Update Rule: To update the state of neuron $i$ , we calculate the weighted sum of inputs from all other neurons:

$h_{i} = \sum_{j} w_{i j} s_{j} = (W s)_{i}$

If $h_{i} > 0$ , the neuron becomes active ( $s_{i} = 1$ ). If $h_{i} < 0$ , the neuron becomes inactive ( $s_{i} = - 1$ ).

Example: Simple Hopfield Network#

Let’s consider a 3-neuron Hopfield network with the following weight matrix:

\begin{array}{r} W = (\begin{array}{c} 0 & 1 & - 1 \\ 1 & 0 & 1 \\ - 1 & 1 & 0 \end{array}) \end{array}

Initial State#

Assume the initial state of the network is:

\begin{array}{r} s = (\begin{array}{c} 1 \\ - 1 \\ 1 \end{array}) \end{array}

Update Neurons#

We update each neuron based on the weighted inputs:

Neuron 1: The input to neuron 1 is:

$h_{1} = W_{11} s_{1} + W_{12} s_{2} + W_{13} s_{3} = 0 \cdot 1 + 1 \cdot (- 1) + (- 1) \cdot 1 = - 2$

Since $h_{1} < 0$ , the new state of neuron 1 is $s_{1} = - 1$ .
Neuron 2: The input to neuron 2 is:

$h_{2} = W_{21} s_{1} + W_{22} s_{2} + W_{23} s_{3} = 1 \cdot 1 + 0 \cdot (- 1) + 1 \cdot 1 = 2$

Since $h_{2} > 0$ , the new state of neuron 2 is $s_{2} = 1$ .
Neuron 3: The input to neuron 3 is:

$h_{3} = W_{31} s_{1} + W_{32} s_{2} + W_{33} s_{3} = (- 1) \cdot 1 + 1 \cdot (- 1) + 0 \cdot 1 = - 2$

Since $h_{3} < 0$ , the new state of neuron 3 is $s_{3} = - 1$ .

New State#

After one round of updates, the new state of the network is:

\begin{array}{r} s = (\begin{array}{c} - 1 \\ 1 \\ - 1 \end{array}) \end{array}

Storing Patterns and Energy#

A Hopfield network stores specific patterns by adjusting the weights, and it evolves to minimize its energy. The energy of the network is defined as:

E = - \frac{1}{2} \sum_{i, j} w_{i j} s_{i} s_{j} = - \frac{1}{2} s^{T} W s

As the network updates, it reduces its energy until it reaches a stable state, which corresponds to one of the stored patterns.

Summary#

Neurons are represented by a state vector $s$ .
Weights between neurons are represented by a matrix $W$ .
Neurons update their states based on the weighted sum of inputs, which is a matrix-vector multiplication.
The network evolves to stable patterns by minimizing its energy function.

This is a basic introduction to the Hopfield network using matrix and vector operations.