Hopfield Networks for Associative Memory: Examples with Python Implementation

Hopfield networks are a type of recurrent neural network, named after John Hopfield who was awarded the Nobel Prize in Physics in 2024. This article will give you an idea of how an Hopfield network with binary neurons works with examples.

Hopfield networks are primarily used for associative memory, a process of retrieving a complete pattern from an incomplete or noisy input. This makes them particularly useful for applications like pattern recognition, image processing, and optimization problems.

The Hopfield Network Architecture

A Hopfield network consists of a set of interconnected neurons. We consider binary neurons here. Each neuron is either in an “on” or “off” state, represented by 1 or 0, respectively (sometimes 1 and -1 are used and it is just a matter of shifting and scaling). The connections between neurons are symmetric, meaning the weight of the connection from neuron i to neuron j is the same as the weight from neuron j to neuron i.

The Update Rule

The state of each neuron is updated synchronously based on the states of its connected neurons and the corresponding weights. The update rule is as follows:

where:

• x_j(t+1) is the state of neuron i at the next time step
• w_jk is the weight of the connection between neuron j and k, j ≠ k
• x_j(t) is the state of neuron j at the current time step
• I_j is the external input to neuron j

Lyapunov function ( ‘Energy function’)

Given weights and the state of the Hopfield network, the Lyapunov function is given by

Lyapunov functions have been widely used in applications such as stability and convergence analysis. The Lyapunov approach is based on the physical idea that the energy of an isolated system decreases.

Python Implementation

Let’s implement a Hopfield network in Python using NumPy for numerical operations. For illustration purpose, we consider a network with 4 nodes, and no external input I.

Here weights is a symmetric matrix with zero traces (no self-connection) and initial_state and new_state are binary vectors. (Comment the print commands if you do not want to track the evolution of the network!)

import numpy as np

def hopfield_network(weights, initial_state, iter=100):
  """
  Implements a Hopfield network.
  Args:
    weights: A numpy array representing the connection weights.
    initial_state: A numpy array representing the initial state of the network.
    iter: The number of iterations to run the network.
  Returns:
    The final state of the network.
  """
  print(initial_state)
  for _ in range(iterations):
    n = len(initial_state)
    new_state = np.max(np.stack((np.zeros(n,dtype=int),np.sign(np.dot(weights, initial_state)))),0)
    initial_state = new_state
    print(new_state)

  return new_state

To use the Hopfield network, we need to define the connection weights and the initial state. For a small network, a few iterations will bring the system to its steady state.

# example weight and initial state
n = 4
weights = np.array([[0, 1, -1, 1],
                   [1, 0, 1, -1],
                   [-1, 1, 0, 1],
                   [1, -1, 1, 0]])

initial_state = np.array([1, 1, 0, 0])

# implement a Hopfield network
final_state = hopfield_network(weights, initial_state, 8)

Here’s the evolution of the network states. The initial state is a stable equilibrium.

[1 1 0 0]
[1 1 0 0]
[1 1 0 0]
[1 1 0 0]
[1 1 0 0]
[1 1 0 0]
[1 1 0 0]
[1 1 0 0]
[1 1 0 0]

Let’s look at another example with a different initial state. The system oscillates between [0 1 0 1] and [1 0 1 0].

initial_state = np.array([1, 0, 0, 0])

# implement a Hopfield network
final_state = hopfield_network(weights, initial_state, 8)

[1 0 0 0]
[0 1 0 1]
[1 0 1 0]
[0 1 0 1]
[1 0 1 0]
[0 1 0 1]
[1 0 1 0]
[0 1 0 1]
[1 0 1 0]

Now let’s check the Lyapunov function for all network states with weights given. To this end, we list all binary states given the number of nodes.

import itertools

def list_binary_states(n):
  """Lists all possible states of n binary neurons.

  Args:
    n: The number of neurons.

  Returns:
    A list of lists, where each inner list represents a possible state of the neurons.
  """

  states = list(itertools.product([0, 1], repeat=n))
  return [list(s) for s in states]

# Example usage:
n = 4
all_states = list_binary_states(n)
print(all_states)

Here’s the function to compute the Lyapunov function given weights and the state of the Hopfield network.

import numpy as np

def lyapunov_function(weights, states):
  """Computes the Lyapunov function for a Hopfield network.

  Args:
    weights: A numpy array representing the weights of the network.
    states: A numpy array representing the states of the neurons.

  Returns:
    The value of the Lyapunov function.
  """

  return -0.5 * np.dot(states.T, np.dot(weights, states))

# Example usage:
weights = np.array([[0, 1, -1, 1],
                   [1, 0, 1, -1],
                   [-1, 1, 0, 1],
                   [1, -1, 1, 0]])
state = np.array([1, 0, 0, 0])

lyapunov_value = lyapunov_function(weights, state)
print(lyapunov_value)

Let’s compute the Lyapunov function for all states

# compute a Lyapunov function
states = list_binary_states(n)
for state in states:
  print(state,':',lyapunov_function(weights, state))

The Lyapunov function cannot increase, but can stay the same or decrease. The system does not necessarily reach the absolute minimum because it may not be reachable. The Lyapunov function for all 16 states:

[0 0 0 0] : -0.0
[0 0 0 1] : -0.0
[0 0 1 0] : -0.0
[0 0 1 1] : -1.0
[0 1 0 0] : -0.0
[0 1 0 1] : 1.0
[0 1 1 0] : -1.0
[0 1 1 1] : -1.0
[1 0 0 0] : -0.0
[1 0 0 1] : -1.0
[1 0 1 0] : 1.0
[1 0 1 1] : -1.0
[1 1 0 0] : -1.0
[1 1 0 1] : -1.0
[1 1 1 0] : -1.0
[1 1 1 1] : -2.0

Training the Hopfield Network

To train a Hopfield network, we need to store a set of patterns as stable states. This is done by setting the connection weights as follows:

def train_hopfield_network(patterns):
  """
  Trains a Hopfield network.
  Args:
    patterns: A list of patterns to store in the network.
  Returns:
    The trained connection weights.
  """
  N = patterns[0].shape[0]
  weights = np.zeros((N, N))
  for pattern in patterns:
    weights += np.outer(pattern, pattern)
  np.fill_diagonal(weights, 0)
  return weights

Let’s look at an example

patterns = np.array([[1, 1, 1, 0]])

new_weights = train_hopfield_network(patterns)
print(new_weights)

The new weights are

[[0. 1. 1. 0.]
 [1. 0. 1. 0.]
 [1. 1. 0. 0.]
 [0. 0. 0. 0.]]

In essence, neurons 1, 2 and 3 are connected together to achieve the stored pattern.

Let’s look at the evolution of the network with the new, trained weights. The initial state is

new_final_state = hopfeld_network(new_weights, initial_state, 8)

[1 0 0 0]
[0. 1. 1. 0.]
[1. 1. 1. 0.]
[1. 1. 1. 0.]
[1. 1. 1. 0.]
[1. 1. 1. 0.]
[1. 1. 1. 0.]
[1. 1. 1. 0.]
[1. 1. 1. 0.]

Yay, the pattern is stored. A network may store more than one pattern, and which pattern to exhibit depends on the initial state. The system may also oscillate and converge to spurious states that are not among the stored patterns. Larger networks have more interesting dynamics. The dynamics would be more complex with neurons with continuous state values, external inputs and self-connection.

Applications of Hopfield Networks

• Pattern recognition: Hopfield networks can be used to recognize patterns, even when they are noisy or incomplete.
• Image processing: They can be used for tasks like image denoising and image restoration.
• Optimization problems: Hopfield networks can be used to solve optimization problems, such as the traveling salesman problem.

Limitations of Hopfield Networks

• Capacity: Hopfield networks have limited capacity, meaning they can only store a certain number of patterns without errors.
• Spurious states: The network may converge to spurious states that are not among the stored patterns.