TinyML —Recurrent Neural Networks

From mathematical foundations to edge implementation

Social media:

👨🏽‍💻 Github: TinyML | Embedded Machine Learning Hub 👷🏾 Linkedin: Thommas Kevin | LinkedIn 🧑‍🎓Scholar: ‪Thommas Kevin Sales Flores‬ — ‪Google Académico‬ 📽 Youtube: Thommas Kevin — YouTube 👨🏻‍🏫 Research group: Conecta.ai (ufrn.br)

SUMMARY

1 — Introduction 2 — Mathematical Foundations 3 — TinyML Implementation 3.1 — Exemplo 1: RNN Regression 3.2 — Exemplo 2: RNN Binary Classification 3.3 — Exemplo 3: RNN Multiclass Classification 3.4— Exemplo 4: RNN Seq2Seq

Find for “Recurrent Neural Networks” and Give me a Star 🌟 in TinyML | Embedded Machine Learning Hub

1 — Introduction

Recurrent Neural Networks (RNNs) are a class of neural network architectures designed to process data that arrives in ordered sequences. Unlike feedforward networks, which map a fixed-size input vector to an output in a single forward pass, an RNN maintains an internal state, commonly called the hidden state, that is updated at each position in the input sequence. This hidden state acts as a compressed representation of all information the network has encountered up to the current time step, enabling the model to condition its output on the full history of the sequence rather than on a single observation in isolation.

This document develops the mathematical foundations of RNNs, beginning with the limitations of memoryless feedforward architectures and progressing to the derivation of the vanilla RNN state equations, the Long Short-Term Memory (LSTM) gate mechanism, the Gated Recurrent Unit (GRU), the backpropagation through time (BPTT) training algorithm, and the decomposition of sequence-to-vector and sequence-to-sequence computation patterns. The final section explains how recurrent inference can be mapped to efficient embedded C implementations suitable for TinyML deployment on microcontrollers.

1.1—Why Sequential Memory Matters

Consider a microcontroller embedded in an industrial machine that monitors a vibration sensor sampled at 100 Hz. At any given moment, the raw sensor reading conveys limited information: a single amplitude value does not indicate whether the machine is accelerating, decelerating, exhibiting a fault signature, or operating normally. The information required to make that determination is distributed across a window of recent observations.

A feedforward neural network applied to a single sample at each time step has no access to that window. It must either receive a fixed-size block of past observations concatenated into its input, which requires buffering and limits adaptability to variable-length events, or it must discard temporal context entirely and accept reduced accuracy. A recurrent network resolves this tension by maintaining a hidden state that is updated at each time step, effectively implementing a learned and compact summary of the relevant past. This property is the core functional advantage of RNNs for time series classification, anomaly detection, speech processing, natural language understanding, and any task where the meaning of the current observation depends on its history.

Sequential memory in RNNs manifests in two forms that are practically distinct. Short-term dependencies span a small number of time steps and are captured reliably by even the vanilla RNN formulation. Long-term dependencies span tens to hundreds of time steps and are the primary motivation for gated architectures such as the LSTM and the GRU, which introduce explicit mechanisms to control what information is retained, updated, or discarded at each step of the sequence.

1.2 — The Temporal Limitation of Feedforward Neural Networks

A standard fully connected feedforward neural network with L hidden layers computes a stateless mapping from a fixed-dimensional input x∈Rdx​ to an output y^​:

This architecture is memoryless: the output y^​ depends only on the current input x and the fixed weight matrices {W(l)}. There is no mechanism by which an observation at time t−1 can influence the computation at time t unless it is explicitly included in the input vector at time t.

Two workarounds are commonly applied in practice, both of which carry significant limitations. The first is a sliding-window approach, in which the input at each time step is a concatenation of the kk most recent observations: x**twin​=[xt−k+1⊤​,…,x**t⊤​]⊤. This approach fixes the context length at k, requires k times the input dimensionality, and cannot generalize to events whose duration is variable or unknown. The second is a handcrafted feature extraction step, in which domain-specific statistics such as mean, variance, spectral power, and zero-crossing rate are computed over a window and fed to the network. This approach depends on prior knowledge of which features are relevant, discards the raw temporal structure, and is not end-to-end differentiable.

Recurrent Neural Networks eliminate the need for both workarounds by parameterizing a learned state transition that integrates information from the full past sequence into a fixed-size hidden vector at each time step.

1.3 — From Feedforward to RNN

There is a fundamental distinction between stateless feedforward computation and stateful recurrent computation. A feedforward network processes each input independently, with no shared information across time steps. A recurrent network connects each time step to the next through a learned hidden state, creating a directed cycle in the computational graph that gives the architecture its name.

Figure 1 — The transition from feedforward to recurrent computation. A feedforward network (left) processes each input independently with no temporal memory. An RNN (right), shown unrolled over three time steps, passes a hidden state htht​ from each step to the next, enabling the output at time tt to depend on the full input history x1​,…,xt​.

Recurrent Neural Networks occupy the stateful end of the neural computation spectrum. The hidden state **h**t​ serves as a learned, compressed representation of the input history, and the recurrent weight matrix governs how that representation is updated at each new time step. Training the recurrent weights to produce useful hidden state representations for a specific task is the central objective of RNN learning, accomplished by unrolling the recurrence over time and applying gradient-based optimization through the resulting computation graph.

The remainder of this document develops the mathematical framework that enables this approach.

2 — Mathematical Foundations

This section develops the mathematical foundations of Recurrent Neural Networks in full. We begin with the vanilla RNN state equations, which provide the theoretical basis for sequential computation. The section concludes with a step-by-step numerical walkthrough that makes each equation concrete.

2.1 — The Vanilla RNN: State Equations and Computational Graph

The vanilla RNN defines a parameterized state transition function that maps the current input and the previous hidden state to a new hidden state, and a separate output function that maps the hidden state to the network output.

2.1.1 — The Hidden State Update

Given an input sequence x1​,x2​,…,x**T​ with xt​∈Rdx​, the hidden state h**t​∈Rdh​ at time step t is computed as:

where:

• Whh​∈Rdh​×dh​ is the recurrent weight matrix, which governs the transition from the previous hidden state to the current one,
• Wxh​∈Rdh​×dx​ is the input weight matrix, which projects the current input into the hidden state space,
• b**h​∈Rdh​ is the hidden bias vector**,
• tanh is the hyperbolic tangent activation function applied elementwise, which maps the pre-activation to the interval (−1,1).

The initial hidden state h0​ is typically set to the zero vector. This initialization carries no trainable parameters and is the standard default unless the task requires a learned initial state.

2.1.2 — The Output Equation

The output at time step t is computed from the current hidden state through a linear projection followed by a task-appropriate transformation:

2.1.3 — The Unrolled Computational Graph

The recurrence in the hidden state update can be visualized by unrolling the network over T time steps, creating a directed acyclic graph in which each time step tt is a separate node. The unrolled graph uses the same weight matrices {Whh​,Wxh​,Why​} at every step, reflecting the parameter sharing that is the defining property of recurrent computation. This sharing allows the same network to process sequences of arbitrary length without increasing the number of parameters.

The total number of learnable parameters in a vanilla RNN is:

which is independent of the sequence length T.

Figure 2 — The vanilla RNN unrolled over four time steps. Input xt​ is projected into the hidden state ht​ via Wxh​. The hidden state is carried forward via WhhWhh​, and the output y^​t​ is produced via Why​. All three weight matrices are shared across all time steps, enabling the network to process sequences of any length.

2.2 — Architecture: Input, Hidden State, and Output

This section describes the functional role of each architectural component of an RNN, the choice of hidden state dimensionality, and the principal input and output configurations used in practice.

2.2.1 — The Role of the Hidden State

The hidden state h**t​∈Rdh​ is the central quantity in an RNN. It serves simultaneously as the internal memory of the network and as the input to the output layer. At each time step, the hidden state integrates three sources of information: the previous hidden state ht−1​ (historical context), the current input xt​ (new observation), and the bias b**h​ (learned offset). The tanh⁡tanh nonlinearity bounds the hidden state in (−1,1)dh​, which stabilizes the recurrence and prevents the hidden state from growing without bound.

The dimensionality dh​ is the primary capacity hyperparameter of an RNN. A larger dh​ allows the hidden state to encode more complex sequential patterns but increases the parameter count quadratically through Whh​ and the per-step computation linearly in dh​. For TinyML deployment, dh​ is constrained by the available SRAM, which must store **h**t​ at every inference step, and the available compute budget per time step.

2.2.2 — Input Configuration

The input **x**t​∈Rdx​ at each time step can represent any fixed-size feature vector. Common TinyML input types include scalar sensor readings (dx​=1), multi-axis inertial measurement unit data (dx​=3 or dx​=6), and short-time spectral frames from an audio front-end (dx​=40 for mel-filterbank features). The same weight matrix Wxh​ is applied at every time step, so the network can process sequences of any length T with the same parameter set.

2.2.3 — Output Configurations

RNNs support several input-output configurations depending on the task:

• Many-to-one: The output is produced only at the final time step T: y^​=g(Why​h**T​+b**y​). This configuration is used for sequence classification and regression tasks where a single label or value is assigned to the entire input sequence.
• Many-to-many (synchronized): An output is produced at every time step: y^​t​=g(Why​h**t​+b**y​) for t=1,…,T. This configuration is used for sequence labeling tasks such as activity recognition with per-frame labels.
• Many-to-many (encoder-decoder): An encoder RNN reads the full input sequence and compresses it into a context vector, which is passed to a decoder RNN that generates an output sequence of potentially different length. This configuration is used for machine translation and sequence transduction.

For TinyML applications, the many-to-one configuration is the most common, as it produces a single decision per inference window and minimizes output processing overhead.

Figure 3 — The three principal RNN output configurations. Many-to-one (top) produces a single output from the final hidden state, used for sequence classification. Many-to-many synchronized (middle) produces one output per time step, used for sequence labeling. Encoder-decoder (bottom) uses a context vector to bridge two separate RNNs, used for sequence transduction.

2.2.4 — Stacked RNNs

Multiple RNN layers can be stacked to increase model capacity. In a two-layer stacked RNN, the hidden state sequence of the first layer serves as the input sequence to the second layer:

Stacking increases the depth of the representation at the cost of additional parameters and computation. For TinyML targets, a single-layer RNN with moderate dh​ is the standard configuration, and stacking is applied only when the task complexity and available compute budget justify it.

2.5 — The Training Process: Backpropagation Through Time

Training an RNN requires computing the gradient of a loss function with respect to all weight matrices {Whh​,Wxh​,Why​}. Because the same weights appear at every time step of the unrolled graph, the gradient must accumulate contributions from all time steps, a procedure known as Backpropagation Through Time (BPTT).

2.5.1 — The Loss Function

For a many-to-one sequence classification task, the loss is computed from the output at the final time step:

For a many-to-many task with one target per time step, the loss sums contributions across all steps:

Common task losses are mean squared error for regression, binary cross-entropy for binary classification, and categorical cross-entropy for multiclass classification.

2.5.2 — Gradient Flow Through the Recurrence

The gradient of the loss with respect to the hidden state at time step t receives contributions from the output at time tt and from the hidden state at time t+1:

where the factor (1−h**t+12​) is the derivative of the tanh⁡tanh activation at time step t+1t+1. The term Whh⊤​∂L/∂h**t+1​ carries the gradient backward from step t+1t+1 to step tt through the recurrent connection.

The gradient with respect to the recurrent weight matrix is the sum over all time steps:

2.5.3 — The Vanishing and Exploding Gradient Problems

When the product of Whh​ and the ⁡tanh Jacobian (1−h2) has spectral radius less than one, the gradient magnitude decreases exponentially as it propagates from step T back to step 1. This is the vanishing gradient problem: the weights governing early time steps receive negligible gradient signal, preventing the network from learning long-range dependencies. When the spectral radius exceeds one, gradients grow exponentially: the exploding gradient problem. The standard mitigation for exploding gradients is gradient clipping, which rescales the gradient vector to a maximum norm:

where c is the clipping threshold (typically c∈[1,5]).

Figure 4 — Backpropagation through time for a five-step vanilla RNN. The forward pass (teal arrows) computes hidden states left to right. The backward pass (red arrows) propagates gradients right to left through repeated multiplication by Whh⊤​ and the tanh⁡tanh Jacobian. The progressive attenuation of backward arrows illustrates the vanishing gradient problem that motivates gated architectures.

2.5.4 — Truncated BPTT

Full BPTT over sequences of length T requires storing the complete hidden state history {h1​,…,h**T​} in memory during the forward pass. For long sequences on resource-constrained hardware, this is prohibitive. Truncated BPTT* divides the sequence into non-overlapping segments of length k, runs a full forward and backward pass within each segment, and carries the final hidden state forward between segments as a fixed initialization. This trades some gradient accuracy for a constant memory footprint of O(k⋅dh​) regardless of the total sequence length T*.

2.6 — Sequence Representation and Many-to-One, Many-to-Many Configurations

This section analyzes how information is accumulated across the time dimension of an RNN and describes the relationship between hidden state dynamics and downstream task performance.

2.6.1 — Information Accumulation in the Hidden State

At each time step t, the hidden state hth**t​ is a nonlinear function of the entire input prefix x1​,…,xt​. The recurrent weight matrix Whh​ determines how much of the previous hidden state is retained versus overwritten at each step. For the vanilla RNN, the effective influence of an early input xτ​ on the final hidden state h**T​ is proportional to (Whh​)T−τ modulated by the tanh⁡tanh Jacobians at each intervening step. For large T−τ, this influence decays with the spectral radius of WhhWhh​, which is the dynamical origin of the vanishing gradient problem and its forward-pass counterpart: the vanishing influence of distant inputs on the current hidden state.

2.6.2 — Sequence Length and Hidden State Dimensionality

The hidden state dimension dhdh​ controls the information capacity of the RNN memory. In practice, the choice of dh​ is guided by the following heuristics for TinyML:

• dh​∈[8,32]: suitable for simple univariate time series classification with short sequences (T≤50),
• dh​∈[32,128]: suitable for multivariate sensor fusion or moderate-length sequences (T≤200),
• dh​>128: generally exceeds the SRAM budget of most microcontrollers without aggressive quantization.

2.6.3 — Bidirectional RNNs

A bidirectional RNN processes the input sequence in both the forward direction (t=1,…,T) and the backward direction (t=T,…,1), producing two hidden state sequences h→th**t​ and h←th**t​ that are concatenated at each step:

Bidirectional processing allows the network to use both past and future context at each position. However, it requires the full input sequence to be available before any output is produced, which is incompatible with real-time streaming inference on embedded devices. For TinyML applications, unidirectional RNNs that process the sequence causally are the standard choice.

Figure 5 — Unidirectional versus bidirectional RNN. The unidirectional RNN (top) processes the sequence causally and can produce outputs in real time, making it suitable for embedded streaming inference. The bidirectional RNN (bottom) combines forward and backward hidden states, requiring the complete sequence before any output is produced, which is incompatible with real-time TinyML deployment.

2.6.4 — Out-of-Distribution Detection via Hidden State Statistics

RNNs do not provide explicit uncertainty estimates by default. However, the hidden state norm ∥**h**T​∥ and the maximum absolute output logit can serve as proxy confidence measures. Inputs that lie far from the training distribution tend to drive the hidden state to regions of activation space that were rarely visited during training, producing norms or logits that deviate from the distribution observed on the validation set. A threshold applied to these statistics provides a low-cost out-of-distribution detector that requires no additional parameters and adds only a norm computation to the inference loop:

where hlow​ and hhigh​ are calibrated on a held-out validation set.

Figure 6 — Out-of-distribution detection using hidden state norm statistics. In-distribution inputs (blue) produce hidden state norms within the calibrated band [hlow​,hhigh​] (dashed lines). Out-of-distribution inputs (orange) fall outside this band, enabling a parameter-free OOD flag directly from the final hidden state norm.

3 — TinyML Implementation

With this example you can implement the machine learning algorithm in ESP32, Arduino, Arduino Portenta H7 with Vision Shield, Raspberry and other different microcontrollers or IoT devices.

3.1 — Clone repository (🌟Give me a Star)

Find for “Recurrent Neural Networks” in TinyML | Embedded Machine Learning Hub

3.2 — Install the libraries listed in the requirements.txt file

!pip install -r requirements.txt

3.3 — Importing Libraries

import sys, os
sys.path.append('34_RNN')   # adjust if running from a different directory

import torch
import torch.optim as optim
import numpy as np
from sklearn.datasets import make_moons, make_blobs
from sklearn.model_selection import train_test_split

from model  import RNNModel
from layers import get_activation
from losses import compute_loss, LOSS_NAMES
from utils  import (
    export_to_json,
    train_model,
    plot_training_history,
    plot_regression_uncertainty,
    plot_decision_boundary,
    plot_sequence_prediction,
)
from cpp_generator import generate_ino

print('Available loss functions:', LOSS_NAMES)

os.makedirs('json_model',   exist_ok=True)
os.makedirs('arduino_code', exist_ok=True)

3.4 — Example 1: Regression

def train_regression():
    print('=== 1D Regression (Elman RNN, Huber loss) ===')
    torch.manual_seed(42)

    # ---- Data ----
    x1 = torch.linspace(-4, -1.5, 60)
    x2 = torch.linspace( 1.5,  4, 60)
    X_train_1d = torch.cat([x1, x2])                  # (120,)
    y_train    = torch.sin(X_train_1d) + 0.1 * torch.randn_like(X_train_1d)
    X_test_1d  = torch.linspace(-5, 5, 200)
    y_test     = torch.sin(X_test_1d)

    # RNN input: (batch, seq_len=1, input_size=1)
    X_train = X_train_1d.view(-1, 1, 1)
    X_test  = X_test_1d.view(-1, 1, 1)

    print('X_train shape:', X_train.shape, '  y_train shape:', y_train.shape)
    print('Sample X_train[0]:', X_train[0].item(), '  y_train[0]:', y_train[0].item())

    # ---- Architecture ----
    model = RNNModel(
        recurrent_layers=[
            {'input_size': 1,  'hidden_size': 64, 'activation': 'tanh'},
            {'input_size': 64, 'hidden_size': 64, 'activation': 'tanh'},
        ],
        dense_layers=[
            {'out_features': 32, 'activation': 'relu'},
            {'out_features': 1,  'activation': 'linear'},
        ],
    )

    optimizer = optim.Adam(model.parameters(), lr=0.01)

    # ---- Train ----
    # Change loss_name to 'mse', 'mae', 'rmse', or 'huber'
    history = train_model(
        model, X_train, y_train, optimizer,
        loss_name='huber', epochs=1500, print_every=300,
        delta=0.5,   # Huber delta (only used by huber loss)
    )
    plot_training_history(history, loss_name='huber')

    # ---- Visualize ----
    plot_regression_uncertainty(
        model, X_train_1d.view(-1, 1), y_train.view(-1, 1),
        X_test_1d.view(-1, 1), y_test.view(-1, 1),
        title='Elman RNN — 1D Regression (notice the gap uncertainty)',
        seq_len=1,
    )

    # ---- Export ----
    export_to_json(model, 'json_model/regression_model.json')
    generate_ino(
        'json_model/regression_model.json',
        'arduino_code/regression_ino',
        board='esp32', task='regression',
    )
    return model

model_reg = train_regression()

3.4.1 — Deploy in Microcontroller

/*
 * RNN Model -- Arduino verification sketch
 * Generated automatically -- do not edit the weights.
 *
 * VERIFICATION GUIDE
 * -------------------
 * Input  seq_len    : 4
 * Input  input_size : 1
 * Input values (same order as the flat array below):
 *   t=0: [1.76405239]
 *   t=1: [0.40015721]
 *   t=2: [0.97873801]
 *   t=3: [2.24089313]
 *
 * Expected value    : 0.86626846
 *
 * Upload this sketch, open Serial Monitor at 115200 baud,
 * and confirm the printed value matches the expected value
 * above to at least 5 decimal places.
 *
 * Acceptable tolerance: +/-0.00002  (float32 rounding)
 */

#include "RNNModel.h"

RNNModel model;

void setup() {
    Serial.begin(115200);
    while (!Serial);  // Wait for Serial on native-USB boards

    const int SEQ_LEN    = 4;
    const int INPUT_SIZE = 1;

    // Verification input (auto-generated by Python exporter)
    float input[SEQ_LEN * INPUT_SIZE] = {
        1.76405239f, 0.40015721f, 0.97873801f, 2.24089313f
    };

    float output = model.predict(input, SEQ_LEN);

    // Expected value  : 0.86626846
    Serial.print("Predicted value  : "); Serial.println(output, 8);
}

void loop() {
    // Nothing to do here
}

3.5 — Example 2: Binary Classification

def train_binary():
    print('=== Binary Classification — Moons (Elman RNN, BCE loss) ===')
    torch.manual_seed(42)

    # ---- Data ----
    X, y = make_moons(n_samples=300, noise=0.15, random_state=42)
    X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=42)

    print('X_train[0]:', X_tr[0], '  y_train[0]:', y_tr[0])

    # (batch, seq_len=1, input_size=2)
    X_t = torch.FloatTensor(X_tr).unsqueeze(1)
    y_t = torch.FloatTensor(y_tr).view(-1, 1)

    # ---- Architecture ----
    model = RNNModel(
        recurrent_layers=[
            {'input_size': 2,  'hidden_size': 32, 'activation': 'tanh'},
            {'input_size': 32, 'hidden_size': 32, 'activation': 'tanh'},
        ],
        dense_layers=[
            {'out_features': 16, 'activation': 'relu'},
            {'out_features': 1,  'activation': 'linear'},   # logit for BCE
        ],
    )

    optimizer = optim.Adam(model.parameters(), lr=0.01)

    # ---- Train ----
    history = train_model(
        model, X_t, y_t, optimizer,
        loss_name='bce', epochs=1000, print_every=200,
    )
    plot_training_history(history, loss_name='bce')

    # ---- Visualize ----
    plot_decision_boundary(
        model, X, y,
        title='Binary Classification with Elman RNN',
        task='binary', seq_len=1,
    )

    # ---- Export ----
    export_to_json(model, 'json_model/binary_model.json')
    generate_ino(
        'json_model/binary_model.json',
        'arduino_code/binary_ino',
        board='esp32', task='binary',
    )
    return model

model_bin = train_binary()

3.5.1 — Deploy in Microcontroller

/*
 * RNN Model -- Arduino verification sketch
 * Generated automatically -- do not edit the weights.
 *
 * VERIFICATION GUIDE
 * -------------------
 * Input  seq_len    : 4
 * Input  input_size : 2
 * Input values (same order as the flat array below):
 *   t=0: [1.76405239, 0.40015721]
 *   t=1: [0.97873801, 2.24089313]
 *   t=2: [1.86755800, -0.97727787]
 *   t=3: [0.95008844, -0.15135720]
 *
 * Expected logit    : 10.61098385
 * Expected class    : 1
 *
 * Upload this sketch, open Serial Monitor at 115200 baud,
 * and confirm the printed value matches the expected value
 * above to at least 5 decimal places.
 *
 * Acceptable tolerance: +/-0.00002  (float32 rounding)
 */

#include "RNNModel.h"

RNNModel model;

void setup() {
    Serial.begin(115200);
    while (!Serial);  // Wait for Serial on native-USB boards

    const int SEQ_LEN    = 4;
    const int INPUT_SIZE = 2;

    // Verification input (auto-generated by Python exporter)
    float input[SEQ_LEN * INPUT_SIZE] = {
        1.76405239f, 0.40015721f, 0.97873801f, 2.24089313f, 1.86755800f, -0.97727787f, 0.95008844f, -0.15135720f
    };

    float output = model.predict(input, SEQ_LEN);

    // Expected logit  : 10.61098385
    // Expected class  : 1
    Serial.print("Predicted logit  : "); Serial.println(output, 8);
    Serial.print("Predicted class  : "); Serial.println(output > 0.0f ? 1 : 0);
}

void loop() {
    // Nothing to do here
}

3.6 — Example 3: Multiclass Classification

def train_multiclass():
    print('=== Multiclass Classification — Blobs (Elman RNN, SCCE loss) ===')
    torch.manual_seed(42)

    # ---- Data ----
    X, y = make_blobs(n_samples=400, centers=3, cluster_std=1.5, random_state=42)
    X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=42)

    print('X_train[0]:', X_tr[0], '  y_train[0]:', y_tr[0])

    X_t = torch.FloatTensor(X_tr).unsqueeze(1)   # (batch, 1, 2)
    y_t = torch.LongTensor(y_tr)                  # integer labels

    # ---- Architecture ----
    model = RNNModel(
        recurrent_layers=[
            {'input_size': 2, 'hidden_size': 32, 'activation': 'tanh'},
        ],
        dense_layers=[
            {'out_features': 16, 'activation': 'relu'},
            {'out_features': 3,  'activation': 'linear'},   # 3 classes
        ],
    )

    optimizer = optim.Adam(model.parameters(), lr=0.01)

    # ---- Train ----
    history = train_model(
        model, X_t, y_t, optimizer,
        loss_name='scce', epochs=1000, print_every=200,
    )
    plot_training_history(history, loss_name='scce')

    # ---- Visualize ----
    plot_decision_boundary(
        model, X, y,
        title='Multiclass Classification (3 classes) with Elman RNN',
        task='multiclass', seq_len=1,
    )

    # ---- Export ----
    export_to_json(model, 'json_model/multiclass_model.json')
    generate_ino(
        'json_model/multiclass_model.json',
        'arduino_code/multiclass_ino',
        board='esp32', task='multiclass',
    )
    return model

model_multi = train_multiclass()

3.6.1 — Deploy in Microcontroller

/*
 * RNN Model -- Arduino verification sketch
 * Generated automatically -- do not edit the weights.
 *
 * VERIFICATION GUIDE
 * -------------------
 * Input  seq_len    : 4
 * Input  input_size : 2
 * Input values (same order as the flat array below):
 *   t=0: [1.76405239, 0.40015721]
 *   t=1: [0.97873801, 2.24089313]
 *   t=2: [1.86755800, -0.97727787]
 *   t=3: [0.95008844, -0.15135720]
 *
 * Expected class    : -3
 *
 * Upload this sketch, open Serial Monitor at 115200 baud,
 * and confirm the printed value matches the expected value
 * above to at least 5 decimal places.
 *
 * Acceptable tolerance: +/-0.00002  (float32 rounding)
 */

#include "RNNModel.h"

RNNModel model;

void setup() {
    Serial.begin(115200);
    while (!Serial);  // Wait for Serial on native-USB boards

    const int SEQ_LEN    = 4;
    const int INPUT_SIZE = 2;

    // Verification input (auto-generated by Python exporter)
    float input[SEQ_LEN * INPUT_SIZE] = {
        1.76405239f, 0.40015721f, 0.97873801f, 2.24089313f, 1.86755800f, -0.97727787f, 0.95008844f, -0.15135720f
    };

    float output = model.predict(input, SEQ_LEN);

    // Expected class  : -3
    Serial.print("Predicted class  : "); Serial.println((int)output);
}

void loop() {
    // Nothing to do here
}

3.7— Example 4: Seq2Seq

def train_seq2seq():
    print('=== Seq-to-Seq Sine Prediction (Elman RNN, MSE loss) ===')
    torch.manual_seed(0)

    T      = 30          # sequence length
    N      = 500         # number of sequences
    noise  = 0.05

    # Each sequence: sin(t) for t in [start, start+T]
    starts = torch.rand(N) * 2 * np.pi
    t_vals = torch.arange(T).float() / T * 2 * np.pi   # (T,)

    X_np = np.stack(
        [np.sin(starts[i].item() + t_vals.numpy()) + noise * np.random.randn(T)
         for i in range(N)]
    )   # (N, T)
    Y_np = np.stack(
        [np.sin(starts[i].item() + t_vals.numpy()) for i in range(N)]
    )   # (N, T) — clean target

    X = torch.FloatTensor(X_np).unsqueeze(-1)   # (N, T, 1)
    Y = torch.FloatTensor(Y_np)                  # (N, T)

    print('X shape:', X.shape, '  Y shape:', Y.shape)

    # ---- Architecture ----
    model = RNNModel(
        recurrent_layers=[
            {'input_size': 1,  'hidden_size': 64, 'activation': 'tanh'},
            {'input_size': 64, 'hidden_size': 32, 'activation': 'tanh'},
        ],
        dense_layers=[
            {'out_features': 1, 'activation': 'linear'},
        ],
    )

    optimizer = optim.Adam(model.parameters(), lr=5e-3)

    # ---- Seq-to-seq training loop ----
    history = []
    epochs  = 500
    for epoch in range(1, epochs + 1):
        model.train()
        optimizer.zero_grad()
        # forward_sequence returns (N, T, 1)
        out  = model.forward_sequence(X).squeeze(-1)   # (N, T)
        loss = compute_loss('mse', out, Y)
        loss.backward()
        optimizer.step()
        if epoch % 100 == 0 or epoch == 1:
            print(f'  Epoch {epoch:>4}/{epochs}  |  MSE = {loss.item():.6f}')
            history.append((epoch, loss.item()))

    plot_training_history(history, loss_name='mse')

    # ---- Visualize predictions on 5 samples ----
    plot_sequence_prediction(
        model, X[:5], Y[:5],
        title='Elman RNN — Seq-to-Seq Sine Denoising',
        n_display=5,
    )

    # ---- Export (only last-step prediction → regression) ----
    export_to_json(model, 'json_model/seq2seq_model.json')
    generate_ino(
        'json_model/seq2seq_model.json',
        'arduino_code/seq2seq_ino',
        board='esp32', task='regression',
    )
    return model

model_s2s = train_seq2seq()

3.7.1 — Deploy in Microcontroller

/*
 * RNN Model -- Arduino verification sketch
 * Generated automatically -- do not edit the weights.
 *
 * VERIFICATION GUIDE
 * -------------------
 * Input  seq_len    : 4
 * Input  input_size : 1
 * Input values (same order as the flat array below):
 *   t=0: [1.76405239]
 *   t=1: [0.40015721]
 *   t=2: [0.97873801]
 *   t=3: [2.24089313]
 *
 * Expected value    : 1.67224765
 *
 * Upload this sketch, open Serial Monitor at 115200 baud,
 * and confirm the printed value matches the expected value
 * above to at least 5 decimal places.
 *
 * Acceptable tolerance: +/-0.00002  (float32 rounding)
 */

#include "RNNModel.h"

RNNModel model;

void setup() {
    Serial.begin(115200);
    while (!Serial);  // Wait for Serial on native-USB boards

    const int SEQ_LEN    = 4;
    const int INPUT_SIZE = 1;

    // Verification input (auto-generated by Python exporter)
    float input[SEQ_LEN * INPUT_SIZE] = {
        1.76405239f, 0.40015721f, 0.97873801f, 2.24089313f
    };

    float output = model.predict(input, SEQ_LEN);

    // Expected value  : 1.67224765
    Serial.print("Predicted value  : "); Serial.println(output, 8);
}

void loop() {
    // Nothing to do here
}

References

[1] Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning Representations by Back-Propagating Errors. Nature, 323(6088), 533–536.

[2] Werbos, P. J. (1990). Backpropagation Through Time: What It Does and How to Do It. Proceedings of the IEEE, 78(10), 1550–1560.

[3] Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014). Learning Phrase Representations Using RNN Encoder-Decoder for Statistical Machine Translation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 1724–1734.

[4] Pascanu, R., Mikolov, T., & Bengio, Y. (2013). On the Difficulty of Training Recurrent Neural Networks. Proceedings of the 30th International Conference on Machine Learning (ICML), 28, 1310–1318.

[5] Graves, A. (2012). Supervised Sequence Labelling with Recurrent Neural Networks. Springer.

[6] Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014). Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. NIPS 2014 Workshop on Deep Learning.

[7] Schuster, M., & Paliwal, K. K. (1997). Bidirectional Recurrent Neural Networks. IEEE Transactions on Signal Processing, 45(11), 2673–2681.

[8] Greff, K., Srivastava, R. K., Koutnik, J., Steunebrink, B. R., & Schmidhuber, J. (2017). LSTM: A Search Space Odyssey. IEEE Transactions on Neural Networks and Learning Systems, 28(10), 2222–2232.

[9] Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press.

[10] Karpathy, A., Johnson, J., & Fei-Fei, L. (2015). Visualizing and Understanding Recurrent Networks. ICLR 2016 Workshop Track.

[11] Lane, N. D., Bhattacharya, S., Georgiev, P., Forlivesi, C., & Kawsar, F. (2015). An Early Resource Characterization of Deep Learning on Wearables, Smartphones and Internet-of-Things Devices. Proceedings of the 2015 International Workshop on Internet of Things towards Applications (IoT-App), 7–12.