Differentiable Time: When Neural Networks Learn They Are Finished

Part One: The Mechanism

The Gating Paradox

Skip connections are among the most important innovations in deep learning. They let information bypass layers, enabling deeper networks and faster inference. But they raise a fundamental question: when should a network use the skip path versus the deep path?

Current approaches use fixed rules or learned gates trained separately from the main network. This creates a paradox. If you gate early in training, the layer never receives gradient signal and never learns its function. If you wait until after training to introduce gating, you need some external decision about when learning is complete. That decision is not differentiable. It sits outside the optimization process.

What if the network could feel the cost of computation and learn when to skip as part of the same optimization that teaches it what to compute?

From Artificial Organisms to Metabolic Pressure

In earlier work, I introduced the Artificial Organism framework for neural architecture design. The core idea is that networks can be decomposed into functional organs with local objectives, coordinated by global objectives, and connected through explicit communication channels.

One principle from that framework is particularly relevant here: Learning-Execution Asymmetry. Once a function is learned, it can often be executed by a much simpler mechanism. A layer that required millions of gradient updates to discover its transformation may need only a fraction of that computation to apply it.

Recent work by Bulla et al. demonstrates that AO principles can achieve remarkable results in practice. Their NeuroBulla framework is likely the fastest model running on consumer hardware, achieving 95ms inference on a 16-core CPU while maintaining competitive accuracy. Their work validates the framework.

The question this paper addresses is complementary: can we make the gating mechanism itself learnable, so the network discovers when to skip rather than relying on fixed heuristics?

The answer requires treating time as a differentiable resource.

Time as Metabolic Cost

In biological organisms, computation has a cost. Neurons consume energy. Longer processing paths consume more. This creates evolutionary pressure toward efficiency: if a reflex can handle a situation, the organism does not engage deliberative reasoning.

We can introduce the same pressure into neural networks by defining a metabolic cost function for each layer. The key is measuring whether a layer is doing meaningful work.

A layer’s activity can be measured by its Jacobian sensitivity: how much does the output change when the input changes slightly? A layer with high sensitivity is actively transforming its input in ways that depend on the specific input. A layer with low sensitivity is applying a nearly constant transformation.

We define an Energy-Latency module that computes cost based on layer activity:

cost = (1 — gate) base_cost + gate skip_cost

Here, gate is a value between 0 and 1 derived from comparing the layer’s Jacobian sensitivity to a learned threshold. When the layer is doing heavy transformation, cost approaches base_cost. When the layer is being bypassed, cost drops to skip_cost.

This cost joins the loss function:

total_loss = task_loss + lambda * metabolic_cost

The network now optimizes for accuracy and efficiency simultaneously.

Measuring Sensitivity with Hutchinson’s Estimator

Computing the full Jacobian matrix for each layer would be prohibitively expensive. For a layer with input dimension d, the Jacobian has d squared entries. We need a cheaper approximation.

Hutchinson’s estimator provides one. The Frobenius norm of a matrix can be approximated stochastically using random vectors. If v is a random vector with entries drawn uniformly from negative one and positive one, then the expected value of the squared norm of Jv equals the squared Frobenius norm of J.

In neural network terms, Jv is a Jacobian-vector product, which automatic differentiation computes efficiently. We draw a small number of random vectors, compute the JVP for each, and average the squared norms to estimate layer sensitivity.

This estimate stays in the computational graph. The optimizer can see how changing layer weights affects sensitivity, which affects the gate, which affects metabolic cost. The entire system is differentiable.

One note on this approximation: the Frobenius norm measures average sensitivity across all directions. A layer could have low Frobenius norm but high spectral norm, meaning it is highly sensitive in one particular direction while being stable in others. For most layers, isotropic sensitivity is a reasonable proxy for completion, but layers with concentrated directional sensitivity may require additional analysis.

Absolute Thresholds with Regularization

The gate should respond to whether sensitivity is below an absolute threshold, not whether it is below a running average.

Why does this matter? Consider a layer that consistently has moderate sensitivity throughout training. A relative gate would never activate because the layer is always near its own average. But moderate sensitivity might still be low enough that gating would be appropriate.

The solution is a learnable threshold parameter. The network discovers what stable enough means for each layer.

However, learnable thresholds introduce a risk: the network could game the system by raising thresholds rather than actually stabilizing its layers. To prevent this, we anchor each threshold to its initial value and add a small regularization penalty for drift. This allows thresholds to adapt while preventing runaway manipulation.

The anchor should be understood as a training scaffold. It prevents threshold drift during optimization but does not represent a privileged notion of what stability means. Future work might anneal this constraint or replace it with a population-level prior across layers.

We also introduce a learnable sharpness parameter that controls how binary the gate becomes. Early in training, soft gates allow gradients to flow through both paths. As training progresses, the network can sharpen its gates to make cleaner decisions.

Observing Emergence

When training a network with differentiable time, you can track gate values across layers throughout training. The pattern that emerges is consistent and interpretable.

Early layers rarely gate. They transform raw input into the network’s internal representation. This transformation remains input-dependent even after convergence.

Middle layers often show the most dramatic transitions. They learn transformations that become stable once the network understands the task.

Late layers vary depending on the task. For classification, late layers often need to remain active to map features to class boundaries. For tasks with more redundant outputs, late layers may also transition.

Watching gates rise during training is watching the network discover its own structure.

Part Two: What It Means

Retirement, Not Skipping

Everything described so far is mechanism. Here is what the mechanism reveals.

When a layer’s gate rises, we say it is being skipped. But that language obscures what is actually happening. The layer is not being skipped because it is unimportant. It is being skipped because it has finished.

A layer with high Jacobian sensitivity is doing input-dependent work. Its output depends on the specific input in ways that matter for the task. This is the signature of active learning: the layer is still discovering how to transform its inputs.

A layer with low Jacobian sensitivity has stabilized. Its transformation has become consistent enough that small input variations produce small output variations. The layer has converged to a function it will apply reliably.

The transition from high to low sensitivity is not about the layer becoming useless. It is about the layer becoming complete. The representation it produces is no longer being refined. It is ready for use.

This reframes what the skip connection actually does. It is not a bypass for unimportant computation. It is a recognition that computation has finished. The network is not avoiding work. It is acknowledging that the work is done.

We might call this representational retirement. The layer has learned its function. Now it can step aside. The skip path is not a shortcut. It is a graduation.

Why Sensitivity Is the Right Signal

Other gating approaches use confidence, entropy, or auxiliary classifiers. Those are output-level metrics. They ask: how certain is the network about its prediction?

Sensitivity asks a different question: is this layer still doing input-dependent transformation?

That is the right question, because it targets the learning process itself rather than its downstream effects. Confidence tells you about the network’s belief. Sensitivity tells you about the network’s activity. A layer can contribute to a confident prediction while still being in flux. A layer can also be completely stable while the network remains uncertain.

Sensitivity measures whether the layer has finished its work, not whether the work was successful. The task loss handles success. Sensitivity handles completion.

The Sensitivity-Importance Mismatch as Discovery

There is a case worth examining: when a layer has low sensitivity but high importance.

This happens when a layer applies a transformation that is critical but frozen. The function no longer varies with input, but downstream computation depends on it heavily. Skipping such a layer hurts task performance even though its Jacobian is small.

This is not a failure of the method. It is a discovery.

These layers have crystallized into fixed operators. They do the same thing regardless of input, but that thing matters. They are candidates for architectural optimization: folding into adjacent layers, compiling into simpler forms, or extracting as reusable components.

The mismatch between sensitivity and importance is a diagnostic signal. It reveals structure that the network has learned but no longer needs to compute dynamically. In biological terms, these are reflexes that have become hardwired.

Detecting these layers is valuable. They represent opportunities for permanent simplification rather than dynamic gating.

Lambda as Cognitive Stance

The hyperparameter lambda controls metabolic pressure. But calling it a hyperparameter undersells what it represents.

Lambda is a prior over epistemic effort.

Low lambda says: assume the world is novel, think carefully, pay the cost of deep computation because you might need it.

High lambda says: assume the world is familiar, react quickly, trust that your learned representations are sufficient.

This connects to curriculum, distribution shift, and continual learning. A network trained under high lambda develops strong reflexes but may struggle with novel inputs. A network trained under low lambda remains flexible but never fully exploits its learned structure.

This also connects to human cognition. Under stress, time pressure, or cognitive load, humans shift toward reflexive processing. Under safety and leisure, humans engage more deliberative reasoning. Lambda is the computational analog of that shift.

The ability to adjust lambda at deployment is not just a speed-accuracy tradeoff. It is a way of telling the network how much to trust its own learning.

Connection to Engrams and Retrieval

This work connects to a broader thesis.

In related work on learned compression, we found that engrams do not inject information into a model. They retrieve it. The engram serves as a cue that activates patterns the model already learned. The heavy lifting happened during training. Execution is pattern matching.

Differentiable time reveals the same structure from a different angle.

Engrams decide what to retrieve. Differentiable time decides whether to compute at all. Both are mechanisms for recognizing when learning has already provided the answer.

Together, they point toward architectures that learn deeply once, then execute cheaply forever, routing around their own complexity when experience permits.

When Is Computation Finished?

The deepest contribution of this work is not a gating mechanism. It is a definition.

Computation is finished when further processing would not change the representation in input-dependent ways.

That definition is precise, measurable, and differentiable. It does not require external stopping criteria. It does not require distinguishing training from inference. It emerges from the same optimization that teaches the network its task.

Most neural network training treats learning and inference as separate phases. You train until some stopping criterion, then you deploy. The decision of when learning is done happens outside the network.

Differentiable time brings that decision inside. The network discovers, layer by layer, when it has learned enough. There is no phase transition. There is only a continuous process where layers retire as their representations stabilize.

This is perhaps the real insight: differentiable time lets the network learn not just functions, but when further computation is epistemically pointless.

Most of intelligence may be learning when not to think.

Open Questions

Several questions remain.

Can lambda itself be learned? An adaptive metabolic pressure that responds to input difficulty could produce networks that think harder when needed and coast when possible.

How does this interact with attention? Attention already implements selective computation. Combining metabolic pressure with attention gating might yield transformers that learn which heads have retired.

What happens in continual learning? When new tasks require reactivating dormant layers, how should the network respond? The threshold regularization provides a path back toward active computation, but the dynamics need study.

Can we detect frozen critical operators automatically? Layers where low sensitivity coincides with high skip-loss-delta are architecturally interesting. Systematic detection could guide architecture search and model compression.

What is the relationship between representational retirement and generalization? Do networks that retire layers cleanly generalize better, or does premature retirement indicate overfitting to training distribution?

Conclusion

The gating paradox arises because we treat the decision of when to skip as external to optimization. By making time a differentiable cost, we bring that decision inside the learning process.

But the mechanism is not the point.

The point is that neural networks can learn when they are finished. Not finished with training. Finished with a representation. Finished with a layer. Finished with a particular transformation.

Skip connections become what they should have been all along: not architectural shortcuts, but learned recognitions that the work is done.

The network learns what to compute. It also learns when computation has become epistemically pointless. Those are the same optimization, the same gradient, the same loss.

That is what differentiable time means.

Appendix: Implementation

The following PyTorch code implements the mechanism described in this paper. It includes Hutchinson’s estimator for Jacobian sensitivity, learnable thresholds with regularization, and a complete training loop.

DifferentiableTimeBlock

This module wraps any layer and adds learned gating based on Jacobian sensitivity.

import torch import torch.nn as nn import torch.nn.functional as F

class DifferentiableTimeBlock(nn.Module):

def init(self, layer, base_cost=1.0, skip_cost=0.01, num_hutchinson_samples=3, gate_threshold_init=0.5, threshold_lr_scale=0.1): super().init() self.layer = layer self.base_cost = base_cost self.skip_cost = skip_cost self.num_samples = num_hutchinson_samples self.threshold_lr_scale = threshold_lr_scale

self.gate_threshold = nn.Parameter(torch.tensor(gate_threshold_init)) self.register_buffer(‘threshold_anchor’, torch.tensor(gate_threshold_init))

self.gate_sharpness = nn.Parameter(torch.tensor(3.0))

self.register_buffer(‘running_sensitivity’, torch.tensor(1.0)) self.register_buffer(‘sensitivity_std’, torch.tensor(0.1)) self.momentum = 0.95

self.current_cost = None self.current_gate = None self.current_sensitivity = None

def estimate_jacobian_norm_squared(self, x, y): batch_size = x.shape[0] estimates = []

for _ in range(self.num_samples): v = torch.randint_like(x, low=0, high=2).float() * 2–1

jvp = torch.autograd.grad( outputs=y, inputs=x, grad_outputs=v, retain_graph=True, create_graph=True )[0]

estimate = (jvp ** 2).sum() / batch_size estimates.append(estimate)

return torch.stack(estimates).mean()

def get_threshold_regularization(self): return (self.gate_threshold — self.threshold_anchor) ** 2

def forward(self, x): orig_requires_grad = x.requires_grad

if self.training: x = x.detach().requiresgrad(True)

deep_out = self.layer(x)

if self.training: sensitivity_sq = self.estimate_jacobian_norm_squared(x, deep_out) sensitivity = torch.sqrt(sensitivity_sq + 1e-8)

with torch.no_grad(): delta = sensitivity — self.running_sensitivity self.running_sensitivity += (1 — self.momentum) delta self.sensitivity_std += (1 — self.momentum) (delta.abs() — self.sensitivity_std) else: sensitivity = self.running_sensitivity

normalized = (self.gate_threshold — sensitivity) / (self.sensitivity_std + 1e-8) gate = torch.sigmoid(self.gate_sharpness * normalized)

with torch.no_grad(): self.gatesharpness.data.clamp(1.0, 10.0) self.gatethreshold.data.clamp(0.01, 2.0)

output = (1 — gate) deep_out + gate x

cost = (1 — gate) self.base_cost + gate self.skip_cost

self.current_cost = cost self.current_gate = gate.detach() self.current_sensitivity = sensitivity.detach()

if not orig_requires_grad and not self.training: output = output.detach()

return output

OrganismNetwork

A complete network built from DifferentiableTimeBlocks.

class OrganismNetwork(nn.Module):

def init(self, input_dim, hidden_dim, output_dim, num_layers=4): super().init()

self.input_proj = nn.Linear(input_dim, hidden_dim)

self.blocks = nn.ModuleList() for i in range(num_layers): layer = nn.Sequential( nn.Linear(hidden_dim, hidden_dim), nn.GELU(), nn.Linear(hidden_dim, hidden_dim), ) skip_cost = 0.01 (1 + i 0.5) self.blocks.append(DifferentiableTimeBlock(layer, skip_cost=skip_cost))

self.output_proj = nn.Linear(hidden_dim, output_dim)

def forward(self, x): x = self.input_proj(x) for block in self.blocks: x = block(x) return self.output_proj(x)

def get_metabolic_cost(self): return sum(block.current_cost for block in self.blocks)

def get_threshold_regularization(self): return sum(block.get_threshold_regularization() for block in self.blocks)

def get_gatesummary(self): return { f’layer{i}’: { ‘gate’: block.current_gate.item(), ‘sensitivity’: block.current_sensitivity.item(), ‘threshold’: block.gate_threshold.item() } for i, block in enumerate(self.blocks) }

Training Loop

def train_step(model, optimizer, data, target, lambda_metabolic=0.02, lambda_threshold=0.01):

model.train() optimizer.zero_grad()

output = model(data)

task_loss = F.cross_entropy(output, target) metabolic_cost = model.get_metabolic_cost() threshold_reg = model.get_threshold_regularization()

total_loss = (task_loss + lambda_metabolic metabolic_cost + lambda_threshold threshold_reg)

total_loss.backward()

for block in model.blocks: if block.gate_threshold.grad is not None: block.gate_threshold.grad *= block.threshold_lr_scale

optimizer.step()

return { ‘task_loss’: task_loss.item(), ‘metabolic_cost’: metabolic_cost.item(), ‘threshold_reg’: threshold_reg.item(), ‘gates’: model.get_gate_summary() }

Skip Importance Estimation

For detecting frozen critical operators where sensitivity is low but importance is high.

def estimate_skip_importance(model, data, target, layer_idx):

model.eval()

with torch.no_grad(): output_normal = model(data) loss_normal = F.cross_entropy(output_normal, target)

def force_skip_hook(module, input, output): return input[0]

handle = model.blocks[layer_idx].register_forward_hook(force_skip_hook)

with torch.no_grad(): output_skipped = model(data) loss_skipped = F.cross_entropy(output_skipped, target)

handle.remove()

return (loss_skipped — loss_normal).item()