Architectural Evolution in Large Language Models: A Deep Dive into Jamba’s Hybrid Transformer-Mamba Design

Below is an unedited Gemini Deep Research review detailing Jamba combines transformers and Mamba architectures for some important benefits.

The Transformer Paradigm and Its Inherent Scaling Challenges

The field of natural language processing was irrevocably altered by the introduction of the Transformer architecture in 2017.1 This model, which eschewed the recurrent connections that defined its predecessors, established a new paradigm based entirely on attention mechanisms, enabling unprecedented levels of parallelism and performance on a wide range of sequence-to-sequence tasks.2 However, the very mechanism that propelled the Transformer to dominance also contained the seeds of its most significant limitations. As models grew in size and the demand for processing ever-longer contexts intensified, these foundational architectural choices presented formidable scaling challenges, creating a clear need for a new generation of models capable of transcending these inherent bottlenecks.4

Deconstructing the Vanilla Transformer: Attention, Encoders, and Positional Awareness

The vanilla Transformer, as detailed in the seminal paper “Attention Is All You Need,” is fundamentally an encoder-decoder architecture.1 The encoder’s role is to process an input sequence and transform it into a rich, contextualized numerical representation. The decoder then takes this representation and generates an output sequence, one element at a time.6 Both the encoder and decoder are composed of a stack of identical layers, with the original paper proposing six layers for each.2 The innovation lies not in the high-level structure but in the internal composition of these layers, which rely on two primary components: the self-attention mechanism and a position-wise feed-forward network.6

At the heart of the Transformer is the Scaled Dot-Product Attention mechanism.6 This function operates on a set of input vectors and produces a new set of vectors where each output is a weighted sum of the inputs. The weights are determined dynamically based on the relationships between the elements in the sequence. For each input vector, three corresponding vectors are generated via learned linear projections: a Query (), a Key (), and a Value ().1 The attention score, which represents the relevance of one element to another, is computed by taking the dot product of the Query vector of the current element with the Key vector of every other element in the sequence. These scores are then scaled by the square root of the dimension of the key vectors () to prevent the dot product from growing too large and pushing the subsequent softmax function into regions with vanishingly small gradients — a critical step for stabilizing training.2 The softmax function normalizes these scores, converting them into a set of positive weights that sum to one. Finally, the output vector for each position is calculated as the weighted sum of all Value vectors in the sequence, using the computed softmax scores as weights.1 The entire operation can be expressed compactly by the equation:

To enhance the model’s ability to focus on different aspects of the input, the vanilla Transformer employs Multi-Head Attention.2 Instead of performing a single attention calculation, this mechanism linearly projects the queries, keys, and values multiple times (e.g., eight “heads” in the original model) and runs the attention function in parallel for each projection. The outputs from each head are then concatenated and linearly projected again to produce the final output. This allows the model to jointly attend to information from different representation subspaces at different positions, capturing a richer set of dependencies.3

A significant challenge for an architecture that processes all sequence elements simultaneously is the lack of inherent knowledge about word order. Unlike Recurrent Neural Networks (RNNs) that process tokens sequentially, the Transformer has no built-in sense of position.2 To address this, the model injects positional information through Positional Encodings. These are vectors that are added to the input embeddings at the bottom of the encoder and decoder stacks. The original paper proposed using sinusoidal functions of different frequencies for these encodings, providing a unique positional signal for each token that the model can learn to interpret.1

Each layer in the encoder and decoder stack also contains a fully connected feed-forward network, applied independently to each position. This network consists of two linear transformations with a Rectified Linear Unit (ReLU) activation in between.2 To facilitate the training of these deep networks, both the self-attention sub-layer and the feed-forward sub-layer in each layer incorporate residual connections followed by layer normalization. The residual connection adds the input of the sub-layer to its output, allowing gradients to flow more easily through the network, while layer normalization stabilizes the activations, improving training speed and performance.2

The Computational Bottleneck: Quadratic Complexity and the KV Cache Problem

The self-attention mechanism, while powerful, is also the Transformer’s Achilles’ heel. Because every token must attend to every other token in the input sequence, the number of computations and the memory required to store the attention scores scale quadratically with the sequence length (), a complexity of .8 For short sequences, this is manageable. However, as the demand for processing entire documents, codebases, or long conversational histories has grown, this quadratic scaling has become a severe computational and financial bottleneck.4

The most acute manifestation of this problem occurs during autoregressive generation, the process of generating an output sequence one token at a time. To generate the next token, the model must have access to the attention information from all previously generated tokens. This is managed by storing the Key () and Value () vectors for every token in the context in a memory buffer known as the Key-Value (KV) cache.4 As the context grows, the size of this KV cache expands linearly. For very long sequences, the memory required to store this cache can become astronomical, far exceeding the capacity of even high-end GPUs.4

This is not a theoretical concern but a hard practical limit. For instance, a vanilla Transformer-based model like Llama-2 7B, if extended to a 256,000-token context, would require an estimated 128 GB of VRAM just for its KV cache.4 This makes its deployment for long-context tasks practically infeasible on all but the most extensive and expensive hardware configurations. Even more optimized models like Mixtral-8x7B still require 32 GB for the same context length.4 This fundamental architectural constraint has created a powerful incentive for the development of alternative architectures. The severity of the problem is evidenced by the entire sub-field of research dedicated to creating workarounds, such as sparse attention, sliding window attention, and hardware-level optimizations like Flash Attention.12 These efforts, while valuable, treat the symptoms of the quadratic complexity problem rather than addressing its root architectural cause.

The Performance Plateau: Diminishing Returns in Long-Context Processing

The challenges posed by the Transformer architecture extend beyond raw computational cost. There are two additional critical issues: performance degradation at long contexts and slow inference speed. Many models that advertise very large theoretical context windows often exhibit a significant drop in performance when tested on tasks that require recalling or reasoning over information located far back in the context.13 The model may effectively “lose track” of distant information, rendering the extended context window less useful in practice.

Furthermore, the Transformer architecture is inherently slow during inference. Because it lacks a compact, single summary state like an RNN, the generation of each new token requires a full computational pass over the entire KV cache of the preceding context.4 This means that as the sequence gets longer, the time required to generate each subsequent token increases. This leads to low throughput and high latency, which are unacceptable for many real-time applications.

The confluence of these factors — prohibitive memory requirements, degrading performance, and slow inference — has created an application and innovation bottleneck. The very architecture that enabled the current generation of AI has also constrained the feasibility of more advanced applications. Complex, multi-step agentic workflows, deep analysis of large document corpora, and truly long-form conversational AI require models that can handle massive contexts efficiently and effectively. The economic and practical limitations of the vanilla Transformer have made these next-generation applications prohibitively expensive or non-performant, setting the stage for a paradigm shift in model architecture.

Alternative Paradigms for Sequential Data

In response to the scaling limitations of the Transformer, researchers have explored alternative architectural paradigms. Two of the most promising and impactful are State Space Models (SSMs), particularly the Mamba variant, and the Mixture-of-Experts (MoE) technique. These two approaches address orthogonal scaling challenges: Mamba targets the problem of processing long sequences efficiently (horizontal scaling), while MoE targets the problem of increasing model capacity without a proportional increase in computational cost (vertical scaling). Their maturation as viable, powerful techniques provided the necessary components for a new synthesis in model design.

State Space Models (SSMs): The Mamba Architecture and Linear-Time Processing

State Space Models are a class of models inspired by classical control theory that are well-suited for modeling continuous-time systems and sequences.16 Traditionally, they have been used in fields like signal processing. Recent work has adapted them for deep learning, culminating in the Mamba architecture, which made SSMs competitive with Transformers on language tasks for the first time.16

The core innovation of Mamba is a selection mechanism that makes the SSM’s parameters input-dependent. In prior SSMs, the state transition dynamics were fixed after training. Mamba introduces a mechanism that allows the model to selectively propagate or forget information based on the current input token. This content-based reasoning capability was a key weakness of earlier SSMs and is crucial for handling discrete data like text.16

The primary advantage of the Mamba architecture is its remarkable efficiency. It processes sequences with linear-time complexity () and generates new tokens with constant-time inference ( memory), as it only needs to maintain a fixed-size hidden state rather than a growing KV cache.9 This directly addresses the quadratic bottleneck of the Transformer. As a result, Mamba-based models can achieve significantly higher throughput — up to 5x that of a similarly sized Transformer — and have a much smaller memory footprint during inference.16

However, despite these impressive efficiency gains, pure SSM-based models have historically struggled to match the raw performance and quality of top-tier Transformer models on all tasks.5 While they excel at handling long-range dependencies, they can sometimes lag in tasks that require precise, all-to-all comparisons across the entire context, a task for which the Transformer’s self-attention mechanism is perfectly suited.15 This created a clear trade-off: efficiency versus peak quality.

Mixture-of-Experts (MoE): Decoupling Model Size from Computational Cost

The Mixture-of-Experts (MoE) paradigm offers a solution to a different scaling problem: how to make models larger and more knowledgeable without making them proportionally slower and more expensive to run. MoE is a form of conditional computation that replaces a standard, dense feed-forward network (or MLP) with a sparse equivalent.4

An MoE layer consists of two main components: a set of “expert” networks (each typically a small MLP) and a “gating network” or “router”.9 For each input token, the router dynamically selects a small subset of the available experts to process that token — for example, the top two most relevant experts. The final output is then a weighted combination of the outputs from the selected experts.19

The benefit of this approach is profound: it allows for a dramatic increase in the total number of parameters in the model while keeping the number of active parameters — those used for computation on any given token — constant and manageable.4 This decouples the model’s capacity from its computational cost. A model can have hundreds of billions or even trillions of total parameters, representing a vast store of knowledge, but the inference cost can be equivalent to that of a much smaller, dense model. The initial Jamba model provides a clear example of this principle in action, with 52 billion total available parameters but only 12 billion active parameters used during a forward pass.4 This technique has been successfully used in large-scale Transformer models like Mixtral to achieve high performance with manageable inference costs.19

The emergence of Mamba as a viable, highly efficient alternative for sequence modeling and the validation of MoE as a proven technique for scaling model capacity created a unique moment in AI research. These were not competing ideas but complementary solutions to distinct problems. The true innovation was to recognize that these powerful, independent architectural primitives could be synthesized into a single, hybrid architecture where the strengths of one component could compensate for the weaknesses of another, creating a system superior to any of its individual parts.

The Jamba Architecture: A Synergistic Integration

The Jamba architecture represents a landmark in the evolution of large language models, moving beyond the monolithic Transformer design to create a novel, hybrid system. Developed by AI21 Labs, Jamba is the first production-grade model to successfully integrate three distinct architectural paradigms: the Transformer’s attention mechanism, the Mamba State Space Model, and the Mixture-of-Experts technique.4 This synergistic combination is not merely an academic exercise; it is a carefully engineered solution designed to create a new Pareto frontier, simultaneously optimizing for model quality, computational efficiency, and long-context capability.4

The Hybrid Block: Interleaving Attention and Mamba Layers

The fundamental novelty of Jamba lies in its hybrid decoder architecture, which is constructed from a repeating sequence of what the creators term “Jamba blocks”.4 Unlike a pure Transformer or a pure Mamba model, a Jamba block contains a strategic mix of layer types. Specifically, it interleaves standard Transformer layers, which contain a self-attention module, with Mamba layers.11 Each of these primary layers (either attention-based or Mamba-based) is followed by a multi-layer perceptron (MLP), which itself can be a standard dense network or an MoE layer.10

This “inter-layer hybrid” approach is the architectural core of Jamba.23 By interleaving the two types of layers, the model is designed to harness the distinct advantages of each. It retains the powerful reasoning and high-quality performance of the Transformer’s attention mechanism, which excels at making global, all-to-all comparisons within a sequence.17 Simultaneously, it incorporates the linear-time complexity and extreme efficiency of Mamba layers, which are superior at handling very long sequences with a minimal memory footprint.5 The result is a composite architecture that seeks to achieve the best of both worlds: the quality of a Transformer with the efficiency of an SSM.

Architectural Blueprint: Ratios, Placement, and the Role of MoE

The specific implementation of the Jamba architecture is governed by a set of configurable hyperparameters that allow the design to be tailored for specific resource constraints and performance objectives.11 The initial publicly released Jamba model, designed to be powerful yet fit within a single 80GB GPU, provides a concrete example of these design choices in practice 4:

Attention-to-Mamba Ratio (): The model employs a ratio of 1:7. This means that for every eight sequential layers within a Jamba block, one will be a Transformer attention layer, and the other seven will be Mamba layers.4 This heavily Mamba-biased ratio was selected as an optimal point to maximize compute efficiency and throughput while retaining enough attention capacity to ensure high model quality.4

Mixture-of-Experts (MoE) Configuration:

• Frequency (): MoE is not applied to every MLP. Instead, it is used in every other layer (), alternating between sparse MoE layers and standard dense MLP layers.11
• Number of Experts (): Each MoE layer contains a total of 16 distinct expert networks ().11
• Top-K Routing (): For any given token, the router selects the top 2 most relevant experts to process it ().11

Overall Structure: The model is composed of a total of 8 Jamba blocks, each containing this mix of layers. The total parameter count is approximately 52 billion, but due to the MoE implementation, only 12 billion parameters are active during any single forward pass.11

This carefully calibrated blueprint allows Jamba to achieve its remarkable efficiency. By heavily favoring Mamba layers over attention layers, the model drastically reduces the size of the performance-limiting KV cache. The architecture aims for an 8x smaller KV cache compared to a vanilla Transformer of similar size, which is the key to its long-context capabilities.5

Analysis of Design Trade-offs: Insights from Ablation Studies

The final Jamba design was not arbitrary but the result of extensive experimentation and ablation studies that explored the complex trade-offs between different architectural choices.4 These studies revealed several non-obvious principles that are critical to the success of such a hybrid model.

One of the most important findings relates to the ratio of attention to Mamba layers. The experiments showed a clear trade-off: increasing the proportion of Transformer layers (e.g., to a 1:1 ratio) generally improves model quality on standard benchmarks but comes at a significant cost to efficiency and throughput. Conversely, a higher proportion of Mamba layers improves efficiency. The range of a 1:5 to 1:7 ratio was identified as a “sweet spot” on the Pareto frontier, offering a compelling balance between high quality and high efficiency.23

Perhaps the most surprising and crucial design principle discovered was related to the placement of the layers. The ablation studies revealed a stark rule: one should “never place Transformer blocks at the front” of the model.23 This strongly suggests a functional specialization within the architecture. It implies that the optimal structure involves a division of labor: the initial layers, which are all Mamba layers in the Jamba design, act as highly efficient processors and feature extractors. They can scan the entire long input sequence linearly, compressing relevant information into their hidden states. The sparsely interspersed Transformer layers, placed later in the network, can then perform their more computationally expensive global attention operations on these already-processed, more condensed representations. This structure acts as a highly efficient pipeline, using the right tool for the right job at each stage of processing, rather than being a simple, uniform mixture of components.

Another profound finding emerged when comparing different versions of the Mamba component. While Mamba-2 is a more advanced and performant version of the SSM architecture when used in isolation, the researchers found that in the context of the Jamba hybrid, the combination of the older Mamba-1 with attention layers actually outperformed the Mamba-2 and attention combination.25 The hypothesis for this counterintuitive result is that the presence of the global attention layers renders some of Mamba-2’s key advantages (such as the ability to use a much larger state size) less significant or even redundant. The attention layers can already pool information from the entire context, fulfilling the role that an expanded Mamba state would otherwise play. This discovery is a powerful lesson in complex systems design: optimizing a component in isolation does not guarantee optimal performance for the system as a whole. The interaction effects between components can create a new, emergent optimum. This suggests that the future of AI architecture lies not in simply plugging together the “best” individual parts, but in the co-design and deep, synergistic optimization of hybrid systems.

A Comparative Analysis of Jamba’s Performance and Benefits

The theoretical advantages of Jamba’s hybrid architecture are substantiated by a wealth of empirical data demonstrating its superior performance across multiple dimensions: efficiency, long-context handling, and quality. A direct comparison with leading Transformer-based models reveals that Jamba does not merely represent an incremental improvement but establishes a new performance frontier, achieving the quality of large-scale models with the resource footprint and speed of much smaller ones. This constitutes a Pareto improvement, pushing beyond the traditional trade-offs that have defined LLM design.

Efficiency and Throughput: A New Pareto Frontier

The most immediate and quantifiable benefit of the Jamba architecture is its radical improvement in computational and memory efficiency. By strategically replacing the majority of quadratic-scaling attention layers with linear-scaling Mamba layers, Jamba fundamentally alters the resource requirements for large-scale language modeling.

This is most evident in the reduction of the KV cache size, the primary memory bottleneck for long-context Transformers. As shown in the table below, Jamba’s memory requirement for its KV cache at a 256K context length is an order of magnitude smaller than that of comparable Transformer-based models.

This 8x reduction compared to Mixtral and 32x reduction compared to a hypothetical Llama-2 7B at that context length is a direct consequence of the 1:7 attention-to-Mamba ratio. This dramatic memory saving is what enables Jamba to perform feats like processing up to 140,000 tokens of context on a single 80GB GPU, making high-performance, long-context AI accessible without massive hardware clusters.18

This efficiency translates directly into superior processing speed, or throughput. Jamba consistently demonstrates higher throughput than Transformer models, especially as the context length grows. Reports indicate that Jamba can achieve up to 3x the throughput of Mixtral-8x7B on long contexts.5 Performance evaluations of Jamba 1.5 Mini show this scaling advantage in action.

Data synthesized from performance results reported in.26 Throughput measured on identical hardware.

As the table illustrates, while Jamba 1.5 Mini is competitive with Llama 3.1 8B on short contexts, its performance advantage becomes stark at very long contexts. Jamba’s throughput degrades only slightly, whereas the throughput of the pure Transformer model drops by nearly 50%. This demonstrates how the Mamba layers allow Jamba to escape the performance penalty that Transformers pay for processing long sequences.

Long-Context Mastery: Validating the 256K Token Window

Jamba’s headline feature is its massive 256,000-token context window, equivalent to approximately 800 pages of text.11 However, in an industry where advertised context lengths often do not reflect real-world capability, Jamba’s key differentiator is that its context window is not just a theoretical maximum but an effective one.

The model’s long-context prowess has been validated on rigorous, task-based benchmarks like RULER, which evaluates a model’s ability to perform complex reasoning tasks such as retrieval, multi-hop tracing, and aggregation over long documents.28 Jamba is reportedly the only open-weight model to substantiate its 256K claim on the RULER benchmark, maintaining high performance across the entire context span.9 This contrasts sharply with many other long-context models whose performance degrades significantly as they approach the limits of their context window, highlighting a growing gap between advertised and effective context length.14

This reliable long-context capability has profound implications for practical applications. For complex Retrieval-Augmented Generation (RAG) systems, Jamba’s ability to ingest entire documents in a single pass can obviate the need for intricate and often error-prone document chunking and retrieval pipelines.24 This simplifies development, reduces latency, and can lead to more accurate, contextually grounded responses by allowing the model to see the full picture at once.

Quality and Reasoning Capabilities

The critical question for any efficiency-focused architecture is whether its gains come at the expense of performance quality. Jamba’s results on standard academic and industry benchmarks demonstrate that it makes no such compromise. The hybrid architecture, augmented with MoE, allows it to achieve quality that is competitive with, and in some cases superior to, state-of-the-art Transformer models of similar and even larger active parameter counts.

The initial Jamba release was shown to perform comparably to the much larger Llama-2 70B and the similarly sized Mixtral-8x7B on a range of benchmarks.4 Subsequent releases have continued this trend, establishing Jamba as a top-tier model in its size class.

Benchmark scores for Jamba sourced from.21 Scores for Mixtral and Llama-2 are from their respective release papers for comparison.

While Mixtral shows stronger performance on reasoning-heavy benchmarks like MMLU and GSM8K in this comparison, Jamba demonstrates highly competitive results across the board, particularly on commonsense reasoning tasks like HellaSwag and WinoGrande. More recent versions, such as Jamba 1.5 Mini, have further closed this gap, achieving a score of 46.1 on the challenging Arena Hard benchmark, surpassing larger competitors.24 Jamba 1.5 Large achieves a score of 65.4, outpacing even Llama 3.1 405B.24

Taken together, these three dimensions of performance — efficiency, long-context handling, and quality — paint a clear picture. Jamba is not making a trade-off along the existing performance-vs-efficiency curve. Instead, its hybrid architecture has shifted the curve entirely, creating a model that is simultaneously faster, more memory-efficient, better at long-context tasks, and competitive on quality benchmarks. It proves that it is possible to break free from the scaling limitations of the pure Transformer paradigm without sacrificing performance.

Broader Implications and Future Architectural Trajectories

The introduction of the Jamba architecture is more than just the release of a new model; it signals a potential inflection point in the design philosophy of large language models. By successfully demonstrating the viability of a production-grade hybrid architecture, Jamba challenges the long-held dominance of the pure Transformer design and opens up new avenues for research, application development, and the very economics of artificial intelligence. Its success has broader implications that touch upon the accessibility of AI, the future of model architecture, and the enabling of next-generation AI systems.

The Economic Impact of Hybrid Architectures: Redefining AI Accessibility

One of the most significant consequences of Jamba’s efficiency is its impact on the economics of deploying advanced AI.31 For years, the capabilities of state-of-the-art models have been inextricably linked to massive, centralized data centers and enormous computational budgets. The quadratic scaling of Transformers meant that long-context processing was a luxury reserved for the largest technology companies and research institutions.

Jamba’s architecture fundamentally alters this equation. By enabling a model with a 256K context window and over 50 billion parameters to run effectively on a single, commercially available GPU, it dramatically lowers the barrier to entry for developing and deploying sophisticated AI applications.15 This democratization of capability can be analogized to the historic shift from mainframes to personal computers in the 1980s.31 It empowers a broader community of developers, startups, and researchers to experiment with and build applications that were previously out of reach, fostering a more decentralized and innovative AI ecosystem. This shift could lead to a proliferation of on-device and edge-computing AI solutions, enhancing data privacy and enabling offline resilience.31

Beyond Transformers: The Viability of Hybrid Models as the Next Standard

Jamba serves as a powerful proof-of-concept that the future of LLM architecture is unlikely to be a single “Transformer-killer” that replaces attention with another monolithic mechanism. Instead, the future appears to be one of sophisticated hybridization, where models are constructed from a diverse toolkit of computational primitives — including attention, state space models, and potentially others — each chosen for its specific strengths.17

The success of Jamba’s inter-layer hybrid design validates this approach at production scale, proving that different mechanisms can be combined synergistically to create a whole that is greater than the sum of its parts.18 This opens the door for a new wave of architectural experimentation. Researchers can now explore a vast design space of different hybrid configurations: varying the ratios of components, exploring parallel (intra-layer) versus sequential (inter-layer) fusion, and integrating other novel primitives.23 Jamba has effectively broken the architectural monoculture and demonstrated that there are multiple viable paths to high-performance AI.

This architectural shift could also trigger a co-evolution of AI hardware and software. The AI ecosystem has been heavily optimized for the massive matrix multiplications that dominate Transformer workloads. The rise of hybrid models like Jamba, which rely on different computational patterns like the parallel scans used by Mamba, may drive the development of next-generation AI accelerators and software libraries that are designed to efficiently handle a more diverse set of operations. The modifications made to the vLLM inference library to support Jamba’s specific quantization method is an early example of this trend, where new architectures necessitate and inspire new software optimizations.25

Concluding Analysis and Future Research Directions

In conclusion, the Jamba architecture represents a significant and compelling step forward in the design of large language models. Its core innovation — the synergistic integration of Transformer, Mamba, and Mixture-of-Experts layers — directly addresses the most pressing limitations of the vanilla Transformer paradigm. The result is a model that achieves a new Pareto frontier of performance, delivering state-of-the-art quality and unprecedented long-context capabilities with a fraction of the computational and memory resources.

Jamba’s architecture is particularly well-suited to power the next generation of agentic AI systems. These autonomous agents require a persistent, low-latency, and context-aware “controller” to orchestrate complex tasks — a role for which the high cost and latency of pure Transformers are ill-suited. Jamba’s unique combination of efficiency and long-context mastery makes it an ideal enabling technology for this high-value future application of AI, potentially unlocking the widespread adoption of sophisticated, autonomous workflows.29

The path forward for architectural research is now richer and more varied. Future work will undoubtedly explore the vast design space that Jamba has opened up. This includes further investigation into optimal hybrid configurations, the application of these architectures to other modalities beyond text, and a deeper study of the complex, emergent interactions between different computational primitives. Jamba has not only delivered a powerful new class of models but has also provided a new blueprint for innovation, suggesting that the next great leaps in AI will come not from a single breakthrough, but from the thoughtful and creative synthesis of many.

Works cited

1. Vanilla Transformers — Time Series with Deep Learning Quick Bite, accessed October 9, 2025, https://dl.leima.is/transformers/transformers.vanilla/
2. A Deep Dive Into the Transformer Architecture — The Development …, accessed October 9, 2025, https://www.exxactcorp.com/blog/Deep-Learning/a-deep-dive-into-the-transformer-architecture-the-development-of-transformer-models
3. Demystifying Transformer Architecture in Large Language Models — TrueFoundry, accessed October 9, 2025, https://www.truefoundry.com/blog/transformer-architecture
4. Jamba: A Hybrid Transformer-Mamba Language Model — arXiv, accessed October 9, 2025, https://arxiv.org/pdf/2403.19887
5. Jamba: A Hybrid Transformer-Mamba Language Model — arXiv, accessed October 9, 2025, https://arxiv.org/html/2403.19887v2
6. Overview of the Transformer Architecture — Paperspace Blog, accessed October 9, 2025, https://blog.paperspace.com/learning-in-latent-spaces-improves-the-predictive-accuracy-of-deep-neural-operators/
7. Transformer (deep learning architecture) — Wikipedia, accessed October 9, 2025, https://en.wikipedia.org/wiki/Transformer_(deep_learning_architecture)
8. Vanilla Transformer — Nixtla, accessed October 9, 2025, https://nixtlaverse.nixtla.io/neuralforecast/models.vanillatransformer.html
9. Jamba: Redefining Long-Context AI Performance | by Srujananjali — Medium, accessed October 9, 2025, https://medium.com/@srujananjali888/jamba-redefining-long-context-ai-performance-f477306c17e1
10. JAMBA: HYBRID TRANSFORMER-MAMBA LANGUAGE MODELS — OpenReview, accessed October 9, 2025, https://openreview.net/pdf?id=JFPaD7lpBD
11. Jamba: A Hybrid Transformer-Mamba Language Model — arXiv, accessed October 9, 2025, https://arxiv.org/html/2403.19887v1
12. Are the new architectures Mamba and Jamba better or worse than current existing Transformer architectures. : r/LocalLLaMA — Reddit, accessed October 9, 2025, https://www.reddit.com/r/LocalLLaMA/comments/1lluwee/are_the_new_architectures_mamba_and_jamba_better/
13. Open-Source 3B Inference Model for Mobile Phones: Faster Than Qwen 3–4B and High — Speed with Ultra — Long Context, accessed October 9, 2025, https://eu.36kr.com/en/p/3501820455459718
14. NoLiMa: Long-Context Evaluation Beyond Literal Matching — Finally a good benchmark that shows just how bad LLM performance is at long context. Massive drop at just 32k context for all models. — Reddit, accessed October 9, 2025, https://www.reddit.com/r/LocalLLaMA/comments/1io3hn2/nolima_longcontext_evaluation_beyond_literal/
15. Jamba: A hybrid Transformer-Mamba Language Model | Clio AI, accessed October 9, 2025, https://www.clioapp.ai/research/jamba-xformer-mamba-language-model
16. [2312.00752] Mamba: Linear-Time Sequence Modeling with Selective State Spaces — arXiv, accessed October 9, 2025, https://arxiv.org/abs/2312.00752
17. AI21 Introduces Jamba 1.6, Raising the Bar for Accuracy and Speed in Open Models, accessed October 9, 2025, https://www.prnewswire.com/news-releases/ai21-introduces-jamba-1-6--raising-the-bar-for-accuracy-and-speed-in-open-models-302394382.html
18. Jamba: The LLM with Mamba Mentality — Gradient Flow, accessed October 9, 2025, https://gradientflow.com/ai21labs-jamba/
19. Mixtral of Experts — arXiv, accessed October 9, 2025, https://arxiv.org/pdf/2401.04088
20. [2401.04088] Mixtral of Experts — arXiv, accessed October 9, 2025, https://arxiv.org/abs/2401.04088
21. ai21labs/Jamba-v0.1 — Hugging Face, accessed October 9, 2025, https://huggingface.co/ai21labs/Jamba-v0.1
22. Introducing Jamba — Hybrid Transformer Mamba with MoE : r/LocalLLaMA — Reddit, accessed October 9, 2025, https://www.reddit.com/r/LocalLLaMA/comments/1bpx9sh/introducing_jamba_hybrid_transformer_mamba_with/
23. Hybrid Architectures for Language Models: Systematic Analysis and Design Insights — arXiv, accessed October 9, 2025, https://arxiv.org/html/2510.04800v1
24. Jamba 1.5 LLMs Leverage Hybrid Architecture to Deliver Superior …, accessed October 9, 2025, https://developer.nvidia.com/blog/jamba-1-5-llms-leverage-hybrid-architecture-to-deliver-superior-reasoning-and-long-context-handling/
25. (PDF) Jamba-1.5: Hybrid Transformer-Mamba Models at Scale — ResearchGate, accessed October 9, 2025, https://www.researchgate.net/publication/383308336_Jamba-15_Hybrid_Transformer-Mamba_Models_at_Scale
26. AI21 Labs’ Jamba 1.5 Outpaces Transformers in Long-Text Processing — DeepLearning.AI, accessed October 9, 2025, https://www.deeplearning.ai/the-batch/ai21-labs-jamba-1-5-outpaces-transformers-in-long-text-processing/
27. AI21 Labs Jamba-Instruct model is now available in Amazon Bedrock | Artificial Intelligence, accessed October 9, 2025, https://aws.amazon.com/blogs/machine-learning/ai21-labs-jamba-instruct-model-is-now-available-in-amazon-bedrock/
28. Applying Jamba-Instruct to Long Context Use Cases in Snowflake Cortex AI — Medium, accessed October 9, 2025, https://medium.com/snowflake/applying-jamba-instruct-to-long-context-use-cases-in-snowflake-cortex-ai-fcbbb9fc2df3
29. AI21 Introduces the Jamba Model Family: The most powerful and efficient long-context models for the enterprise — PR Newswire, accessed October 9, 2025, https://www.prnewswire.com/news-releases/ai21-introduces-the-jamba-model-family-the-most-powerful-and-efficient-long-context-models-for-the-enterprise-302228192.html
30. LongCodeBench: Evaluating Coding LLMs at 1M Context Windows — arXiv, accessed October 9, 2025, https://arxiv.org/html/2505.07897v2
31. A121 Labs’ Jamba Reasoning 3B is a powerful tiny model that promises to transform AI economics — SiliconANGLE, accessed October 9, 2025, https://siliconangle.com/2025/10/08/a121-labs-jamba-reasoning-3b-powerful-tiny-model-promises-transform-ai-economics/
32. Beyond Transformers: How AI21’s Jamba 1.5 is Redefining Generative AI | Walden Catalyst, accessed October 9, 2025, https://waldencatalyst.com/blog/beyond-transformers-how-ai21s-jamba-1-5-is-redefining-generative-ai
33. TAI #104; LLM progress beyond transformers with Samba? — Towards AI, accessed October 9, 2025, https://towardsai.net/p/artificial-intelligence/tai-104-llm-progress-beyond-transformers-with-samba
34. AI21 Introduces the Jamba Model Family: The most powerful and efficient long-context models for the enterprise | Radical Data Science, accessed October 9, 2025, https://radicaldatascience.wordpress.com/2024/08/22/ai21-introduces-the-jamba-model-family-the-most-powerful-and-efficient-long-context-models-for-the-enterprise/