In a remarkable development, a pioneering AI laboratory has strayed from conventional methodologies by introducing an entire code model that doesn't utilize Transformers: Mistral’s Codestral, which is based on the latest version of Mamba, known as Mamba2.

While this may seem inconsequential at first glance, it marks a significant departure since the debut of ChatGPT, as no leading lab had attempted such an innovation before, given that most existing models share a striking resemblance.

Could this signal the dawn of a new approach to large-scale AI training?

> Understanding AI is futile unless it enhances decision-making. This is the aim of my newsletter, which caters to AI analysts, strategists, investors, and leaders, addressing the most urgent inquiries in the AI field:

> Are we in a bubble? Is AI genuinely intelligent? Why is Taiwan so pivotal?

> In essence, it provides a comprehensive weekly overview of the technological, geopolitical, and economic landscape of the industry in an accessible format.

> ? Subscribe today below:

# The Transformer Dilemma

Despite models like ChatGPT or Claude appearing impressively human-like, their operation diverges significantly from human cognition.

Human brains maintain a compressed representation of past experiences and knowledge, often termed a 'world model,' which enables us to navigate our environment effectively.

In simple terms, we don't retain every experience; we focus on the significant moments and let the rest fade away.

For example, while reading a book, we don't recall every detail from the initial chapters when progressing to the next; instead, we cultivate a broad understanding of the narrative, key characters, and essential facts that our minds deem necessary for future context.

This contrasts sharply with how Large Language Models (LLMs) function, particularly well-known models like ChatGPT or Claude.

These models possess a memory (or state) that allows retrospective analysis. However, this memory lacks compression.

In essence, Transformers operate similarly to humans carrying an entire book filled with every detail from their lives for reference.

Yet, the situation is even more dire, as this ‘memory’ is, in reality, a ‘cache.’ The model doesn't genuinely remember anything; it simply ‘revisits’ the content.

Using the book analogy, ChatGPT must revisit all prior pages to process the next one. This issue compounds, as it needs to do this for every new word on the subsequent page.

> Conceptually, this outlines its functionality. However, computationally, the cache (known as KV Cache) alleviates some of this revisiting by retaining data in memory, thus avoiding recomputation.

> It serves as a memory, albeit one that includes all past events, regardless of their significance.

This duality presents both advantages and drawbacks.

• On one hand, the ability to revisit every detail grants these models high expressiveness and globality, allowing them to remember crucial facts that may seem trivial yet are pivotal to the storyline.
• Conversely, these models struggle with efficiency when processing vast amounts of data, as accumulating this data demands significant computational resources and memory.

This is where the Mamba2 architecture comes into play.

# The Revolutionary Compressor

As previously mentioned, Mamba architectures, first introduced in late 2023 and now enhanced, represent a novel approach to sequence modeling (predicting sequences from sequences, akin to ChatGPT’s text processing) that compresses state.

This means that for each new word encountered in a text, the model must evaluate its significance and either update its memory or discard it entirely.

Returning to our book example, the name of a character's sister might be worth storing, while a character stuttering the word ‘um’ likely isn't.

With a fixed memory size, the model aims to retain only essential information, filtering out the noise—much like human cognition.

> Conversely, this means that some potentially critical information may be forgotten, just as humans do.

However, for AI applications, the advantages are clear: unlike Transformers, which become increasingly complex as input sequences grow, Mamba models maintain consistent memory requirements relative to sequence length, as each new prediction relies on compressed memory and the latest predicted word.

> For those interested in mathematics, this resembles a Markov Decision Process, as the model's subsequent prediction relies solely on its current state and the last prediction, rather than prior time steps.

> Historically, Mamba models have struggled with non-Markovian dependencies, meaning they often lack the ability to make useful predictions when relying on forgotten past information.

> This has led many researchers to explore hybrid architectures that combine Mamba and Transformers, where Mamba handles decoding (due to its efficiency), and Transformers function as fact retrievers thanks to their extensive ‘memory.’ However, this falls outside the primary focus of this article.

In contrast, Transformers require quadratic computation and memory, rendering long sequences nearly unmanageable (hence the sequence size limitations seen in ChatGPT and similar models).

Despite Mamba's clear advantages, its adoption has been sluggish. Mamba2 aims to rectify this with Codestral.

# Transitioning from Mamba to MatMuls

While the earlier iteration of Mamba was conceptually appealing, it essentially operated as a recurrent neural network (similar to RNNs, which were the previous benchmark before Transformers).

> Although they are RNNs (sequential workloads), rendering them theoretically inefficient for GPU execution (which excels at parallel workloads), they utilized a “hardware-aware” approach to optimize efficiency on GPUs, with mixed results.

On the other hand, Transformers are specifically designed for GPUs. They primarily perform matrix multiplications, which are the operations that GPUs excel at (these operations have historically been utilized for rendering in gaming, the primary GPU application).

To address the challenges researchers faced with the original Mamba's GPU implementation, Mamba2 applies principles that convert these networks into matrix multiplications as well.

Thus, even though these models are fundamentally sequential, they function almost identically to Transformers when executed on GPUs, yielding a ‘best of both worlds’ outcome: models that, through state compression, are more efficient than Transformers while being optimized for GPU performance.

But do they deliver results? Absolutely.

Released under the open-source Apache 2.0 license, Mistral has unveiled Codestral Mamba, a model that outperforms any other of its size and, when scaled to 22B parameters, surpasses the state-of-the-art for code, outdoing CodeLlama 34B, which ranks among the top twelve overall models according to Scale AI’s leaderboards.

Notably, Mamba models, with their reduced dependence on sequence length, can manage up to 256k tokens, or 200k words, making them particularly suited for coding tasks that often require a greater token count than typical text tasks.

Are Mamba models poised to dominate the AI landscape? Mistral seems to indicate a positive answer.

# The Dawn of a New Era?

Mamba2 models hold the potential to transform the technological landscape for AI in the near to mid-term. Their power and efficiency address one of the primary challenges in AI.

> Before making sweeping claims, we need to see Mamba2 adopted more broadly before drawing any substantial conclusions. Nevertheless, based on their impressive outcomes, they certainly deserve a fair chance.

Importantly, as demonstrated by models like Samba or Jamba, the integration of Mamba with Transformers in hybrid architectures could pave the way for highly expressive yet efficient models becoming standard.

With the introduction of Codestral Mamba, numerous labs beyond Mistral, such as AI21 (Jamba) or Microsoft (Samba), will likely broaden their focus beyond just Transformers.

The rationale for this shift is evident.

As we enter an era of data augmentation, where models learn to produce more nuanced responses (i.e., more tokens per output), computationally intensive inference tasks are set to become the norm.

In this context, if we don't develop more efficient language modeling techniques, AI will turn into a domain heavily reliant on capital. Competing will become impossible without a company generating billions in quarterly cash flow, similar to the major tech firms.

However, the risks associated with the current norm also affect these tech giants, which are investing substantial resources in a technology that hasn't yet validated its worth, based on the assumption that Transformers will remain the ‘sole option.’

> Nonetheless, a company has secured $120 million to develop an ASIC for Transformers, a chip specifically designed to run Transformers, and nothing else. This exemplifies the blind faith the industry has in this algorithmic model.

But consider this: what if we discover that AI isn't as costly as previously thought? Would we still require so many GPUs?

I believe the markets have yet to account for more efficient architectures.

While they are currently reflecting a lack of demand based on last month's performance of the leading tech firms and the clear shift toward smaller capitalizations, I doubt they are considering the possibility that new algorithmic advancements, such as Mamba2, could radically alter unit economics.

If this realization occurs, executives in major tech firms might find themselves in a drastically different landscape than the one they inhabit now.

What are your thoughts?

> For inquiries related to AI strategy or analysis, please contact [email protected].

> If you enjoyed this article, I share similar insights in a clearer and more detailed manner for free on my LinkedIn.

> Feel free to connect with me on X.