What kind of Intelligence should AI have?
Cool visions on new architectures, LLM optimizations, fine-tuning and paradigm shifts
What kind of Intelligence Should AI Have?
Cool visions on new architectures, LLM optimizations, fine-tuning, and paradigm shifts
is this the super-intelligence we want to achieve?
This picture stormed the socials few weeks ago.
The Advertising company did a terrible job (in my opinion), but it can be that Corporations are loving it…
What do you think?
The increasing number of new models and architecture are moving forward the AGI narrative. Are we almost there?
Maybe not, but a few questions are open: not related to the Transformers architecture though, but more primitive.
Who is going to be in Control of the AGI? Who will be the owner?
Before we move on: feeling overwhelmed by the AI hype? “The PoorGPU Guy” on Substack is here to help!
I cut through the noise and deliver weekly updates on the most exciting developments in Generative AI.
Want to get your hands dirty? The paid version offers practical Python tutorials for applying AI in your daily life.
I decided to publish from now on only on my Medium Publication, and on my Substack.
So if you don’t want to miss new articles, you can:
- Join my totally free weekly Substack newsletter here
- Subscribe to receive an email every time I publish something new
- Follow my publication https://medium.com/artificial-intel-ligence-playground
Back to the topic, and quoting Michael Heine
… before we lament the end of human dominance, we should ask ourselves a few critical questions. Who controls this new superintelligence? A nonprofit consortium dedicated to maximizing the well-being of all? Or a corporation whose top priority is quarterly figures and shareholder returns?
The question is not whether AGI is coming. But who will own it. And who will buy the “license to play God” in the future
I strongly suggest you read his article: it is a must
[embed]The Multi Trillion-Dollar Question: Who owns the future of AGI?ai.gopubby.com
What kind of Intelligence should AI have?
The LLM as we have them… are not AGI, yet.
Why?
Because for now Generative AI is focused on tasks.
I am not saying that this is not an outstanding feat! On the contrary, the Transformers architecture bloomed like no others.
But is this the right path? Should we re-start from scratch?
This is a general sentiment in the research community: and we have evidence from several clues.
- RWKV architecture change to remove the attention mechanism from the computation demands
- Fourier applications on Neural Network, also pointing toward the tokenization efficiency to solve the attention bottleneck
- Mixture of Experts and gating mechanisms
- RNN and even Diffusion to replace the Transformers (LlaDa — Large Language Diffusion Models)
Plenty of explorations from all over the place. I usually follow all these innovative approaches through the Hugging Face Papers newsletter. People are amazing!
[embed]Daily Papers - Hugging Face Your daily dose of AI research from AKhuggingface.co
I want to focus on all the papers and applications related to an architectural shift. It doesn’t matter if it is a big revolution, or a small movement… what does matter, though, is that we can see glimpses of a change (first of all in the mindset)”.
Theory of Mind
Let's remove the Tokenizer
Paper - "Ultimate Guide to Fine-Tuning LLMs"A bunch of researches are pointing to this direction, that I also believe is the core of the question: if we are looking for AGI, what kind of Intelligence should we refer to?
image from Explore Theory-of-Mind: Program-Guided Adversarial Data Generation for Theory of Mind Reasoning.
A Theory of Mind
This week The Fundamental AI Research (FAIR) team at Meta released an interesting paper titled *Explore Theory-of-Mind: Program-Guided Adversarial Data Generation for Theory of Mind Reasoning.*
Do large language models (LLMs) like AI chatbots understand what other people are thinking? Many researchers have tried to test this by using simple tests and stories. However, these tests might not be challenging enough and could give a false idea of how well the AI actually understands people.
To improve this, a new method called ExploreToMhas been created. ExploreToMgenerates a lot of different and complex stories to test the AI’s ability to understand what others are thinking. These stories are more challenging and varied than the ones used before.
When tested with
ExploreToM, even the best AI models like Llama-3.1–70B and GPT-4o did very poorly, getting only about 5% of the questions right. This shows that these AIs still have a long way to go in understanding people’s thoughts.
Using the data from ExploreToMto train the AI can significantly improve its performance on older, simpler tests. ExploreToMalso helps identify the specific areas where the AI is weak, such as keeping track of what’s happening in a story or having a balanced set of training data.
You can read the paper here.
you can watch the Video here
Let’s remove the Tokenizer
The Tokenization is basically the first step in creating a LLM. This week Meta studied not a way to optimize the LLM generation, but the LLM creation!
I was following few weeks ago an amazing lecture Stanford CS229 I Machine Learning I Building Large Language Models LLM: Yann Dubois, PhD Student at Stanford, gives concise overview of building a ChatGPT-like model, covering both pre-training (language modeling) and post-training (SFT/RLHF). For each component, it explores common practices in data collection, algorithms, and evaluation methods.
At the very beginning of the GPT models creation there is tokenization. Tokenization is a crucial process in natural language processing that involves breaking down text into smaller units called tokens. These tokens can be words, subwords, or even characters. The primary goal is to convert text into a format that a machine learning model can understand and work with effectively.
One of the main reasons we need tokenization is to handle typos and variations. If you tokenize by words, a typo or a variation in how a word is written (e.g., “color” vs. “colour”) can create issues (we don’t have a token for it).
Another important reason for tokenization is to handle languages that do not use spaces between words, or even to handle numbers for Math understanding.
However, it is also worth noting that in some cases, we can live without tokenization. Character-level models, for instance, can work directly with characters instead of tokens. These models offer several advantages. They are flexible and can handle any sequence of characters, including typos and variations. Additionally, there is no need for a separate tokenization step, which can simplify the pipeline.
But they are expensive in terms of computation resource and time!
One common method of tokenization is Byte Pair Encoding (BPE). The process starts with a large corpus of text, where each character is treated as a token. The next step is to merge the most common pairs of tokens. This process is repeated, continuing to merge common pairs until the desired vocabulary size is reached.
Byte Latent Transformer: Patches Scale Better Than Tokens
The guys at Meta FAIR just released a new paper that is going to make a change (in the near future): **Byte Latent Transformer: Patches Scale Better Than Tokens.**
The Byte Latent Transformer (BLT), is a new byte-level LLM architecture that, for the first time, matches tokenization-based LLM performance at scale with significant improvements in inference efficiency and robustness.
Meta FAIR team created a new type of language model called the Byte Latent Transformer (BLT). This model works directly with the smallest units of text, called bytes, instead of using pre-defined groups of words or characters (tokens).
Here’s how it works: BLT takes the text and breaks it into smaller chunks, or “patches.” The size of these patches can change depending on how complex the text is. It is like having a Tokenless LLM: if the entropy of the next byte is high, it starts a new patch; if it’s low, it continues the current patch. Transformer then processes the patch representation instead of tokens
We tested BLT by training it on a huge amount of data — 4 trillion bytes — and found that it can handle this data without needing a fixed set of words or characters.
So it looks like, for the same amount of computing resources, BLT performs better than models that use tokens. It does this by growing both the size of the patches and the size of the model itself. This makes BLT a more efficient and robust choice for handling large amounts of text.
You can read the paper here.
And just few days ago… BOOM
SuperBPE encodes text more efficiently than BPE and the gap grows with vocabulary size!
SuperBPE: Space Travel for Language Models
University of Washington, NVIDIA and Allen Institute for AI just released a new implementation of the BPE tokenization that is promising!
We introduce a family of superword tokenizers that encode the same text using up to 33% fewer tokens than a BPE tokenizer of the same size. 8B models trained with our tokenizer are not only more efficient during inference, but also outperform the baseline on a large suite of downstream tasks, including +8.2% on MMLU.
Tokenizers are the interface through which language models interact with the world. Beginning from a blank slate, they learn the shape and form of language as sequences of tokens, becoming ever more capable. But in the end, the tokenizer will still govern the mapping between text and computation.
Today, tokenization universally occurs at the level of subwords.
But for modern language models, does it really make sense to limit tokens to parts of words? Whitespace is not a consistent delimiter of meaning — multi-word expressions (“by the way”) function semantically as single units, and different languages vary in the number of words needed to express a concept (“spacesuit helmet” is “Raumanzughelm” in German). At the extreme, languages such as Chinese do not use whitespace at all. Tokens in these languages span multiple words and even entire sentences, yet this has seemingly not hindered LMs from learning these languages.
We extend tokenization beyond subwords by introducing SuperBPE, an algorithm that produces tokenizers including both subword and “superword” tokens.
You can read the paper here.
image from the paper
Paper — “Ultimate Guide to Fine-Tuning LLMs”
Here a Consolidated insights on LLM fine-tuning — a long read across 114 pages. This is basically a free BOOK, but in reality is an official paper called “Ultimate Guide to Fine-Tuning LLMs”
To be precise, the paper, last revision, was published end of October 2024 as The Ultimate Guide to Fine-Tuning LLMs from Basics to Breakthroughs: An Exhaustive Review of Technologies, Research, Best Practices, Applied Research Challenges and Opportunities.
The paper outlines the historical evolution of LLMs from traditional Natural Language Processing (NLP) models to their pivotal role in AI. A comparison of fine-tuning methodologies, including supervised, unsupervised, and instruction-based approaches, highlights their applicability to different tasks.
There is a structured seven-stage pipeline for fine-tuning LLMs, spanning data preparation, model initialization, hyperparameter tuning, and model deployment. Emphasis is placed on managing imbalanced datasets and optimization techniques.
Parameter-efficient methods like Low-Rank Adaptation (LoRA) and Half Fine-Tuning are explored for balancing computational efficiency with performance.
Advanced techniques such as memory fine-tuning, Mixture of Experts (MoE), and Mixture of Agents (MoA) are discussed for leveraging specialized networks and multi-agent collaboration.
Novel approaches like Proximal Policy Optimization (PPO) and Direct Preference Optimization (DPO), which align LLMs with human preferences, alongside pruning and routing optimizations to improve efficiency.
Validation frameworks, post-deployment monitoring, and inference optimization, with attention to deploying LLMs on distributed and cloud-based platforms.You can download the book (ehm… paper) from here.
Conclusions
If you are reading all of this, it means that you are curious about Artificial Intelligence.
And like me, you have a Mixture Of Expectations (a different MoE) between optimism and doom-skynet.
The best way to be ready… is to stay updated and follow the main AI labs’ research: because we should never forget that every epochal change comes from a revolutionary theory.
I hope you enjoyed the article. If this story provided value and you wish to show a little support, you could:
- Clap a lot of times for this story
- Highlight the parts more relevant to be remembered (it will be easier for you to find them later and for me to write better articles)
- Join my totally free weekly Substack newsletter here
- Sign up for a Medium membership ($5/month to read unlimited Medium stories)
- Follow me on Medium
- Read my latest articles at https://medium.com/@fabio.matricardi
- Follow my publication https://medium.com/artificial-intel-ligence-playground
If you want to read more, here are some ideas:
Resources and references:
[embed]SuperBPE - a UW Collection SuperBPE tokenizers and models trained with themhuggingface.co
[embed]Paper page - Large Language Diffusion Models Join the discussion on this paper pagehuggingface.co
메타데이터
- post_id
- 14502f8e9385
- slug
- what-kind-of-intelligence-should-ai-have-14502f8e9385
- url
- https://pub.towardsai.net/what-kind-of-intelligence-should-ai-have-14502f8e9385
- canonical_url
- https://pub.towardsai.net/what-kind-of-intelligence-should-ai-have-14502f8e9385
- author_url
- https://medium.com/@fabio.matricardi
- status
- ok
- fetched_at
- 2026-07-31 13:52:40
