AI ART HUM MDA MI Micheal Bee The Memento Machine How we spent more than a year building an outboard mind for an AI from first principles — and the strange experience of finishing most of…
HUM AI MI Micheal Bee Distilling a Teacher into a Student with Doc-to-LoRA Training Large language models are mostly feed-forward weight. In a transformer, the attention layers do the routing, but the feed-forward layers…
AI HUM SPT SCI SH shreeyash Why I Stopped Running Experiments Myself And Let an AI Agent Optimize My TTS Model Overnight —… Karpathy’s autoresearch loop turned a week of manual benchmarking into a single overnight run. Here’s the idea, and what it actually…
HUM AI MI Micheal Bee HyperPEER, Part Four: The Recursion That Wasn’t A clean negative result about making a small model think twice
HUM AI MI Micheal Bee HyperPEER Part Three: The Hard-Corpus Test, and What the Toy Taught Third and last in a series. Part one showed a hypernetwork can write a transformer’s experts on demand instead of retrieving them from a…
HUM AI SOC SCI MI Micheal Bee HyperPEER Part Two: We Put Recursion on Trial, and the Control Group Won A follow-up to “What If the Experts Don’t Have to Exist?” — new experiments on the generated-expert model: what we tried, why we tried it…
HUM AI SCI MI Micheal Bee What If the Experts Don’t Have to Exist? HyperPEER: replacing a stored expert bank with a hypernetwork that writes the experts on demand. One day of experiments on a single…
HUM AI MI Micheal Bee Recursive Reasoning on a Budget: Reproducing Tiny Recursive Models on One Consumer GPU There is a comforting myth in machine learning that the interesting results are locked behind data centers. You need a thousand GPUs, a…
HUM AI MI Micheal Bee The Graph Inside Attention Full attention is a simple, expensive idea: every token in a sequence is allowed to look at every other token. For a sequence of length n…
HUM MDA AI MI Micheal Bee Beyond DeepSeek’s Engram: What If Compressed Attention Didn’t Have to Be Lossy? DeepSeek has shipped two ideas in the last year that I keep coming back to. The first is Engram, from their paper on Conditional Memory via…
HUM AI MI Micheal Bee I Ran Sakana’s Doc-to-LoRA on a Single GPU. The Prompt Still Matters. If I tell you Bill is my best friend and then ask who my best friend is, you don’t need to memorize that Bill is my best friend in order to…
HUM AI MI Micheal Bee When Smart Routing Falls Apart A negative result for engram-based adapter routing, and why the failure is more useful than the fix would have been.
AI MED HUM TCH MI Micheal Bee · GoPenAI Why RAG Goes Off the Rails Even When Retrieval Works: A Training-Format Diagnosis Retrieval-augmented generation has become standard infrastructure for AI applications that need access to specific information beyond a…
HUM AI MI Micheal Bee How Subquadratic Might Have Done It: O(N) Linear Cost While Maintaining the Cartesian Product Subquadratic Inc. launched their SubQ model last week, claiming linear scaling and a 12-million-token context window with quality…
ART HUM AI MI Micheal Bee An Intuition for One Million Experts DeepMind published a paper in 2024 called Mixture of A Million Experts. The architecture they introduced is called PEER, which stands for…
HUM AI MI Micheal Bee Subquadratic Raised $29M for Linear Attention. Here’s What I Built in Two Hours The title is misleading. Let me set the record straight before going further.
ART HUM AI SPR MI Micheal Bee Three Walls and What’s Past Them: HRS as Single-Passage Architecture with Working Memory The Hierarchical Routed Sinkformer stores per-passage information as rank-128 LoRA adapters over a frozen base model. A learned projection…
SPT HUM AI SCI MI Micheal Bee HRS Research Summary A memory-aid document covering the experimental program toward a foundation-level model that runs on consumer hardware. Each section…
HUM AI MI Micheal Bee How to Think About Transformers at the Operational Level Most people who work with transformers know them through frameworks. You write PyTorch code. You import a model from Hugging Face. You…
HUM AI SCI MI Micheal Bee Retrain or Forget: A Simple Experiment That Changes How We Build AI Memory This is a report on a small experiment with a clean result, an instructive null result, and a methodological lesson about how to think…
HUM ART AI MI Micheal Bee We Solved Part of the Memory Problem. The Hard Part Is Next. We built a working sketch of addressable memory. It demonstrates that the storage and routing parts of the memory problem are tractable. It…
HUM AI MI Micheal Bee The Identity Autoencoder Gate: Per-Passage Adapter Libraries for Bit-Identical Test-Time Memory Michael Bonsignore
HUM AI MI Micheal Bee The Bonsignore Kernel We built a custom GPU operator that replaces the dot product in attention with a learned scoring function. It’s 25% faster than PyTorch at…
SCI HUM AI MI Micheal Bee Learn the Kernel The attention scoring function is not fundamental. It is an inductive bias that shapes how representation space is organized. Different…
HUM AI MI Micheal Bee Your “Clean” Training Data Isn’t Clean Shannon entropy reveals that 6% of a curated LLM corpus is garbage — and another slice is dead weight
HUM AI MI Micheal Bee PEER + Engram: 1.71 Perplexity at 510M Parameters on a Consumer GPU Michael Bonsignore