Loom/poly and Lucy: Building a Local Golang AI Engine That Runs BitNet, Qwen, SmolLM, and More

I’ve been working on Loom, a local AI runtime written primarily in Go. The goal is simple but ambitious: make AI models run locally, efficiently, and openly, without depending on Python servers, cloud APIs, or heavyweight framework stacks.

The newest work has been focused on loom/poly, the core neural engine, and lucy, the local interactive chat/runtime layer built on top of it.

This update is a big step forward: Lucy can now load and run several Hugging Face model families directly from the local HF cache, including Qwen, SmolLM2, 1bitLLM BitNet models, quantized local models, and now Microsoft’s official microsoft/bitnet-b1.58-2B-4T packed BitNet checkpoint.

You can find the project here:

https://github.com/openfluke/loom

And demos / experiments here:

https://www.youtube.com/@openfluke

What Loom Is

Loom is a local AI engine and neural runtime built around poly, a Go-based tensor and model execution system.

Instead of treating AI as something that only happens through Python notebooks, CUDA servers, or remote API calls, Loom explores a different direction:

AI as local infrastructure.

That means:

• Local-first inference
• CPU and GPU execution paths
• Hugging Face model loading
• Quantization experiments
• WebGPU and native Go runtime work
• Deterministic execution goals
• Edge-device friendly model experimentation
• Runtime visibility into memory, weights, telemetry, and model behavior

The project is still experimental, but it is moving toward a local AI engine that can ingest real model formats and run them without being glued to Python.

What Lucy Does

Lucy is the interactive runtime layer for Loom.

It scans the Hugging Face cache, lets you select models, loads model config/tokenizer/safetensors, builds a transformer graph in poly, and runs local chat.

Current Lucy model support includes:

• microsoft/bitnet-b1.58-2B-4T
• 1bitLLM/bitnet_b1_58-large
• 1bitLLM/bitnet_b1_58-3B
• Qwen/Qwen3-0.6B
• Qwen/Qwen3-1.7B
• Qwen/Qwen3-4B
• Qwen/Qwen3-8B
• HuggingFaceTB/SmolLM2-135M-Instruct
• HuggingFaceTB/SmolLM2-360M-Instruct
• HuggingFaceTB/SmolLM2-1.7B-Instruct
• local Loom quantized models such as Qwen and SmolLM variants
• experimental CPU ternary PTQ for ordinary FP32 models

Lucy also has an approved-model downloader path, so models can be pulled into the Hugging Face hub layout and used locally.

The Big New Piece: Microsoft BitNet in Go

The most interesting recent milestone is support for Microsoft’s official packed BitNet model:

microsoft/bitnet-b1.58-2B-4T

This model is different from ordinary FP16/BF16 transformer checkpoints. It ships packed 1.58-bit ternary weights using U8 tensors. That means the weights are already compressed into a BitNet-specific format.

Loom/poly now understands that format.

The runtime can:

• load Microsoft’s packed safetensors checkpoint
• decode the packed U8 BitNet weights
• preserve the model’s weight_scale tensors
• run CPU-only packed ternary inference
• use the correct tokenizer and chat template
• handle BitNet-specific relu2 activation
• support BitNet inner normalization tensors such as attention and FFN sub-norms

This is not just “calling a model from a library.” It is model format integration work at the runtime level.

Why BitNet Matters

BitNet b1.58 is important because it changes the shape of the inference problem.

Instead of storing transformer weights as FP16 or BF16, BitNet uses ternary weights:

-1, 0, +1

That makes the model dramatically smaller outside embeddings, and it opens the door to faster CPU inference because many multiplications can become add/subtract style operations.

In practice, doing this correctly is not just “convert weights to 1-bit.” A working BitNet runtime needs to care about:

• packed tensor layout
• row ordering
• activation quantization
• per-token scaling
• weight scale restoration
• tokenizer behavior
• prompt templates
• tied embeddings / LM head behavior
• memory release after packing
• CPU kernel layout

Loom/poly now has a working path for this in Go.

Why Go?

Most AI infrastructure today is Python-first. That makes sense for research, but it is not always ideal for local systems, embedded tools, desktop apps, or long-running runtimes.

Go gives Loom a different set of tradeoffs:

• simple native binaries
• easy concurrency
• predictable deployment
• good systems tooling
• no mandatory Python process
• practical networking and telemetry
• clean integration with local apps

Loom is exploring what a Golang AI engine can look like when it owns the runtime instead of wrapping another one.

Search terms this project now genuinely touches:

• Golang AI engine
• Go LLM runtime
• local AI inference in Go
• BitNet inference in Go
• CPU-only LLM inference
• Hugging Face safetensors Go runtime
• local transformer engine
• Go neural network runtime
• WebGPU AI runtime
• edge AI engine

Current Performance

On a MacBook Air CPU, Lucy is now able to run microsoft/bitnet-b1.58-2B-4T locally and produce coherent chat output.

It is not yet as optimized as bitnet.cpp, and that matters. Microsoft’s official C++ runtime is heavily optimized for this model family.

But the important thing is that Loom/poly now has a native Go path that can load the official packed Hugging Face checkpoint and run it directly.

That is a meaningful runtime milestone.

What Works Now

The current Loom/poly + Lucy stack supports:

• Hugging Face cache discovery
• safetensors loading
• LLaMA-style decoder block construction
• Qwen-style GQA model dimensions
• SmolLM2 chat models
• BitNet packed ternary inference
• Microsoft offline packed BitNet weights
• CPU-only inference
• experimental GPU/WebGPU paths
• memory footprint reporting
• tokenizer special-token masking
• chat templates
• approved model downloads
• experimental low-bit model morphing / PTQ

It is still experimental, but it is now running real models instead of toy networks.

What This Means

The direction of Loom is local AI sovereignty.

Not “send a prompt to a cloud API.”

Not “only run through Python.”

Not “wait for a giant framework to support your hardware.”

The goal is to understand and control the runtime all the way down:

• how weights are loaded
• how tensors are packed
• how models execute
• how much memory they use
• how CPU/GPU dispatch works
• how telemetry flows through the system
• how local agents can run without cloud dependency

Lucy is becoming the proving ground for that.

It gives Loom a real interactive test loop: load a model, talk to it, see if the math is right, see if the tokenizer is right, see if memory is sane, and improve the engine.

The Road Ahead

The next work is obvious:

• faster packed ternary CPU kernels
• better Unicode/tokenizer decoding
• lower memory use for embeddings and tied LM heads
• more quantized model formats
• better GPU/WebGPU integration
• more stable generation quality across model families
• cleaner model download and runtime UX
• broader support for local edge AI experiments

But the foundation is now there.

Loom is becoming a real local AI runtime in Go.

If you care about local AI, Golang AI engines, BitNet, edge inference, or running models without cloud APIs, check it out:

https://github.com/openfluke/loom

Demos and experiments:

https://www.youtube.com/@openfluke