Running oMLX on Apple Silicon — Installation, Models, and Shell Management

Part 2 of 6: Getting the inference server running and keeping it under control

This article covers the practical setup of oMLX — the MLX-based inference server that powers the whole stack. By the end you will have a running server, three capable models downloaded, and a set of fish shell functions to start, stop, and check it without thinking.

If you have not read Part 1, the short version: oMLX uses Apple’s MLX framework, which treats CPU and GPU memory as a single unified pool. This makes it faster and more memory-efficient than Metal-based alternatives like Ollama on the same hardware.

Prerequisites

• Apple Silicon Mac (M1 or later)
• macOS 15 Sequoia or later (MLX requirement)
• Homebrew
• fish shell (or adapt the functions to zsh/bash)
• 32GB+ unified memory recommended for 26B+ models

Step 1 — Install oMLX

brew tap jundot/omlx https://github.com/jundot/omlx

brew install omlx

This installs the omlx CLI. Verify it is there:

which omlx

omlx — version

Optional: install MCP support so oMLX can run MCP servers directly (needed if you want to wire MCPs into oMLX itself rather than through the client):

/opt/homebrew/opt/omlx/libexec/bin/pip install mcp

Upgrade later:

brew update && brew upgrade omlx

Step 2 — Choose a port

The default oMLX port is 8000. I chose 11436 for two reasons:

1. Port 11435 was already in use on my machine.
2. Ollama runs on 11434 — keeping oMLX at 11436 puts them in the same neighbourhood and makes them easy to remember.

This port decision is baked into the fish functions below. Change the number in one place if you need a different port.

Step 3 — Fish shell management functions

Running omlx serve directly works, but you will type it often and want clean lifecycle management (start, stop, status, log tailing). Three fish functions handle this.

omlx-up

Start the server in the background, log to ~/.omlx/logs/server.log:

# ~/.config/fish/functions/omlx-up.fish

function omlx-up

— description “Start oMLX inference server on port 11436”

set -l port 11436

set -l model_dir ~/.omlx/models

set -l log_file ~/.omlx/logs/server.log

set -l api_key “YOUR_OMLX_API_KEY” # generate with: openssl rand -hex 12

# Check if already running

if pgrep -f “omlx serve” > /dev/null 2>&1

echo “oMLX is already running. Use omlx-status to check or omlx-down to stop.”

return 1

end

mkdir -p (dirname $log_file)

mkdir -p $model_dir

echo “Starting oMLX on port $port…”

nohup omlx serve — model-dir $model_dir — port $port — api-key $api_key >> $log_file 2>&1 &

sleep 2

if pgrep -f “omlx serve” > /dev/null 2>&1

echo “oMLX started (PID “(pgrep -f “omlx serve”)”)”

echo “Admin UI: http://localhost:$port/admin"

else

echo “oMLX failed to start. Check logs: tail -30 $log_file”

return 1

end

end

omlx-status

Process check + HTTP health check + last 30 log lines:

~/.config/fish/functions/omlx-status.fish

function omlx-status

— description “Check oMLX server status”

set -l port 11436

echo “=== Process ===”

if pgrep -f “omlx serve” > /dev/null 2>&1

echo “Running (PID “(pgrep -f “omlx serve”)”)”

else

echo “Not running”

end

echo “”

echo “=== HTTP health ===”

curl -sf http://127.0.0.1:$port/health && echo “OK” || echo “Not responding”

echo “”

echo “=== Last 30 log lines ===”

tail -30 ~/.omlx/logs/server.log

end

omlx-down

Graceful stop, SIGKILL fallback after 5 seconds:

# ~/.config/fish/functions/omlx-down.fish

function omlx-down — description “Stop oMLX inference server”

if not pgrep -f “omlx serve” > /dev/null 2>&1

echo “oMLX is not running.”

return 0

end

echo “Stopping oMLX…”

pkill -SIGTERM -f “omlx serve”

set -l waited 0

while pgrep -f “omlx serve” > /dev/null 2>&1; and test $waited -lt 5

sleep 1

set waited (math $waited + 1)

end

if pgrep -f “omlx serve” > /dev/null 2>&1

echo “Did not stop cleanly — sending SIGKILL”

pkill -SIGKILL -f “omlx serve”

else

echo “oMLX stopped.”

end

end

Tab completions

# ~/.config/fish/completions/omlx.fish

complete -c omlx-up -s p -l port -d “Port to bind (default: 11436)”

complete -c omlx-up -s m -l model-dir -d “Model directory”

complete -c omlx-up -s h -l help -d “Show help”

complete -c omlx-down -f -d “Stop oMLX inference server”

complete -c omlx-status -f -d “Check oMLX server status”

Step 4 — API key

The — api-key flag requires every API request to include:

Authorization: Bearer <your-key>

The key in the function above (YOUR_OMLX_API_KEY) is a placeholder — generate your own with openssl rand -hex 12 and replace it. The key is only enforced locally, but it is good hygiene: it prevents other processes on your machine from accidentally hitting the API and burning through model context.

Step 5 — Start the server and open the admin UI

omlx-up

Then open **http://localhost:11436/admin** in your browser. You should see the admin dashboard.

Step 6 — Choose models for your hardware

The right model depends on how much unified memory your Mac has. The rule of thumb for 4-bit quantised MLX models is roughly 0.5 GB per billion parameters for weights, plus the OS, oMLX process, and KV cache (plan for ~5–6 GB overhead).

Memory tiers at a glance

MoE models are an exception. Qwen3.6–35B-A3B-OptiQ-4bit is a Mixture-of-Experts model: it has 35B total parameters but only ~3B are active per token. Its memory footprint (~20 GB) is closer to a 14B dense model while its quality is closer to a full 35B. On 32 GB this model fits (barely) with a reduced context window of 32k–48k; on 24 GB it does not fit reliably.

16 GB — recommended models

At 16 GB, load one model at a time. oMLX’s LRU eviction will swap models automatically, but with only ~10 GB available you will feel it.

24 GB — recommended models

27B at 3-bit — quality dip vs 4-bit but fits in 24 GB

At 24 GB you can run one 14B model plus the embedding model simultaneously, or keep two 7–9B models loaded.

Context window: keep at 32k–64k. 128k KV cache at this tier consumes too much of your available memory headroom.

32 GB — recommended models

At 32 GB, run one large model at a time plus the embedding model. Context window: 64k is comfortable; 128k will strain memory.

64 GB — the setup in this series

All four fit simultaneously. Context window: 128k is fine.

How to download

In the admin dashboard at http://localhost:11436/admin, navigate to Models → Downloader. Search for the model name, check the file size, and click download. Models go to ~/.omlx/models/. Expect a few minutes per model.

Step 7 — Widen the context window

The default oMLX context window is 32,768 tokens. That is too small for real agent work — agent prompts include tool schemas, previous tool results, and conversation history, which can easily exceed 32k before the actual task begins.

In the admin UI: Settings → Global Settings

Set both Max Context Window and Max Tokens to the same value. Changes apply immediately — no restart needed.

Step 8 — Verify the API

curl http://127.0.0.1:11436/v1/models \

-H “Authorization: Bearer YOUR_OMLX_API_KEY”

Expected response:

{

“object”: “list”,

“data”: [

{“id”: “Qwen3.6–27B-4bit”, “object”: “model”, “owned_by”: “omlx”},

{“id”: “Qwen3.6–35B-A3B-OptiQ-4bit”, “object”: “model”, “owned_by”: “omlx”},

{“id”: “gemma-4–26b-a4b-it-4bit”, “object”: “model”, “owned_by”: “omlx”}

]

}

Send a test completion:

curl http://127.0.0.1:11436/v1/chat/completions \

-H “Authorization: Bearer YOUR_OMLX_API_KEY” \

-H “Content-Type: application/json” \

-d ‘{

“model”: “Qwen3.6–35B-A3B-OptiQ-4bit”,

“messages”: [{“role”: “user”, “content”: “Say hello in one sentence.”}],

“max_tokens”: 50

}’

If you get a completion back, the server is working correctly.

Step 9 — Run as a background service (optional)

If you want oMLX to start automatically at login:

brew services start omlx

Logs go to $(brew — prefix)/var/log/omlx.log.

I prefer the fish functions because I do not always want the inference server running — it holds memory. But for a machine dedicated to AI work, the service approach is cleaner.

What you have now

• oMLX running on port 11436, OpenAI-compatible API
• Three 26–35B models available
• 128k context window
• Fish functions to manage the server lifecycle
• API access verified

The next step is giving the model access to the web.