Sovereign AI: Running Useful, Self-Hosted Agents on Modest Hardware

Start here: your first local model in ten minutes

You need a computer with 8 GB of memory and an internet connection for the first download. No graphics card is needed. Every command below was run exactly as written on a 2018 mini PC (Intel i5-8500T, 8 GB, no GPU, Rocky Linux 10) with Ollama 0.40.10, on 2026-10-23.

1. Install Ollama

Ollama is a free program that downloads models and runs them on your computer. On Linux:

curl -fsSL https://ollama.com/install.sh | sh

It installs in about 30 seconds, starts itself in the background and says the API is available at 127.0.0.1:11434. A warning that no GPU was found is normal on a computer without a graphics card. If the installer says it needs zstd, install the zstd package with your package manager and run it again.

On a Mac or Windows PC, download Ollama from ollama.com/download instead. Those two were not tested here.

2. Ask your first question

ollama run qwen3.5:4b --think=false "Write one shell command that counts the .log files under ./data, recursively."

The first time, this downloads the model, a 3.3 GB file (49 seconds on a fast connection). Then it answers in a few seconds:

find ./data -name "*.log" | wc -l

On the mini PC that took 3.6 seconds, at about 6 tokens per second (a token is about three quarters of a word).

3. Use it from other programs

curl http://localhost:11434/v1/chat/completions -d '{"model":"qwen3.5:4b","reasoning_effort":"none","messages":[{"role":"user","content":"Say OK and nothing else."}]}'

The reply has the same shape as OpenAI's API, with "content":"OK" inside. Any tool that can talk to OpenAI can use your own model instead: point it at http://localhost:11434/v1.

4. Give it hands

A model only writes text. A harness turns that text into actions: it reads files, runs commands and checks the result. One command installs OpenCode, a free coding agent, and starts it with your model:

ollama launch opencode --model qwen3.5:4b --yes

On the mini PC, OpenCode started with the local model already selected. In a separate test it finished a one-line task ("list the three largest files in this directory") in a few minutes. That is slow, and it all ran on one small machine. Small models do best on short jobs with a clear result you can check.

5. If you have more memory

With 16 GB or more, or a graphics card, try the larger qwen3.5:9b. On the 8 GB mini PC it fit in memory but wrote half a token per second, too slow to use. Section 3, Getting started, covers llama.cpp and the rest of the tested setup.

To remove the model later: ollama rm qwen3.5:4b. To stop Ollama running in the background on Linux: sudo systemctl stop ollama.

1. What sovereign AI means and why

Governments use "sovereign AI" for national models and datacenters. The same idea works for one person or one team. Sovereign AI, as used here, means four things are yours:

• The model: open weights in a file on your disk.
• The hardware: machines you control.
• The data: prompts and files stay on your network.
• The rules: you decide when the model changes.

A model is a file of numbers. llama.cpp is a free program that loads that file and answers over the same kind of API that Anthropic and OpenAI offer. A harness (pi, OpenCode, Claude Code) is the program that sends your request to the model, runs the commands the model writes, and sends the results back. Using a cloud model for one hard task can still be a sovereign choice when you make it on purpose.

The seven sections below give the reasons to run your own models, then what you give up.

Privacy

A prompt sent to Claude or ChatGPT, and every file attached to it, is processed on the provider's servers. A local model processes the same prompt in the memory of the machine that runs it. In every local test on this page, prompts crossed only one home network, from the machine running the agent to the machine running the model.

Your ideas

Code, notes and documents pasted into a hosted model leave your control and fall under the provider's terms. Consumer chat plans can use your conversations for training unless you opt out, and you trust the provider with work you have not published. Paid fine-tunes are exposed too. OpenAI's deprecation notice of 2026-04-22 also removes fine-tuned models on 2026-10-23, ft-gpt-3.5-turbo among them, so a fine-tune a customer paid to train stops answering on that date. A model file on your own disk has no shutdown date: the 2023 Llama 2 7B file used below (4.08 GB, sha256 08a5566d61d7cb6b420c3e4387a39e0078e1f2fe5f055f3a03887385304d4bfa) ran on a 2021 phone in October 2026.

Security

Running the model locally does not make an agent safe. Simon Willison calls the risk the lethal trifecta: an agent that can read private data, also reads untrusted text, and can send data out can be tricked into leaking that data. Removing any one of the three closes the path. The software supply chain carries risk too. LiteLLM 1.82.7 and 1.82.8, published to PyPI on 2026-03-24, carried a credential stealer; the web-search test below used 1.82.6, installed from a hash-checked wheel. The model card for nl2sh-1.5b, a small shell-command model, reports that 11.0% of its outputs on adversarial prompts would destroy data.

The same three abilities are what make an agent useful, so the fix is to split the work. Give each job only two of the three, and ask before anything leaves the machine. An agent that reads the web gets no secrets. An agent that holds secrets reads no outside text. Anything that sends data out, such as an email, a push or a web request, waits for your yes. For risky jobs, run the agent in a sandbox on a throwaway copy of your files; a bad command can still delete what is inside, so keep nothing there you would miss.

Dependence

When a vendor runs every layer for you, you never learn how the layers work, and leaving gets harder every month. Running locally shows how much a tool depends on its vendor's servers. Claude Code completed 3 of 3 shell tasks against a local model. Its web search stopped working: llama-server offers no search tool, and the local model answered a question about llama.cpp's latest release with an invented tag, "v0.2.2, 2024-06-12". Claude Code also sent a 15,680-token system prompt with its request. For Qwen3.5-2B, reading that prompt takes at least 23 minutes on the 2021 phone, 3 minutes on a 2018 mini PC, 51 seconds on the 2024 laptop's CPU and 17 seconds on its built-in GPU. These are floors: they stretch reading speeds measured on 512 to 2,861-token prompts to a 15,680-token one, and longer prompts read slower (on the phone, a 16K context cut reading speed about 16%). The measured speeds: 11.2 tokens per second on the phone, about 85 on the mini PC for a real 2,861-token agent prompt, 308 on the laptop CPU and 914 on its GPU.

Cost

Hosted models charge per token, and prices can rise once you depend on them. Subscription plans also change: lower limits, or features moved to a higher tier. A local model costs hardware you may already own, plus electricity. The Claude comparison below (246 shell tasks, 150 homelab questions, 12 desktop tasks and pilot runs) cost $0.74 at API prices, as reported by Claude Code; the runs used a subscription. The local hardware dates from 2018 to 2024, and its electricity was not metered in these tests. Buying new memory got expensive: a Reddit comment put 96 GB of laptop DDR5 at $200 in October 2025, and a 2026-08-18 post reported 128 GB of DDR5 at $3,399.

Model drift

A hosted model can change or disappear under a working setup. OpenAI's deprecations page lists 12 API model snapshots, among them gpt-4-turbo, o1 and o3-mini, with a shutdown date of 2026-10-23, each with a suggested substitute. On 2026-10-14 OpenAI announced that GPT-4o, GPT-4o mini, GPT-4.1 and GPT-4.1 mini leave the API on 2027-01-14. A prompt tuned for one model has to be retested on its substitute. A local model file stays byte-identical until you replace it. In the shell tests, both local models passed the same tasks in all 3 runs.

Outages

A hosted service can be down, slow or rate-limited. During the Claude comparison on 2026-10-22, Claude Code hit usage limits more than once, and the affected batches were rerun. The local runs waited only on the model. A model on your own machine keeps answering when the internet connection is down.

What you give up

Small local models are weaker. On 25 questions about a homelab, each given the 5 best-matching passages of the homelab's notes, Claude Sonnet 5.5 answered 13 to 14 correctly and a 7-billion-parameter local model answered 10 to 11. Small machines are slow at reading long prompts. The local models did best on jobs with a checkable result: shell commands, fixes on a running system, and data extraction. Local is not free: it costs hardware, power, and your time to patch systems, update drivers and swap models.

2. The hardware tested

Five devices ran models. None was bought for these tests.

The mini PCs and the laptop ran Linux. The phone ran stock Android with no root; llama.cpp ran from /data/local/tmp over adb. Two other machines supported the tests without serving a model: a Linux desktop that ran the router and the test scripts, and a workstation that ran the Claude comparison.

The three mini PCs boot over the network from one shared image with Warewulf, an open-source tool for provisioning cluster nodes. Disclosure: CIQ, where I work, sells support for Warewulf; these machines run the free open-source release. Warewulf was not benchmarked. It only puts the operating system on the machines.

Terms used below. A token is about three quarters of a word. Reading speed is how fast a machine processes your prompt (llama.cpp calls it prompt processing, pp). Writing speed is how fast it produces the reply (generation, tg). People read roughly 5 to 8 tokens per second (an estimate, not measured here). A parameter is one number inside a model; "4B" means 4 billion. A MoE (mixture of experts) model uses only part of its parameters for each token: Granite 4.0 H Tiny has 6.9B parameters and uses about 1B per token. Q4_K_M and similar names describe quantization, storing each parameter in about 4 to 5 bits instead of 16; gpt-oss-20b has 20.9B parameters and its file is 12.11 GB.

3. Getting started

All local runs used llama.cpp. Release b11784 (commit 493253fe5, published 2026-10-21) produced most numbers on this page. Its prebuilt Linux downloads include a CPU build and a Vulkan build.

3.1 Get llama.cpp

Prebuilt, for the laptop and any x86-64 Linux machine:

# CPU-only build
curl -LO https://github.com/ggml-org/llama.cpp/releases/download/b11784/llama-b11784-bin-ubuntu-x64.tar.gz
# Vulkan build, for a built-in AMD or Intel GPU
curl -LO https://github.com/ggml-org/llama.cpp/releases/download/b11784/llama-b11784-bin-ubuntu-vulkan-x64.tar.gz

mkdir llama-vulkan
tar -xzf llama-b11784-bin-ubuntu-vulkan-x64.tar.gz -C llama-vulkan
find llama-vulkan -name llama-server     # cd into the folder this prints

From source. The first mini PC runs used a source build of commit dd002b9 (GCC 11.5, CPU backend, AVX2); b11784 is the tag to check out now:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout b11784
cmake -B build
cmake --build build --config Release -j 6
ls build/bin/llama-server build/bin/llama-bench

3.2 Run a model

This is the server behind the pi bug fix in section 3.9 (Qwen3.5-4B Q4_K_M, 32,768-token context, thinking off):

./llama-server -m Qwen3.5-4B-Q4_K_M.gguf -c 32768 --reasoning off --jinja -t 8 --port 8080

This is the server used for the Claude Code test (Granite 4.0 H Tiny, laptop CPU, 32,768-token context):

./llama-server -m granite-4.0-h-tiny-Q4_K_M.gguf -c 32768 --reasoning-format none --jinja --port 8916

The Granite file is 4.25 GB on Hugging Face. llama-server can download it on first start with -hf; this form was not the one timed in the tests:

./llama-server -hf unsloth/granite-4.0-h-tiny-GGUF:Q4_K_M -c 32768 --reasoning-format none --jinja --port 8916

Notes from the tests:

• --reasoning-format none was required for Granite 4.0 H Tiny. Without it, Granite answered agent requests in prose and called no tools.
• --jinja makes llama-server use the model's own chat template, which Granite needs for tool calls. Release b11784 turns it on by default; older builds may not, so every command on this page spells it out.
• Qwen3.5 models write hidden "thinking" text by default. Turn it off with --reasoning off on the server, or send "chat_template_kwargs": {"enable_thinking": false} in each request. Before that change, the first Qwen3.5-2B test runs returned empty answers and scored 0.
• llama-server listens on 127.0.0.1 by default. To serve other machines, add --host with a LAN address and --api-key with a key the clients must send.
• Prompt caching is on by default (--cache-prompt). Section 5.5 shows what it saves.

Measure a machine the way the tables below were measured:

./llama-bench -m Qwen3.5-2B-Q4_K_M.gguf -p 512 -n 128 -r 3 -t 6

-p 512 reads a 512-token prompt, -n 128 writes 128 tokens, -r 3 repeats 3 times, and -t sets CPU threads (6 on the mini PCs, 12 on the laptop).

3.3 Point Claude Code at it

llama-server answers Anthropic's Messages API (/v1/messages), so Claude Code can use it with two environment variables. Tested with Claude Code 2.1.273, Granite 4.0 H Tiny on the laptop CPU, 3 of 3 shell tasks completed with real tool calls:

ANTHROPIC_BASE_URL=http://127.0.0.1:8916 ANTHROPIC_API_KEY=dummy-key \
  claude -p --permission-mode acceptEdits --allowedTools Bash \
  "Count the lines in notes.txt and write the number to count.txt"

The key value is ignored by llama-server. Claude Code features that run on Anthropic's servers, such as web search, do not work this way; section 3.8 covers the workaround that was tested. pi and OpenCode (section 3.9) talk to llama-server directly.

3.4 Use the laptop's built-in GPU (Vulkan)

The Radeon 760M runs llama.cpp's Vulkan build through the Mesa RADV driver. Install the driver and check that Vulkan sees the GPU:

# Fedora and Enterprise Linux family
sudo dnf install mesa-vulkan-drivers vulkan-loader vulkan-tools
# Debian and Ubuntu
sudo apt install mesa-vulkan-drivers libvulkan1 vulkan-tools

vulkaninfo --summary

Then run the Vulkan build with all layers on the GPU. The log line should name the GPU (on this laptop: RADV PHOENIX):

./llama-bench -m Qwen3.5-2B-Q4_K_M.gguf -ngl 99 -p 512 -n 128 -r 3 -t 12
./llama-server -m Qwen3.5-4B-Q4_K_M.gguf -ngl 99 -c 8192 --reasoning off --jinja --port 8092

For a CPU comparison on the identical binary, copy the folder and delete libggml-vulkan.so from the copy; llama.cpp then reports the CPU backend. Run only one llama-server per built-in GPU: two at once crashed with a Vulkan "device lost" error.

3.5 Run a model bigger than your RAM

llama.cpp can map the model file from the SSD and read the parts it needs. On an 8 GB machine, turn off weight repacking, or llama.cpp tries to copy the weights into RAM (a 9.7 GB buffer for gpt-oss-20b) and fails to load:

# benchmark (llama-bench spells the flag --repack 0)
./llama-bench -m gpt-oss-20b-MXFP4.gguf --repack 0 -lm mmap -p 128 -n 32 -r 3 -t 6

# server (llama-server spells it --no-repack)
./llama-server -m gpt-oss-120b-MXFP4.gguf --no-repack -lm mmap -t 6 -c 8192 --jinja --port 8080

Close other programs first: free RAM becomes page cache for the model file. Section 5.2 shows two mini PCs differing 2 times on the same file for reasons the tests did not settle.

3.6 Run a model on an Android phone

Tested path: a CPU-only build pushed over adb

llama.cpp's prebuilt Android binaries link OpenCL 3.0, and the S21 Ultra's driver is older, so they did not run. A CPU-only build made with the Android NDK did. The recipe below is the one in llama.cpp's docs/android.md; the exact flags of the tested build were not recorded.

# on a Linux machine with the Android NDK installed
cmake -DCMAKE_TOOLCHAIN_FILE=$ANDROID_NDK/build/cmake/android.toolchain.cmake \
  -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-28 \
  -DGGML_NATIVE=OFF -DGGML_OPENMP=OFF -DGGML_LLAMAFILE=OFF -DLLAMA_OPENSSL=OFF \
  -B build-android
cmake --build build-android --config Release -j
cmake --install build-android --prefix ./android-install --config Release

# copy to the phone (USB debugging on) and start a server with 3 threads
adb shell mkdir -p /data/local/tmp/llama
adb push android-install /data/local/tmp/llama/
adb push Qwen3.5-0.8B-Q8_0.gguf /data/local/tmp/llama/
adb shell "cd /data/local/tmp/llama && LD_LIBRARY_PATH=android-install/lib android-install/bin/llama-server -m Qwen3.5-0.8B-Q8_0.gguf -t 3 -c 8192 --jinja --port 8091"

# on the computer: reach the phone's server at 127.0.0.1:8094
adb forward tcp:8094 tcp:8091

On the Snapdragon 888, 3 threads beat 4 because the phone throttled less. For Qwen3.5-0.8B, the 8-bit file (Q8_0) was the fastest of 4 quantizations at 3 threads (21.8 tokens per second writing, against 18.8 to 20.3); an 8,192-token context cost no speed, and 16,384 cost about 16% of reading speed. For Llama 2 models, pass the official chat template with --chat-template-file; with the default fallback template, Llama 2's output was incoherent.

Termux route (not the path tested here)

Termux is a Linux environment for Android, installed from F-Droid or its GitHub releases. Termux ships a llama-cpp package; its build recipe pins llama.cpp v0.7.0.

pkg update
pkg install llama-cpp
llama-server -m Qwen3.5-0.8B-Q8_0.gguf -t 3 -c 8192 --jinja --port 8091

On Samsung phones, set Termux to unrestricted battery use, or Android may put it to sleep. Watch the battery temperature: the 10-minute test in section 5.1 ended at 40.5 °C, below its 42 °C pause point.

3.7 One address for every machine: llama-swap

llama-swap is a router. Each machine runs its own model under its own name, and clients send every request to the router's single address. Version v265 was tested (binary sha256 cb1f9a94be05afa129d3c17868f1dd2ca710b919333c7b868239dc31575dcfa1), from github.com/mostlygeek/llama-swap/releases. This config follows llama-swap's remote-upstream pattern: cmd: sleep keeps a placeholder process alive, proxy points at the remote llama-server, and checkEndpoint: none skips the local health check.

# llama-swap.yaml (addresses replaced with placeholders)
listen: 127.0.0.1:8890

models:
  minipc-granite-tiny:
    cmd: sleep 999999
    proxy: http://127.0.0.1:18918     # SSH tunnel to the mini PC's llama-server on :8918
    checkEndpoint: none

  laptop-granite-tiny:
    cmd: sleep 999999
    proxy: http://LAPTOP_IP:8916       # laptop llama-server started with --host LAPTOP_IP
    checkEndpoint: none

  phone-qwen08b:
    cmd: sleep 999999
    proxy: http://127.0.0.1:8094       # adb forward tcp:8094 tcp:8091
    checkEndpoint: none

# on the mini PC
./llama-server -m granite-4.0-h-tiny-Q4_K_M.gguf -t 6 --reasoning-format none --jinja --port 8918
# on the router machine
ssh -N -L 18918:127.0.0.1:8918 user@MINI_PC_IP &
adb forward tcp:8094 tcp:8091
./llama-swap -config llama-swap.yaml -listen 127.0.0.1:8890

# any OpenAI-style client
curl http://127.0.0.1:8890/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"minipc-granite-tiny","messages":[{"role":"user","content":"hi"}]}'

llama-swap does not retry on another machine. In 1 failover trial, a request to the stopped laptop's model failed at once with HTTP 502; the same request sent with the mini PC's model name answered 0.7 seconds after the laptop's server was stopped. The client has to pick the other name.

3.8 Web search for a local model: LiteLLM and SearXNG

SearXNG is a self-hosted search engine. LiteLLM (version 1.82.6) sat between the client and llama-server, caught the model's search tool call, ran the search on SearXNG and returned the results to the model. Install the pinned version from a checked wheel:

pip download litellm==1.82.6 --no-deps -d wheels
sha256sum wheels/litellm-1.82.6-*.whl
# expect 164a3ef3e19f309e3cabc199bef3d2045212712fefdfa25fc7f75884a5b5b205
pip install wheels/litellm-1.82.6-*.whl

# litellm-websearch.yaml (keys removed, address replaced)
model_list:
  - model_name: granite-tiny-local
    litellm_params:
      model: openai/granite-4.0-h-tiny
      api_base: http://127.0.0.1:8916/v1
      api_key: dummy-key          # llama-server needs a key header; the value is ignored

litellm_settings:
  callbacks: websearch_interception

websearch_interception:
  search_provider: searxng
  searxng_url: http://SEARXNG_HOST:PORT

Three details from the test. In 1.82.6 the interception callback had to be registered in Python, and the SearXNG connector read api_base from the router's search tool list and ignored searxng_url:

litellm.callbacks = [WebSearchInterceptionLogger(enabled_providers=["openai"])]

SEARCH_TOOLS = [{"search_tool_name": "searxng-search",
  "litellm_params": {"search_provider": "searxng",
                     "model": "searxng/search",
                     "api_base": "http://SEARXNG_HOST:PORT"}}]

The tool offered to the model must be named exactly litellm_web_search:

TOOLS = [{"type": "function", "function": {"name": "litellm_web_search",
  "description": "Search the web",
  "parameters": {"type": "object", "properties": {"query": {"type": "string"}}}}}]

Results, 3 runs with Granite 4.0 H Tiny: 2 runs returned answers grounded in real search results, and 1 returned nothing. The answers cited the llama.cpp GitHub repository instead of the release tag the question asked for. Only SearXNG's GitHub engine answered; its Brave, DuckDuckGo and Startpage engines were rate-limited on that instance. The test called LiteLLM through its OpenAI-style interface. A full Claude Code session routed through LiteLLM was not recorded.

3.9 Open agents: pi and OpenCode

pi and OpenCode are open-source coding agents that talk to any OpenAI-compatible server, so they need no translator in front of llama-server. Both were installed from scratch on the laptop on 2026-10-22. For the shell tasks below, both pointed at this Granite server (CPU-only build, 8 threads); the bug fix used the Qwen3.5-4B server from section 3.2 on the same port:

./llama-server -m granite-4.0-h-tiny-Q4_K_M.gguf -c 32768 --reasoning-format none --jinja --port 8080 -t 8

pi 1.4.0

pi needs Node.js 22.19 or later (the test used 22.23.1). The npm package is @earendil-works/pi-coding-agent; older guides name @mariozechner/pi-coding-agent.

npm install -g --ignore-scripts @earendil-works/pi-coding-agent

# if npm's global prefix is owned by root, install under your home instead
npm install -g --ignore-scripts --prefix ~/.local @earendil-works/pi-coding-agent
# the binary is then ~/.local/bin/pi

Config file: ~/.pi/agent/models.json. The agent directory is ~/.pi/agent, not ~/.pi.

{
  "$schema": "https://pi.dev/schemas/models.schema.json",
  "providers": {
    "llamacpp": {
      "name": "llama.cpp (local)",
      "baseUrl": "http://127.0.0.1:8080/v1",
      "apiKey": "dummy-key",
      "api": "openai-completions",
      "models": [
        {
          "type": "chat",
          "id": "granite",
          "name": "Granite 4.0 H Tiny Q4_K_M",
          "reasoning": false,
          "input": ["text"],
          "cost": { "input": 0, "output": 0, "cacheRead": 0, "cacheWrite": 0 },
          "contextWindow": 32768,
          "maxTokens": 4096
        },
        {
          "type": "chat",
          "id": "qwen3.5-4b",
          "name": "Qwen3.5-4B Q4_K_M",
          "reasoning": false,
          "input": ["text"],
          "cost": { "input": 0, "output": 0, "cacheRead": 0, "cacheWrite": 0 },
          "contextWindow": 32768,
          "maxTokens": 4096
        }
      ]
    }
  }
}

llama-server answers with whichever model it loaded, so the entry you pick should match the server you started. The qwen3.5-4b entry follows the same pattern as the tested Granite entry; its exact file from the bug-fix takes was not saved.

pi --list-models granite        # expect: llamacpp  granite  32.8K  4.1K
pi --print --provider llamacpp --model granite "<task>"

# interactive, as in the bug-fix takes
pi --provider llamacpp --model qwen3.5-4b

OpenCode 1.19.4

The installer is a script piped to bash; read it first, or use the npm package opencode-ai.

curl -fsSL https://opencode.ai/install | bash      # binary: ~/.opencode/bin/opencode
# or
npm i -g opencode-ai

Config file: ~/.config/opencode/opencode.json.

{
  "$schema": "https://opencode.ai/config.json",
  "model": "llamacpp/granite",
  "provider": {
    "llamacpp": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "llama.cpp (local)",
      "options": {
        "baseURL": "http://127.0.0.1:8080/v1",
        "apiKey": "dummy-key"
      },
      "models": {
        "granite": {
          "name": "Granite 4.0 H Tiny Q4_K_M",
          "tool_call": true,
          "limit": { "context": 32768, "output": 4096 }
        }
      }
    }
  }
}

opencode models | grep llamacpp     # expect: llamacpp/granite
opencode run -m llamacpp/granite "<task>"

Results

A real bug from a one-sentence request

A small repository held logstats.py, a log summarizer, and test_logstats.py. It counted server errors with if status > 500:, so a 500 response was missed and pytest reported 2 failed, 2 passed. The request was typed into interactive pi 1.4.0 the way a person would type it: "the tests are failing, can you figure out why and fix it?" No file names, no steps.

Qwen3.5-4B read the files pi showed it, found the off-by-one and wrote if status >= 500:. pi applied the edit with its edit tool and ran pytest: 4 passed. The pass rule was checked outside pi: pytest reported 4 passed and git showed only logstats.py changed. The model decides what to change; the harness reads the files, applies the edit and runs the tests.

Pick the model for the job. Granite scored 39 of 41 on short shell commands and failed this coding job every time; Qwen3.5-4B, trained harder on code, fixed it every time. With a step-by-step prompt and a two-file FizzBuzz repository, Granite did fix a bug once in --print mode, after earlier attempts guessed a wrong file path and once edited toward the broken behavior.

Problems hit during setup:

• OpenCode's first launch failed with "Database is not empty and has no session table". A 14 MB ~/.local/share/opencode/opencode.db left by an older OpenCode install caused it. Moving that file aside fixed it.
• OpenCode with Granite needed exact prompts. Asked vaguely to "run wc and write the number", it wrote wrong file contents, such as the literal text <output_of_wc_command>. Prompts that said to execute an exact command worked.
• Granite called tools in both agents on the first try with --reasoning-format none and --jinja on the server.

4. Harnesses: what makes a small model useful

4.1 A model does nothing on its own

A model is a file of numbers behind an API: text in, text out. It never starts anything. Something has to send it a prompt: you typing, a cron job, or an event such as a new email or a failed build. The harness is that something. It builds the prompt, gives the model its tools, runs the commands the model asks for, feeds the results back and loops. A check decides when the job is done.

• Chat apps are the simplest harness: Ollama's app, LM Studio and Open WebUI.
• Coding agents in the terminal read your code, run commands and edit files. OpenCode, pi, Claude Code and Codex all work with local models. Ollama documents a one-command launch for OpenCode and Claude Code; it was not tested here.
• Editor agents such as Cline, Continue and Zed do the same inside the editor.
• Always-on agents such as Hermes Agent and OpenClaw run on a schedule, keep memory and reach you through chat apps.

Picks after these tests: OpenCode, pi and Hermes Agent. The harness also changes cost. A 2026 ICML paper by Vats and Golev ran the same models through three open harnesses: pass rates were within 8 points, and tokens per solved task differed by up to 40 times.

4.2 The same model passed or failed depending on the setup

The task: backport one upstream Linux kernel fix onto an older kernel tree, graded the same way every run. The benchmark ran 136 runs across models, providers and harnesses. Its repository is not public.

Why, from the benchmark's own write-up:

• Under Codex and Claude Code, all 6 Kimi K3 runs finished with a wrong patch. The 3 Claude Code patches were wrong in the same way.
• One Codex run also stalled twice. Codex ends a turn whenever a reply contains no tool call, so a model that describes its next step instead of calling a tool stops mid-job. That run still failed after two nudges.
• A translation layer between harness and server dropped the reasoning-effort and output-limit settings until they were forced.
• One provider made Kimi K3 write the literal text "[Tool use interrupted]", and replaying that history made it worse.
• The 20 of 20 GLM runs used a patched OpenCode build with a cap on its own summary requests, on a different provider and quantization, so that row does not isolate the harness.

The same Kimi K3 weights on the same servers passed 11 of 15 runs in one harness and 0 of 3 in each of two others.

4.3 Fixes the harness and gateway needed

These fixes came from three months of serving open models to a team: in a locked-down OpenCode build, and in llm-relay, the gateway I run at work. Most people should not have to make them; they are why the choice of harness and gateway matters.

4.4 Desktop jobs on a 2018 mini PC

A small Python agent with one tool, run_command, ran inside an Omarchy desktop virtual machine and sent its requests to Granite 4.0 H Tiny on a 2018 mini PC. A separate check looked at the machine after every job.

• Exact prompts: six jobs were written. Granite on mini PC 3 passed 5, with a median of 6.2 seconds per job. Two Claude models, driven through Claude Code instead of the Python agent, also passed 5, in 5.6 to 5.8 seconds. Both Claude models failed the sixth job the same way: they ran the commands and reported the right answers, but the checker looked for a word the correct output does not contain. That job is not counted, so the score is 5 of 5.
• Casual prompts, recorded with the model on mini PC 1: the agent killed a stuck process and fixed a file's permissions, 2 of 3. Asked to change key=value to mode=fast in a config file, it wrote key=fast and reported success. The check caught it. In an earlier take with a vaguer prompt, it looped on the config until it hit its turn limit.

4.5 Lessons for small models

5. Measured results

All speeds are tokens per second. "Reps" is the number of repetitions behind each number. Where two sessions disagree, both appear.

5.1 Speeds by device and model

What the table shows:

• Memory bandwidth sets writing speed on a CPU. With the same build (dd002b9) and 6 threads on both, the laptop CPU wrote 2.4 to 2.5 times faster than the mini PC on all three models (33.3 against 13.2 for Qwen3.5-2B). Its memory bandwidth is 4.8 times higher.
• The laptop's built-in GPU read prompts 2.7 to 3.3 times faster than its own CPU and wrote 1.4 to 1.9 times faster, on the same binary.
• The MoE model wrote faster than the smaller dense one: Granite 4.0 H Tiny (6.9B total, about 1B active) wrote 15.3 tokens per second on the mini PC, against 13.2 for the 1.9B Qwen3.5-2B.
• The phone served Qwen3.5-0.8B Q8_0 with 3 threads for 10 minutes without a pause: 486 requests in 601 seconds, median writing speed 19.31 tokens per second. The battery went from 28.9 to 40.5 °C, rising about 0.7 °C per minute at minute 9; the test's pause point was 42 °C. 1 run.

5.2 Models bigger than RAM

• gpt-oss-20b (20.9B parameters, 3.6B active per token) ran on an 8 GB mini PC from the SSD. Two mini PCs on two llama.cpp builds measured 6.06 and 2.93 tokens per second. The cause of the 2-times gap is not established. Mini PC 2, the slower one, reported 7.2 GB of memory available before its run, so a memory shortage does not obviously explain it. Qwen3.6-35B-A3B on the mini PCs varied more, from 0.27 to 3.49 tokens per second across 3 sessions, so it is left out of the table.
• gpt-oss-120b (116.83B parameters, 63.39 GB) ran on an 8 GB mini PC at 0.23 to 0.42 tokens per second and produced a correct one-paragraph explanation of mixture-of-experts models after 63 seconds. It runs; it is too slow for conversation.
• On the phone, gpt-oss-120b was not run: its 58 to 63 GB file did not fit in the phone's free storage. The BigMoeOnEdge project (github.com/Helldez/BigMoeOnEdge) publishes 1.3 tokens per second for gpt-oss-120b on a 12 GB OnePlus 15R with its default settings, 2.2 with tuned settings, and 0.09 with plain mmap. Those are the project's numbers and were not reproduced here.
• The phone's battery went from 27.2 to 34.3 °C during the 35B runs.

5.3 Claude against a 2018 mini PC, same tasks

• Granite's 2 misses were real errors. Asked for the largest number in a file, it wrote awk '{m=$1} END {print m}', which prints the last number. On another task it garbled the quoting of a printf command.
• Qwen3.5-2B's 8 misses were also model errors. In 4 it wrote line-break escapes without quotes (printf x\ny\n); bash removes unquoted backslashes, so the output was xnyn. Granite and Claude quoted the escapes and passed. Qwen also piped grep -c into wc -l, used sed -i where printed output was asked for, and gave paste a flag it does not have.
• Sonnet's misses were reply formatting: it wrapped a command in Markdown backticks, which the shell runs as command substitution, or replied with a tool-call fragment. Its best run scored 41.
• The two local models passed the same tasks in every run. Sonnet's 5 runs ranged from 39 to 41.
• Claude received each shell request with tools disabled, and its reply was run and checked by the same scorer as the local models.

5.4 A 2023 model against 2026 models, same phone

The 2026 Qwen3.5-2B has 3.6 times fewer parameters than Llama 2 7B, a 3.2 times smaller file, writes 2.8 times faster, and passed 34 shell tasks against 3. Llama 2 passed only the 3 tasks whose result is a file (append, make a directory, move a file). Most of its replies were prose ("Sure! Here is the shell command...") or backtick-wrapped commands instead of the bare command the task asked for. Probes with Llama 2's official prompt format returned sensible commands, so the low score reflects the model, not a broken setup.

5.5 Prompt caching

Agents resend a long block of instructions with every turn. llama-server keeps the part of the prompt it has already read and skips it next time.

The cached turn is almost all writing: 128 tokens at 12 tokens per second take about 10.7 seconds. Without caching, every agent turn on this mini PC would cost about 23 seconds of reading before the first word.

5.6 Three machines as one, against one laptop

llama.cpp's RPC mode splits one model across machines. The pool was mini PC 1 (head, 8 GB), mini PC 2 (worker, 8 GB, 1 Gb Ethernet) and the laptop (worker, 16 GB, Wi-Fi 6). It served Qwen3.6-35B-A3B UD-Q4_K_XL, a 22.85 GB file that fits no single member's RAM. The laptop alone ran the same file from its SSD.

The pool gave room: it ran a model no member could hold in RAM. It did not give speed: every token crossed the network between machines, and the laptop alone wrote about 1.4 times faster. The pool read a long prompt somewhat faster than the laptop's first pass. Section 6 lists what broke.

Splitting by layers, as llama.cpp's RPC mode does, sends every token through every machine in turn. Splitting within each layer needs machines to trade results many times per token, which takes RDMA-class links, where one machine writes straight into another's memory. Fast links change the result. Jeff Geerling ran Kimi K2 Thinking (1 trillion parameters, 32B active) on four Mac Studios linked by Thunderbolt 5, using RDMA added in macOS 26.2 in December 2025. Exo 1.0 over RDMA wrote 28.3 tokens per second, against 16.4 for llama.cpp over TCP on the same four machines (his post; not reproduced here).

5.7 Homelab questions: how a small model fails

Granite 4.0 H Tiny on mini PC 3 answered 25 questions about a homelab's hardware and network. A keyword search fed it the 5 most relevant chunks of the homelab's notes, with an instruction to copy the answer exactly. An answer passed when it contained the expected string. Scores: 10, 10 and 11 of 25 in 3 runs.

The 15 misses in run 1, by kind:

• 11 gave the right kind of answer with the wrong value. Asked for mini PC 2's cluster address, it gave mini PC 3's address, in all 3 runs. Asked for a mini PC's RAM, it answered "1G" against "8GB".
• 3 gave a different kind of thing. Asked for a CPU model, it named a network chip.
• 1 may be an answer-key error. The key says DDR4-2667; the model said DDR4-2666, which matches the mini PC's own hardware report.

Which chunks the model saw was not recorded, so the split between retrieval misses and model misses is unknown. With the same retrieval, Claude Sonnet 5.5 answered 13 to 14 of the same 25.

5.8 One model, five file sizes

The same model, Granite 4.0 H Tiny, ran at five quantizations on mini PC 2 (8 GB), each scored on the 41 shell tasks.

• Writing speed nearly doubled from Q8_0 to Q2_K, 10.1 to 19.8 tokens per second, because each token reads fewer bytes from memory.
• Reading speed stayed between 95 and 99 tokens per second below Q8_0, because reading is limited by compute.
• Q4_K_M scored 39, one task more than Q8_0, and wrote 50% faster. With one run per file, rounding can flip a borderline task either way, so scores of 37 to 39 do not rank the files. On an 8 GB machine, Q8_0 was the biggest file, the slowest, and no more accurate.
• Shell commands are a forgiving test. Harder reasoning tasks probably lose more at 2 to 3 bits; that was not measured here.

Published: the same cut on a small and a big model

A 2025 study (arXiv 2505.02214, Table 4) quantized Qwen3 models of several sizes and scored them on MMLU, a 57-subject multiple-choice exam with four answers per question, so 25% is a random guess.

At 4 bits the small model keeps 89% of its score and the big one 96%. At 3 bits the 0.6B model falls to a random guess, 25%, while the 32B still scores 55%. The study cuts every layer to the same bits, which is harsher than GGUF k-quants such as Q3_K_M and Q2_K that keep key layers at higher precision. Rule of thumb: start with Q4_K_M, go lower only to fit in memory, and test your own tasks.

6. What did not work, and why

Colibri was slower than llama.cpp when the model nearly fit

Colibri streams model weights from the SSD. On the laptop, its Qwen3.6-35B-A3B (int4, 23.1 GB) wrote about 2.7 tokens per second and took 436.3 seconds to finish a 2,000-token prompt. llama.cpp with a 16.85 GB file of the same model wrote 13.80 tokens per second, about 5 times faster. The files differ in quantization and size, so the comparison is not exact. Colibri's Vulkan mode did not help: with the 760M sharing system memory, its planner found 873 MiB free against a 1.00 GiB reserve and placed 0 of 40 layers on the GPU. Colibri did run an 85 GB model that llama.cpp was not tried on, at about 2.4 tokens per second.

RPC broke repeatedly

llama-bench over the RPC pool hung for more than 35 minutes at 0% CPU and was killed; the speeds in section 5.6 come from the server API instead. The head node crashed twice: once mid-answer on a network send failure, once at load when a worker had exited. A third worker is a known deadlock case (llama.cpp issue 28487), so the phone stayed out of the pool. llama.cpp's own RPC README calls the backend "fragile and insecure" and says never to run it on an open network.

Qwen3.5-4B did not call tools in Claude Code

Its built-in chat template rejected the system messages Claude Code inserts mid-conversation. With --chat-template chatml it ran, but wrote code blocks instead of tool calls, so Claude Code executed nothing. Granite 4.0 H Tiny worked once --reasoning-format none was set. Reddit tests point the same way: one 21-model comparison found a model's tool-call score rose from 0.640 to 0.880 after its parser was fixed.

The 2018 Intel graphics was slower than the CPU

Mini PC 2's Intel UHD 630, through Vulkan, read at 35.6 and wrote at 7.2 tokens per second on Qwen3.5-2B (2 reps). The same mini PC's CPU read at 138.7 and wrote at 13.2 (3 reps, an earlier build). A built-in GPU helps only when it is recent: the 2024 laptop's 760M was 3 times faster than its CPU at reading.

Smaller failures

• The phone's Adreno 660 GPU went unused. llama.cpp's OpenCL support list starts at the Adreno 750, and the prebuilt Android binaries would not load.
• Two llama-servers on one built-in GPU crashed with "device lost". One more device-lost crash happened during a 100-post extraction run; a supervisor script restarted the server.
• On that extraction run (Qwen3.5-4B, laptop GPU, 1 run), 72 of 100 replies were valid JSON, at 848 posts per hour. The first batch returned nothing usable until thinking mode was turned off. llama-server's -jf flag, which constrains output to a JSON schema, was not used in that run.
• With weight repacking on (the default), gpt-oss-20b failed to load on an 8 GB mini PC. --no-repack on llama-server, or --repack 0 on llama-bench, fixed it.
• llama.cpp served Llama 2 on the phone with a fallback template until the official template was passed with --chat-template-file.
• 2 of the 41 original shell answer keys were wrong: one expected another machine's hostname, and one did not match its own input file. Every model lost those 2 points until the keys were fixed. Read your own answer keys before trusting a score.

7. Methods

Software versions

• llama.cpp dd002b9 (2026-10-20): built from source with GCC 11.5, CPU backend, AVX2. First round of mini PC and laptop CPU benchmarks.
• llama.cpp b11784 (commit 493253fe5, 2026-10-21): prebuilt Linux CPU and Vulkan downloads, and a CPU-only Android NDK build. Most numbers on this page.
• llama.cpp e36df60: an older binary already on mini PC 3, used for gpt-oss-120b and one Granite benchmark.
• Colibri v2.0.0 (e0db083), llama-swap v265 (d1e743f), LiteLLM 1.82.6.
• Claude Code 2.1.273 for the local-model test; Claude Code 2.1.308 for the Claude comparison.

Benchmarks

• Small models: llama-bench -p 512 -n 128 -r 3. Threads: 6 on the mini PCs, 12 on the laptop CPU, 3 or 4 on the phone as noted. The quantization runs in section 5.8 added --repack 0 -lm mmap.
• Models from the SSD: llama-bench --repack 0 -lm mmap -p 128 -n 32 -r 3 -t 6, two passes per model. Whether the page cache was emptied between passes was not recorded. One later pass with a recorded cache drop, on an older build, measured Qwen3.6-35B-A3B at 2.50 tokens per second cold against 3.51 warm.
• CPU against GPU on the laptop: one b11784 download, with libggml-vulkan.so removed for the CPU pass. A pass where the "CPU" numbers matched the GPU numbers was thrown out.
• Time to first token and serving speed: llama-server's streaming API with prompts of about 100, 2,000 and 4,000 tokens.

How tasks were checked

• Shell tasks (41): a plain-English request such as "count the lines in notes.txt". The model's command ran in a scratch directory with fixed input files. A checker compared the output, or the resulting file, with an answer key. 3 runs per model (5 for Sonnet). Two original keys were wrong: one expected the hostname of a different machine, and one did not match its own input file. The hostname key now compares with hostname run on the grading machine, and the other key matches its input file. The first scorer also kept only the first line of each reply; the fixed scorer strips Markdown fences and keeps multi-line commands. Every saved answer was rescored with the fixed keys and scorer, without rerunning a model.
• Homelab questions (25): keyword retrieval of the top 5 chunks from a scrubbed copy of a homelab inventory and wiki, an instruction to copy the answer exactly, and a pass when the expected string appeared in the answer.
• Claude: the same task files, keys and checkers. Shell tasks went through claude -p with tools disabled. Claude Haiku 5.5 was also run, but its scorer kept only the first line of each reply and the full replies were not saved, so its scores are left out.

Who ran the tests

An AI agent (Hermes Agent) ran the experiments overnight from written plans and logged every run as one line of JSON. The overnight log holds 817 records: 793 completed, 10 failed and 7 voided. Later runs (big models, Claude, the 2023 model, the RPC pool) were logged the same way. No model ran on the machine that orchestrated the tests. The results were then audited against the log: numbers with conflicting runs appear as ranges or are left out.

8. Reddit trend

Posts about small hardware and posts about agents both grew as a share of r/LocalLLaMA and r/LocalLLM. Small-hardware posts went from 14.1% in October 2025 to 21.7% in October 2026. Agent posts went from 14.1% to 28.0%, after a peak of 30.5% in March 2026.

Method: keyword matching on each post's title and body. A post counts once per category when any keyword matches. Matching counts mentions, not topics: a post that says "laptop" or "budget" once counts as small hardware, and a post that says "agent" once counts as agents. The script below holds both keyword lists; it reads posts-LocalLLaMA.jsonl and posts-LocalLLM.jsonl (one Arctic Shift post object per line) from a reddit2 folder next to it.

"""Monthly share of r/LocalLLaMA and r/LocalLLM posts about small hardware and about agents.

Input: the Arctic Shift full listings in research/reddit2/posts-*.jsonl (2025-10-01 to 2026-10-19).
Small hardware = the phones, sbc_edge and minipc_laptop_cpu theme patterns from the harvest script.
Agents = the AGENT pattern below. A post counts when its title or body matches.
Usage: python3 -I reddit-trend.py
"""
import datetime
import json
import os
import re
from collections import Counter

BASE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "reddit2")
SMALL = [
    # phones
    r"\bphones?\b", r"\bandroid\b", r"\biphone", r"\bios\b", "termux", "pocketpal", r"\bmlc\b", "executorch", "litert",
    "mediapipe", "edge gallery", "snapdragon", "exynos", "dimensity", r"tensor g\d", "adreno", r"\bmali\b", "hexagon",
    "smartphone", r"\btablet", r"\bipad", r"\bmnn\b", "chatterui", "layla", "smolchat", "phonon", r"\bgalaxy s\d",
    # single-board computers and edge devices
    "raspberry", r"\brpi\b", r"\bpi ?[45]\b", "orange ?pi", "rk3588", "rockchip", r"\bnpu\b", "jetson", r"\borin\b",
    "hailo", r"\bcoral\b", r"\bsbc\b", "radxa", "embedded", r"\bedge devices?\b", "esp32", "microcontroller",
    # mini PCs, laptops, CPU-only
    r"mini ?pcs?\b", r"\bn1[05]0\b", r"\bnuc\b", "thinkcentre", "optiplex", "elitedesk", r"old laptop", r"\blaptops?\b",
    "chromebook", "steam deck", r"\bigpu\b", r"integrated (gpu|graphics)", r"\b[78][689]0m\b", "lunar lake",
    "meteor lake", r"cpu[- ]only", r"\bno gpu\b", r"without (a )?gpu", r"\bddr[45]\b", "memory bandwidth",
    "dual[- ]channel", r"\bpotato\b", r"low[- ]end", r"\bbudget\b", r"old hardware", r"e-?waste",
]
SMALL_RE = [re.compile(p, re.I) for p in SMALL]
AGENT = re.compile(r"\bagent(s|ic)?\b|\bharness|opencode|claude code|tool[- ]?call|\bmcp\b|hermes agent|\bpi agent"
                   r"|openhands|\bgoose\b|\baider\b|\bcline\b", re.I)


def main():
    total, small, agents = Counter(), Counter(), Counter()
    for sub in ("LocalLLaMA", "LocalLLM"):
        with open(os.path.join(BASE, f"posts-{sub}.jsonl")) as f:
            for line in f:
                try:
                    post = json.loads(line)
                except ValueError:
                    continue
                month = datetime.datetime.fromtimestamp(int(post["created_utc"]), datetime.timezone.utc).strftime("%Y-%m")
                text = (post.get("title") or "") + "\n" + (post.get("selftext") or "")
                total[month] += 1
                small[month] += any(r.search(text) for r in SMALL_RE)
                agents[month] += bool(AGENT.search(text))
    for month in sorted(total):
        n = total[month]
        print(f"{month} posts {n:6d}  small hardware {100 * small[month] / n:5.1f}%  agents {100 * agents[month] / n:5.1f}%")


if __name__ == "__main__":
    main()

9. Further reading

A review of 85,900 posts from r/LocalLLaMA, r/LocalLLM, r/ollama, r/LocalAIServers and r/RockchipNPU (2025-10-01 to 2026-10-19) shaped these tests. Reddit numbers are self-reported unless marked checked.

Speed tiers and big models on small boards

• 1q5m2n6: a 30B MoE at 8.03 tokens per second on a 16 GB Raspberry Pi 5, at 2.70 bits per weight (ByteShape; their blog checks).
• 1wqcrly: a 2017 laptop with 8 GB and no GPU runs Ling 3.0 Tiny (about 1B active) at about 10 tokens per second.
• 1vjs3sf: Radeon 780M mini PCs run Qwen3.6-35B-A3B at 20 to 30 tokens per second.
• 1rg87bj: "the 'active parameters' trick turned MoE from a datacenter architecture into an embedded one."
• 1pzggbf: prompt reading, not writing, is the bottleneck without a GPU.
• 1x8m0jj: a 552B model at 27 tokens per second on one 24 GB card plus 90 GB of RAM.
• 1s8xuew: "Even at 1.5 tok/s, having a 35B model churning through ... batch data classification overnight on a Pi5 is completely viable."

Gaming GPUs, 12 to 32 GB

Other people's numbers, not measured on the hardware above. Several use tuned builds.

• 1t7l56a: Qwen3.6-35B-A3B on a 12 GB card with plain llama.cpp, about 47 tokens per second writing, with 18 layers of experts kept in system RAM.
• 1t82zxv: the same model on 12 GB at over 80 tokens per second with 128K context, using llama.cpp's multi-token prediction pull request.
• 1t1judm: Qwen3.6-27B at 72 tokens per second on an RTX 3090 with vLLM.
• 1whqwdq: Qwen3.8-27B as a daily coding agent for 30 days, 73.8 tokens per second mean writing speed.
• 1vqsnzi: Qwen3.8-27B at 150 to 200 tokens per second with 262K context on a 32 GB card, ninfer engine.

Phones

• 1uz7rrv: gpt-oss-120b at 1.3 tokens per second on a 12 GB OnePlus 15R with BigMoeOnEdge (README checked).
• 1x3hn1o: the best phone for local models; commenters put a 12 GB iPhone's usable RAM near 7 to 7.5 GB.
• 1sl6931: a Xiaomi 12 Pro as a headless 24/7 server.
• 1x94cax: Phonon turns a phone into an OpenAI-compatible endpoint. Its site (checked) reports 10.54 tokens per second after 15 minutes of work on a Pixel 7.
• 1x4awt8: two phones over USB split a 27B model at 2.8 tokens per second. Read the authenticity comments first.

Clusters and streaming from disk

• Jeff Geerling, 2025-12-18: four Mac Studios with 1.5 TB of combined memory over Thunderbolt 5 RDMA; Kimi K2 Thinking at 28.3 tokens per second with Exo 1.0.
• 1r87ou8: 800 RX 580 GPUs; splitting one model across nodes failed, one model per GPU worked.
• 1wviphh: six ex-mining BC-250 boards on 1 Gb Ethernet at about 28 tokens per second.
• 1vy2ypg: why peer-to-peer inference fails on trust and tail latency; lists 30 projects.
• 1wt3s0s: the Colibri announcement and its reception.
• 1wqwm3o: one 85 GB model on one box: Overspill 2.8 to 3.4, Colibri 1.1 to 1.2, llama.cpp 0.4 to 0.5 tokens per second.

Small models, harnesses and safety

• 1vk5pjt: nl2sh-1.5b (941 MB) scores 0.620 on 300 shell tasks, against 0.613 for Qwen2.5-Coder-7B (card checked).
• 1r4ie8z: 21 models tested on tool judgment; fixing parsers changed the scores.
• 1sdv3lo: PokeClaw drives an Android phone with Gemma 4; top comment (score 572) asks for open source before trusting it.

Things people built

• 1uitkg1: a Raspberry Pi 5 robot arm with wake word, Whisper and Qwen3 1.7B calling a tool.
• 1t73yu9: a 10-inch rack with a local model, NAS, media and offline Wikipedia, idling at 70 W.
• 1qmidlh: a Pi 5 with a button and whisper.cpp that captures ideas offline.
• 1qxka0p: offline semantic search over years of journal entries on a Pi 4.

Primary sources

• llama.cpp: release b11784, server README, Android guide, RPC README.
• Colibri, BigMoeOnEdge, llama-swap.
• OpenAI API deprecations.
• LiteLLM security update, March 2026.

10. About the speaker

Brady Dibble is Director of Product at CIQ. He oversees Rocky Linux from CIQ and runs CIQ's self-hosted AI fleet. In the 30 days to 2026-10-22 he sent 105.6 billion tokens through those self-hosted models, most of it agents re-reading their own context: 94.4% of input tokens came from the prompt cache. Nothing here is a CIQ product pitch. Every local tool on this page is open source or free to use.

Contact: [email protected] · linkedin.com/in/bradydibble · bradydibble.com