Colibrì: Running 744B-Parameter MoE Models in Pure C on Consumer Hardware

Table of Contents

Why This Repo, Why Now

Checking github.com/trending today, the top repository is JustVugg/colibri, described by its maintainers as “tiny engine, immense model.” Its pitch is direct: run frontier Mixture-of-Experts (MoE) models — from 744B up to 2.8T parameters — on consumer and heterogeneous hardware, in pure C with zero engine dependencies, by treating storage, RAM, and VRAM as a single inference hierarchy the project calls “AI memory multitiering.”

The idea behind the trending spike is a genuinely interesting observation the README leads with: a 744B-parameter MoE model only activates ~40B parameters per token, and only ~11 GB of those actually change from token to token (the routed experts). Instead of requiring the whole model to fit in fast memory, Colibrì’s router streams just the experts a given token needs, from whichever tier — VRAM, RAM, or NVMe — currently holds them, and a learning cache pins the hottest experts over time. That lets the project’s reference model, GLM-5.2 (744B parameters), run on a machine with roughly 25 GB of RAM and no GPU at all, according to the README’s own numbers.

Nine model families are documented as supported today: GLM-5.2/5.3 (744B), GLM-5.3-Flash (321B, with vision), Inkling (975B), Kimi K3 (2.8T), DeepSeek V4 Flash (284B), DeepSeek V4.1 Flash (552B, with vision), Qwen3.8-Flash-Next (125B + 51B n-gram), Qwen3.6 (35B-A3B), and OLMoE (7B) — each its own single-C-file engine behind the same coli chat / coli serve / coli web front end. The project frames itself explicitly as “an inference engine you can run today, and an open research platform,” with a documented table of open hypotheses and unresolved experiments rather than blanket performance claims. As always, a trending position reflects visibility and momentum, not a verdict on quality, security, or production-readiness — those are reviewed independently below.

What the Technology Actually Is

Per the repository’s own structure and README, Colibrì’s inference engines are written in pure C (c/colibri.c, c/inkling.c, c/kimi_k3.c, and one .c file per additional model family), building with a plain Makefile and optional CUDA=1 / METAL=1 / VULKAN=1 backend flags. Shared functionality — the safetensors reader, container/quantization decoders, tokenizer, JSON parser, the streaming expert cache, and routing telemetry — lives in shared C headers included by every engine, so a fix in one place reaches all model families at once.

Around the C core:

• coli — the user-facing CLI (Python), the entry point for chat, serve, web, plan, doctor, tune, convert, and mirror subcommands; it picks the correct compiled engine binary based on the target model’s config.json.
• openai_server.py — an OpenAI-compatible HTTP gateway, written using only the Python standard library, that the CLI’s serve/web commands start.
• resource_plan.py — the RAM/VRAM planner behind coli plan and coli doctor.
• web/ — a browser UI described as a “pure OpenAI-API client” (the live dashboard, the “Brain” expert-activity view, and the “Atlas” 3D expert map).
• desktop/ — a Tauri v2 desktop shell wrapping the same web UI.
• docker/ — container images (Dockerfile, a slim variant, and a docker-compose.yml).

At a high level, per the README’s “How it works” section: every layer of every token walks the same five steps — route → union → place → overlap → learn. The router decides which experts a token needs; batched positions read each unique expert once (“batch-union”); the placement layer decides whether that expert currently lives in VRAM, RAM, or on NVMe; I/O is overlapped with compute through an async prefetch pool; and a per-workload learning cache (.coli_usage) records routing heat and pins the hottest experts automatically. The design goal stated explicitly in the README is that placement only ever changes speed, never model semantics — insufficient fast memory should slow the engine down, not silently alter output.

Other documented architectural pieces: MLA attention stores a compressed KV cache (576 floats/token instead of 32,768, a 57× reduction) persisted to disk (.coli_kv) so conversations reopen warm without re-prefilling; a dual-SSD mode can stripe reads across two drives by measured bandwidth; and native speculative decoding (MTP heads, grammar-forced drafts) is implemented with explicit caveats about when it does and doesn’t pay off, backed by linked GitHub issues documenting the measured trade-offs.

Purpose and Use Cases

Per the README’s stated “research mission,” Colibrì’s primary purpose is removing hyperscaler-class hardware as a prerequisite for running frontier-scale open-weight MoE models — running them locally, inspecting the routing behavior in real time, and being able to change the engine code that does it, rather than only accessing similar capability behind a hosted API.

Documented use cases and target users include:

• ML systems researchers and engineers who want to study MoE routing, expert specialization, and memory-tiering strategies on real hardware, using the project’s own “Brain” and “Atlas” visualizations of live expert activity.
• Individuals and small teams who want to run a 744B+-parameter model on hardware they already own — a workstation or a well-specced desktop — instead of renting GPU-hours or paying per-token API costs.
• Local-inference and privacy-conscious users who need an OpenAI-compatible local API (coli serve) to point existing tooling at a self-hosted model.
• Contributors to open inference research, via the project’s documented “open hypotheses” table — e.g., whether routing-history-based caching beats plain LRU, or whether a hardware-aware planner can match a hand-tuned configuration — where the project explicitly invites reproducible, published experiments (including negative results).

It is not positioned as a production-grade, multi-tenant serving stack: the API gateway serves one generation at a time per model process, per the docs.

Installation

Per the project’s README and Quick Start guide.

Prerequisites

• Python 3 (required for the coli launcher and API gateway; the engine itself has no Python runtime dependency at inference time)
• For building from source: gcc or clang with OpenMP
• Disk space for a model — the reference GLM-5.2 int4 container is ~372 GB
• Optional: an NVIDIA GPU (CUDA), Apple Silicon (Metal), or any Vulkan 1.2-capable GPU for the corresponding accelerated backend

Recommended: prebuilt release (Linux, macOS, Windows)

mkdir colibri && tar xzf colibri-v1.8.0-linux-x86_64.tar.gz -C colibri && cd colibri
python3 coli info                         # engine ready ✓

Alternative: build from source

git clone https://github.com/JustVugg/colibri && cd colibri/c
./setup.sh                                # checks gcc/OpenMP, builds, self-tests

To put coli on PATH from a source checkout: pip install -e . (an editable install; the compiled engine still lives in c/).

Get a model

The maintainers’ recommended container for the reference model is the group-scaled (gs64) GLM-5.2 int4 build with an int8 MTP head, published on Hugging Face:

https://huggingface.co/mastouri/GLM-5.2-colibri-int4-g64-with-int8-mtp (~372 GB)

The README explicitly warns against older per-row int4 mirrors, citing measured quality regressions and linked issues (#455, #8). Alternatively, convert from the FP8 source directly:

./coli convert --model /nvme/glm52_i4     # download+convert shard by shard, resumable

Minimal getting-started example

COLI_MODEL=/nvme/glm52_i4 ./coli chat     # RAM budget, cache and MTP auto-detected

Quick Verification

Based on the documented CLI commands:

1. Engine readiness — python3 coli info should print the engine version and confirm the prebuilt engine is next to the launcher.
2. Read-only preflight — COLI_MODEL=/path/to/model ./coli doctor checks readiness without loading the model; ./coli doctor --deep runs a stricter tensors/shards/index/mirror preflight.
3. Placement plan — COLI_MODEL=/path/to/model ./coli plan shows the planned VRAM/RAM/disk placement before running inference.
4. Chat smoke test — COLI_MODEL=/path/to/model ./coli chat should print a startup banner similar to the README’s own example (🐦 colibri v1.11.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU, ✓ ready in 32s) and accept a prompt.
5. API check — COLI_MODEL=/path/to/model COLI_API_KEY=local-secret ./coli serve --host 127.0.0.1 --port 8000 followed by curl http://127.0.0.1:8000/v1/chat/completions (documented in docs/api.md) confirms the OpenAI-compatible gateway is answering requests.

Security Review

Verified from the repository itself:

• No SECURITY.md file was found in the repository root or .github/ directory at the time of writing, so there is no documented private-disclosure process for this project.
• The engine is licensed Apache 2.0; the README separately notes that GLM-5.2 model weights are released by Z.ai under MIT — the engine license and the model-weight license are independent of each other.
• CI (.github/workflows/ci.yml, check.yml) runs the C test suite under AddressSanitizer and UndefinedBehaviorSanitizer on Linux, and a dependency-free make check build/test gate across Linux, Windows (MinGW/UCRT64), and macOS on every push and pull request. Third-party GitHub Actions in the workflows are pinned to a commit SHA (e.g. msys2/setup-msys2@66cd2cce...) rather than a floating tag.
• Release cadence is active: the latest tag is v1.11.0 (2026-09-13, per the repository’s tags and changelog), with a documented history of point releases (v1.6.0 through v1.11.0) and a maintained CHANGELOG.md.
• Per docs/api.md, the built-in API server (coli serve) binds to localhost by default, and the docs explicitly instruct setting COLI_API_KEY “before exposing the server beyond the machine.” A separate --allowed-host / COLI_ALLOWED_HOSTS option is documented specifically as a DNS-rebinding allowlist, independent of CORS and API-key auth.
• No published GitHub Security Advisories were found against this repository at the time of writing; absence of advisories is not proof of absence of vulnerabilities.

Security considerations to weigh yourself (not verified vulnerabilities — documented factors relevant to this class of tool):

• The project recommends downloading multi-hundred-gigabyte pre-converted model containers from third-party Hugging Face accounts (e.g. mastouri/…, nbeerbower/…, Kreuzzelg/…) rather than only from the original model publishers. The README itself documents that some alternative mirrors of the same model measured worse output quality and caused generation bugs — treat any pre-converted container’s provenance and checksum the same way you would any other third-party binary artifact.
• ./setup.sh and the build scripts compile and run local self-tests; review build scripts before running them, as with any source you build locally.
• The engine has no built-in multi-tenant isolation for the API server (it serves one generation at a time per process); if you expose it beyond localhost, the documented API key and allowed-host settings are the controls to rely on — there is no wildcard host matching by design.
• Docker images are provided (docker/Dockerfile, Dockerfile.slim) but were not independently reviewed here; verify base images and any privileges a container grants before running one from an unfamiliar registry tag.

Practical recommendations:

• Pin a specific release tag or commit when building from source rather than always building main.
• Verify checksums/provenance of any pre-converted Hugging Face model container before loading multi-hundred-gigabyte weights, and prefer containers referenced directly from the project’s own README over ones found elsewhere.
• Always set COLI_API_KEY and use --allowed-host before exposing coli serve/coli web beyond a single trusted machine.
• Review Dockerfile/Dockerfile.slim before building container images for production or shared-infrastructure use.
• Run an initial evaluation in an isolated environment before trusting the engine with sensitive prompts or production traffic.

Should You Use It?

Strengths: a genuinely novel memory-tiering approach backed by concrete architectural detail (learning cache, dual-SSD striping, compressed KV state), a dependency-free pure-C engine core, an OpenAI-compatible API with documented localhost-by-default binding and an API-key/allowed-host model for safer exposure, sanitizer-covered CI across three platforms, and an unusually candid research posture — the README documents open hypotheses and negative results rather than only headline numbers.

Limitations: running the reference model requires hundreds of gigabytes of disk for weights, decode speed is fundamentally disk-bound on machines without full expert residency, no SECURITY.md or documented vulnerability-disclosure process currently exists for the project, and it explicitly positions itself as a research platform with “no SLA on speed” rather than a hardened production serving stack.

Good fit for: researchers and engineers who want to run frontier-scale open-weight MoE models locally without hyperscaler-class GPU budgets, teams evaluating memory-tiering and expert-caching strategies for their own inference infrastructure, and developers who want an OpenAI-compatible local API for privacy-sensitive or offline workloads.

Evaluate alternatives if: you need a hardened, multi-tenant production inference server with an established vulnerability-disclosure process, you cannot provision the disk space the target model requires, or your workload needs guaranteed low-latency serving rather than a best-effort, disk-tiered pipeline.

Primary sources used: GitHub repository · README · Quick Start guide · API docs · CHANGELOG · CI workflow · Tags