Moxel
A CPU-first runtime and file format for large mixture-of-experts models on high-memory UMA machines.
The machine I am targeting
The target is a high-memory UMA desktop, specifically the Ryzen AI Max+ class of hardware with 128 GB of shared LPDDR5X memory and fast NVMe. That machine can hold models that would normally need several discrete GPUs. Capacity is not the same as throughput, though. A naive runtime spends its time moving experts, missing caches, and waiting on storage.
Moxel is an attempt to design the file format and runtime around that hardware instead of pretending it is a slow GPU server.
The format
The .moxl format stores model data in 2 MiB aligned shards with a table of contents, typed metadata, quantization data, calibration hints, and optional auxiliary files. The repacker ingests source weights, normalizes them, quantizes them, builds shards, and records the information the runtime needs later.
The Rust workspace separates common types, math, format handling, repacking, and runtime work. The format and math layers are already the most mature parts. The runtime is still the hard part.
The runtime ideas
Cost-Aware Gating adjusts router scores only when candidates are close enough that residency cost should matter. Router-Ahead Prefetch tries to predict expert demand early enough to move cold shards before the token reaches them. Dynamic-K reduces active experts when a guarded quality margin allows it. Delta-Zip targets the KV cache by encoding temporal deltas with an adaptive bit width.
None of those ideas matter in isolation. The scheduler has to coordinate DRAM residency, NVMe promotion, cache pressure, expert churn, and tail latency without making routing quality collapse.
Targets, not results
The current README records design targets of 37 to 45 tokens per second for a 120B-class MoE model, a 20 to 40 percent reduction in p95 latency against a naive baseline, and a 5 to 6.5 times smaller KV footprint at long context. Those numbers are targets for the Framework Desktop test machine. They are not published benchmark results yet.
Current state
The foundation, math, format structures, and zero-copy readers are in place. The repacker has working ingestion, quantization, and aligned shard construction. The end-to-end runtime, scheduling experiments, and benchmark evidence are still being built.