The WASTE inference engine

Running Kimi K3 2.78T parameter with 29GB of RAM

Today we are releasing WASTE, an open-source inference engine designed to run models whose weights are substantially larger than the memory available on the host machine.

The first model fully supported by WASTE is Kimi K3, a 2.78 trillion parameter mixture-of-experts model. We converted the original 1.42 TB checkpoint into a 982 GB container and ran the complete model on a MacBook Pro with 64 GB of unified memory. This is not a distilled, pruned, or reduced variant.

A typical execution currently looks like this:

$ waste run ~/models/k3.waste "What is the capital of Italy?"

waste: no --budget, using 46.24 GB of 64.00 GB
       (expert cache 17.56 GB)

The capital of Italy is Rome.

[16 tokens, 49.31 s, 0.32 tok/s]

The current performance, approximately one third of a token per second, is not yet suitable for most interactive applications. However, the result demonstrates that available RAM does not necessarily have to impose a hard limit on the total size of a model that can be executed locally.

WASTE is an initial step in a broader effort to make increasingly capable models available on hardware controlled by the people and organizations using them.

The goal is to provide greater control over infrastructure costs, data privacy, availability and deployment.

Moving the memory boundary

Kimi K3 is a mixture-of-experts model. Although it contains 896 routed experts, only 16 experts are selected in each layer for a given token.

Most of the model weights are therefore inactive at any particular point in the computation.

WASTE takes advantage of this property by separating the model into two main parts.

The model trunk remains resident in memory. The expert weights remain on NVMe storage and are read when selected by the router. Any remaining RAM is used as a bounded expert cache.

This changes the primary constraint. Instead of requiring the entire model to fit in memory, the system needs to retrieve the active portion of the model within an acceptable amount of time.

For K3, each token accesses approximately 17 GB of expert data. This makes storage bandwidth and cache behavior central to performance.

WASTE stores experts in a layout designed around this access pattern. The matrices required for one expert are adjacent and can be loaded with a single positional read. The engine bypasses the operating system page cache and manages expert caching directly, so its behavior remains representative even when the complete model is much larger than memory.

The runtime is written in C and has no external inference dependencies. Python is used for conversion and validation, but not during inference.

The project currently includes:

• an embeddable C API;

• a command-line client;

• an OpenAI-compatible HTTP server;

• CPU implementations for ARM and x86 systems;

• support for macOS, Linux and Windows;

• text and image input for Kimi K3.

The same engine also runs Kimi-Linear 48B from a 19 GB container at approximately 8.9 tokens per second on the same machine.

Developing through measurements

One of the most useful parts of this project has been the development process itself.

The repository includes a document called What we know, and how we know it. It records not only successful optimizations, but also assumptions and proposed designs that were later disproved by measurements.

We adopted a simple working rule:

Before starting a long or expensive operation, run a smaller real-world test capable of invalidating the underlying assumption.

This was particularly important because many operations in this project are expensive. Downloading the source model involves 1.42 TB of data. A complete conversion produces almost one terabyte of output. Changes to the expert format can require converting the model again.

Testing an assumption first is considerably less expensive than discovering a problem at the end of that process.

Storage placement

We initially planned to run the converted model from an external NVMe drive.

The nominal performance of the drive appeared sufficient, but conventional storage benchmarks did not represent the pattern used by WASTE. The engine reads expert-sized records from many different positions rather than processing one continuous file sequentially.

We therefore wrote a benchmark that reproduced the actual access pattern.

On the target machine, the external SSD reached approximately 0.94 GB/s. The internal SSD reached 12.78 GB/s.

The main limitation was the USB bridge in the external enclosure rather than the NVMe device itself.

As a result, the external drive is used for downloading and staging the source checkpoint, while the converted WASTE container is placed on the internal SSD.

This measurement changed an important deployment decision before the full model was downloaded and converted.

Working with the released architecture

Development started before the complete K3 checkpoint became publicly available. Early feasibility work was based on the published information and on measurements taken from Kimi-Linear, the closest available model from the same family.

When the K3 weights were released on July 27, 2026, we reviewed the actual configuration, tensor index and weight shapes.

The most important finding was that K3 uses a latent mixture-of-experts architecture.

The model has a hidden width of 7,168, but its routed experts operate on a projected width of 3,584. This reconciles the announced parameter count with the size of the released checkpoint and substantially reduces the amount of expert data that must be read for each token compared with a full-width implementation.

The released model also differed from our assumptions in several other ways, including its number of layers, attention heads, shared experts and attention implementation.

Those differences required changes to both the runtime and the memory model.

The initial projection based on the smaller model suggested that K3 might reach between 1 and 1.5 tokens per second on the target hardware. The final measured performance was closer to 0.3 tokens per second.

The earlier estimate did not sufficiently account for the size of the resident trunk, the resulting reduction in available cache memory, and the computational cost outside storage I/O.

The earlier projection remains in the repository together with an explanation of why it was inaccurate. Keeping that history makes the current limitations and the next optimization targets easier to understand.

Understanding the cache

Expert caching turned out to have a less intuitive behavior than expected.

K3 accesses approximately 17 GB of expert records for each token. If the expert cache is smaller than this working set, most entries are evicted before they can be reused by the next token.

Below this threshold, the measured cache hit rate is effectively zero.

Once the cache exceeds one token’s working set, reuse begins to provide a measurable benefit. However, increasing the cache indefinitely does not improve performance.

On the 64 GB test machine, larger memory budgets increased the reported cache hit rate while making inference substantially slower. The operating system began compressing and paging memory. An entry reported as present in the WASTE cache could therefore require a page fault before it was usable.

The engine remained within its configured allocation, but the machine as a whole was under memory pressure.

For this reason, WASTE does not attempt to consume all available memory. Its automatic planner allocates cache in multiples of the token working set and maintains a margin for the operating system and other processes.

On the test machine, a total budget of approximately 46 GB produced better results than larger allocations.

Approaches that did not improve the system

Several plausible approaches were tested and rejected.

A three-bit model trunk

Reducing the trunk from four bits to three freed more than six gigabytes of memory and increased the expert-cache hit rate.

It nevertheless reduced overall performance because the additional unpacking cost exceeded the I/O savings. More importantly, generation quality collapsed.

K3 was trained with quantization-aware training for its expert weights, but not for the rest of the model. The trunk therefore did not tolerate the same level of compression.

Variable precision by expert

The original format design allowed some experts to use three bits and others to use two, based on their relative importance.

Measurements showed that the quantization error was unusually uniform across experts and layers. An optimized allocation performed only marginally better than a random one.

Routing frequency does vary, but reducing the precision of infrequently used experts mainly saves disk capacity rather than the bytes read during normal inference. Disk capacity was not the main constraint.

The mechanism was therefore not implemented.

GPU and Metal execution

A Metal backend was implemented and verified for correctness. On the tested workload it was 22% slower than the CPU path.

WASTE performs many relatively small and dependent matrix-vector operations. This is not necessarily a favorable workload for GPU execution, particularly when the CPU implementation already operates near the system’s memory-bandwidth limits.

Metal remains available as an optional backend, but it is disabled by default.

Index-layout changes

A revised expert index layout produced a 1.44× improvement in an isolated microbenchmark.

After converting a complete 19 GB model to the new layout, the end-to-end engine showed no meaningful improvement. The microbenchmark did not accurately represent the effects of multiple threads sharing the processor cache.

This was a useful reminder that a valid local measurement may still be irrelevant to the complete system.

The optimizations that mattered

The first correct version of the engine expanded quantized expert weights into floating-point values before using them.

Profiling showed that dequantization accounted for most of the expert processing time.

WASTE now performs the matrix-vector operation directly from the residual vector quantization representation. It builds partial dot-product tables for each codebook and uses those tables when processing the expert rows.

The full expert matrix is never materialized.

This approach was adapted from work previously done in sqlite-vector, where lookup tables are used to accelerate operations over quantized vectors.

Together with threading, SIMD kernels and the removal of repeated table construction, this reduced Kimi-Linear inference from 2.15 seconds per token in the first correct implementation to approximately 0.12 seconds per token.

After each optimization, the output was compared with a PyTorch reference implementation. The final logits remain within a small numerical tolerance, with the same most likely tokens and the same top-token ordering.

Correctness has remained a requirement throughout the optimization process.

Why we built WASTE

Cloud inference is appropriate for many applications. It provides access to specialized hardware, predictable deployment and models that may be impractical to operate independently.

At the same time, relying exclusively on remote inference introduces trade-offs.

Sensitive information may leave the local environment. Applications depend on network connectivity and external service availability. Usage costs are tied to token volume. Providers retain control over rate limits, model availability and version changes.

Local inference offers a different set of properties:

• prompts and results can remain on the device;

• the application can continue operating without a network connection;

• infrastructure costs can be based on owned hardware rather than token consumption;

• model deployment and upgrades remain under the operator’s control.

The name WASTE originally referred to the underused compute, memory and storage already available on local machines while inference is routinely moved to remote infrastructure.

The acronym—Weight-Aware Streaming Tensor Engine—was chosen later.

The intention is not to argue that all inference should move away from the cloud. It is to make local execution a practical option in cases where privacy, cost control, offline availability or infrastructure ownership are important.

K3 as a reference point

Kimi K3 is intentionally a demanding first target.

Its converted container occupies 982 GB. Each token requires access to approximately 17 GB of expert data. Its resident trunk consumes most of the minimum memory budget, and the internal SSD is already operating close to its measured throughput limit.

These constraints make it useful as a reference case.

Problems that are difficult to observe on smaller models become visible at this scale. Memory accounting errors become gigabytes. Ineffective caching strategies become immediately measurable. Storage placement becomes part of the inference architecture.

Smaller models are significantly less demanding.

Kimi-Linear 48B requires a 19 GB container, can open with less than 2 GB of RAM and reaches approximately 8.9 tokens per second with an 8 GB budget on the same hardware.

This is closer to the initial practical use case for WASTE: running capable local models inside applications, agents and private infrastructure without requiring a large inference stack.

K3 helps establish how far the same architecture can be extended.

What comes next

WASTE is the first release in a planned series of work focused on local model execution.

The current priorities include:

• improving the performance of expert retrieval and processing;

• reducing the number of bytes read for each token;

• supporting additional model architectures;

• improving prefetching and cache persistence;

• simplifying model conversion and distribution;

• publishing pre-converted containers;

• improving support for systems with larger unified-memory configurations.

The immediate objective is not limited to making K3 faster. It is to develop a general runtime and model format that can support a range of local hardware, from relatively small systems to workstations with large amounts of memory and fast storage.

Some of the approaches we test will not produce useful results. As with this first release, we intend to document those findings rather than present only the successful experiments.

At its current speed, K3 running through WASTE is primarily a technical result rather than a general-purpose interactive deployment.

It nevertheless establishes that a model can be executed locally even when its total weights are many times larger than available memory.

Future work is now focused on improving efficiency, usability and model coverage.

WASTE is available under the Apache 2.0 license:

github.com/sqliteai/waste

This release represents the first step toward giving developers and organizations a practical way to run increasingly capable models on hardware they own, with direct control over data, operating costs and deployment.