Lizard-LLM

From model file to usable product

Lizard-LLM connects a GGUF model, a Windows machine and a local inference service. I led the architecture and hands-on delivery during my Qendryx contract. The engineering scope extends from model onboarding and provider selection to native execution, telemetry and recovery.

Problem and operating context

Machines expose different CPU capabilities, GPU backends and memory budgets. Parameter count alone cannot establish whether a model will run reliably. Model identity, context length, quantization and supported execution paths must agree before activation.

My responsibility

I connected the application workflow to the runtime: artifact verification, provider policy, native process control, graph construction, memory accounting and observable serving behavior. Product usability and numerical execution were parts of the same delivery responsibility.

System architecture

The application and local API sit above worker processes and provider policy. Lizard Native and Caterpillar are distinct execution approaches. Caterpillar represents computation as a C++ graph plan with allocation and backend scheduling. The portfolio records Caterpillar as stable, Lizard Native as experimental and Ostrich as isolated research at its review date.

Technical decisions

Model identity stays attached to execution conditions. A common serving interface does not imply identical internals across providers. CPU kernels and D3D12 execution require explicit capability checks, while graph planning separates logical computation from placement and allocation.

Implementation details

The runtime work includes GGUF metadata, decoder graph construction, quantized CPU kernels, D3D12 dispatch, resident weights and KV-cache accounting. Cache cost depends on context, layers, heads and representation. Resource calculations support a machine-specific fit decision rather than a generic promise based on installed RAM.

Production challenges

Process startup is not proof that a model is ready. Activation, request execution, cancellation and teardown need distinct states. A failed provider or exhausted memory budget should leave an explanation and a recoverable user workflow. Residency and synchronization must be measured across the full request, not inferred from one local operation counter.

Evaluation and observability

A benchmark is a contract: retain the model artifact, quantization, hardware, prompt, context, generation length, threads, provider and revision. Separate prefill and decode, cold and warm behavior, correctness and throughput. Telemetry should explain the selected path and the reason for a failure.

Measured evidence

The supplied second-edition portfolio records historical local decode reaching 9.67 tokens/second on an Intel i7-1165G7 reference machine. The corrected llama.cpp control remained faster at 12.50 tokens/second. These are historical results under the book's recorded conditions, not a new benchmark campaign or a general performance ranking. See PDF pages 26–29 for methodology and qualifications.

Ostrich experiments also retained negative results. Per-node dispatch measured 10.71 tokens/second against 10.14 for the compared scheduling experiment in the cited historical record. An experiment that does not justify a new default remains useful engineering evidence.

Technology stack

C++, TypeScript, GGUF, quantized kernels, D3D12, CPU/GPU routing, local process control and an OpenAI-compatible serving interface. Repository references in the supplied PDF identify implementation boundaries and historical records.

Diagrams and demonstrations

The original application-layer, graph-plan and performance figures are reproduced below. Product films show the communication around the product; they are not substitutes for benchmark records. The full 40-page portfolio includes the remaining execution, memory and recovery diagrams.

Lessons and next steps

The strongest result is the connection between precise model identity, an executable hardware plan and behavior the user can understand. Further performance work should preserve the corrected baseline, test numerical correctness and change one scheduling or memory hypothesis at a time. External large-model GPU reports are context, not Lizard measurements.