CodingAgent

▶ Launch ConsoleSign in

■ LOCAL LLMs // SOVEREIGN INFERENCE

Your code never has to leave the machine

Local runtimes are first-class targets: private repositories route strictly to local models, with hardware-aware dispatch across VRAM, context size and task history.

Ollama

LOCAL

Developer-Local Inference

Instant local model serving with automatic GPU acceleration across Metal, CUDA, and ROCm. Perfect for single-developer workstations.

Specs

Fast quantization (Q4/Q5), 32k context, automatic model pulling

Hardware

16GB-64GB unified memory or 12GB+ VRAM GPU

Privacy

100% offline, zero network telemetry

vLLM

LOCAL

High-Throughput Self-Hosted Serving

Continuous batching and PagedAttention for serving multiple concurrent agent subroutines across shared on-premise GPU clusters.

Specs

PagedAttention v2, tensor parallelism, OpenAI API compatibility

Hardware

NVIDIA A100 / H100 or multi-RTX 4090 cluster

Privacy

On-premise enterprise VPC isolation

llama.cpp

LOCAL

Portable GGUF Bare-Metal Runtime

Lightweight C/C++ engine executing quantized GGUF weights with zero dependencies and efficient CPU/RAM fallback.

Specs

GGUF format, mmap zero-copy loading, CPU NEON/AVX2 acceleration

Hardware

Runs on standard laptops, edge devices, and CI runners

Privacy

Air-gapped verified, zero external dependencies

LM Studio

LOCAL

Desktop Model Endpoint

Visual desktop application providing interactive model experimentation, hardware utilization monitoring, and local HTTP endpoint.

Specs

GUI catalog, HuggingFace direct download, local port 1234 API

Hardware

Apple Silicon Mac or Windows/Linux gaming PC

Privacy

Localhost loopback only

Routing dimensions

Four rules decide where inference runs: the repository's privacy classification, real GPU memory headroom, the required context window, and each model's empirical verification history.

Privacy Tier

POLICY

If code is labeled Confidential or Air-Gapped, routing locks strictly to local models; external cloud APIs are hard-disabled.

VRAM Fit

RULE

Inspects real-time GPU memory headroom: dispatches 7B-14B models for fast localized edits, reserving 32B+ for complex refactors.

Context Size

RULE

Evaluates required context length from AST search. Short fixes fit standard 8k windows; deep call graph migrations use 32k-128k.

Quality & Task History

RULE

Routes based on historical verification pass rates for specific languages (e.g. Qwen2.5-Coder for Python, DeepSeek-Coder for TypeScript).