Sovereign Tooling Map: World-Class or Wire It In

Every leaderboard-winning technique maps to a sovereign stack component. When a component doesn't support a technique at world-class level, we don't skip it — we find or build the capability and wire it into apr CLI commands.

5.1 Tooling Coverage Matrix

5.2 Gap 1: DPO/ORPO Preference Optimization (CRITICAL)

Why world-class: DPO is the single most impactful post-training technique for leaderboards. Merged + DPO models "completely dominate" HF leaderboard rankings. Without DPO, we compete with one hand tied.

Current state: make align routes through apr finetune --method dpo which connects to entrenar's loss functions. A dedicated apr align subcommand is planned (GH-8).

Current implementation:

# DPO alignment via make align (routes through apr finetune)
make align CHECKPOINT=model.apr PREFS_DATA=prefs.jsonl ALIGN_METHOD=dpo

# Equivalent direct command
apr finetune model.apr --method dpo --data prefs.jsonl \
    --output aligned.apr --verbose

Remaining wire-in plan:

Component: entrenar
  Add: src/dpo/mod.rs — DPO loss (β-scaled log-ratio of policy vs reference)
  Add: src/dpo/data.rs — preference pair loader (chosen/rejected format)
  Add: src/dpo/orpo.rs — ORPO variant (no reference model needed)

Component: alimentar
  Add: Preference pair generation from execution feedback
    alimentar generate-preferences \
      --model model.apr \
      --problems humaneval.jsonl \
      --n-samples 10 \
      --judge execution \
      -o preference-pairs.jsonl

Component: Ground truth corpus
  Use: hf-ground-truth-corpus, algorithm-competition-corpus
    → Source of verified correct/incorrect code pairs for DPO training

Acceptance criterion: apr align --method dpo produces a model with ≥2% higher HumanEval+ than the input model after 3 epochs.

5.3 Gap 2: Code Execution Sandbox (CRITICAL)

Why world-class: HumanEval and MBPP require executing generated code against test cases. Without execution, we can't compute pass@k — we can only measure perplexity, which doesn't correlate well with code correctness.

Current state: aprender has no sandboxed code execution. Generated completions must be evaluated externally.

Wire-in plan (two options):

Option A: External EvalPlus harness (short-term, pragmatic)
  apr eval model.apr --data humaneval.jsonl --n-samples 10 \
    --output-completions completions/ --json
  # Then externally: evalplus.evaluate --samples completions/
  # This is what everyone does — even Google and Meta use external harnesses

Option B: WASM sandbox (long-term, sovereign)
  Component: realizar or new crate
  Add: Embedded WASM runtime (wasmtime) for safe code execution
    apr eval model.apr --data humaneval.jsonl \
      --sandbox wasm --timeout 10s --json
  Advantage: Fully sovereign, no Python dependency even for eval
  Risk: Python test cases require Python-in-WASM (CPython compiled to WASM)

Decision: Option A for v1.0 (get on the leaderboard), Option B as stretch goal. Neither compromises the "zero Python" claim for the model pipeline — eval is a separate concern.

5.4 Gap 3: N-Sampling + Reranking Pipeline

Why world-class: Generating N=10-50 completions and selecting the best one boosts effective pass@1 by 10-30%. This is the single most impactful inference-time technique.

Current state: aprender can generate multiple completions via temperature sampling. Missing: batched generation, reranking logic, majority voting.

Wire-in plan:

Component: aprender (apr-cli)
  Extend: `apr eval --n-samples N --rerank strategy`
    Strategies: logprob (sum of log-probabilities), majority (output voting),
                execution (run and pick passing code — requires sandbox)

Component: realizar
  Already supports: batched generation, concurrent requests
  Need: expose batch generation for N completions per prompt efficiently

Component: alimentar
  Add: Result aggregation and voting logic for N-sample outputs

5.5 Gap 4: Synthetic Training Data Pipeline

Why world-class: Qwen2.5-Coder, Phi-4, and NVIDIA OCR-Nemotron all credit large-scale synthetic data as core to their success. Without high-quality synthetic training data, fine-tuning is limited to existing datasets.

Current state: apr chat --batch can generate completions. alimentar handles data loading and quality scoring. Ground-truth corpora exist (hf-ground-truth-corpus, algorithm-competition-corpus). Missing: end-to-end curation pipeline.

Wire-in plan:

Component: alimentar
  CLI pipeline:
    # 1. Generate raw synthetic code from teacher
    apr chat teacher.apr --batch problems.txt --n-samples 5 \
      --temperature 0.8 --json > raw-synthetic.jsonl

    # 2. Quality-filter with alimentar
    alimentar quality raw-synthetic.jsonl --min-score 80 \
      -o filtered-synthetic.jsonl

    # 3. Decontaminate against eval benchmarks
    alimentar drift raw-synthetic.jsonl \
      --reference humaneval.jsonl mbpp.jsonl \
      --overlap-threshold 0.01 \
      -o clean-synthetic.jsonl

    # 4. Balance and split
    alimentar convert clean-synthetic.jsonl \
      -o training-data.parquet

Component: Ground truth corpora
  hf-ground-truth-corpus → HuggingFace API patterns, transformer implementations
  algorithm-competition-corpus → Algorithm problems with verified solutions
  → Both feed into fine-tuning data mix

5.6 Gap 5: Prompt Strategy Engine

Why world-class: SCoT prompting improves HumanEval pass@1 by up to 13.79%. Few-shot exemplars add 3-8%. The prompt template matters as much as the model weights.

Current state: PROMPT_STRATEGY is implemented in scripts/eval-pass-at-k.sh with 4 built-in strategies. The upstream apr run --chat provides raw chat template support.

Implemented in eval pipeline:

# All 5 strategies work via Makefile targets (best: few-shot 87.20%):
make eval-humaneval CHECKPOINT=m.apr PROMPT_STRATEGY=standard
make eval-humaneval CHECKPOINT=m.apr PROMPT_STRATEGY=scot
make eval-humaneval CHECKPOINT=m.apr PROMPT_STRATEGY=few-shot
make eval-humaneval CHECKPOINT=m.apr PROMPT_STRATEGY=cgo

Built-in strategies (with aliases):

Remaining wire-in for upstream apr:

Component: realizar
  Already supports: chat templates (ChatML, LLaMA2, Mistral, Phi, Alpaca)
  Need: expose template composition for eval pipeline

5.7 Sovereign Stack Version Requirements

All gap closures must use published crates from crates.io. No git dependencies.

5.8 The Decision Rule

When we find a gap:

1. Can an existing sovereign crate do it? → Wire it in via apr CLI. No new crates.
2. Does a sovereign crate need a new module? → Add it to that crate, publish to crates.io, bump apr-leaderboard's dependency.
3. Is it fundamentally outside the stack's scope? → Use an external tool (e.g., EvalPlus for code execution) and document the boundary explicitly.
4. Is it a research problem with no clear solution? → Add to §21 Open Questions. Don't block the pipeline.

Hard rule: We never add a Python dependency. We never add a C/C++ FFI dependency. GPU compute is wgpu (primary, any vendor, pure Rust) with optional CUDA backend for hardware where wgpu support lags (e.g., Blackwell sm_121). No GPU vendor lock-in. If the sovereign stack can't do it in pure Rust, we either build it or scope it out with an explicit boundary.

5.9 Parity Check: Ludwig Feature Coverage

Ludwig (ludwig.ai) is the state-of-the-art declarative ML framework. Every feature Ludwig ships, the sovereign stack must match or exceed — in pure Rust, with zero Python. This is the parity bar.

5.9.1 Feature-by-Feature Parity Matrix

Training & Fine-tuning:

Optimizers:

Distributed Training:

Quantization:

Inference & Generation:

Serving & Deployment:

Data Processing:

Hyperparameter Optimization:

Model Architecture:

Experiment Tracking:

Explainability & Visualization:

Correctness & Quality (sovereign stack advantages):

5.9.2 Summary: Where We Exceed, Where We Must Close Gaps

We have parity in 24+ areas: LoRA, QLoRA, full fine-tuning, AdamW/Adam/SGD, gradient accumulation, mixed precision, early stopping, LR scheduling, all sampling strategies, beam search, REST serving, HF upload, data loading, preprocessing, train/val/test splits, HPO (grid/random/TPE/ASHA), feature importance.

We exceed Ludwig in 16+ areas (updated): speculative decoding, PagedAttention, continuous batching, streaming API, OpenAI-compatible serving, compile-to-binary, multi-format export (ONNX/CoreML/OpenVINO), data quality scoring, drift detection, imbalance detection, Prometheus metrics, TUI monitoring, provable contracts, deterministic builds, format forensics, checkpointing (18 verified contracts: atomic writes, NaN scan, filtered loading, round-trip determinism, provenance chain — vs Ludwig's basic callback).

Gaps to close (9 items):

Recently closed gaps:

• Multi-node training → GPU-SHARE Phase 3: SSH cluster config, job placement, checkpoint coordination (143 tests)
• Automatic batch size selection → VRAM guard + ledger prevents OOM, --vram planning
• Experiment tracking → entrenar TUI monitor + JSONL event logging + checkpoint metadata

Out of scope (not needed for leaderboard): ECD architecture, GBM/LightGBM, multi-modal (text+image+audio), Triton export, TorchScript. These serve Ludwig's "general ML framework" positioning. We are a purpose-built leaderboard pipeline, not a general framework.

5.10 GPU Compute Architecture: PTX JIT vs Pre-compiled Kernels

5.10.1 Why PTX JIT (Not nvcc)

PyTorch ships fat binaries — pre-compiled SASS (GPU machine code) for every supported architecture (sm_70, sm_80, sm_86, sm_89, sm_90). At runtime, the CUDA driver selects the matching SASS — zero JIT, instant startup. This requires nvcc (NVIDIA's proprietary compiler) and the CUDA toolkit (~2+ GB) at build time.

trueno-gpu takes a fundamentally different approach: PTX string templates embedded in Rust. PTX (Parallel Thread Execution) is NVIDIA's stable intermediate assembly language. trueno-gpu writes CUDA kernels directly as PTX strings in Rust source code, compiled into the apr binary by cargo build — no nvcc, no CUDA toolkit, no C/C++ FFI.

At runtime, the CUDA driver JIT-compiles PTX to device-specific SASS for whatever GPU is present. This is the same mechanism PyTorch uses as a fallback for unsupported architectures — trueno-gpu uses it as the primary path.

5.10.2 Trade-offs

5.10.3 Amortization via Batch Mode

The --batch-jsonl flag is the architectural answer to JIT overhead. For a 164-problem HumanEval eval:

• Without batch: 80s JIT × 164 invocations = 3.6 hours of JIT alone
• With batch: 80s JIT × 1 load = 80s total JIT, then pure inference

Amortized JIT cost per problem: <0.5s. The sovereignty benefit (zero external toolchain, forward GPU compatibility) far outweighs the one-time startup cost.

5.10.4 Blackwell sm_121 and the Try 1/Try 2 Pattern

On Blackwell (sm_121), the CUDA 13.0 driver has a JIT bug: it rejects PTX with .target sm_121 (error 300, CUDA_ERROR_INVALID_SOURCE). The GH-480 fix implements a defensive fallback:

1. Try 1: Compile PTX with explicit .target sm_121 — fails (error 300)
2. Try 2: Compile with cuModuleLoadData (no explicit target) — succeeds

This Try 1 → Try 2 pattern is a driver workaround, not a design choice. When NVIDIA fixes the sm_121 JIT in a future driver, Try 1 will succeed and the fallback becomes dead code. The PTX post-processor (GH-480) also patches backward bra LABEL instructions to @%p_jw bra LABEL for sm_121 compatibility.

5.10.5 FP8 Architecture Guard (GH-542)

FP8 E4M3 GEMM kernels (Ada/Hopper-specific) cause CUDA_ERROR_ILLEGAL_ADDRESS on Blackwell, poisoning the CUDA context. Fix: detect_fp8_prefill() uses cc >= 89 && cc < 100 to auto-disable FP8 on Blackwell. Provable contract: gpu-context-health-v1.yaml (3 proof obligations, 3 falsification tests).

Five-whys: (1) Why crash? FP8 warmup writes invalid memory on sm_121. (2) Why invalid? FP8 E4M3 cuBLASLt kernels are Ada/Hopper-specific. (3) Why enabled? cc >= 89 without upper bound. (4) Why no bound? Blackwell didn't exist when written. (5) Fix: cc < 100 guard in 3 files (commit a4bcd908).