Implementation Status

Tracking table mapping spec sections to apr-leaderboard implementation. Updated as code lands.

19.1 Orchestration Targets (§6.2)

apr-leaderboard is a thin orchestrator — a Makefile + shell scripts — that calls apr CLI subcommands. There is no Rust source code; all ML operations are delegated to aprender.

19.2 Shell Scripts

19.3 Quality Metrics

19.4 Config Templates (§4)

19.4.1 GPU Sharing Infrastructure (entrenar)

The GPU-SHARE specification is fully implemented in entrenar with 143 tests across all modules.

CLI flags: --wait-gpu, --vram, --experimental-mps, --gpu-share, --adapters, --adapters-config

19.5 apr CLI Subcommand Availability

All ML operations are provided by apr CLI v0.4.11. Verified via make verify:

19.6 Dogfooding Findings

End-to-end dogfooding with real model import and inference. See also §22 for detailed findings.

19.6.1 GGUF vs SafeTensors Import Path

SafeTensors imports produce F16/BF16 tensors that realizar cannot run inference on (fused matmul only supports Q4_0/Q8_0/Q4_K/Q5_K/Q6_K). GGUF import (pre-quantized Q4_K_M) is the working path — produces runnable models with embedded tokenizer.

19.6.2 GPU Inference Status

GPU inference uses wgpu (Vulkan/Metal/DX12) or CUDA (optional). GPU is mandatory for production eval.

Status (2026-03-28): FIXED — single-prompt AND batch mode working via wgpu.

• Single-prompt apr run --gpu: wgpu (Vulkan), cosine=0.999863, token-for-token parity.
• Batch --batch-jsonl: GH-560 FIXED (2026-03-28) — two bugs: FFN buffer overflow in trueno (attn_out_buf was hidden_dim=3584, needs intermediate_dim=18944; fix: ffn_silu_buf) + KV cache pre-filled length in realizar (vec![0.0; ...] → Vec::with_capacity() + clear()). Verified on gx10: identical output to CPU, 1.1-2.0 tok/s on 7B. Contract-bound: gpu-weight-residency-v1 + gpu-multi-backend-parity-v1.
• CPU batch (default): Proven reliable, ~3 hours for 164 HumanEval, 84.76-85.98% pass@1.

The CUDA cosine=-0.005 on sm_121 (GH-559) is NOT a JIT bug — falsification proved the PTX and JIT are both correct. Individual kernels produce correct results (RMSNorm diff=5e-7, Q GEMV ~1%). The -0.005 cosine is from FP32 accumulation ordering differences (GPU parallel vs CPU sequential) compounding through 28 layers × 10+ operations. wgpu avoids this by using the same accumulation order as CPU (cosine=0.999863).

See §25 (GPU Compute Architecture) for full specification, provable contracts, and roadmap.

Diagnostic trail (2026-03-25 → 2026-03-27):

Corrected root cause (2026-03-27): ~0.1% FP32 rounding per kernel × 280 operations → (1.001)^280 = 1.32 → cosine=-0.005. Individual kernels are correct (RMSNorm diff=5e-7, Q GEMV ~1%). PyTorch avoids this via TF32/FP64 accumulators. wgpu avoids it with sequential accumulation matching CPU.

Active tickets:

• GH-560: CLOSED (2026-03-28) — wgpu batch fully working. Two-bug fix: trueno e24a6f6c + realizar e600bbff.
• GH-561: IN PROGRESS — FP64 accumulators in NF4 GEMM forward + backward. Forward NF4 GEMM fixed previously (trueno 9e021c35, 81a9c16f). Backward GEMM (6 variants) now also fixed with f64 accumulators — training verified on gx10: loss 13.61→12.02, no NaN. Remaining: other kernels (RMSNorm backward, softmax backward, etc.) still use f32 accumulators but are lower priority — training converges without them.

19.6.3 apr serve for .apr Files

apr serve loads .apr models but the HTTP server doesn't bind. Serve may only be implemented for raw GGUF files. apr run works correctly for single-prompt inference.

19.6.4 Pipeline Ordering Validation

Recipe B (merge-alchemist) correctly emits a warning:

WARNING: Merge without finetune: merging untrained variants is suboptimal.

The §10 golden ordering enforcement works. The pipeline allows violation but warns.

19.6.5 Real Inference Verified

apr run checkpoints/qwen2.5-coder-1.5b-q4k.apr "def fibonacci(n):" --max-tokens 128 generates real Python code (Fibonacci implementation). GPU mandatory for production eval.

19.6.6 GPU Sharing Spec Complete

All three phases of the GPU-SHARE specification implemented and tested:

• Phase 1: VRAM guard prevents OOM crashes. Ledger uses flock + atomic JSON write for crash safety. Wait queue polls until VRAM budget is available. MPS available as --experimental-mps opt-in.
• Phase 2: Multi-adapter pipeline loads base model once, trains N LoRA adapters concurrently (3x VRAM savings for 3 adapters). Round-robin and priority scheduling. TOML config via --adapters-config.
• Phase 3: Cluster config (YAML), job placement (VRAM-aware scoring), SSH transport (real std::process::Command, not stubs), checkpoint coordination with leaderboard, health check via SSH.

143 GPU tests pass. Zero SATD. Examples: gpu_ledger, multi_adapter_training, cluster_training.

19.6.7 QA Gate (2026-03-05)

apr qa on Qwen2.5-Coder-1.5B-Instruct Q4K: 6 PASS (capability, tensor contract, metadata, golden output, throughput, perf regression), 1 FAIL (format parity — GH-13: .apr-wrapped GGUF not recognized), 5 SKIP (no CUDA).

19.6.8 Perplexity Baseline (2026-03-05)

apr eval --dataset wikitext-2: perplexity 6.63, cross-entropy 1.89. Throughput: 2.5 tok/s on CPU, 385ms TTFT.

19.6.9 MBPP Eval (2026-03-29)

MBPP result: 74.80% pass@1 (374/500) few-shot 7B Q4K. Duplicate MBPP eval runs on Intel were killed — were burning 32 cores for 4 days with no additional value over the completed result.

19.6.10 Tokenizer Preservation Fix — GH-580 (2026-04-03)

Problem: All merge/quantize pipeline outputs lost embedded tokenizer, producing dead models that fail with PMAT-172 ERROR: APR file missing embedded tokenizer.

Five Whys:

1. Why can't distilled model run inference? Missing tokenizer.
2. Why missing? run_merge() used AprWriter (v1) which creates empty tokenizer.
3. Why empty? AprWriter v1 only writes weight tensors, not metadata sections.
4. Why not v2? Original code predated AprV2Writer.
5. Why not caught? apr check passes (validates weights), but apr run fails (needs tokenizer for encoding).

Fix (GH-580): Read base model with AprV2Reader, clone metadata (preserving tokenizer), use AprV2Writer for output. Also supports SafeTensors adapter input from wgpu training pipeline. Contract: tokenizer-preservation-v1.yaml.

Impact: Unblocks PMAT-007 eval, PMAT-008 DPO merge, PMAT-010 TIES merge. All merge operations now produce runnable models.

19.6.11 PMAT-007 Distillation Pipeline Complete (2026-04-03)

Full text-based distillation pipeline ran on gx10:

1. 99 teacher completions generated (32B model)
2. Combined with instruct corpus (15,326 lines)
3. QLoRA training: 7B on combined data, rank=32
4. Adapter exported: 40 MB safetensors
5. Merged into base 7B model (GH-580 fix)
6. Quantized to Q4K (6.2 GB)

Awaiting: HumanEval + MBPP evaluation of distilled Q4K model.