PMAT Roadmap

Work item dependency graph and critical path to AC-022 (leaderboard submission gate).

27.1 Work Item Summary

27.2 Dependency DAG

PMAT-006 (DONE: 85.37% baseline)
├── PMAT-007 (DONE: adapter trained, merged, Q4K — awaiting eval)
│   └── PMAT-010 (PENDING: TIES merge)
│       └── PMAT-011 (PENDING: final artifact → AC-022)
├── PMAT-014 (IN PROGRESS: N-sampling preference pairs)
│   └── PMAT-008 (READY: DPO contract v2.0, pipeline defined)
│       └── PMAT-010 (PENDING: TIES merge)
└── PMAT-037 (DONE: wgpu training verified, 13 KAIZEN fixes)

PMAT-017 (DONE: 56 Makefile targets)

27.3 Critical Path

The shortest path to AC-022 (leaderboard submission):

PMAT-014 → PMAT-008 → PMAT-010 → PMAT-011 → AC-022
  (pairs)    (DPO)     (merge)    (quantize)   (gate)

Parallel track: PMAT-007 (distillation) feeds into PMAT-010 independently.

Critical Path Estimates

27.4 AC Coverage by PMAT

27.5 Contract Coverage

Each PMAT item has associated provable contracts:

Total: 28 contract YAMLs, 98 proof obligations, 98 falsification tests, 10 Kani harnesses. Makefile gate: 59/60 passing.

27.6 Gap Analysis

MBPP Gap (3.8pp to AC-022)

Current: 76.2% → Target: 80.0%

Conservative estimate: DPO alone should close 2-3pp, combined with distillation gets to 80%+.

Blocked Items

27.7 GH-580: Tokenizer Preservation Fix (2026-04-03)

Root cause: run_merge() used AprWriter (v1) which creates empty tokenizer. Base model is APR v2 with tokenizer in AprV2Metadata.custom HashMap.

Fix: Read base model with AprV2Reader, clone metadata (preserving tokenizer), use AprV2Writer for output. Also supports SafeTensors adapter input (wgpu training pipeline).

Impact: Unblocks PMAT-007 eval (distilled model can now run inference), PMAT-008 (DPO merge), PMAT-010 (TIES merge). All merge operations now preserve embedded tokenizer.

Contract: tokenizer-preservation-v1.yaml — 2 equations, 3 proof obligations, 3 falsification tests.

27.8 PMAT-007 Pipeline Artifacts (2026-04-03)

Status (2026-04-03 18:39): GH-580 merge fix VERIFIED. Additionally, LoRA merge had a critical bug — element-wise multiply instead of matrix multiply (Hadamard product instead of GEMM). Five-whys traced to a "simplified" comment in merge engine. Fix: proper triple-loop GEMM computing B^T @ A^T with d_in/d_out inferred from flat arrays + rank. Fix deployed to gx10. All previous merged models (v1, v2) are invalid — must re-merge with corrected binary.

Next step: Re-merge distilled model after PMAT-014 N-sampling completes. Merge OOM-killed twice on gx10 (49 GB peak + 18 GB N-sampling exceeds 119 GB unified memory). Auto-merge pipeline (PID 1886069) queued — runs automatically when N-sampling finishes. Pipeline: merge → apr check → quantize Q4K → inference test.

N-sampling (PMAT-014): Running on gx10 with base 7B Q4K. 1157/1640 prompts completed (70.5%) as of 2026-04-04. Rate: ~47 prompts/hour. ETA: ~10h remaining. Work dir: /tmp/tmp.4izwh76p7m preserved with APR_KEEP_WORKDIR=1.

27.9 LoRA Merge Matmul Fix (2026-04-03)

Root cause: MergeEngine::merge() used element-wise multiply a[i%len]*b[i%len] (Hadamard product) instead of matrix multiply B @ A (GEMM). This produced garbage weight deltas that corrupted every merged model.

Five whys:

1. Why garbage inference? Model weights corrupted after LoRA merge
2. Why corrupted? MergeEngine::merge() produced wrong weight deltas
3. Why wrong deltas? Used a[i%len]*b[i%len] (element-wise) not B@A (matmul)
4. Why element-wise? Comment said "Simplified: just add scaled A and B values"
5. Why not caught? No matrix multiply unit test, garbage only visible at inference

Fix: Replaced with proper GEMM — infer d_in/d_out from flat arrays + rank, compute B^T @ A^T with triple loop. O(d_out × d_in × rank) per tensor. Handles both standard and transposed LoRA conventions.

Impact: All PMAT-007 merged models must be regenerated. Critical path unchanged — merge takes minutes once N-sampling finishes.

27.10 Contract Coverage Update (2026-04-03)

3 new provable contracts written:

Updated totals: 28 contracts, 98 proof obligations, 98 falsification tests, 10 Kani harnesses.

AC verification update: 19/29 verified (66%). Newly verified: AC-009 (Q4K size), AC-014 (HF parity), AC-023 (INT4 loss, 32B 1.65pp < 2pp), AC-025 (data quality, 0 duplicates, 0 short responses).