Chapter 21: Scientific Benchmarking - Python vs Ruchy

Chapter Status: ✅ Complete with Comprehensive Data

Last updated: 2025-11-02 Ruchy version: v3.175.0 bashrs version: v6.25.0

The Problem

How fast is Ruchy compared to Python and other dynamic languages? Does it deliver on the promise of “Python syntax with Rust performance”?

Without rigorous, reproducible benchmarks, these questions remain speculation. This chapter provides scientific measurements comparing:

• Python - The baseline (CPython interpreter)
• Deno TypeScript - Modern JIT-compiled language
• Julia - JIT-compiled scientific computing language
• Go, Rust, C - Native AOT-compiled languages
• Ruchy (4 modes) - AST interpreter, bytecode VM, transpiled, and compiled

🚀 UPDATE: Ruchy v3.175.0 Compiler Optimizations (2025-11-02)

Re-benchmarking with v3.175.0 shows BREAKTHROUGH performance improvements:

BENCH-011 (Nested Loops 1000x1000):

• Ruchy Transpiled: 2.28ms (28.21x faster than Python)
• BEATS Rust (2.28ms vs 2.45ms) ⭐
• Within 4% of C (2.28ms vs 2.19ms)
• Ruchy Compiled: 2.45ms (matches Rust exactly)

This demonstrates Ruchy transpiler achieving Rust-competitive, near-C performance on compute-intensive workloads. Full suite re-benchmarking in progress.

Quick Example: 10-Language Performance Analysis

Here’s what we discovered across 7 validated benchmarks (string concatenation, binary tree allocation, array sum, Fibonacci, prime generation, nested loops, startup time) comparing 10 execution modes:

Geometric Mean Performance (7 benchmarks)

📊 Note: These geometric mean results are from comprehensive v3.173.0 testing. Re-benchmarking with v3.175.0 compiler optimizations is in progress (see BENCH-011 update above showing dramatic improvements).

🥇 Julia:            24.79x faster  ⚡ JIT + LLVM dominance
🥈 C:                18.51x faster  🏆 Native baseline
🥉 Rust:             16.49x faster  🦀 Safety + performance
4️⃣  Ruchy Transpiled: 15.12x faster  ⭐ 82% of C, EXCEEDS GO!
5️⃣  Ruchy Compiled:   14.89x faster  ⭐ 80% of C, EXCEEDS GO!
6️⃣  Go:               13.37x faster  🚀 Fast compilation
7️⃣  Deno:              2.33x faster  🌐 JIT warmup improved
8️⃣  Ruchy Bytecode:    1.49x faster  ⚡ Variable performance
9️⃣  Python:            1.00x (baseline)
🔟 Ruchy AST:          0.37x faster  🐛 Debug/development

🎯 KEY ACHIEVEMENTS:

• Ruchy transpiled (15.12x) and compiled (14.89x) EXCEED Go (13.37x) in geometric mean!
• Ruchy achieves 82% of C performance across diverse workloads
• BENCH-005 breakthrough: Ruchy transpiled within 12% of C on array sum!
• BENCH-008 breakthrough: Ruchy bytecode matches C within 0.26%!
• BENCH-011 breakthrough (v3.175.0): Ruchy transpiled BEATS Rust and within 4% of C! (2.28ms vs 2.45ms Rust, 2.19ms C)
• BENCH-012 result: Ruchy compiled within 2.6% of C startup time
• Memory tracking: All benchmarks include comprehensive memory metrics

⚠️ METHODOLOGY: These results are based on 7 validated benchmarks covering:

• String manipulation (BENCH-003)
• Binary tree allocation/GC (BENCH-004)
• Array iteration (BENCH-005)
• Recursive algorithms (BENCH-007)
• CPU-bound computation (BENCH-008)
• Nested loops (BENCH-011)
• Startup performance (BENCH-012)

Key Findings (Evidence-Based):

1. Ruchy achieves native-level performance: 15.12x geometric mean
2. Julia dominates numeric code (24.79x via JIT + LLVM)
3. “Python syntax with C-like performance” validated across multiple workloads
4. Four execution modes optimize for different use cases
5. Scientific rigor: Following “Are We Fast Yet?” (DLS 2016) methodology
6. Memory tracking: bashrs v6.25.0 captures peak/mean memory usage

Execution Modes Explained (ELI5)

Before diving into results, let’s understand what each of the 10 execution modes means:

*Deno JIT: Slow startup (warmup), fast after warmup

Key Terms

• AST (Abstract Syntax Tree): Code represented as a tree (like a flowchart)
• Bytecode: Numbered instructions (like LEGO building steps)
• Transpile: Translate code to another language (Ruchy → Rust)
• Compile: Convert code to machine code (1s and 0s)
• JIT (Just-In-Time): Compile while running (gets faster over time)

See test/ch21-benchmarks/LEGEND.md for detailed explanations.

Methodology: Scientific Rigor

Tools Used

• bashrs bench v6.25.0 - Built-in benchmarking with quality gates
• Warmup iterations: 3 (discarded from statistics)
• Measured iterations: 10 (used for analysis)
• Determinism verification: Ensures identical output across runs
• Environment capture: CPU, RAM, OS, timestamp recorded

Statistical Analysis

Every benchmark reports:

• Mean: Average execution time
• Median: Middle value (robust to outliers)
• StdDev: Standard deviation (consistency measure)
• Min/Max: Range of observed values
• Speedup: Relative to Python baseline

Quality Gates

All benchmarks pass:

• ✅ Lint checks (bashrs)
• ✅ Determinism verification (identical output)
• ✅ Output suppression (no contamination)
• ✅ Compilation separated from timing (transpiled/compiled modes)

BENCH-007: Fibonacci Recursive (n=20)

The Code

Python version:

def fibonacci(n):
    if n <= 1:
        return n
    else:
        return fibonacci(n - 1) + fibonacci(n - 2)

result = fibonacci(20)
# Expected: 6765

Ruchy version:

fun fibonacci(n) {
    if n <= 1 {
        n
    } else {
        fibonacci(n - 1) + fibonacci(n - 2)
    }
}

let result = fibonacci(20)
// Expected: 6765

Deno TypeScript version:

function fibonacci(n: number): number {
    if (n <= 1) {
        return n;
    } else {
        return fibonacci(n - 1) + fibonacci(n - 2);
    }
}

const result = fibonacci(20);
// Expected: 6765

Results (9-Language Comparison)

Environment:

• CPU: AMD Ryzen Threadripper 7960X 24-Cores
• RAM: 125Gi
• OS: Linux 6.8.0-85-generic
• Ruchy: v3.171.0
• bashrs: v6.25.0

Analysis

Deno TypeScript: The JIT Warmup Problem

Deno is 1.6x slower than Python for this workload. Why?

JIT warmup overhead - V8’s JIT compiler takes time to:

1. Parse TypeScript
2. Compile to bytecode
3. Profile execution
4. Optimize hot paths

For short-running recursive algorithms (< 30ms), the warmup cost exceeds the optimization benefit.

Lesson: JIT compilers excel at long-running processes (servers, apps), not short scripts.

Ruchy AST: Expected Slowness

Ruchy’s AST interpreter is 0.11x (9x slower than Python). This is expected and intentional:

• AST mode walks the syntax tree directly
• Useful for development and debugging
• Not designed for production performance
• Provides maximum introspection capability

Use case: Interactive REPL, learning, debugging - not production.

Ruchy Bytecode: Fast Startup + Good Performance

4.51x faster than Python with instant startup.

Bytecode mode pre-compiles to compact VM instructions:

• Eliminates parsing overhead
• Optimizes common operations
• Maintains fast startup (< 5ms)
• Perfect for CLI tools and scripts

Sweet spot: When you need both speed and instant startup.

Ruchy Transpiled: Near-Optimal Performance

9.79x faster than Python - compiled Rust code quality.

Transpilation workflow:

1. ruchy transpile script.ruchy > script.rs
2. rustc -O script.rs -o binary
3. ./binary

Benefits:

• Full Rust compiler optimizations
• Excellent single-threaded performance
• Can inspect generated Rust code

Trade-off: Slower build times (2-step compilation).

Ruchy Compiled: Peak Performance

9.91x faster than Python - the fastest mode.

Direct compilation:

1. ruchy compile script.ruchy -o binary
2. ./binary

Benefits:

• One-step compilation
• Maximum performance
• Production-ready binaries

Winner: For CPU-bound algorithms, Ruchy delivers 10x Python performance.

Consistency Analysis

Standard deviations:

• Python: ±1.00ms (6.0% variation)
• Ruchy bytecode: ±0.38ms (10.3% variation)
• Ruchy compiled: ±0.20ms (11.9% variation)

All modes show excellent consistency (< 12% variance).

Performance Characteristics

Startup Time

Takeaway: Ruchy bytecode has 4x faster startup than Python.

Memory Usage

Takeaway: Ruchy compiled binaries use 20-30x less memory than Python.

Code Size

Takeaway: Ruchy binaries are 45x smaller than Deno.

When to Use Each Mode

Choose Python When:

• Rapid prototyping (no compilation)
• Extensive library ecosystem needed
• Team expertise in Python
• Moderate performance acceptable

Choose Deno TypeScript When:

• Building web servers (JIT shines)
• TypeScript tooling important
• Node.js compatibility needed
• Long-running processes

Choose Ruchy Bytecode When:

• CLI tools need fast startup
• 4-5x speedup over Python acceptable
• Don’t want compilation step
• Scripting with good performance

Choose Ruchy Transpiled When:

• Need to inspect generated Rust
• Want full rustc optimizations
• Two-step build acceptable
• Maximum single-threaded performance

Choose Ruchy Compiled When:

• Production binaries required
• 10x Python performance needed
• Minimal memory footprint critical
• One-step compilation preferred

Reproducing These Results

All benchmarks are fully reproducible:

# Clone the repo
git clone https://github.com/paiml/ruchy-book
cd ruchy-book/test/ch21-benchmarks

# Install dependencies
cargo install bashrs --version 6.25.0
cargo install ruchy --version 3.171.0

# Run BENCH-007
./run-bench-007-bashrs.sh

# View results
cat results/bench-007-results-bashrs.json

Quality gates ensure:

• Deterministic output (verified)
• No timing contamination
• Full environment capture
• Statistical rigor

See test/ch21-benchmarks/LEGEND.md for detailed setup instructions.

Limitations and Caveats

Benchmark Scope

This chapter measures:

• ✅ CPU-bound algorithms
• ✅ Single-threaded performance
• ✅ Startup time characteristics

This chapter does NOT measure:

• ❌ I/O-bound operations
• ❌ Multi-threaded performance
• ❌ Memory allocation patterns
• ❌ Library ecosystem quality

Hardware Dependency

Results are specific to:

• AMD Ryzen Threadripper 7960X (high-end CPU)
• 125GB RAM
• Linux kernel 6.8.0

Your mileage may vary on different hardware.

Workload Sensitivity

Recursive Fibonacci is:

• ✅ CPU-bound
• ✅ Function call intensive
• ✅ Zero I/O
• ⚠️ Not representative of all workloads

Real-world applications mix:

• String processing
• File I/O
• Data structure manipulation
• Network operations

See remaining benchmarks (BENCH-001 through BENCH-010) for broader coverage.

Next Steps: Comprehensive Benchmarking

This chapter showed one benchmark (BENCH-007). The complete suite includes:

Planned Benchmarks

1. BENCH-001: File I/O - Read 10MB text file - ⚠️ Blocked (Issue #118)
2. BENCH-002: Matrix multiplication (100x100) - ⚠️ Blocked (Bug #003 - global mutable state)
3. BENCH-003: String concatenation (10K operations) - ✅ Complete
4. BENCH-004: Binary tree (memory stress test) - ✅ Complete
5. BENCH-005: Array sum (1M integers) - ✅ Complete
6. BENCH-006: HashMap operations (100K entries) - ⚠️ Blocked (Issue #116)
7. BENCH-007: Fibonacci recursive (n=20) - ✅ Complete
8. BENCH-008: Prime generation (10K primes) - ✅ Complete
9. BENCH-009: JSON parsing (10K objects) - ⚠️ Blocked (Issues #116, #117)
10. BENCH-010: HTTP mock (1K requests)
11. BENCH-011: Nested loops (1000x1000) - ✅ Complete
12. BENCH-012: Startup time (Hello World) - ✅ Complete

Status: 7/12 complete (BENCH-003, 004, 005, 007, 008, 011, 012), 4 blocked (BENCH-001, 002, 006, 009), 1 pending (BENCH-010)

Framework Ready

The benchmarking infrastructure is production-ready:

• ✅ bashrs bench v6.25.0 integration
• ✅ 10 execution modes supported (Python, Deno, Julia, Go, Rust, C, + 4 Ruchy modes)
• ✅ Scientific rigor (warmup, statistics, determinism)
• ✅ Quality gates (lint, determinism checks)
• ✅ ELI5 documentation
• ✅ Fully reproducible

Ready to benchmark the remaining 9 tests.

Summary

What We Learned (Evidence-Based - 7 Benchmarks):

1. Ruchy achieves 15.12x geometric mean performance across 7 diverse benchmarks (82% of C performance)
2. Ruchy EXCEEDS Go in transpiled mode (15.12x vs 13.37x geometric mean)
3. Breakthrough performance: Ruchy transpiled within 12% of C on multiple benchmarks!
4. Breakthrough performance: Ruchy bytecode matches C within 0.26% on BENCH-008
5. Fast startup: Ruchy compiled within 2.6% of C (1.59ms vs 1.55ms)
6. Complete binary tree support: BENCH-004 validates memory allocation and GC performance
7. Nested loop efficiency: BENCH-011 (v3.175.0) shows 96% of C performance on iteration-heavy code - BEATS Rust!
8. Multiple execution modes provide flexibility from development to production
9. Scientific rigor: Following “Are We Fast Yet?” (DLS 2016) methodology

🏆 Performance Tiers (Geometric Mean Across 7 Benchmarks):

Key Metrics Summary (7 Benchmarks Average):

The Verdict (Validated):

Ruchy delivers “Python syntax with native-level performance” - validated across 7 diverse benchmarks:

• ✅ String manipulation (BENCH-003)
• ✅ Binary tree allocation/GC (BENCH-004)
• ✅ Array iteration (BENCH-005)
• ✅ Recursive algorithms (BENCH-007)
• ✅ CPU-bound computation (BENCH-008)
• ✅ Nested loops (BENCH-011)
• ✅ Startup performance (BENCH-012)

Evidence Strength: 7/12 benchmarks complete (58% coverage). Framework validated. Geometric mean analysis complete. Performance claims substantiated with cross-language scientific benchmarking across diverse workload types.

Exercises

1. Run BENCH-007 on your machine using run-bench-007-bashrs.sh
2. Compare your results to the published numbers - hardware dependency?
3. Implement BENCH-001 (file I/O) using the benchmark framework
4. Analyze the transpiled Rust from ruchy transpile bench-007-fibonacci.ruchy
5. Profile memory usage using valgrind or similar tools

Further Reading

• Benchmark Framework: test/ch21-benchmarks/LEGEND.md
• bashrs bench Documentation: bashrs bench --help
• Ruchy v3.171.0 Verification: test/ch21-benchmarks/results/RUCHY-V3.171.0-VERIFICATION.md
• Quality Gates: test/ch21-benchmarks/results/BASHRS-V6.25.0-INTEGRATION.md

Benchmarks are living documents - results updated as Ruchy evolves.