# PR: Add Command-Line Benchmark Mode to GrowNet Runner

## Title

benchmark: add CLI benchmarking mode for cross-language GrowNet runtime comparison

---

## Summary

This PR adds a **benchmark mode** to the GrowNet command-line runner so we can compare implementation performance across languages using the same scenarios and arguments.

The goal is to answer questions like:

- Which implementation is fastest for `single_neuron`?
- How does `two_layer` compare in Java vs Python?
- What is the latency distribution for `growth`?
- What is the throughput of each implementation under the same scenario and same input settings?

This benchmark mode should be integrated into the existing command-line runner, while remaining clearly separate from trace mode.

---

## Important architectural rule

Respect GrowNet's language-reference model:

- **Java** = core semantic / architectural reference
- **Python** = Python-language-style reference
- **Mojo** should track Python almost one-to-one
- **C++** follows Java as much as practical

Benchmarking must compare implementations fairly, but it must **not** force Python or Mojo to become Java-like.

The purpose of benchmarking is visibility and measurement, not semantic distortion.

---

## Required reading before coding

Read these before implementation:

1. `docs/READ_ORDER.md`
2. `docs/LANGUAGE_REFERENCE_POLICY.md`
3. `docs/CODING_STYLE_MUST_READ.md`
4. `docs/GOLDEN_RULE.md`
5. `docs/GROWTH.md`
6. `docs/CREATION_AND_GROWTH_POINTS.md`
7. `docs/contracts/grownet.contract.v5.json`
8. `docs/GrowNet_Design_Spec_V5.md`
9. `docs/TRACE_RUNNER.md`
10. `docs/TRACE_PARITY.md`
11. `docs/PARITY_DECISION_CHECKLIST.md`

---

## Coding style requirements

These are mandatory:

- Follow `docs/CODING_STYLE_MUST_READ.md`
- No single- or double-character identifiers anywhere, including loops
- Keep Python explicit and readable
- Avoid clever one-liners
- Preserve determinism
- Keep the benchmark harness simple and obvious
- Do not introduce speculative redesigns
- Do not change runtime semantics just to improve benchmark numbers

---

## Scope

### Implement in this PR

1. benchmark mode in the existing CLI runner
2. benchmark support in Java and Python runners
3. comparable machine-readable benchmark output
4. human-readable summary output
5. latency and throughput statistics
6. docs explaining how to run and compare benchmarks
7. focused tests for benchmark plumbing and output shape

### Do not implement in this PR

- Mojo benchmark runtime unless it already exists and is trivial to wire
- C++ benchmark runtime unless it already exists and is trivial to wire
- benchmark-driven runtime rewrites
- profiler integration
- visualization UI
- broad architecture refactors

---

# Main design goal

Benchmarking should be a **separate execution mode**, not mixed into trace mode.

## Preferred model

- `trace` mode = explain what happened
- `benchmark` mode = measure how fast it happened

These can coexist, but benchmark mode should default to **trace off** because tracing dramatically changes performance.

---

# Phase 1 — extend CLI surface

## 1. Add benchmark flags to the command-line runner

Extend `./grownet.sh` and backend runner parsing to support:

```text
--benchmark
--benchmark-format text|jsonl
--benchmark-warmup 100
--benchmark-iterations 1000
--benchmark-repeat 5
--benchmark-scenario single_neuron|two_layer|image_2d|growth
--benchmark-value 0.42
--benchmark-output stdout|path/to/file
```

### Notes

- `--benchmark-scenario` may reuse `--scenario` if you prefer one shared argument surface
- keep the interface clean and not redundant
- if shared args are reused, document it clearly

### Preferred behavior

If `--benchmark` is present:

- run warmup iterations first
- then measured iterations
- then emit aggregate stats

---

# Phase 2 — benchmark execution model

## 2. Benchmark Java and Python first

Implement benchmark support for:

- Java
- Python

Do not block this PR on Mojo or C++.

If the launcher accepts Mojo/C++, benchmark mode should return a clean “not implemented” message for those languages unless they are already available.

## 3. Use deterministic benchmark scenarios

Benchmarking must use deterministic scenario construction.

For the same command line, each implementation must benchmark:

- the same scenario
- the same value(s)
- the same tick count
- the same warmup count
- the same measured iteration count

## 4. Measure scenario execution, not process startup

Benchmarking should primarily measure the runtime path of the scenario, not the overhead of launching a fresh process per iteration.

### Preferred model

Inside each backend runner:

- build scenario setup once where appropriate
- or rebuild per iteration if that is the true intended workload
- but make the choice explicit and consistent per scenario

### Important

Document what exactly is being benchmarked:

- full scenario construction + execution
or
- execution only after setup
or
- both, as separate benchmark modes

### Recommended approach

Start with:

- **end-to-end per-run scenario execution benchmark**
- explicit and simple
- easiest to reason about across languages

If you later want setup-vs-execution split, add it in a future PR.

---

# Phase 3 — benchmark statistics

## 5. Compute useful latency stats

For each benchmark run, compute at minimum:

- count
- min latency
- max latency
- mean latency
- median / p50
- p95
- p99
- total elapsed wall time
- throughput (iterations per second)

### Units

Use a clear unit consistently.

Preferred:
- per-iteration latency in **microseconds**
- total elapsed in **milliseconds**
- throughput in **iterations/sec**

## 6. Repeat support

If `--benchmark-repeat N` is provided:

- execute the benchmark N times
- emit per-run stats
- emit an overall summary

This helps reduce noise.

---

# Phase 4 — benchmark output formats

## 7. Text output

Default human-readable benchmark output should look something like:

```text
[benchmark.start] lang=java scenario=single_neuron warmup=100 iterations=1000 repeat=5 trace=false

[benchmark.result] lang=java scenario=single_neuron run=1 count=1000 min_us=8.200 p50_us=9.100 p95_us=11.400 p99_us=13.800 max_us=20.100 mean_us=9.540 throughput_per_sec=104822.3

[benchmark.result] lang=java scenario=single_neuron run=2 count=1000 min_us=8.000 p50_us=9.000 p95_us=11.200 p99_us=13.500 max_us=19.700 mean_us=9.420 throughput_per_sec=105991.8

[benchmark.summary] lang=java scenario=single_neuron runs=5 best_p50_us=8.900 best_p99_us=13.300 best_throughput_per_sec=106220.4 overall_mean_us=9.470
```

## 8. JSONL output

Machine-readable output should be supported with:

```text
--benchmark-format jsonl
```

Each line should be a JSON object.

Example:

```json
{"event":"benchmark.start","lang":"python","scenario":"two_layer","warmup":100,"iterations":1000,"repeat":5,"trace":false}
{"event":"benchmark.result","lang":"python","scenario":"two_layer","run":1,"count":1000,"min_us":31.4,"p50_us":35.1,"p95_us":44.8,"p99_us":50.3,"max_us":71.2,"mean_us":36.8,"throughput_per_sec":27173.9}
{"event":"benchmark.summary","lang":"python","scenario":"two_layer","runs":5,"overall_mean_us":36.5,"best_p50_us":34.8,"best_p99_us":49.6,"best_throughput_per_sec":27502.7}
```

---

# Phase 5 — benchmark implementation details

## 9. Add benchmark helpers in Java

Add a small benchmark utility in the Java debug/runner area.

Possible files:

```text
src/java/ai/nektron/grownet/debug/BenchmarkStats.java
src/java/ai/nektron/grownet/debug/BenchmarkRunner.java
```

Responsibilities:

- collect per-iteration timings
- compute percentiles
- format text/jsonl
- reuse existing scenario runners where practical

## 10. Add benchmark helpers in Python

Add similar support in Python, for example:

```text
src/python/debug/benchmark_stats.py
src/python/debug/benchmark_runner.py
```

Responsibilities mirror Java.

## 11. Use monotonic timers

Use monotonic high-resolution timing in each language.

Examples:
- Java: `System.nanoTime()`
- Python: `time.perf_counter_ns()`

Do not use wall-clock APIs for per-iteration timing.

---

# Phase 6 — scenario benchmarking rules

## 12. Support at least these scenarios initially

- `single_neuron`
- `two_layer`
- `growth`

### Optional / provisional
- `image_2d`

If `image_2d` is not mature enough yet for fair apples-to-apples benchmarks, mark it as provisional in docs.

## 13. Keep benchmark semantics fair

Benchmark runs must not accidentally compare different workloads across languages.

For each scenario, define clearly whether the benchmark measures:

- one tick
- N ticks
- scenario setup + one tick
- scenario setup + N ticks

And keep it the same across Java and Python.

---

# Phase 7 — docs and usage

## 14. Update trace/runner docs

Update `docs/TRACE_RUNNER.md` to include a benchmark section.

Make it clear:

- trace mode explains behavior
- benchmark mode measures performance
- trace should usually be off during benchmarking
- JSONL benchmark output is suitable for later aggregation/comparison

## 15. Add a dedicated benchmark doc

Create:

```text
docs/BENCHMARK_RUNNER.md
```

Include:

- supported flags
- example commands
- explanation of warmup / iterations / repeat
- text vs JSONL output
- how to compare Java and Python fairly
- warning that benchmark numbers depend on machine, Python version, JVM state, etc.

---

# Phase 8 — tests

## 16. Add focused benchmark tests

At minimum, test:

- benchmark CLI argument parsing
- benchmark output contains required fields
- JSONL benchmark output parses
- benchmark summary emits for Java
- benchmark summary emits for Python

### Important

Do not write fragile performance-threshold tests like:
- “must be faster than X”
- “Python must be slower than Java”

Test shape and consistency, not absolute speed.

---

# Phase 9 — optional comparison helper

## 17. Optional but useful: add a small comparison script

If small and clean, add a helper like:

```text
tools/benchmark/compare_benchmark_results.py
```

Purpose:
- read JSONL benchmark outputs
- compare runs by language/scenario
- print a compact comparison table

This is optional for this PR. Do it only if it stays small and obvious.

---

## Acceptance criteria

This PR is complete when:

1. `./grownet.sh` supports benchmark mode
2. Java benchmark mode works
3. Python benchmark mode works
4. benchmark output includes:
   - min
   - p50
   - p95
   - p99
   - max
   - mean
   - throughput
5. text and JSONL benchmark output are both supported
6. docs explain how to run benchmarks fairly
7. tests verify benchmark plumbing/output shape
8. no runtime semantics were changed just to chase benchmark numbers

---

## Suggested commit structure

1. add CLI benchmark flags and parsing
2. add Java benchmark runtime
3. add Python benchmark runtime
4. add docs and tests
5. optional comparison helper

---

## Final response expected from Codex

At the end, provide:

1. summary of what changed
2. files changed
3. example benchmark commands
4. sample output for Java and Python
5. whether `image_2d` is fully benchmark-ready or provisional
6. tests run
7. any follow-up ideas for Mojo / C++ benchmark support
