# BENCHMARK_RUNNER.md

`./grownet.sh` supports a benchmark mode for deterministic Java and Python scenario comparison.

## Scope

Benchmark mode is for runtime measurement, not trace explanation.

- Benchmark mode reuses the existing scenario runner.
- Python benchmark mode can also run config-driven experiments via `--config`.
- If a Python config enables Focus & Anchoring V1, the benchmark measures that focused path as part of the configured workload.
- The measured workload is end-to-end per iteration:
  - build the selected scenario
  - run `--ticks` ticks
  - collect benchmark timings inside one backend process
- Warmup runs execute before measured iterations.
- Benchmark mode defaults to trace off.

## Supported Languages

- `--lang java`
- `--lang python`

The launcher also accepts `--lang mojo` and `--lang cpp`, but benchmark mode returns the same deterministic not-implemented message as trace mode:

- Exit code: `2`
- Message: `GrowNet trace runner: --lang <lang> is not implemented in this repository.`

## Shared Scenario Flags

- `--scenario single_neuron|two_layer|image_2d|growth`
- `--value 0.42`
- `--ticks 1`

## Benchmark Flags

- `--benchmark`
- `--benchmark-format text|jsonl`
- `--benchmark-warmup 100`
- `--benchmark-iterations 1000`
- `--benchmark-repeat 5`
- `--benchmark-output stdout|path/to/file.log`
- `--config path/to/experiment.json`
- `--input path/to/input_file`
- `--input-type sequence|tensor|image|audio`
- `--output path/to/output_file_or_dir`

## Output Fields

Each measured run reports:

- `count`
- `min_us`
- `p50_us`
- `p95_us`
- `p99_us`
- `max_us`
- `mean_us`
- `total_elapsed_ms`
- `throughput_per_sec`

Summary output reports:

- `runs`
- `overall_mean_us`
- `overall_elapsed_ms`
- `best_p50_us`
- `best_p99_us`
- `best_throughput_per_sec`

Latency values are per iteration in microseconds. Total elapsed values are milliseconds. Throughput is iterations per second.

## Output Formats

### Text

Default output is human-readable text:

```text
[benchmark.start] lang=java scenario=single_neuron ticks=1 warmup=100 iterations=1000 repeat=5 trace=false
[benchmark.result] lang=java scenario=single_neuron run=1 count=1000 min_us=8.200 p50_us=9.100 p95_us=11.400 p99_us=13.800 max_us=20.100 mean_us=9.540 total_elapsed_ms=9.866 throughput_per_sec=101357.900
[benchmark.summary] lang=java scenario=single_neuron runs=5 overall_mean_us=9.470 overall_elapsed_ms=48.991 best_p50_us=8.900 best_p99_us=13.300 best_throughput_per_sec=106220.400
```

### JSONL

Use `--benchmark-format jsonl` for machine-readable output:

```json
{"event":"benchmark.start","lang":"python","scenario":"two_layer","ticks":1,"warmup":100,"iterations":1000,"repeat":5,"trace":false}
{"event":"benchmark.result","lang":"python","scenario":"two_layer","run":1,"count":1000,"min_us":31.4,"p50_us":35.1,"p95_us":44.8,"p99_us":50.3,"max_us":71.2,"mean_us":36.8,"total_elapsed_ms":37.1,"throughput_per_sec":26954.2}
{"event":"benchmark.summary","lang":"python","scenario":"two_layer","runs":5,"overall_mean_us":36.5,"overall_elapsed_ms":184.7,"best_p50_us":34.8,"best_p99_us":49.6,"best_throughput_per_sec":27502.7}
```

## Example Commands

```bash
./grownet.sh --lang java --benchmark --scenario single_neuron --benchmark-warmup 25 --benchmark-iterations 250 --benchmark-repeat 3
./grownet.sh --lang python --benchmark --scenario two_layer --value 0.42 --benchmark-warmup 25 --benchmark-iterations 250 --benchmark-repeat 3
./grownet.sh --lang java --benchmark --benchmark-format jsonl --scenario growth --value 0.42 --ticks 2
./grownet.sh --lang python --benchmark --benchmark-format jsonl --scenario image_2d --benchmark-warmup 10 --benchmark-iterations 50 --benchmark-repeat 3
./grownet.sh --lang python --benchmark --config experiments/single_scalar_sequence.json --benchmark-warmup 25 --benchmark-iterations 250 --benchmark-repeat 3
```

## Fair Comparison Notes

For a fair Java/Python comparison:

- run both commands on the same machine
- keep `--scenario`, `--value`, `--ticks`, `--benchmark-warmup`, `--benchmark-iterations`, and `--benchmark-repeat` identical
- for config-driven Python runs, keep the experiment config and any `--input` override identical
- leave tracing off unless you are intentionally measuring trace overhead
- repeat runs and compare the distribution, not a single best number

Benchmark numbers depend on:

- machine hardware
- OS scheduling
- Python version
- JVM warmup and JIT state
- background load

## Trace Interaction

If you pass `--trace` together with `--benchmark`:

- trace processing remains part of the measured workload
- benchmark output still owns stdout by default
- trace-to-stdout is suppressed so benchmark output stays parseable
- use `--out path/to/file.log` if you want trace logs written separately during benchmarking

## Scenario Notes

- `single_neuron`, `two_layer`, and `growth` are the primary benchmark scenarios.
- `image_2d` is available but provisional for strict cross-language comparison because the Java benchmark scenario is still intentionally lightweight rather than a richer retina-style workload.
- Config-driven benchmarking is Python-first in this PR. Use `docs/EXPERIMENT_RUNNER.md` for the shared config model and input-source rules.
