Skip to main content

Benchmarks

Configure how Artemis measures your code's performance from the Settings → Benchmarks tab. Benchmark results are what Artemis uses to compare versions and decide which changes are genuine improvements.

Artemis automatically records runtime, CPU, and memory for every benchmark run — you don't need to configure anything for those. Custom metrics (throughput, accuracy, error rate, inference time, …) are optional and let you track what matters most to your project.

Preparing a benchmark​

To add custom metrics, open the Benchmarks tab and click Prepare benchmark to launch the guided wizard. It walks you through five steps: decide what to measure, write a results file, check it parses, run the benchmark, and view your metrics.

Preparing a benchmark — the five-step flow

  1. Prepare benchmark — from the Benchmarks tab, start the wizard.
  2. Decide what to measure — pick the numbers worth tracking (throughput, accuracy, error rate, …) and tick I know which metrics to track. Runtime, CPU, and memory are already covered for you.
  3. Write a results file — make your benchmark write an artemis_results.json (or .csv) file to the project root just before it exits. The wizard gives you a ready-made prompt you can copy (📋) and hand to a coding agent; once your benchmark writes the file, tick My benchmark writes artemis_results.json….
  4. Check the file parses — paste a sample of your results file. Artemis runs the same checks it uses on real benchmark output and shows exactly how it will read it (e.g. "Valid JSON — 1 measurement across 4 metrics"). When it's valid, tick My file passed validation.
  5. Run benchmark & view your metrics — run the benchmark; your custom metrics are registered and appear on the Benchmarks tab. Their values populate as Artemis benchmarks code versions during Optimise and Discover runs.

The results file​

Artemis reads a file named exactly artemis_results.json or artemis_results.csv from the project root (the working directory — not a subfolder), written by your benchmark right before it exits.

Rules Artemis enforces:

  • The file must be named exactly artemis_results.json or artemis_results.csv and live in the project root.
  • Every metric value must be a finite number (integer or float) — no strings, booleans, null, or nested objects.
  • Use clear, consistent metric names across runs (e.g. throughput, accuracy, error_rate, inference_ms).
  • One measurement → a single JSON object: 
    { "throughput": 4500, "error_rate": 0.02 }
  • Multiple measurements → a JSON array of objects, or a CSV with a header row and one numeric row per measurement.
  • If both a .json and a .csv exist, Artemis uses the JSON.
tip

If your benchmark runs from a subfolder, copy the results file back to the project root before it exits. See Custom Metrics for the full schema and more examples.

Example: wrapping a benchmark that prints to the console​

If your benchmark tool prints its numbers to the terminal instead of writing a file, wrap it in a small script that parses the output and writes artemis_results.json. Commit the wrapper to your repo and set it as the Benchmark command in Runner and Scripts.

For example, a wrapper that runs a throughput benchmark and captures the reported MB/s figures produces a single-object results file like:

{
  "XXH32_MBps": 8926.2,
  "XXH64_MBps": 17770.1,
  "XXH3_64b_MBps": 59029.4,
  "XXH128_MBps": 57612.7
}

On the next run, Artemis reads the file, registers each key as a custom metric, and starts tracking it across versions alongside the built-in runtime, CPU, and memory.

Next Steps​

Last updated on by Rozhin Khalilian