Skip to main content

Metrics

Project Settings → Metrics lists the custom metrics your benchmark records. Benchmark results are what Artemis uses to compare Versions and Branches with the Baseline.

Artemis already records runtime, CPU and memory whenever benchmark commands run. Custom metrics (throughput, accuracy, error rate, inference time and so on) are optional and measure what matters to your project.

The metrics table​

Each custom metric shows its name, Direction (Higher is better or Lower is better), Aggregator (mean, median, min, max or p95), Unit, and a summary of what has been recorded so far. Edit a metric to change its direction, aggregator or unit. Direction and aggregator change how every existing and future reading of that metric is judged and summarised.

A metric appears here after a benchmark has written it at least once.

Add metrics​

Click Add metrics to open Create a script to track metrics. Describe what you want to measure, for example API latency while keeping responses correct, and choose one of three routes:

  1. Use an Artemis agent. An agent prepares the benchmark and the Script on a Branch, runs it, and reports the metrics it recorded. It needs an online Runner.
  2. Use Artemis CLI and your local agent. Download benchmark-prepare-changeset.md and give it to your own coding agent. It prepares the benchmark in your checkout and saves the result into an Artemis Branch through the CLI.
  3. Prepare a Git branch without Artemis CLI. Download benchmark-prepare-git.md. Your agent prepares and verifies the benchmark, then commits and pushes a Git branch that you select in Artemis.

In every route, the benchmark reports its numbers by writing a results file.

The results file​

After each benchmark iteration, the Runner reads artemis_results.json or artemis_results.csv from the build directory, which is the root of the staged code, or any folder below it.

  • JSON takes precedence over CSV, and the first file found of that type is used. Avoid leaving several competing results files around.
  • JSON holds one object, or a list of objects, with metric names as keys and numeric values: 
    { "throughput": 4500, "error_rate": 0.02 }
  • CSV uses metric names as column headers and one row per measurement, not name,value pairs.
  • Every value must be a finite number. No strings, booleans, null or nested objects.
  • Keep metric names and units the same across runs.
  • Write a fresh file on every iteration. Each benchmark iteration is read separately, so do not append earlier samples again.

Only the benchmark phase collects metrics. Numbers printed to the console stay in the logs, and a command that exits successfully without writing the file records no custom metrics. See Custom Metrics for more examples.

Example: wrapping a benchmark that prints to the console​

If your benchmark tool prints its numbers instead of writing a file, wrap it in a small script that parses the output and writes artemis_results.json. Commit the wrapper to your repository and use it as a benchmark command in your Script.

For example, a wrapper that runs a throughput benchmark and captures the reported MB/s figures produces:

{
  "XXH32_MBps": 8926.2,
  "XXH64_MBps": 17770.1,
  "XXH3_64b_MBps": 59029.4,
  "XXH128_MBps": 57612.7
}

On the next run Artemis reads the file, registers each key as a custom metric, and tracks it across Versions alongside runtime, CPU and memory.

Next steps​

Last updated on by Mike Basios