Custom Metrics

Track domain-specific measurements such as throughput, accuracy, error rates, or anything else your benchmark produces. Artemis already records runtime, CPU and memory. Custom metrics measure what matters to your project, so you can compare those values across Versions of your code.

How Artemis reads your metrics

Your benchmark writes a results file, and Artemis picks it up after every benchmark iteration.

1. Write a results file. Have your benchmark write artemis_results.json or artemis_results.csv in the build directory (the root of the staged code) or a folder below it.
2. Run the benchmark. Put the benchmark in the benchmark phase of a Script and run it from a Branch's Scripts tab, or let a Discovery Run run it. Each benchmark iteration is read separately.
3. Artemis picks it up. After each iteration the Runner reads the file and stores every value. If both a .json and a .csv exist, JSON takes precedence.
4. Compare across Versions. Recorded values appear in the Project's Settings → Metrics and in the Discovery and Branch charts, so you can see how each Version moves the numbers.

File format

Use any metric names you want. Each object (JSON) or row (CSV) is one measurement. Repeated benchmark runs build up a distribution.

artemis_results.json

[
  { "throughput": 4500, "error_rate": 0.02 },
  { "throughput": 4620, "error_rate": 0.01 },
  { "throughput": 4480, "error_rate": 0.03 }
]

artemis_results.csv

throughput,error_rate
4500,0.02
4620,0.01
4480,0.03

Numbers only

Every value must be a finite number. Keep metric names consistent across measurements so they line up into one series.

The full set of rules Artemis applies to the file is in Metrics. In short: keep the name exact, write one file per iteration and use consistent metric names across runs.

Set it up with your coding agent

Custom metrics need a small change to your benchmark code. Paste this prompt into your own coding agent (Claude Code, Cursor, Copilot and so on) inside your repository. It reads your benchmark and adds the artemis_results output. The Project's Settings → Metrics → Add metrics also offers maintained prompts for this.

I'm setting up **custom metrics** for TurinTech Artemis.

Artemis runs my benchmark on a runner and, after each run, reads a results file from the repository to track performance metrics across versions of my code.

Please update my benchmark so that, right before it exits, it writes the key metrics it measures to an `artemis_results.json` (or `artemis_results.csv`) file in the project root.

Rules Artemis enforces:
- The file must be named exactly `artemis_results.json` or `artemis_results.csv` and sit in the directory the benchmark command starts in (the repository root) or a folder below it.
- Every metric value must be a finite number (integer or float). No strings, booleans, null, or nested objects.
- Use clear, consistent metric names across runs (e.g. throughput, accuracy, error_rate, inference_ms).
- One run -> a single JSON object: {"throughput": 4500, "error_rate": 0.02}
- Multiple measurements -> a JSON array of objects, or a CSV with a header row and one numeric row per measurement.
- If both a .json and a .csv exist, Artemis uses the JSON.

Steps:
1. Read my benchmark code and list the numeric metrics it already computes.
2. Pick the ones worth tracking and add code to write them to artemis_results.json before the benchmark exits.
3. If the benchmark runs outside the repository, copy the results file back into it before exiting.
4. Run the benchmark locally and confirm the file appears in the project root with numeric values.
5. Show me the diff.

Prefer a guided, end-to-end setup?

Install the Artemis agent skills and ask your coding agent to use repo-command-setup. It derives and verifies the commands Artemis needs, including numeric artemis_results metrics. See Artemis Skills.

Running outside the repository

If your benchmark runs somewhere outside the staged code, such as a separate clone or a container mount, copy the results file back before the command exits.

# Run your benchmark wherever it lives, then copy the
# results file back to the project root before exiting.
ORIG=$(pwd)
cd /path/to/clone
./run_bench.sh
cp artemis_results.json "$ORIG/"

Artemis only looks in the build directory and the folders below it, so the file has to land there. Capturing $(pwd) first is the simplest way to copy it back from wherever your benchmark ran.

What you get

Once values are recorded, each metric can be compared across Versions and Branches with its spread, so you can spot noise and outliers. Report the number of observations and their spread, not only the best result, when you judge a small improvement.