Skip to main content

Verifying commands on a runner

note

This tutorial needs Artemis CLI 1.1.18 or later. If artemis version shows an older version, run the installer again with --version 1.1.18.

Goal​

This example assumes you have completed Get started with Particle Life: skills loaded, CLI authenticated, Git key available, and a compatible runner online.

Unlike the basic example, you will not treat local compile/test/benchmark success as the gate. You will fork a teaching branch, import it, iterate command strings on the runner with changeset validate, fix a broken harness in Git, pull the project to the new commit, re-validate, and run a short Discovery.

What you will learn​

  • how to try different compile, test, and benchmark command strings on a runner;
  • how to read a validation's log (artemis changeset validation logs) when exit codes alone are not enough;
  • how to update the project's code after editing the harness (project pull); and
  • why a fresh empty changeset is required after the project's gitHash moves.

Teaching seed​

Use the public lab branch:

https://github.com/turintech/particle-life
branch: lab/runner-commands
commit: 00f6b2d484d05be9cb6e652573e4953b3cc1b9e5

That revision keeps CMake, the simulation library, and checksum tests. ./build/particle_life benchmark prints fps= to stdout only. tools/benchmark.py exists but writes artemis_results.json under tools/ instead of the repository root.

1. Fork and clone the lab branch​

Ask the assistant to fork turintech/particle-life under your account (explicit permission required), then clone your fork and check out lab/runner-commands at the commit above.

gh repo fork turintech/particle-life --clone=true
cd particle-life
git fetch origin lab/runner-commands
git checkout lab/runner-commands
git rev-parse HEAD   # expect 00f6b2d484d05be9cb6e652573e4953b3cc1b9e5

Checkpoint: the checkout is your fork, on lab/runner-commands, at the lab commit.

2. Import the fork​

Skill: project-import

Import the fork URL and lab/runner-commands branch — not upstream main:

artemis project import \
  --git-url https://github.com/<your-account>/particle-life \
  --key-id <key-id> \
  --branch lab/runner-commands \
  --name particle-life-runner-commands-lab

Wait until importedStatus is success, then confirm the imported gitHash matches the lab commit.

Checkpoint: Artemis project UUID exists and points at the teaching seed. Do not treat a local build as verification.

3. Level 1 — try commands on the runner​

Skill: repo-command-setup (§5b)

Discovery runs the commands saved in a script. Save a first attempt as the project's default script, create an empty Branch, then validate the Branch's original code on your runner:

artemis --output-format json project scripts create --project <project-id> \
  --name "Particle Life benchmark" --default \
  --setup-cmd "make -j\$(nproc)" \
  --setup-cmd "ctest" \
  --benchmark-cmd "./build/particle_life benchmark 2200 30"
# → capture <script-id>

artemis --output-format json changeset create --project <project-id> --name "runner check"
# → capture <changeset-id>

artemis --output-format json changeset validate <changeset-id> \
  --project <project-id> --version original --script <script-id> \
  --runner <runner-name> --wait

Expect failures. Change the script's commands and validate again until the build and test commands work:

artemis project scripts update <script-id> --project <project-id> \
  --setup-cmd "cmake -S . -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build --parallel" \
  --setup-cmd "ctest --test-dir build --output-on-failure" \
  --benchmark-cmd "python3 tools/benchmark.py --no-visualize"

artemis --output-format json changeset validate <changeset-id> \
  --project <project-id> --version original --script <script-id> \
  --runner <runner-name> --wait

Keep using the same empty Branch while only the script changes. Take the validation's id from the validate response, and read its log to see what the benchmark did:

artemis changeset validation logs <validation-id> --project <project-id>

The build and test commands should reach exitCode: 0. The harness still writes metrics under tools/, so the Artemis results contract is not met even when the benchmark exits with zero. Exit codes don't show metric values or where the results file was written; the validation log does.

Checkpoint: you have working build and test commands, and the validation log shows that the harness writes to the wrong path.

4. Level 2 — edit, push, and update project code​

Fix tools/benchmark.py so it writes numeric {"simulation_fps": ...} to artemis_results.json at the repository root (not under tools/). Keep the harness headless.

git add tools/benchmark.py
git commit -m "Write simulation_fps to root artemis_results.json"
git push origin lab/runner-commands

Update the Artemis project to the new commit with project pull:

artemis project compare <project-id>
artemis project pull <project-id>

Wait until the project's gitHash matches your fix commit.

Create a new empty changeset after the pull. The previous changeset's original stays on the old SHA.

artemis --output-format json changeset create --project <project-id> --name "runner check after fix"
# → capture <new-changeset-id>

artemis --output-format json changeset validate <new-changeset-id> \
  --project <project-id> --version original --script <script-id> \
  --runner <runner-name> --wait

artemis changeset validation logs <validation-id> --project <project-id>

Confirm every command shows exitCode: 0 and the logs show a root-level numeric simulation_fps write.

Checkpoint: the project's original code includes the fixed harness, and runner validation proves the script's commands.

5. Run a short Discovery​

Skills: discovery-start, then discovery-inspect

artemis discovery create \
  --project <project-id> \
  --task "Maximize simulation_fps without changing simulation behavior or weakening the correctness tests." \
  --versions 5 \
  --script <script-id> \
  --target-files src/simulation.cpp \
  --target-files src/simulation.hpp \
  --runner <runner-name> \
  --mode automatic \
  --model <model-catalogue-uuid>

The run uses the script you verified on the runner. Monitor with discovery get / versions list until the baseline is finalised and at least one version has measured simulation_fps.

Checkpoint: Discovery ran on a runner-verified script against your fixed harness; rank versions by measured simulation_fps, not only by the Composite score.

What to expect​

  • Changing the script's commands is cheap and doesn't change the project's code, so reuse the empty Branch.
  • Script or source fixes require push → project pull → a new empty changeset before re-validation.
  • changeset validate reports exit codes, resource use and each recorded metric's mean. Read artemis changeset validation logs to see where the results file was written.
  • Local success on your laptop is useful rehearsal, not proof the selected runner can run the same commands.

Optional next steps​

  • Steer a running discovery to redirect a live run and expand its budget.
  • Make a repository ready when the repository has no harness at all.
  • Adapt the same runner verify → fix → pull loop to your own repository when local and runner environments differ.
Last updated on by Eoin