vms: record provenance, steps and an evidence page with each run #19

Merged

asxpi merged 3 commits from vms/run-provenance into main

2026-11-09 12:49:02 +00:00

asxpi commented

2026-11-09 12:38:47 +00:00

Member

Every VM run recorded from now on explains itself, so a human checking a bug in #9 sees how aim was driven and can re-run it.

run.toml: harness, harness_commit (full sha), harness_dirty, command (as typed), peers; the exact script copied in as scenario.py.
steps.log: every click (position and OCR'd text), key, typed text (passwords as a placeholder), shot, tone share, test-card reading, file hash, RTP digest, notification and PASS/FAIL, with UTC times.
Side files kept: notify-<vm>.log, rtp.log, web-rtp.log.
EVIDENCE.md (plain markdown) from vms/EVIDENCE.template.md: header filled at record time; the judge fills what proves it (≥2 file:line quotes), expected/seen/spec, deciding screenshots, control run.
interop.py check: runs with harness need all of the above and no placeholder left; older runs unchanged. finish.zsh/commit.zsh refuse unfilled evidence.
New run names show both sides: <date>-<time>-<SPEC>-aim-vs-<peers>-<server>.

Verified with one real --record (XEP-0308 vs Gajim on ejabberd, not committed): its filled EVIDENCE.md passes check, no password in any file. pytest 46 passed, 979 runs OK, nix flake check passes.

🤖 Generated with Claude Code

Every VM run recorded from now on explains itself, so a human checking a bug in #9 sees how aim was driven and can re-run it. - run.toml: `harness`, `harness_commit` (full sha), `harness_dirty`, `command` (as typed), `peers`; the exact script copied in as `scenario.py`. - `steps.log`: every click (position and OCR'd text), key, typed text (passwords as a placeholder), shot, tone share, test-card reading, file hash, RTP digest, notification and PASS/FAIL, with UTC times. - Side files kept: `notify-<vm>.log`, `rtp.log`, `web-rtp.log`. - `EVIDENCE.md` (plain markdown) from `vms/EVIDENCE.template.md`: header filled at record time; the judge fills what proves it (≥2 `file:line` quotes), expected/seen/spec, deciding screenshots, control run. - `interop.py check`: runs with `harness` need all of the above and no placeholder left; older runs unchanged. `finish.zsh`/`commit.zsh` refuse unfilled evidence. - New run names show both sides: `<date>-<time>-<SPEC>-aim-vs-<peers>-<server>`. Verified with one real `--record` (XEP-0308 vs Gajim on ejabberd, not committed): its filled EVIDENCE.md passes check, no password in any file. pytest 46 passed, 979 runs OK, `nix flake check` passes. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

asxpi added 3 commits

2026-11-09 12:38:47 +00:00

vms: record provenance, steps and an evidence page with each run 971f626c03

Bugs in issue #9 cite runs whose driver scripts were never kept, so a
person checking one can't see how aim was driven or re-run it. Each
--record run now carries what made it:

- session.json (and so run.toml, via finish) gets harness,
  harness_commit, harness_dirty, command and peers; vmrun.zsh passes
  the command as typed, with the knobs that matter, and an explore
  variant that records nothing.
- The script is copied in as scenario.py.
- interopvm.step_log writes steps.log: every click (position and the
  OCR'd text it hit), key, typed text (the password only as a
  placeholder), shot, tone share, test card, file hash, RTP digest and
  notification seen, and the PASS/FAIL/SKIP/INFO lines.
- The side files that stayed in the out dir come along: notify-<vm>.log,
  rtp.log (all rtp_stats/call_diag digests) and web-rtp.log.
- EVIDENCE.md is written from vms/EVIDENCE.template.md with the header
  filled (versions, harness@commit link, re-run command, steps); the
  judge fills the proof, expected/seen, screenshots and control.

New run dirs name both sides, <SPEC>-aim-vs-<peers>-<server>; nothing
parses the name beyond its date-time prefix. Everything copied into the
run is redacted again with the accounts' passwords. finish.zsh asks for
EVIDENCE.md to be filled; commit.zsh refuses a run that fails check or
still has placeholders.

Assisted-by: Claude:opus-5-5

report: check VM runs' provenance and EVIDENCE.md 74096d748d

A VM run recorded with provenance (a harness key) must have a full
harness_commit, command, peers, scenario.py and steps.log in [files],
and an EVIDENCE.md with no placeholder left, its verdict row matching
run.toml and at least two file:line quotes. EVIDENCE.md is not hashed
(the judge writes it after finish and may amend it after review) but
is scanned for secrets and passwords like the logs. finish writes the
session's provenance and driven = "vm" into run.toml.

Older runs have no harness key and stay valid as they are.

Assisted-by: Claude:opus-5-5

report: point finish at EVIDENCE.md when check fails

/ check (pull_request) Successful in 9m56s

Details

80401ac5d1

finish ended with "Next: git add ... && git commit" even when its own
check had just failed, which for a VM run is now the normal case: the
judge fills EVIDENCE.md after finish, once server.log is in. Say what
to do instead.

Assisted-by: Claude:opus-5-5

asxpi merged commit 1c2478518a into main

2026-11-09 12:49:02 +00:00

asxpi referenced this pull request from a commit

2026-11-09 12:49:03 +00:00

Merge pull request 'vms: record provenance, steps and an evidence page with each run' (#19) from vms/run-provenance into main

asxpi referenced this pull request

2026-11-09 14:48:35 +00:00

vm: batch 0, evidence re-recordings for #9 #20

asxpi referenced this pull request

2026-11-09 15:02:30 +00:00

WIP: Verify anotherim-js/libtacky bugs from the GUI VM runs (rc9 / v1.2.3, 20 found) #9