another-im/interop
Watch
9
0
Fork
You've already forked interop
 0
Code Issues 5 Pull requests 1 Releases 7 Activity Actions

tools/ai: harness cleanup, scenario lint and a fake-tackyd test #7

Merged
asxpi merged 9 commits from harness-cleanup into main 2026-11-08 02:07:33 +00:00
Conversation 0 Commits 9 Files changed 115 +1281 -169
asxpi commented 2026-11-08 01:34:30 +00:00
Member
Copy link

Harness cleanup from a separate session, plus a fix:

  • tools/ai/common.py: holds the scenarios' shared bookkeeping.
  • Scenarios: what pyflakes finds is fixed, and they are linted in CI.
  • tools/ai/test_harness.py: tests the harness against a fake tackyd-json.
  • interop.py: says what a fail is down to, and clears a run's state at finish.
  • The broken run is dropped: runs/20261107-184109-XEP-0384-slixmpp-ejabberd (only its run.toml was committed, and it is a pass without the OMEMO markers).
  • main is merged in, including the fast check (#5).

This branch is built on ai-runs-2026-11-07, so it also carries #4's two C3.1 commits. Merge #4 first, or merge this one and close #4.

Locally: pyflakes clean, 46 tests pass, report --check clean. The PR changes tools/interop.py, so CI runs the full check over all runs.

Harness cleanup from a separate session, plus a fix: - **`tools/ai/common.py`:** holds the scenarios' shared bookkeeping. - **Scenarios:** what pyflakes finds is fixed, and they are linted in CI. - **`tools/ai/test_harness.py`:** tests the harness against a fake tackyd-json. - **`interop.py`:** says what a fail is down to, and clears a run's state at finish. - **The broken run is dropped:** `runs/20261107-184109-XEP-0384-slixmpp-ejabberd` (only its run.toml was committed, and it is a pass without the OMEMO markers). - **`main` is merged in,** including the fast check (#5). This branch is built on `ai-runs-2026-11-07`, so it also carries #4's two C3.1 commits. Merge #4 first, or merge this one and close #4. Locally: pyflakes clean, 46 tests pass, `report --check` clean. The PR changes `tools/interop.py`, so CI runs the full check over all runs. 
A draft holding U+0001 never left the device: chatsync's PublishDraft
wrote it uncleaned and the writer refused it, escaping as a bgerror.
repro-C3.1.py sets two such drafts (tokenless and as a request), then a
second tackyd-json session on the same store reads the drafts node;
ai1a's other device (slixmpp) watches and restores the node and ai1b
observes. v1.2.1 fails (no publish, bgerror), fix-C3.1 (libtacky
fix/C3.1-draft-xml-text 040cb864) passes. Both runs pass check.

AI-RESULTS.md is not regenerated here: report would also pick up
another agent's uncommitted XEP-0384 run.

Assisted-by: Claude:opus-5-5
AI-RESULTS: regenerate with the C3.1 runs
All checks were successful
/ check (pull_request) Successful in 14m29s
Details
2d66c2215c
 
The C3.1 commit added two runs but left the reports as they were.

Assisted-by: Claude:opus-5-5
Every agent-written scenario carried its own check(), info(), until(),
muc_service(), body_of() and mark() because the prompt forbade touching
the harness: 181 copies of check() alone across 202 files. Put them in
common.py as Report (PASS/FAIL/INFO lines, spec-rules.txt, RESULT) and
plain helpers, and give wire()/iq_answer() a since= taken from mark(), so
a reused iq id from an earlier daemon session in the same run no longer
matches. XEP-0184.py, the model scenario, now uses them; the other
scenarios keep their copies until touched.

Assisted-by: Claude:fable-5-1
112 findings: unused imports and locals, f-strings without placeholders,
`global ok` in functions that only read it. None changes what a scenario
checks; the dropped assignments were never read. nix run .#check now
lints tools/ai/scenarios too, so the next batch stays clean.

Assisted-by: Claude:fable-5-1
tacky.py (framing, request/notify dispatch, event listeners, the XML
console log, the fresh-run store, pending requests on exit) and
common.py (since-scoped iq_answer, Report, until, tcl_list) had no
tests; nix run .#check ran only tools/test_interop.py. fake_tackyd.py
speaks the daemon's framed JSON enough to sign in and answer a few
methods, and TACKYD may now be a command line so the tests can point
Tacky at it without a shebang.

Assisted-by: Claude:fable-5-1
A fail run looked the same whether libtacky, the server, the peer or
the scenario was at fault, so AI-RESULTS.md counted server bugs as
libtacky's. finish takes --cause libtacky|server|peer|harness (libtacky
when not given), check validates it, and the report shows it beside
each fail and counts fails by cause. The ten existing fails whose bug
text names the server first are tagged server; a grep of the rest found
no harness- or peer-caused verdicts.

finish also drops everything in the run's .state but session.json: a
fresh run's client profiles and tackyd store keep the account's
password and FAST token, and stayed on disk indefinitely.

Assisted-by: Claude:fable-5-1
Assisted-by: Claude:opus-5-5
runs: drop the XEP-0384 ejabberd run 184109
Some checks failed
/ check (pull_request) Failing after 9m57s
Details
944da57837
 
ace4de3 added only its run.toml: the logs it lists were never
committed (their LFS objects are not on the server), and it is a pass
whose logs lack the OMEMO markers, so interop check fails on it. It
was left out on purpose when the S4 runs were committed (8a1fcea).

Assisted-by: Claude:opus-5-5
runs: drop five superseded runs committed without their logs
All checks were successful
/ check (pull_request) Successful in 9m37s
Details
3ce58f2117
 
ace4de3 also committed the run.toml of five runs that were left out
on purpose (superseded or incomplete): 093611 drafts, 181318 XEP-0483,
191208 and 191231 XEP-0060, 205307 XEP-0472. Their logs were never
committed, so interop check fails on all of them (check #82).

Assisted-by: Claude:opus-5-5
asxpi merged commit d9a9d9ccc2 into main 2026-11-08 02:07:33 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
another-im/interop!7
No description provided.