Batch 7 harness: headless calls, libtacky against libtacky, also/scenario, shared fake server

asxpi commented

2026-11-10 22:08:05 +00:00

Member

The harness batch 7 (#29) needs before its agents start: they write scenarios, not harness code. Base main ba63ec0e; to be merged after batch 6 (#25).

H1, calls headless: Tacky(..., media_backend="remote") and common.Page, which plays the page: canned SDP (sdp(): audio, audio+video, data channel), CANDIDATES, media state, and a data pipe between two pages. Fixtures from libtacky's test_calls_remote.tcl.
H2, libtacky against libtacky: each Tacky writes <stem>.log, <stem>-events.log and <stem>-debug.log. interop start --under-test tackyd --client tackyd --peer-as JID makes a second tackyd the peer (peer-tackyd.log). check, normalize and the matrix ("a second libtacky" column) know it.
H3, a second device: Tacky(state_dir=…) gives the same account a second store.
H4, also = [...]: finish --also SPEC, validated by check, counted in FEATURES.md, matrix.md, AI-RESULTS.md and gaps.
scenario = "0166-2": finish --scenario. Each scenario of a feature keeps its own newest run, and a cell shows a fail while any of them fails, so a newer pass of one scenario no longer hides another's fail.
H5, shared fake server: tools/ai/fake.py, now imported by XEP-0215.py and RFC-5929.py (fake parts pass unchanged).
#29's decisions:
- ai_ok = true in all 145 features/*.toml
- the 11 rows without a scenario point at #29's scenario ids
- XEP-0453 gets an na reason on ejabberd and anotherim-server
- PROMPT.md describes the new options
Secrets: tackyd's events logs now redact TURN credential values.

Checks: pyflakes clean, 67 tests pass, interop check 1495 runs OK, report --check clean.

Smoke runs on prosody.asxp.io (ai1a/ai1b, outside runs/): a video call between two tackyds up to <Active> with equal SAS; 50 kB over the data channel, kept verified; the XEP-0215 and RFC-5929 fake parts. The stray OMEMO device a probe left on ai1b was wiped afterwards.

The harness work was written by Claude Fable 5.2 and finished and checked by Claude Opus 5.5.

🤖 Generated with Claude Code

The harness batch 7 (#29) needs before its agents start: they write scenarios, not harness code. Base main `ba63ec0e`; to be merged after batch 6 (#25). - **H1, calls headless:** `Tacky(..., media_backend="remote")` and `common.Page`, which plays the page: canned SDP (`sdp()`: audio, audio+video, data channel), `CANDIDATES`, media state, and a data pipe between two pages. Fixtures from libtacky's `test_calls_remote.tcl`. - **H2, libtacky against libtacky:** each `Tacky` writes `<stem>.log`, `<stem>-events.log` and `<stem>-debug.log`. `interop start --under-test tackyd --client tackyd --peer-as JID` makes a second tackyd the peer (`peer-tackyd.log`). `check`, `normalize` and the matrix ("a second libtacky" column) know it. - **H3, a second device:** `Tacky(state_dir=…)` gives the same account a second store. - **H4, `also = [...]`:** `finish --also SPEC`, validated by `check`, counted in FEATURES.md, matrix.md, AI-RESULTS.md and `gaps`. - **`scenario = "0166-2"`:** `finish --scenario`. Each scenario of a feature keeps its own newest run, and a cell shows a fail while any of them fails, so a newer pass of one scenario no longer hides another's fail. - **H5, shared fake server:** `tools/ai/fake.py`, now imported by XEP-0215.py and RFC-5929.py (fake parts pass unchanged). - **#29's decisions:** - `ai_ok = true` in all 145 `features/*.toml` - the 11 rows without a scenario point at #29's scenario ids - XEP-0453 gets an `na` reason on ejabberd and anotherim-server - PROMPT.md describes the new options - **Secrets:** tackyd's events logs now redact TURN `credential` values. **Checks:** pyflakes clean, 67 tests pass, `interop check` 1495 runs OK, `report --check` clean. **Smoke runs** on prosody.asxp.io (ai1a/ai1b, outside runs/): a video call between two tackyds up to `<Active>` with equal SAS; 50 kB over the data channel, kept verified; the XEP-0215 and RFC-5929 fake parts. The stray OMEMO device a probe left on ai1b was wiped afterwards. The harness work was written by Claude Fable 5.2 and finished and checked by Claude Opus 5.5. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

asxpi added 5 commits

2026-11-10 22:08:05 +00:00

ai: run several tackyds in one run, and play the page of its media backend 09ffa92ab4

Batch 7 (#29) needs three things the harness could not do. Jingle calls
never started headless: the interop build of tackyd-json has no native
media backend, so `--media-backend auto` opens none. A second libtacky
could not be the peer, because every instance wrote client-events.log
and client-debug.log. And a second device of the account under test had
no store of its own.

Tacky takes media_backend (the `remote` backend opens in the same binary
and leaves the peer connection to the host), state_dir, and names its
events and debug logs after log_name. common.py gains Page, the page
behind a tackyd on the remote backend: it answers calls <MediaOpen> with
canned SDP (sdp(): audio, audio+video, a data channel), trickles
candidates, reports the connection state and carries a transfer's bytes
between two pages; befriend_tacky makes two tackyds contacts.

The audio section of sdp() is libtacky's own CHROME_OFFER fixture
(tests/client/test_calls_remote.tcl). Checked against prosody.asxp.io
with two tackyds: a video call up to <Active> with equal SAS on both
sides, and a 50 kB file over the data channel, kept verified.

Assisted-by: Claude:fable-5-2

interop: count agents' runs for every feature, and for the specs they exercise 836960f42b

Decided on #29 (2026-11-11): an agent's run may count for every row of
features.md, so ai_ok is true in all 208 feature files and for rows
imported later. FEATURES.md no longer shows "no run" for the 98 rows that
had low-confidence runs only. VERIFIED is still a person's pass, and a
feature with a display or media part is still run with confidence low.

Batch 7's scenarios also need the bookkeeping to tell more apart:

- `also = [...]` in run.toml (finish --also): a call run exercises
  XEP-0293, 0294, 0338 and 0339 at once, and counts for each in
  FEATURES.md, matrix.md, AI-RESULTS.md and gaps.
- `scenario = "0166-2"` (finish --scenario): a feature now has several
  scenarios against the same peer, and the newest run used to hide the
  others. The report keeps the newest run of each and shows a fail while
  any of them fails. A run without a scenario still replaces everything
  before it, so the reports of existing runs do not change.
- tackyd against a second tackyd: `start --client tackyd --under-test
  tackyd --peer-as JID` makes a run with a peer (peer-tackyd.log wanted
  by check, read by normalize, its own column in matrix.md).

A page's ICE servers come with TURN credentials (calls <MediaOpen> in a
tackyd's events log): redact them.

Assisted-by: Claude:fable-5-2

ai: share one fake server between scenarios cc53bfa54a

XEP-0215.py and RFC-5929.py each carried their own local TLS server
(direct TLS on 127.0.0.1, a pinned certificate, PLAIN, scripted
answers). Batch 7 needs the same for XEP-0048's private XML fallback,
XEP-0068's forms, XEP-0488's tokens and urn:tacky:storage:0, none of
which a test server offers. Move it to tools/ai/fake.py and import it.

Both scenarios' fake parts pass unchanged in the pinned ai shell.

Assisted-by: Claude:fable-5-2
Assisted-by: Claude:opus-5-5

features: point the rows without a scenario at interop#29 e1ebd4a891

The 11 rows batch 7 writes new scenarios for (XEP-0068, 0176, 0260,
0261, 0293, 0294, 0320, 0338, 0339, 0343, 0453) still had the import's
TODO steps. Name the scenario ids of #29's feature list, which holds
the checks and hostile cases. XEP-0453 is a document, not a wire
feature: its one offline check is recorded on Prosody, the other two
servers get an na reason.

Assisted-by: Claude:opus-5-5

ai: brief agents on the batch 7 harness

/ check (pull_request) Successful in 18m6s

Details

f301db7c7c

Every feature now has ai_ok = true, so confidence is the orchestrator's
call per feature instead of following ai_ok. Name the new options a
run can use: --scenario, --also, a second tackyd as peer or device,
the remote media backend with common.Page, and the shared fake server.

Assisted-by: Claude:opus-5-5

/ check (pull_request) Successful in 18m6s

Details

This pull request can be merged automatically.

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin ai/batch-7-harness:ai/batch-7-harness

git switch ai/batch-7-harness

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main

git merge --no-ff ai/batch-7-harness

git switch ai/batch-7-harness

git rebase main

git switch main

git merge --ff-only ai/batch-7-harness

git switch ai/batch-7-harness

git rebase main

git switch main

git merge --no-ff ai/batch-7-harness

git switch main

git merge --squash ai/batch-7-harness

git switch main

git merge --ff-only ai/batch-7-harness

git switch main

git merge ai/batch-7-harness

git push origin main

Rows
Columns

Batch 7 harness: headless calls, libtacky against libtacky, also/scenario, shared fake server #30

Checkout

Merge