AnotherIM interop

Hand tests of libtacky's features, through the AnotherIM clients, against every server and third-party client we care about. A person does each test; the logs of every party are the proof.

• FEATURES.md: libtacky's feature list with each row's result and links to its proof.
• matrix.md: feature × client × server, latest run per cell (clients with a run only; the headless libtacky × slixmpp runs of #1 included, as 🤖).

Both are generated from runs/; never edit them by hand.

For testers with an AI agent

Before you start, from asxpi (privately, never through git or chat logs): ~/.config/interop/accounts.toml (the test accounts), and a test build of AnotherIM with the XML log (or use a release and save its XML console by hand). Add your SSH public key as infra/keys/testers/<you>.pub in a PR, so runs can fetch the server's log. Then paste this to your agent (Claude Code or similar), in an empty folder:

You run the bookkeeping of hand interop tests for libtacky; I only use the two
app windows and tell you what I see. Repo: https://dev.another.im/another-im/interop
(read its README.md and infra/README.md first; follow them over anything here).

Setup, once:
1. git clone https://dev.another.im/another-im/interop && cd interop.
   For SSH to dev.another.im (port 3022) force IPv4 (ssh -4 / GIT_SSH_COMMAND="ssh -4").
2. Everything runs inside `nix develop` (clients, git-lfs, tools); git lfs install --local.
3. Check ~/.config/interop/accounts.toml exists (mode 600). Never print it whole,
   never commit it, never paste passwords anywhere but to me in this terminal.
4. Ask me for: the path of the AnotherIM AppImage (export
   ANOTHERIM="appimage-run <path>"), the libtacky version it is built on
   (INTEROP_LIBTACKY, e.g. v1.2.0-rc2), and which server to use (INTEROP_SERVER:
   prosody, ejabberd or anotherim-server). Check the server answers on 5222
   before starting; if not, tell me (servers are started by maintainers).

Per test (one feature, one peer client, one server):
5. Pick the feature with me: `interop todo --server <server>` lists untested
   cells, P0 first.
   If features/<SPEC>.toml still has TODO steps, write numbered steps, the
   expected result and log markers into it first and show them to me.
6. `interop start <SPEC> --client <peer>`: it opens AnotherIM (alice@server)
   and the peer (bob@server) with logging; give me the accounts it prints and
   the steps, then wait.
7. While I test, read the run's logs (client.log is JSON lines of AnotherIM's
   stanzas; peer-<client>.log; the server's comes at finish) and check the wire
   matches the steps: the right elements, ids, errors (`interop normalize <run>`
   puts every party's stanzas on one timeline). Ask me what each side showed.
   Do not decide pass/fail for me: I confirm what I saw. Only when I say this is
   an AI-only run, on a feature with ai_ok = true, finish it with
   --verified-by ai; it shows as 🤖 and never makes a feature VERIFIED.
8. `interop finish <run> --verdict pass|fail|na --notes "<what both sides showed,
   anything odd>"` (a fail needs --issue). Then check: no password from
   accounts.toml appears in any file of the run (also base64 SASL forms), and
   `interop check` passes.
9. `interop report`, then commit only that run folder plus FEATURES.md and
   matrix.md on a branch (git-lfs must be on PATH), push to your fork and open a
   PR. Commit messages: short imperative subject, body saying why.
10. Anything odd beyond the feature (a dropped message, a server error): tell me
   with the log lines; file an issue only when I say so.

At the end of the day, remind me to stop the servers we used (see "Servers:
on in the morning, off at night" in the README); I do it, or ask a maintainer.

Never: touch infra/ or nix/, start or stop servers yourself, commit secrets or
profiles (~/.local/share/interop), push to main, or edit a finished run's logs.

Servers: on in the morning, off at night

The test servers (prosody, ejabberd, anotherim) cost money while they run and keep their state (accounts, archives, certificates) when stopped: down backs it up and destroys the server, up creates it again at the same address and restores it. So stop the ones you used at the end of the day, and start them when you begin, unless someone else is still testing on them (ask in the team chat).

With write access to this repo, from Forgejo: Actions → servers → Run workflow → server, down (or up). From a terminal:

fj -H dev.another.im actions dispatch servers main -r another-im/interop -I server=prosody -I action=down
fj -H dev.another.im actions dispatch servers main -r another-im/interop -I server=prosody -I action=up

up takes about 6 minutes; the server is ready when the workflow is green. Which are running: matrix.md does not say; Actions → servers shows the last run per server, or ask a maintainer. Without write access, ask one.

Layout

clients.toml     clients on the matrix: under-test (AnotherIM builds) and peers, how each logs stanzas
servers.toml     Prosody, ejabberd, anotherim-server: versions, domains, how to fetch their logs
features/        one file per features.md row: kind, hand steps, expected result, log markers, n/a cells
runs/<id>/       one test = one proof bundle (logs in Git LFS)
  run.toml         who, when, which versions, verdict, sha256 of every file
  client.log       XML console of the client under test
  peer-<id>.log    stanza log of the peer client
  server.log       server log for the test's time window
tools/interop.py import | session | check | report
vms/             subflake: GUI clients in NixOS VMs, driven and screenshotted (vms/README.md)

kind = "server" features (sign-in, SM, MAM, upload…) are checked on each under-test build against each server. kind = "peer" features are checked against each peer client, on each server. An AnotherIM build may be its own peer.

Doing a test

Once per machine:

nix develop                 # nix develop .#android adds the Android SDK and emulator (several GB)
export ANOTHERIM="appimage-run $HOME/Apps/anotherim/another-im-tauri-0.8.0-rc2-x86_64.AppImage"
export INTEROP_LIBTACKY=v1.2.0-rc2          # the app version is read from the file name
export INTEROP_SERVER=prosody                # optional default

Other builds under test: the Flatpak (--under-test anotherim-flatpak, after flatpak install --user --bundle of the release's .flatpak; not published yet, its build fails on the builder) and macOS (--under-test anotherim-macos, ANOTHERIM=/Applications/another.im.app/Contents/MacOS/anotherim). Test builds with the XML log are this repo's releases anotherim-<tag>-xmllog, built from every anotherim-js release (build-anotherim workflow): with anotherim-js's build option XML_LOG=1 from v0.8.0-rc7 (its own releases are built without it), with patches/anotherim-js before that.

The test accounts (alice, bob, carol on each server) are in ~/.config/interop/accounts.toml, never in git. Then, per test:

interop session XEP-0308 --client gajim

The client under test signs in as alice@<server>, the peer as bob@<server> (printed with their passwords). Each client keeps one profile per account under ~/.local/share/interop/profiles, so you sign in once per client and server; features whose test is the sign-in itself (fresh = true, or --fresh) start from empty profiles. AnotherIM writes its XML console to the run's client.log by itself (ANOTHERIM_XML_LOG, builds with it); the peer's log and the server's are collected too. You do the steps, press Enter, give a verdict; secrets are redacted, every file hashed into run.toml, and the run's private state (fresh profiles, stores) cleared. Then:

interop check runs/<id> && interop report
git add runs/<id> FEATURES.md matrix.md && git commit -m "XEP-0308: gajim on prosody, pass"

Open a PR. CI rejects bundles with missing logs, changed files, unredacted secrets, or (once a feature's markers are reviewed) a pass whose logs lack them. A fail needs an issue link and says what it is down to (--cause: libtacky, the server, the peer client or the harness; AI-RESULTS.md counts them).

Priorities and AI-verified runs

Each feature has a priority (P0: sign-in, messaging, encryption, rooms, files, calls; P2: rare or experimental) and interop todo lists untested cells in that order. Features whose result the stanza logs alone prove have ai_ok = true; an agent may run those without a person (interop finish --verified-by ai). Such runs show as 🤖 in FEATURES.md and matrix.md and never count toward VERIFIED, which needs a person's pass on every cell; interop check refuses an AI run on a feature without ai_ok unless it has confidence = "low". An agent driving GUI clients in VMs (vms/) adds driven = "vm" to run.toml: AI-RESULTS.md tags those 🖥, and matrix.md shows such a run in its client's cell as 🤖✅/🤖❌ until a person's run exists for that cell. Name screenshots shot-<n>-<client>-<what>.png (aim = anotherim-js).

Rules

• One test per run. Fresh accounts, never personal ones.
• A pass means you saw it work on both ends, not only that the stanza went out.
• n/a goes in the feature's [na] table with a reason (e.g. MUC/Sub on Prosody), not as a run.
• A row of FEATURES.md is VERIFIED when every cell it applies to passed. libtacky's maintainer decides what becomes 🟢 DONE upstream.

Not done yet

• Shared servers on the runner host: domains, invites and log_cmd in servers.toml.
• Steps, expected results and reviewed markers for each feature (all TODO).
• Checking the stanza logging of the clients marked log_verified = false.
• A Claude review of each submitted bundle (do the logs back the verdict?).
• The 200 agent scenarios predate common.Report and the shared helpers and carry their own copies; new ones use common.py (the prompt says so), old ones are converted when touched.