message, caption: move mention offsets past the characters XML cannot carry #25

Closed

asxpi wants to merge 1 commit from fix/C3-mention-offsets-after-strip into develop

asxpi commented

2026-11-07 21:57:39 +00:00

Member

What and why

send, send_file, send_album and set_caption strip what XML 1.0 cannot
carry (XmlText), then check the caller's mentions against the cleaned text
without moving them. A mention after a stripped character went out pointing
past its words (XEP-0426 §2). The new XmlTextMentions moves each begin/end
back by the characters dropped before it. send applies it before the Matrix
route and the main path, and CaptionMentions before its leading-space shift.

Tests

test_message_mentions.tcl: mentions-send-moves-past-the-characters-xml-cannot-carry
test_caption.tcl: caption-mentions-move-past-the-characters-xml-cannot-carry

Both fail on develop. With the fix, test_message_mentions passes 40/40,
test_caption 18/18 and test_message 618/618.

Verified

Rebased on develop 56d29734; test_message_mentions, test_caption and test_message pass (679 total, 0 failed).

On Prosody 13, slot 1, running XEP-0426.py unchanged:

develop as of v1.2.0, R9 fails (begin='30' end='34' covers ee e): https://dev.another.im/another-im/interop/src/branch/ai-runs-2026-11-07/runs/20261107-215610-XEP-0426-slixmpp-prosody
this branch, R1-R9 pass (begin='28' end='32' is @Bee): https://dev.another.im/another-im/interop/src/branch/ai-runs-2026-11-07/runs/20261107-215616-XEP-0426-slixmpp-prosody

Also reproduced by hand in anotherim-js 0.8.0-beta5 in an OMEMO chat (see #24).

Fixes #24

## What and why `send`, `send_file`, `send_album` and `set_caption` strip what XML 1.0 cannot carry (`XmlText`), then check the caller's mentions against the cleaned text without moving them. A mention after a stripped character went out pointing past its words (XEP-0426 §2). The new `XmlTextMentions` moves each `begin`/`end` back by the characters dropped before it. `send` applies it before the Matrix route and the main path, and `CaptionMentions` before its leading-space shift. ## Tests - `test_message_mentions.tcl`: `mentions-send-moves-past-the-characters-xml-cannot-carry` - `test_caption.tcl`: `caption-mentions-move-past-the-characters-xml-cannot-carry` Both fail on develop. With the fix, test_message_mentions passes 40/40, test_caption 18/18 and test_message 618/618. ## Verified Rebased on develop 56d29734; test_message_mentions, test_caption and test_message pass (679 total, 0 failed). On Prosody 13, slot 1, running `XEP-0426.py` unchanged: - develop as of v1.2.0, R9 fails (`begin='30' end='34'` covers `ee e`): https://dev.another.im/another-im/interop/src/branch/ai-runs-2026-11-07/runs/20261107-215610-XEP-0426-slixmpp-prosody - this branch, R1-R9 pass (`begin='28' end='32'` is `@Bee`): https://dev.another.im/another-im/interop/src/branch/ai-runs-2026-11-07/runs/20261107-215616-XEP-0426-slixmpp-prosody Also reproduced by hand in anotherim-js 0.8.0-beta5 in an OMEMO chat (see #24). Fixes #24

asxpi added 1 commit

2026-11-07 21:57:39 +00:00

message: move mention offsets past the characters XmlText drops d4f25ef582

send, send_file, send_album and set_caption strip what XML 1.0 cannot
carry (C0 controls, lone surrogates, U+FFFE, U+FFFF) with XmlText, then
check the caller's mentions against the cleaned text without moving
them. A body with a U+0001 and a U+FFFE before a mention of "@Bee" went
out with a <reference/> two characters too far, marking "ee e" (interop
runs 20261107-131221, -132640, -153155, XEP-0426).

XmlTextMentions moves each mention's begin and end back by the
characters dropped before it, so the reference marks the words it
marked in the text the host gave (XEP-0426 §2 counts in the body
sent). send applies it before the Matrix route too; CaptionMentions
applies it before its leading-space shift.

Assisted-by: Claude:opus-5-5

asxpi referenced this pull request from another-im/interop

2026-11-07 21:58:03 +00:00

Verify libtacky bugs from the AI interop runs (v1.2.0-rc2, 71 found) #1

asxpi commented

2026-11-08 00:19:36 +00:00

Author

Member

Fixed upstream in v1.2.1 by 67d627cf ("message: mentions move past the characters XML cannot carry, so a mention still marks the words it marked (interop C3)"). It has the same code and tests as this branch (git patch-id), so this is closed as a duplicate.

Found by hand since filing, in anotherim-js develop with libtacky develop 56d29734:

The bug also hits OMEMO chats.
Second symptom: when the mention is the last thing in the message, the stripped characters push its end past the body, message send refuses the whole send, and the message silently isn't sent.
With the fix, the same message goes out with begin='3' end='21' (exactly @Nick).

Fixed upstream in v1.2.1 by 67d627cf ("message: mentions move past the characters XML cannot carry, so a mention still marks the words it marked (interop C3)"). It has the same code and tests as this branch (`git patch-id`), so this is closed as a duplicate. Found by hand since filing, in anotherim-js `develop` with libtacky `develop` 56d29734: - The bug also hits **OMEMO chats**. - **Second symptom:** when the mention is the last thing in the message, the stripped characters push its end past the body, `message send` refuses the whole send, and the message silently isn't sent. - With the fix, the same message goes out with `begin='3' end='21'` (exactly `@Nick`).

asxpi closed this pull request

2026-11-08 00:19:36 +00:00