Evals & Deep Dives
Cross-implementation conformance
A test that only one library can run proves little about the spec. The scenarios below assert behavior derivable from cited ECMA-376 clauses — not from any library's internals — and run unchanged against every registered implementation, in the tradition of wpt.fyi for the web platform.
The suite lives in open-agreements/docx-platform-tests: each scenario is an XML input plus assertions, each implementation participates through a small adapter CLI, and an adapter may decline an operation it cannot perform (reported as Unsupported, the analog of wpt's NOTRUN). A gap in the matrix is information about the library, not a failure of the suite.
| Scenario | safe-docx v0.15.0+git.459051c072da | python-docx v1.2.0 | libreoffice vLibreOffice 24.2.7.2 420(Build:2) | openxml-sdk vDocumentFormat.OpenXml 3.5.1 | dolanmiu-docx v9.7.1 | superdoc-sdk v1.16.1 | docx-rs vdocx-rs 0.4.20 |
|---|---|---|---|---|---|---|---|
acceptDeletedParagraphMarkMergesParagraphs Accepting a deleted paragraph mark merges the paragraph with the following paragraph ECMA-376 edition 5, Part 1 § 17.13.5.15 (del (Deleted Paragraph)) | Pass | Unsupported python-docx has no tracked-changes (revision) API | Pass | Unsupported the Open XML SDK is a typed DOM over the package with no accept/reject-revisions API; implementing one would be an adapter-side algorithm | Unsupported dolanmiu/docx can generate revision markup but exposes no API to accept or reject existing tracked changes | Unsupported SuperDoc SDK does not expose paragraph-mark tracked changes through trackChanges.list | Unsupported docx-rs can generate revision markup but exposes no API to accept or reject existing tracked changes |
acceptDeletionsRemovesDelContent Accepting deletions removes w:del wrappers and their content ECMA-376 edition 5, Part 1 § 17.13.5.14 (del (Deleted Run Content)) | Pass | Unsupported python-docx has no tracked-changes (revision) API | pass-divergent | Unsupported the Open XML SDK is a typed DOM over the package with no accept/reject-revisions API; implementing one would be an adapter-side algorithm | Unsupported dolanmiu/docx can generate revision markup but exposes no API to accept or reject existing tracked changes | pass-divergent | Unsupported docx-rs can generate revision markup but exposes no API to accept or reject existing tracked changes |
acceptFormattingChangeKeepsNewRunProperties Accepting formatting changes keeps current run properties ECMA-376 edition 5, Part 1 § 17.13.5.31 (rPrChange (Revision Information for Run Properties)) | Pass | Unsupported python-docx has no tracked-changes (revision) API | pass-divergent | Unsupported the Open XML SDK is a typed DOM over the package with no accept/reject-revisions API; implementing one would be an adapter-side algorithm | Unsupported dolanmiu/docx can generate revision markup but exposes no API to accept or reject existing tracked changes | pass-divergent | Unsupported docx-rs can generate revision markup but exposes no API to accept or reject existing tracked changes |
acceptInsertionsUnwrapsInsWrappers Accepting insertions unwraps w:ins and keeps run content ECMA-376 edition 5, Part 1 § 17.13.5.18 (ins (Inserted Run Content)) | Pass | Unsupported python-docx has no tracked-changes (revision) API | pass-divergent | Unsupported the Open XML SDK is a typed DOM over the package with no accept/reject-revisions API; implementing one would be an adapter-side algorithm | Unsupported dolanmiu/docx can generate revision markup but exposes no API to accept or reject existing tracked changes | pass-divergent | Unsupported docx-rs can generate revision markup but exposes no API to accept or reject existing tracked changes |
acceptNestedDeletionInsideInsertion Accepting nested deletion inside insertion keeps surviving inserted text ECMA-376 edition 5, Part 1 § 17.13.5.18 (ins (Inserted Run Content)) | Pass | Unsupported python-docx has no tracked-changes (revision) API | Pass | Unsupported the Open XML SDK is a typed DOM over the package with no accept/reject-revisions API; implementing one would be an adapter-side algorithm | Unsupported dolanmiu/docx can generate revision markup but exposes no API to accept or reject existing tracked changes | Pass | Unsupported docx-rs can generate revision markup but exposes no API to accept or reject existing tracked changes |
rejectDeletionsRestoresDelContent Rejecting deletions restores w:delText content to body text ECMA-376 edition 5, Part 1 § 17.13.5.14 (del (Deleted Run Content)) | Pass | Unsupported python-docx has no tracked-changes (revision) API | pass-divergent | Unsupported the Open XML SDK is a typed DOM over the package with no accept/reject-revisions API; implementing one would be an adapter-side algorithm | Unsupported dolanmiu/docx can generate revision markup but exposes no API to accept or reject existing tracked changes | pass-divergent | Unsupported docx-rs can generate revision markup but exposes no API to accept or reject existing tracked changes |
rejectFormattingChangeRestoresPriorRunProperties Rejecting formatting changes restores prior run properties ECMA-376 edition 5, Part 1 § 17.13.5.31 (rPrChange (Revision Information for Run Properties)) | Pass | Unsupported python-docx has no tracked-changes (revision) API | pass-divergent | Unsupported the Open XML SDK is a typed DOM over the package with no accept/reject-revisions API; implementing one would be an adapter-side algorithm | Unsupported dolanmiu/docx can generate revision markup but exposes no API to accept or reject existing tracked changes | pass-divergent | Unsupported docx-rs can generate revision markup but exposes no API to accept or reject existing tracked changes |
rejectInsertedParagraphMarkMergesParagraphs Rejecting an inserted paragraph mark removes the mark and merges the paragraph with the following paragraph ECMA-376 edition 5, Part 1 § 17.13.5.20 (ins (Inserted Paragraph)) | Pass | Unsupported python-docx has no tracked-changes (revision) API | Pass | Unsupported the Open XML SDK is a typed DOM over the package with no accept/reject-revisions API; implementing one would be an adapter-side algorithm | Unsupported dolanmiu/docx can generate revision markup but exposes no API to accept or reject existing tracked changes | Unsupported SuperDoc SDK does not expose paragraph-mark tracked changes through trackChanges.list | Unsupported docx-rs can generate revision markup but exposes no API to accept or reject existing tracked changes |
rejectInsertionsRemovesInsContent Rejecting insertions removes w:ins wrappers and their run content ECMA-376 edition 5, Part 1 § 17.13.5.18 (ins (Inserted Run Content)) | Pass | Unsupported python-docx has no tracked-changes (revision) API | pass-divergent | Unsupported the Open XML SDK is a typed DOM over the package with no accept/reject-revisions API; implementing one would be an adapter-side algorithm | Unsupported dolanmiu/docx can generate revision markup but exposes no API to accept or reject existing tracked changes | pass-divergent | Unsupported docx-rs can generate revision markup but exposes no API to accept or reject existing tracked changes |
rejectNestedDeletionInsideInsertion Rejecting nested deletion inside insertion removes the whole insertion ECMA-376 edition 5, Part 1 § 17.13.5.18 (ins (Inserted Run Content)) | Pass | Unsupported python-docx has no tracked-changes (revision) API | Pass | Unsupported the Open XML SDK is a typed DOM over the package with no accept/reject-revisions API; implementing one would be an adapter-side algorithm | Unsupported dolanmiu/docx can generate revision markup but exposes no API to accept or reject existing tracked changes | Pass | Unsupported docx-rs can generate revision markup but exposes no API to accept or reject existing tracked changes |
replaceFirstOccurrencePreservesOffsets Replacing the first occurrence places the new text at the matched offset ECMA-376 edition 5, Part 1 § 17.3.3.31 (t (Text)) | Pass | Pass | Pass | Pass | Unsupported dolanmiu/docx patchDocument targets explicit patch placeholders; protocol requires arbitrary paragraph-local literal search, which would be an adapter-side algorithm | Pass | Unsupported docx-rs exposes no public existing-document text replacement API; implementing this would require adapter-side XML surgery |
replaceTextAcrossRunBoundary Replacing text spanning a run boundary places the replacement at the matched offset ECMA-376 edition 5, Part 1 § 17.3.3.31 (t (Text)) | Pass | Unsupported match spans run boundaries; the python-docx adapter only performs intra-run replacement (glue, not algorithms) | Pass | Unsupported first occurrence spans w:t boundaries; the openxml-sdk adapter only performs intra-w:t replacement (glue, not algorithms) | Unsupported dolanmiu/docx patchDocument targets explicit patch placeholders; protocol requires arbitrary paragraph-local literal search, which would be an adapter-side algorithm | Pass | Unsupported docx-rs exposes no public existing-document text replacement API; implementing this would require adapter-side XML surgery |
Results from the suite run of 2026-07-06 (DSL 1.2, adapter protocol v1). The snapshot is refreshed by a maintainer running npm run refresh:conformance-explorer; the build never fetches at CI time, so pages stay deterministic.
Reading the matrix
- Pass — the adapter's output satisfied every assertion in the scenario.
- Unsupported — the adapter declined the operation with a stated reason. The suite's contribution rules forbid implementing missing library capabilities inside an adapter, so this measures the library, not the adapter author.
- Fail / Error — the output violated an assertion, or the adapter crashed; per-assertion detail is in the suite's published
results/latest.json.
Per-implementation comparisons
Each compared library gets its own page: the matrix slice for that pair, plus what every cell means for a reader's document workflow.
Where to dig deeper
- Safe DOCX conformance explorer — browse the same pinned result set from ECMA section, schema declaration, or capability/operation identities.
- docx-platform-tests — scenario DSL, adapter protocol, and how to register another implementation (Apache-2.0).
- Suite-owned results matrix — the neutral view published from the suite's own CI, with the raw
results/latest.json. - safe-docx evals — the per-primitive scenario pages this matrix complements.
- safe-docx vs python-docx — where each tool sits in the stack.