John O'Keefe
b10bf3e8c7
The progress documentation described GET/POST /api/progress/:id — endpoints that do not exist in the router — while the endpoint every client actually uses (GET/PUT/DELETE /api/media-items/:id/progress) had no field-level docs at all. New: - progress/position-contract.md: the canonical position model — server as position authority, the three-tier submission (percentage / context_text / epubcfi), ingest verification and healing, the restore handles, the OPF spine numbering hazard (canonical CFI spine steps include linear="no" items; clients resolve documents by anchor_href and land by css_selector + char_offset, never by spine step), the offset currencies (UTF-16 at the wire, runes internal), and context_text rules. - progress/get_media_progress.md and update_media_progress.md: the real endpoints with full field tables, conditionality of the restore handles, the first-page anti-clobber guard, and healed-response semantics. - progress/delete_media_progress.md: the real DELETE route. - koreader/sync_progress.md and koreader/get_metadata.md rewritten to the actual payloads: the plugin sends a single-book array whose "epubcfi" field is a CRE xpointer; the metadata response navigates via koreader_xpointer (canonical CFI converted back to CRE), with page as the canonical locator for fixed-layout books. Removed: the five files documenting the nonexistent /api/progress/:id GET/POST/DELETE endpoints. Kept get_progress_history.md (that route exists). The legacy developer/api-reference.md and the indexed api/api-reference.md progress sections now match the wire and link the new docs; the duplicate "Universal Progress" section points at Reading Progress.
136 lines
6.9 KiB
Markdown
136 lines
6.9 KiB
Markdown
# Position Contract
|
|
|
|
How reading positions are represented, submitted, verified, and restored
|
|
across clients. This is the contract every client speaks — readium-based
|
|
apps, web (foliate), KOReader, and Kobo devices alike.
|
|
|
|
**The server is the position authority.** Every submission is
|
|
independently verified against the book itself before it is stored; a
|
|
client whose locator math is wrong cannot poison the stored position.
|
|
In exchange, the server hands back handles clients can apply directly,
|
|
so they never have to parse or trust CFIs themselves.
|
|
|
|
## The three-tier submission
|
|
|
|
Clients submit what their renderer can reliably observe. All three tiers
|
|
are accepted on `PUT /api/media-items/:id/progress` (and on the KOReader
|
|
device endpoints):
|
|
|
|
| Tier | Field | Currency | Notes |
|
|
| ------ | -------------- | ----------------------------------------------------- | ----- |
|
|
| 1 | `percentage` | Float 0..1 of the whole book | Universal. The only field every client can supply. |
|
|
| 2 | `context_text` | Up to 100 chars, whitespace-normalized, starting at the anchor | The verification anchor. The server extracts the text at the submitted structural anchor and compares. |
|
|
| 3 | `epubcfi` | Standard wrapped CFI, terminals in UTF-16 code units | Optional structural anchor. KOReader submits a CRE xpointer and the server converts it. |
|
|
|
|
Tier 2 is what makes the system self-correcting: percentages alone
|
|
cannot distinguish "the reader's locator is right" from "the reader
|
|
silently reported wherever it is currently scrolled".
|
|
|
|
## Verification and healing (ingest side)
|
|
|
|
On every progress save for a convertible reflowable book that carries a
|
|
`context_text`, the server:
|
|
|
|
1. Parses the submitted `epubcfi` (if any) and resolves it against the
|
|
book's own XHTML.
|
|
2. Extracts the text at the resolved anchor and cross-checks it with the
|
|
submitted `context_text`.
|
|
3. On a mismatch — or an unresolvable anchor, or no anchor at all —
|
|
heals the position by text search, using the submitted `percentage`
|
|
to disambiguate repeated phrases. The healed CFI and a recomputed
|
|
percentage are stored in place of the submitted ones.
|
|
4. Refreshes the book-wide `character_offset` column from the verified
|
|
anchor on every verified save, so it never goes stale behind the
|
|
anchor.
|
|
|
|
If verification fails outright (book file unreadable, context not found
|
|
and percentage cannot disambiguate), the error is logged and the
|
|
submission is stored as-is — verification never rejects a save, it only
|
|
corrects.
|
|
|
|
## The restore handles
|
|
|
|
`GET /api/media-items/:id/progress` serves, alongside the raw stored
|
|
fields, the server-derived handles for reflowable books with a
|
|
resolvable anchor:
|
|
|
|
| Field | Currency | Meaning |
|
|
| ------------- | ----------------------- | ------- |
|
|
| `anchor_href` | — | The spine document the anchor lives in (e.g. `1984.xhtml`). **This is how a client finds the right resource.** |
|
|
| `css_selector`| — | Body-relative `tag:nth-child(k)` chain of the anchor's block element (e.g. `body>div:nth-child(4)>p:nth-child(29)`). |
|
|
| `char_offset` | UTF-16 code units | Offset of the anchor within the concatenated text of that block. |
|
|
| `epubcfi` | UTF-16 terminals | The stored canonical CFI — re-served healed if the GET-time re-verification improved it. |
|
|
| `context_text`| — | The stored verification context (≤100 normalized chars from the anchor). |
|
|
|
|
**Recommended restore sequence for a client:**
|
|
|
|
1. Open the book and jump to the stored `percentage` (coarse floor).
|
|
2. Resolve `anchor_href` against your own spine (an ends-with match on
|
|
document hrefs) and open that document if you are not already there.
|
|
3. Query `css_selector` in that document, walk its text nodes counting
|
|
UTF-16 units to `char_offset`, and scroll that position into view.
|
|
4. If the anchor cannot be measured (renderer-specific laziness), the
|
|
percentage floor stands.
|
|
|
|
## SPINE NUMBERING HAZARD — read this before parsing a stored CFI
|
|
|
|
**`epubcfi` spine steps index the OPF spine AS WRITTEN, including
|
|
`linear="no"` items.** Several rendering engines (notably readium)
|
|
number their reading order EXCLUDING `linear="no"` items. When a book's
|
|
cover (or any other item) is `linear="no"`, the two numberings differ by
|
|
a constant offset from that item onward — a client resolving a stored
|
|
CFI's spine step against its own numbering lands in the WRONG DOCUMENT.
|
|
|
|
This is not hypothetical: "1984" epubs commonly have a `linear="no"`
|
|
cover, which makes readium spine 0 = OPF spine 1. The server heals such
|
|
numbering mismatches at ingest (that is what the context check is for),
|
|
but the durable rule for client authors is:
|
|
|
|
> **Never resolve a stored CFI's spine step yourself.** Resolve the
|
|
> document by `anchor_href`, then land with `css_selector` +
|
|
> `char_offset`. Treat the CFI as opaque server currency.
|
|
|
|
## Offset currencies
|
|
|
|
Two counting systems are in play, and they are deliberately kept apart:
|
|
|
|
| Quantity | Currency | Why |
|
|
| -------- | -------- | --- |
|
|
| CFI terminal offsets (`…/1:456`) | UTF-16 code units | The EPUB CFI spec, and what every client observes (JavaScript `.length`). |
|
|
| `char_offset` (block-relative handle) | UTF-16 code units | Same reason — clients walk DOM text with JS semantics. |
|
|
| KOReader CRE `text().N` offsets | UTF-16 code units | crengine is UCS-16 internally. |
|
|
| `character_offset` (book-wide column) | Unicode runes | Internal, consistent with `total_characters` and the percentage derivations. |
|
|
|
|
For all-BMP text the two currencies are identical. They diverge on
|
|
astral-plane characters (emoji, rare CJK ideographs): one rune, two
|
|
UTF-16 units. The server converts at every wire boundary; internal
|
|
arithmetic never crosses.
|
|
|
|
## `context_text` rules
|
|
|
|
- Starts at the anchor position (it may begin mid-word).
|
|
- Whitespace-normalized (all runs of whitespace collapse to single
|
|
spaces).
|
|
- At most 100 characters.
|
|
- Comparison is containment-based (client and server suffixes of the
|
|
same block verify in either direction); contexts shorter than 12
|
|
chars never match.
|
|
|
|
## Engine-specific notes
|
|
|
|
- **readium-based clients** (the Android app): submit
|
|
percentage + `context_text` + a CFI generated from the laid-out
|
|
WebView. Their locally-generated CFI spine steps use readium
|
|
numbering — the server heals the difference; nothing to do.
|
|
- **Web (foliate)**: submit all three tiers; foliate CFIs are the same
|
|
currency the server stores.
|
|
- **KOReader**: submits a CRE xpointer in the `epubcfi` field of the
|
|
device payload (historical field name; it is a CRE xpointer, not a
|
|
CFI). The server converts CRE → canonical at ingest and canonical →
|
|
CRE on pull (`koreader_xpointer` in the metadata response).
|
|
- **Kobo**: submits kepub CFI locators, converted server-side the same
|
|
way.
|
|
- **Fixed-layout content** (PDF/CBZ): the page index is the canonical
|
|
locator; CFI/xpointer are meaningless and neither submitted nor
|
|
served.
|