From 9b44bf6f5e1ee368025eecdf422429a4d12ddfdb Mon Sep 17 00:00:00 2001 From: KodeStar Date: Sat, 14 Nov 2026 12:39:01 +0000 Subject: [PATCH 1/9] Document meta-sync's Audible search fallback for region searches When libex's live store search does not answer, meta-sync now asks Audible's catalogue the same title and author search and keeps a hit only when it states the production's SKU group in the store asked. Describe the trigger, the separate SYNC_AUDIBLE_SEARCH_BUDGET, the dead-store rule, the new cycle-record fields and the /healthz degraded reason. --- docs-developers/meta/sync-bot.md | 74 +++++++++++++++++++++++++++++--- 1 file changed, 68 insertions(+), 6 deletions(-) diff --git a/docs-developers/meta/sync-bot.md b/docs-developers/meta/sync-bot.md index 4a44ffcd..abf4ab6b 100644 --- a/docs-developers/meta/sync-bot.md +++ b/docs-developers/meta/sync-bot.md @@ -23,7 +23,8 @@ It serves no public traffic. It talks only to: - libexdb.com; - api.github.com; - Audible's public catalog API (`api.audible.*`), only when libex says it cannot - reach Audible, and not at all with `SYNC_AUDIBLE_FALLBACK=off`; + reach Audible or its store search does not answer, and not at all with + `SYNC_AUDIBLE_FALLBACK=off`; - metaserve's intake queue, with [the reported-ASIN lane](#the-reported-asin-lane) enabled; - librivox.org's API, for [the LibriVox lane](#the-librivox-lane), unless @@ -254,8 +255,65 @@ A store search that has no answer within 5 seconds is not waited for. libex fetches a store it has not seen while the search waits, which can take half a minute. The service asks it again once, with the full timeout, at the end of the cycle's discovery, so a copy it finds still lands in that cycle's pull request. -A store that fails that retry too is skipped for the rest of the cycle, and the -region follow-up asks it again later. +A store that fails that retry too is asked of Audible's own catalogue, as +described below. When Audible cannot answer either, or the fallback is off, the +store is skipped for that production, and the region follow-up asks it again +later. + +#### When libex's search does not answer + +libex's live search can stop answering for every store at once. On 2026-11-14 +it answered `upstream_unavailable` after about 30 seconds in every store, while +its mirror still answered. Every cycle that day ended with hundreds of +unanswered searches, and a new preorder was written and announced with its US +store alone. Audible's own catalogue answered the same searches at once, and it +states each product's SKU group (its `sku_lite`). So, with +`SYNC_AUDIBLE_FALLBACK` on, a region search libex does not answer is asked of +Audible instead. + +- **What sends a search to Audible.** No answer from libex: a timeout, a server + fault, libex saying it cannot reach Audible, or a store on hold. A 404 is an + answer (the store does not sell the book), and so is a search libex rejects. + Neither goes to Audible. +- **When it is asked.** In the retry pass at the end of discovery, never + inline. A search whose libex retry does not answer either is asked of Audible + in the same store, by the production's title and first author. A store on + hold sends libex nothing, so its searches go to Audible alone. +- **What counts as a copy.** A hit is kept only when Audible states the + production's own SKU group, in the store asked. That is the same rule a libex + search follows, however loose Audible's title search is. An answer with no + product of that group is the store answering without the book, so the + follow-up's backoff applies to it as usual. The rows go through + `metaimport libex` like every other, stamped `libex-import`. +- **Its own budget.** At most 1,200 Audible searches a cycle + (`SYNC_AUDIBLE_SEARCH_BUDGET`). This is a separate limit from the follow-up's + 1,200 searches and from the 300 record and series requests the rest of the + cycle may ask Audible. It shares their breaker (a 429 or 403 opens it at once, + as do three faults in a row) and their 500ms gap between requests. The + creation path's searches are asked first, then the follow-up's in their + priority order. A search the budget or the breaker cannot cover stays a + failure of that store, never a store without the book, so the follow-up asks + it again later. +- **A dead store is not asked again.** After three searches of one store in a + row that got no answer, the store's next search waits libex's full timeout + instead of 5 seconds. A store libex has not seen yet is slow on a healthy day, + not dead, and this one wait tells the two apart. Only when that search does + not answer either is the store dead for the rest of the cycle: libex is not + asked about it again, and its searches go straight to Audible. Any answer, + copies or a 404, starts the count over. A store on hold never becomes dead. +- **Only while Audible can be asked.** Once the breaker is open, a search left + to Audible alone could only fail. From then on held stores fail at once, no + store becomes dead, and the service behaves as with the fallback off. + +A cycle whose region searches were made and not one was answered, by libex nor +by Audible, is degraded: the cycle log says so, the pull request body says so, +and `/healthz` reports it (see [Operating it](#operating-it)). The mark is set +even when the cycle then fails, and it stays until a later cycle's searches are +answered. A cycle that makes no region search at all leaves it standing, since +it says nothing about whether the searches answer now. The mark is kept in +memory, so a restart clears it. The cycle record counts the searches Audible +answered, in all and by store, and the cycle log has a line for the Audible +part of the retry pass. A store often lists a book later than the others, and a search can fail. So each cycle also runs the **region follow-up**. It takes the catalogued @@ -922,13 +980,16 @@ to `127.0.0.1`. That binding is what makes the missing auth safe, since | Endpoint | What it does | |---|---| -| `GET /healthz` | `{"status":"ok"}` or `{"status":"degraded","detail":"..."}`. Always 200 once the process is up. This is a **liveness** check: a service whose last cycle failed is still alive and will try again. | +| `GET /healthz` | `{"status":"ok"}` or `{"status":"degraded","detail":"..."}`. Always 200 once the process is up. This is a **liveness** check: a service whose last cycle failed is still alive and will try again. The `detail` names one reason, in this order: the last problem the service met (a failed cycle, for example), the parked pull requests at the limit, a throttle, then [region searches nobody answered](#when-libexs-search-does-not-answer). | | `GET /status` | The whole observable state. Its fields are listed under this table. It never reports the lane's address or token. | | `POST /run` | Run a cycle now. 202 when queued, 409 when a cycle is already running or queued. The service never stacks a request behind a running cycle. With the steward on, it also needs `Authorization: Bearer` with the token in `/data/admin-token`, since the steward's agent shares the container's loopback. | `GET /status` reports: -- the last cycle; +- the last cycle. Its `region_search_audible` and `audible_searches_by_store` + count the region searches Audible's catalogue answered (in all and by store), + and `region_searches_unanswered` is true when the cycle made region searches + and none was answered; - the open pull request and the parked pull requests; - how many pull requests the ladder has excluded, how many in a row, and the last reason; @@ -977,7 +1038,8 @@ config surface is exactly this table. Only `GITHUB_TOKEN` is required. | `SYNC_REGIONS` | all 11 | `us uk ca au de fr es it jp in br`, exactly the meta schema's region enum. | | `SYNC_DRY_RUN` | off | Do everything except push and open a pull request. Leaves the diff in the clone and prints the pull request body. | | `LIBEX_BASE` | `https://libexdb.com` | For tests and local fakes. | -| `SYNC_AUDIBLE_FALLBACK` | on | Ask Audible's public catalog API for a book record or series listing that libex says it cannot give right now. `off` keeps the service to libex alone. | +| `SYNC_AUDIBLE_FALLBACK` | on | Ask Audible's public catalog API for a book record or series listing that libex says it cannot give right now, and for a region search libex's search does not answer. `off` keeps the service to libex alone. | +| `SYNC_AUDIBLE_SEARCH_BUDGET` | `1200` | Region searches a cycle may ask Audible's catalogue when libex's search does not answer ([when libex's search does not answer](#when-libexs-search-does-not-answer)). Separate from the 300 record and series requests. Past it a search stays a failure of that store, and the region follow-up asks it again. `0` asks none. Only with `SYNC_AUDIBLE_FALLBACK` on. | | `SYNC_TOOLS_DIR` | `/app/bin` | Where `metaimport`, `metafmt` and `metacheck` live. Empty means resolve on `PATH`. | | `SYNC_INTAKE_URL` | - | metaserve's address, for [the reported-ASIN lane](#the-reported-asin-lane). The lane runs only with this and the token both set. https, or plain http to a loopback host only. | | `SYNC_INTAKE_TOKEN` | - | The Bearer token of the intake queue's `GET` (metaserve's `METASERVE_INTAKE_TOKEN`). | From 2c1609d1096e011e479d942770bf251c8eff5f25 Mon Sep 17 00:00:00 2001 From: KodeStar Date: Sat, 14 Nov 2026 12:40:01 +0000 Subject: [PATCH 2/9] Document the Discord bot's hold on new watch-feed entries meta-bot now waits NOTIFY_HOLD (6h by default, 0 to 24h) before the DM for a new feed entry, so meta-sync's later cycles can add the book's other stores first. Describe the hold, how stop, replace and pause treat held entries, and tell readers of the User Guide that DMs come a few hours after a book appears. --- docs-developers/meta/discord-bot.md | 27 ++++++++++++++++++++++----- docs-users/community/meta-site.md | 4 +++- 2 files changed, 25 insertions(+), 6 deletions(-) diff --git a/docs-developers/meta/discord-bot.md b/docs-developers/meta/discord-bot.md index a63fbc97..653c837d 100644 --- a/docs-developers/meta/discord-bot.md +++ b/docs-developers/meta/discord-bot.md @@ -15,10 +15,22 @@ the equivalent JSON Feed and notifies on new item IDs or changed entry content. A preorder becoming a release has a new ID, so both events notify. The first fetch is a silent baseline; reordering and entries aging out do not notify. +A new item ID waits `NOTIFY_HOLD` (6 hours by default) before its DM. +meta-sync often adds a book's other stores in a later cycle, so the wait lets +the linked page fill in first. If the entry changes while it waits, the one DM +carries the latest version, and the wait still counts from when the ID first +appeared. A change to an entry that is not waiting, such as a date correction +to one already sent or in the baseline, goes out at once. Polling continues +while entries wait, and `status` counts them as queued. `NOTIFY_HOLD=0` sends +new entries at once and releases anything still waiting. + Members can DM `status`, `pause`, `resume`, `stop` or `help`. `stop` deletes their -subscription and pending messages. Replacing a URL also clears that user's old -pending messages. Members who leave the configured server are removed on the -next successful membership check. Blocked DMs pause delivery until `resume`. +subscription, pending messages and waiting entries. Replacing a URL also clears +that user's old pending messages; a waiting entry the new list still contains +keeps its wait and arrives once due, and the rest are dropped. `pause` keeps +waiting entries, which arrive after `resume` once due. Members who leave the +configured server are removed on the next successful membership check. Blocked +DMs pause delivery until `resume`. ## Community announcements @@ -30,6 +42,7 @@ character sheets. Announcements wait until both conditions hold: There is no further wait after release: recaps and character sheets are spoiler gated on the site, so an announcement is useful to listeners of the current book. +`NOTIFY_HOLD` applies to feed DMs only, never to announcements. Core composes the community repository into its data release, so the bot waits for publication after the community merge. A release date stating only a year, @@ -92,6 +105,7 @@ check, `libexdb.com`. | `META_BASE_URL` | Allowed metadata HTTPS origin; defaults to `https://meta.audiosilo.app` | | `LIBEX_BASE_URL` | HTTPS origin for the release-day spot check; defaults to `https://libexdb.com`, `off` disables it | | `POLL_INTERVAL` | Feed polling interval, default `5m` | +| `NOTIFY_HOLD` | How long a new feed entry waits before its DM, default `6h`, from `0` (off) to `24h` | | `DATA_DIR` | State directory, `/data` in Compose | | `HEALTH_ADDR` | Health listener, unexposed port 8080 in Compose | @@ -113,8 +127,11 @@ entries that have already left that feed. Community scans run hourly; pending release-date checks are bounded per poll. Commands and polling are serialized, so a large scan can delay a reply. -Delivery retries survive restarts. A crash between Discord accepting a message -and its local acknowledgment can produce a duplicate. Back up `state.json` while +Delivery retries survive restarts, and so do waiting entries with the time +each first appeared. A waiting entry is saved like a pending message and joins +the queue once its wait is over and anything already queued has been sent. A +crash between Discord accepting a message and its local acknowledgment can +produce a duplicate. Back up `state.json` while the bot is stopped; deleting the volume loses subscriptions and delivery history. Corrupt state and failed state writes stop the service instead of silently resetting it. The container health check checks connectivity and worker progress. diff --git a/docs-users/community/meta-site.md b/docs-users/community/meta-site.md index 0a9c378e..b7d20c6c 100644 --- a/docs-users/community/meta-site.md +++ b/docs-users/community/meta-site.md @@ -140,7 +140,9 @@ To get them as Discord messages, use the **AudioSilo Discord bot**: **Direct Messages**). 3. Send the AudioSilo bot a direct message with your **Atom feed** URL. -New books then arrive as DMs. Send `help` for the commands. `stop` deletes +New books then arrive as DMs, a few hours after they first appear in your feed. +The wait gives the book's page time to list all its stores. Send `help` for the +commands. `stop` deletes everything the bot keeps about you. Two things to know: From d87d386c620e7a26ceb3e27189853fde8bdde845 Mon Sep 17 00:00:00 2001 From: KodeStar Date: Sat, 14 Nov 2026 12:42:06 +0000 Subject: [PATCH 3/9] Give the Audible search fallback its own subsection after the follow-up The fallback's description sat in the middle of 'Every store's copy of a book', so the region follow-up's paragraphs fell under its heading and its bullets named the follow-up before the page introduced it. Move it to the end of that section, name who does what, give the measured counts, and say in the User Guide that the DM wait lets the book's page fill in. --- docs-developers/meta/sync-bot.md | 121 +++++++++++++++--------------- docs-users/community/meta-site.md | 5 +- 2 files changed, 64 insertions(+), 62 deletions(-) diff --git a/docs-developers/meta/sync-bot.md b/docs-developers/meta/sync-bot.md index abf4ab6b..ea41846d 100644 --- a/docs-developers/meta/sync-bot.md +++ b/docs-developers/meta/sync-bot.md @@ -255,66 +255,12 @@ A store search that has no answer within 5 seconds is not waited for. libex fetches a store it has not seen while the search waits, which can take half a minute. The service asks it again once, with the full timeout, at the end of the cycle's discovery, so a copy it finds still lands in that cycle's pull request. -A store that fails that retry too is asked of Audible's own catalogue, as -described below. When Audible cannot answer either, or the fallback is off, the -store is skipped for that production, and the region follow-up asks it again +When that retry fails too, the service asks Audible's own catalogue +([when libex's search does not answer](#when-libexs-search-does-not-answer)). +When Audible cannot answer either, or the fallback is off, the service writes +the production without that store, and the region follow-up asks it again later. -#### When libex's search does not answer - -libex's live search can stop answering for every store at once. On 2026-11-14 -it answered `upstream_unavailable` after about 30 seconds in every store, while -its mirror still answered. Every cycle that day ended with hundreds of -unanswered searches, and a new preorder was written and announced with its US -store alone. Audible's own catalogue answered the same searches at once, and it -states each product's SKU group (its `sku_lite`). So, with -`SYNC_AUDIBLE_FALLBACK` on, a region search libex does not answer is asked of -Audible instead. - -- **What sends a search to Audible.** No answer from libex: a timeout, a server - fault, libex saying it cannot reach Audible, or a store on hold. A 404 is an - answer (the store does not sell the book), and so is a search libex rejects. - Neither goes to Audible. -- **When it is asked.** In the retry pass at the end of discovery, never - inline. A search whose libex retry does not answer either is asked of Audible - in the same store, by the production's title and first author. A store on - hold sends libex nothing, so its searches go to Audible alone. -- **What counts as a copy.** A hit is kept only when Audible states the - production's own SKU group, in the store asked. That is the same rule a libex - search follows, however loose Audible's title search is. An answer with no - product of that group is the store answering without the book, so the - follow-up's backoff applies to it as usual. The rows go through - `metaimport libex` like every other, stamped `libex-import`. -- **Its own budget.** At most 1,200 Audible searches a cycle - (`SYNC_AUDIBLE_SEARCH_BUDGET`). This is a separate limit from the follow-up's - 1,200 searches and from the 300 record and series requests the rest of the - cycle may ask Audible. It shares their breaker (a 429 or 403 opens it at once, - as do three faults in a row) and their 500ms gap between requests. The - creation path's searches are asked first, then the follow-up's in their - priority order. A search the budget or the breaker cannot cover stays a - failure of that store, never a store without the book, so the follow-up asks - it again later. -- **A dead store is not asked again.** After three searches of one store in a - row that got no answer, the store's next search waits libex's full timeout - instead of 5 seconds. A store libex has not seen yet is slow on a healthy day, - not dead, and this one wait tells the two apart. Only when that search does - not answer either is the store dead for the rest of the cycle: libex is not - asked about it again, and its searches go straight to Audible. Any answer, - copies or a 404, starts the count over. A store on hold never becomes dead. -- **Only while Audible can be asked.** Once the breaker is open, a search left - to Audible alone could only fail. From then on held stores fail at once, no - store becomes dead, and the service behaves as with the fallback off. - -A cycle whose region searches were made and not one was answered, by libex nor -by Audible, is degraded: the cycle log says so, the pull request body says so, -and `/healthz` reports it (see [Operating it](#operating-it)). The mark is set -even when the cycle then fails, and it stays until a later cycle's searches are -answered. A cycle that makes no region search at all leaves it standing, since -it says nothing about whether the searches answer now. The mark is kept in -memory, so a restart clears it. The cycle record counts the searches Audible -answered, in all and by store, and the cycle log has a line for the Audible -part of the retry pass. - A store often lists a book later than the others, and a search can fail. So each cycle also runs the **region follow-up**. It takes the catalogued recordings that are released in the last 60 days, not released yet, or added in @@ -350,6 +296,63 @@ or, when a memo entry keeps the copy out, once that entry expires. A copy outside the follow-up's window, of a book in a catalogued series, still arrives through its own marketplace's new-release or coming-soon feed. +### When libex's search does not answer + +libex's live search can stop answering for every store at once. On 2026-11-14 +it answered `upstream_unavailable` after about 30 seconds in every store, while +its mirror still answered. Every cycle that day ended with 644 to 1,073 +unanswered searches, and a new preorder was written and announced with its US +store alone. Audible's own catalogue answered the same searches at once, and it +states each product's SKU group (its `sku_lite`). So, with +`SYNC_AUDIBLE_FALLBACK` on, the service asks Audible a region search that libex +does not answer. + +- **What sends a search to Audible.** No answer from libex, which means a + timeout, a server fault, libex saying it cannot reach Audible, or a store on + hold. A 404 is an answer (the store does not sell the book), and so is a + search libex rejects. Neither goes to Audible. +- **When the service asks.** In the retry pass at the end of discovery, never + inline. When a search's libex retry does not answer either, the service + searches the same store's Audible marketplace by the production's title and + first author. A store on hold sends libex nothing, so its searches go to + Audible alone. +- **What counts as a copy.** The service keeps a hit only when Audible states + the production's own SKU group, in the store asked. That is the same rule a + libex search follows, however loose Audible's title search is. An answer with + no product of that group is the store answering without the book, so the + follow-up's backoff applies to it as usual. The rows go through + `metaimport libex` like every other, stamped `libex-import`. +- **Its own budget.** A cycle asks Audible at most 1,200 searches + (`SYNC_AUDIBLE_SEARCH_BUDGET`). This limit is separate from the follow-up's + 1,200 searches and from the 300 record and series requests the rest of the + cycle may ask Audible. It shares their breaker (a 429 or 403 opens it at + once, as do three faults in a row) and their 500ms gap between requests. The + creation path's searches go first, then the follow-up's in their priority + order. A search the budget or the breaker cannot cover stays a failure of + that store, never a store without the book, so the follow-up asks it again + later. +- **A dead store is not asked again.** After three searches of one store in a + row that got no answer, the service gives the store's next search libex's + full timeout instead of 5 seconds. A store libex has not seen yet is slow on + a healthy day, not dead, and this one wait tells the two apart. Only when + that search does not answer either is the store dead for the rest of the + cycle. The service then asks libex nothing more about it, and sends its + searches straight to Audible. Any answer, copies or a 404, starts the count + over. A store on hold never becomes dead. +- **Only while Audible can be asked.** Once the breaker is open, a search left + to Audible alone could only fail. From then on held stores fail at once, no + store becomes dead, and the service behaves as with the fallback off. + +A cycle that made region searches and got not one answer, from libex or from +Audible, is degraded. The cycle log and the pull request body say so, and +`/healthz` reports it (see [Operating it](#operating-it)). The service sets the +mark even when the cycle then fails, and keeps it until a later cycle's +searches are answered. A cycle that makes no region search at all leaves it +standing, since it says nothing about whether the searches answer now. The mark +lives in memory, so a restart clears it. The cycle record counts the searches +Audible answered, in all and by store, and the cycle log has a line for the +Audible part of the retry pass. + ## The reported-ASIN lane One of the bot's two other ways in. When enabled, the lane reads metaserve's @@ -980,7 +983,7 @@ to `127.0.0.1`. That binding is what makes the missing auth safe, since | Endpoint | What it does | |---|---| -| `GET /healthz` | `{"status":"ok"}` or `{"status":"degraded","detail":"..."}`. Always 200 once the process is up. This is a **liveness** check: a service whose last cycle failed is still alive and will try again. The `detail` names one reason, in this order: the last problem the service met (a failed cycle, for example), the parked pull requests at the limit, a throttle, then [region searches nobody answered](#when-libexs-search-does-not-answer). | +| `GET /healthz` | `{"status":"ok"}` or `{"status":"degraded","detail":"..."}`. Always 200 once the process is up. This is a **liveness** check: a service whose last cycle failed is still alive and will try again. The sync's `detail` names one reason, in this order: the last problem the service met (a failed cycle, for example), the parked pull requests at the limit, a throttle, then [region searches nobody answered](#when-libexs-search-does-not-answer). | | `GET /status` | The whole observable state. Its fields are listed under this table. It never reports the lane's address or token. | | `POST /run` | Run a cycle now. 202 when queued, 409 when a cycle is already running or queued. The service never stacks a request behind a running cycle. With the steward on, it also needs `Authorization: Bearer` with the token in `/data/admin-token`, since the steward's agent shares the container's loopback. | diff --git a/docs-users/community/meta-site.md b/docs-users/community/meta-site.md index b7d20c6c..1631f3d7 100644 --- a/docs-users/community/meta-site.md +++ b/docs-users/community/meta-site.md @@ -141,9 +141,8 @@ To get them as Discord messages, use the **AudioSilo Discord bot**: 3. Send the AudioSilo bot a direct message with your **Atom feed** URL. New books then arrive as DMs, a few hours after they first appear in your feed. -The wait gives the book's page time to list all its stores. Send `help` for the -commands. `stop` deletes -everything the bot keeps about you. +The wait gives the book's page time to fill in. Send `help` for the commands. +`stop` deletes everything the bot keeps about you. Two things to know: From 9eb73d42bef62dabbf3b8f46dc151fc199b85042 Mon Sep 17 00:00:00 2001 From: KodeStar Date: Sat, 14 Nov 2026 12:47:43 +0000 Subject: [PATCH 4/9] Define a store on hold and trim the repeated search fallback rules The search fallback section used 'a store on hold' without saying what it is, named the Audible breaker in a way that clashed with the escalation ladder's circuit breaker, and restated the retry order, the budget's failure rule and the cycle record's counters in several places. Define the hold once, name the breaker, keep each rule in one place, list an unanswered search among the external errors, and say that a copy is proven by the SKU group libex or Audible's catalogue states. Part of the Audible search fallback work. --- docs-developers/meta/contributing-data.md | 2 +- docs-developers/meta/sync-bot.md | 44 ++++++++++++----------- 2 files changed, 25 insertions(+), 21 deletions(-) diff --git a/docs-developers/meta/contributing-data.md b/docs-developers/meta/contributing-data.md index c1586570..c5e56069 100644 --- a/docs-developers/meta/contributing-data.md +++ b/docs-developers/meta/contributing-data.md @@ -610,7 +610,7 @@ exclusive planning modes: `{"asin","anchor"}` line per row. Each listed row is another marketplace's copy of the production the catalogued `anchor` ASIN belongs to, as the caller proved it. [The series-completion bot](./sync-bot.md#every-stores-copy-of-a-book) -proves it by libex SKU group. The import merges the row's ASIN onto the anchor's +proves it by SKU group, as libex or Audible's catalogue states it. The import merges the row's ASIN onto the anchor's recording and resolves nothing from the row's credits or series, so it mints no person, work or series. It still refuses a row whose anchor is not a catalogued retail recording (`copy-anchor-unknown`, a LibriVox recording included), whose diff --git a/docs-developers/meta/sync-bot.md b/docs-developers/meta/sync-bot.md index ea41846d..dc3dd4c5 100644 --- a/docs-developers/meta/sync-bot.md +++ b/docs-developers/meta/sync-bot.md @@ -23,7 +23,7 @@ It serves no public traffic. It talks only to: - libexdb.com; - api.github.com; - Audible's public catalog API (`api.audible.*`), only when libex says it cannot - reach Audible or its store search does not answer, and not at all with + reach Audible or libex's store search does not answer, and not at all with `SYNC_AUDIBLE_FALLBACK=off`; - metaserve's intake queue, with [the reported-ASIN lane](#the-reported-asin-lane) enabled; @@ -231,6 +231,9 @@ A few properties of the completion pass are worth knowing. and `GET /status` counts them on the last cycle. - A news row libex returns no record for is counted. - A chapters payload libex refuses is counted and logged. + - A store search nobody answers leaves that store for the region follow-up. + A cycle in which not one region search was answered is marked degraded + ([when libex's search does not answer](#when-libexs-search-does-not-answer)). - **`metacheck` is the gate on the way out.** A cycle whose `metafmt` or `metacheck` pass fails discards its whole tree back to the base commit and opens nothing. @@ -298,22 +301,22 @@ through its own marketplace's new-release or coming-soon feed. ### When libex's search does not answer -libex's live search can stop answering for every store at once. On 2026-11-14 -it answered `upstream_unavailable` after about 30 seconds in every store, while -its mirror still answered. Every cycle that day ended with 644 to 1,073 -unanswered searches, and a new preorder was written and announced with its US -store alone. Audible's own catalogue answered the same searches at once, and it -states each product's SKU group (its `sku_lite`). So, with +libex's live search can stop answering for every store at once, as it did on +2026-11-14, while Audible's own catalogue still answers the same searches. +Audible states each product's SKU group (its `sku_lite`). So, with `SYNC_AUDIBLE_FALLBACK` on, the service asks Audible a region search that libex does not answer. +A store is **on hold** when libex has said it cannot reach that store's Audible +marketplace: the service sends libex no search of that store for 5 minutes, or +for as long as libex's `Retry-After` asks. + - **What sends a search to Audible.** No answer from libex, which means a timeout, a server fault, libex saying it cannot reach Audible, or a store on hold. A 404 is an answer (the store does not sell the book), and so is a search libex rejects. Neither goes to Audible. -- **When the service asks.** In the retry pass at the end of discovery, never - inline. When a search's libex retry does not answer either, the service - searches the same store's Audible marketplace by the production's title and +- **How the service asks.** In the retry pass at the end of discovery, it + searches the store's own Audible marketplace by the production's title and first author. A store on hold sends libex nothing, so its searches go to Audible alone. - **What counts as a copy.** The service keeps a hit only when Audible states @@ -325,21 +328,22 @@ does not answer. - **Its own budget.** A cycle asks Audible at most 1,200 searches (`SYNC_AUDIBLE_SEARCH_BUDGET`). This limit is separate from the follow-up's 1,200 searches and from the 300 record and series requests the rest of the - cycle may ask Audible. It shares their breaker (a 429 or 403 opens it at - once, as do three faults in a row) and their 500ms gap between requests. The + cycle may ask Audible. It shares their Audible breaker, which stops every + Audible request for the rest of the cycle (a 429 or 403 opens it at once, as + do three faults in a row), and their 500ms gap between requests. The creation path's searches go first, then the follow-up's in their priority order. A search the budget or the breaker cannot cover stays a failure of that store, never a store without the book, so the follow-up asks it again later. -- **A dead store is not asked again.** After three searches of one store in a +- **When a store counts as dead.** After three searches of one store in a row that got no answer, the service gives the store's next search libex's full timeout instead of 5 seconds. A store libex has not seen yet is slow on - a healthy day, not dead, and this one wait tells the two apart. Only when - that search does not answer either is the store dead for the rest of the - cycle. The service then asks libex nothing more about it, and sends its + a healthy day, not dead, and this one wait tells the two apart. If that + search gets no answer either, the store is dead for the rest of the cycle. + The service then asks libex nothing more about it, and sends its searches straight to Audible. Any answer, copies or a 404, starts the count over. A store on hold never becomes dead. -- **Only while Audible can be asked.** Once the breaker is open, a search left +- **Only while Audible can be asked.** Once the Audible breaker is open, a search left to Audible alone could only fail. From then on held stores fail at once, no store becomes dead, and the service behaves as with the fallback off. @@ -350,7 +354,7 @@ mark even when the cycle then fails, and keeps it until a later cycle's searches are answered. A cycle that makes no region search at all leaves it standing, since it says nothing about whether the searches answer now. The mark lives in memory, so a restart clears it. The cycle record counts the searches -Audible answered, in all and by store, and the cycle log has a line for the +Audible answered (see `GET /status`), and the cycle log has a line for the Audible part of the retry pass. ## The reported-ASIN lane @@ -1041,8 +1045,8 @@ config surface is exactly this table. Only `GITHUB_TOKEN` is required. | `SYNC_REGIONS` | all 11 | `us uk ca au de fr es it jp in br`, exactly the meta schema's region enum. | | `SYNC_DRY_RUN` | off | Do everything except push and open a pull request. Leaves the diff in the clone and prints the pull request body. | | `LIBEX_BASE` | `https://libexdb.com` | For tests and local fakes. | -| `SYNC_AUDIBLE_FALLBACK` | on | Ask Audible's public catalog API for a book record or series listing that libex says it cannot give right now, and for a region search libex's search does not answer. `off` keeps the service to libex alone. | -| `SYNC_AUDIBLE_SEARCH_BUDGET` | `1200` | Region searches a cycle may ask Audible's catalogue when libex's search does not answer ([when libex's search does not answer](#when-libexs-search-does-not-answer)). Separate from the 300 record and series requests. Past it a search stays a failure of that store, and the region follow-up asks it again. `0` asks none. Only with `SYNC_AUDIBLE_FALLBACK` on. | +| `SYNC_AUDIBLE_FALLBACK` | on | Ask Audible's public catalog API for a book record or series listing that libex says it cannot give right now, and for a region search libex does not answer. `off` keeps the service to libex alone. | +| `SYNC_AUDIBLE_SEARCH_BUDGET` | `1200` | Region searches a cycle may ask Audible's catalogue [when libex's search does not answer](#when-libexs-search-does-not-answer), apart from the 300 record and series requests. `0` asks none. Only with `SYNC_AUDIBLE_FALLBACK` on. | | `SYNC_TOOLS_DIR` | `/app/bin` | Where `metaimport`, `metafmt` and `metacheck` live. Empty means resolve on `PATH`. | | `SYNC_INTAKE_URL` | - | metaserve's address, for [the reported-ASIN lane](#the-reported-asin-lane). The lane runs only with this and the token both set. https, or plain http to a loopback host only. | | `SYNC_INTAKE_TOKEN` | - | The Bearer token of the intake queue's `GET` (metaserve's `METASERVE_INTAKE_TOKEN`). | From 00f6770ad7ed80fb29a2a8558711bd3c6a1e99cd Mon Sep 17 00:00:00 2001 From: KodeStar Date: Sat, 14 Nov 2026 12:47:43 +0000 Subject: [PATCH 5/9] Link the bot's notify hold to the region follow-up and tidy its wording Say that status counts waiting entries among its queued notifications (they join the delivery queue only once due), state the replacement rule in plain order, link the reason for the hold to meta-sync's region follow-up, and give the state backup its own paragraph. --- docs-developers/meta/discord-bot.md | 37 ++++++++++++++++------------- 1 file changed, 20 insertions(+), 17 deletions(-) diff --git a/docs-developers/meta/discord-bot.md b/docs-developers/meta/discord-bot.md index 653c837d..874a3727 100644 --- a/docs-developers/meta/discord-bot.md +++ b/docs-developers/meta/discord-bot.md @@ -1,6 +1,6 @@ --- title: Discord notification bot -description: Deploy the private AudioSilo watchlist bot and understand its feed notifications and delayed community announcements. +description: Deploy the private AudioSilo watchlist bot and understand its held feed notifications and delayed community announcements. --- ## Purpose @@ -16,21 +16,22 @@ A preorder becoming a release has a new ID, so both events notify. The first fetch is a silent baseline; reordering and entries aging out do not notify. A new item ID waits `NOTIFY_HOLD` (6 hours by default) before its DM. -meta-sync often adds a book's other stores in a later cycle, so the wait lets -the linked page fill in first. If the entry changes while it waits, the one DM -carries the latest version, and the wait still counts from when the ID first -appeared. A change to an entry that is not waiting, such as a date correction -to one already sent or in the baseline, goes out at once. Polling continues -while entries wait, and `status` counts them as queued. `NOTIFY_HOLD=0` sends -new entries at once and releases anything still waiting. +[meta-sync](./sync-bot.md#every-stores-copy-of-a-book) often adds a book's +other stores in a later cycle, so the wait lets the linked page fill in first. +If the entry changes while it waits, the one DM carries the latest version, and +the wait still counts from when the ID first appeared. A change to an entry +that is not waiting, such as a date correction to one already sent or in the +baseline, goes out at once. Polling continues while entries wait, and `status` +counts them among its queued notifications. `NOTIFY_HOLD=0` sends new entries +at once and releases anything still waiting. Members can DM `status`, `pause`, `resume`, `stop` or `help`. `stop` deletes their subscription, pending messages and waiting entries. Replacing a URL also clears -that user's old pending messages; a waiting entry the new list still contains -keeps its wait and arrives once due, and the rest are dropped. `pause` keeps -waiting entries, which arrive after `resume` once due. Members who leave the -configured server are removed on the next successful membership check. Blocked -DMs pause delivery until `resume`. +that user's old pending messages and drops the waiting entries the new list +does not contain. The others keep their wait. `pause` keeps waiting entries, +which arrive after `resume` once due. Members who leave the configured server +are removed on the next successful membership check. Blocked DMs pause delivery +until `resume`. ## Community announcements @@ -131,7 +132,9 @@ Delivery retries survive restarts, and so do waiting entries with the time each first appeared. A waiting entry is saved like a pending message and joins the queue once its wait is over and anything already queued has been sent. A crash between Discord accepting a message and its local acknowledgment can -produce a duplicate. Back up `state.json` while -the bot is stopped; deleting the volume loses subscriptions and delivery history. -Corrupt state and failed state writes stop the service instead of silently -resetting it. The container health check checks connectivity and worker progress. +produce a duplicate. + +Back up `state.json` while the bot is stopped; deleting the volume loses +subscriptions and delivery history. Corrupt state and failed state writes stop +the service instead of silently resetting it. The container health check checks +connectivity and worker progress. From c32db84228612e28b629c2508fba3ff634217dca Mon Sep 17 00:00:00 2001 From: KodeStar Date: Sat, 14 Nov 2026 12:52:53 +0000 Subject: [PATCH 6/9] State the search fallback's edge cases as the sync applies them A rejected libex search is not counted as answered, so it is no longer called a plain answer; a store's hold is capped at an hour whatever Retry-After asks; a store that stays slow gets the full timeout once in every four searches, not once a cycle; a dead store is asked of libex again once the Audible breaker opens; and a search budget of 0 turns the whole search fallback off, the held-store and dead-store routing included. Part of the Audible search fallback work. --- docs-developers/meta/contributing-data.md | 5 ++-- docs-developers/meta/sync-bot.md | 29 +++++++++++++---------- 2 files changed, 19 insertions(+), 15 deletions(-) diff --git a/docs-developers/meta/contributing-data.md b/docs-developers/meta/contributing-data.md index c5e56069..518f4f87 100644 --- a/docs-developers/meta/contributing-data.md +++ b/docs-developers/meta/contributing-data.md @@ -610,8 +610,9 @@ exclusive planning modes: `{"asin","anchor"}` line per row. Each listed row is another marketplace's copy of the production the catalogued `anchor` ASIN belongs to, as the caller proved it. [The series-completion bot](./sync-bot.md#every-stores-copy-of-a-book) -proves it by SKU group, as libex or Audible's catalogue states it. The import merges the row's ASIN onto the anchor's -recording and resolves nothing from the row's credits or series, so it mints no +proves it by SKU group, as libex or Audible's catalogue states it. The import +merges the row's ASIN onto the anchor's recording and resolves nothing from the +row's credits or series, so it mints no person, work or series. It still refuses a row whose anchor is not a catalogued retail recording (`copy-anchor-unknown`, a LibriVox recording included), whose language is not the recording's (`copy-language`), whose runtime does not match diff --git a/docs-developers/meta/sync-bot.md b/docs-developers/meta/sync-bot.md index dc3dd4c5..56ee7890 100644 --- a/docs-developers/meta/sync-bot.md +++ b/docs-developers/meta/sync-bot.md @@ -258,7 +258,7 @@ A store search that has no answer within 5 seconds is not waited for. libex fetches a store it has not seen while the search waits, which can take half a minute. The service asks it again once, with the full timeout, at the end of the cycle's discovery, so a copy it finds still lands in that cycle's pull request. -When that retry fails too, the service asks Audible's own catalogue +When that retry gets no answer either, the service asks Audible's own catalogue ([when libex's search does not answer](#when-libexs-search-does-not-answer)). When Audible cannot answer either, or the fallback is off, the service writes the production without that store, and the region follow-up asks it again @@ -309,12 +309,13 @@ does not answer. A store is **on hold** when libex has said it cannot reach that store's Audible marketplace: the service sends libex no search of that store for 5 minutes, or -for as long as libex's `Retry-After` asks. +for as long as libex's `Retry-After` asks, at most an hour. - **What sends a search to Audible.** No answer from libex, which means a timeout, a server fault, libex saying it cannot reach Audible, or a store on - hold. A 404 is an answer (the store does not sell the book), and so is a - search libex rejects. Neither goes to Audible. + hold. A 404 is an answer (the store does not sell the book). A search libex + rejects is an answer no retry changes, and fails that store alone. Neither + goes to Audible. - **How the service asks.** In the retry pass at the end of discovery, it searches the store's own Audible marketplace by the production's title and first author. A store on hold sends libex nothing, so its searches go to @@ -338,14 +339,16 @@ for as long as libex's `Retry-After` asks. - **When a store counts as dead.** After three searches of one store in a row that got no answer, the service gives the store's next search libex's full timeout instead of 5 seconds. A store libex has not seen yet is slow on - a healthy day, not dead, and this one wait tells the two apart. If that - search gets no answer either, the store is dead for the rest of the cycle. - The service then asks libex nothing more about it, and sends its - searches straight to Audible. Any answer, copies or a 404, starts the count - over. A store on hold never becomes dead. -- **Only while Audible can be asked.** Once the Audible breaker is open, a search left - to Audible alone could only fail. From then on held stores fail at once, no - store becomes dead, and the service behaves as with the fallback off. + a healthy day, not dead, and this wait tells the two apart. If that search + gets no answer either, the store is dead for the rest of the cycle. The + service then asks libex nothing more about it, and sends its searches + straight to Audible. Any answer, copies or a 404, starts the count over, so + a store that stays slow gets the full timeout once in every four searches. + A store on hold never becomes dead. +- **Only while Audible can be asked.** Once the Audible breaker is open, a + search left to Audible alone could only fail. From then on held stores fail + at once, no store is dead (libex is asked about a dead store again), and the + service behaves as with the fallback off. A cycle that made region searches and got not one answer, from libex or from Audible, is degraded. The cycle log and the pull request body say so, and @@ -1046,7 +1049,7 @@ config surface is exactly this table. Only `GITHUB_TOKEN` is required. | `SYNC_DRY_RUN` | off | Do everything except push and open a pull request. Leaves the diff in the clone and prints the pull request body. | | `LIBEX_BASE` | `https://libexdb.com` | For tests and local fakes. | | `SYNC_AUDIBLE_FALLBACK` | on | Ask Audible's public catalog API for a book record or series listing that libex says it cannot give right now, and for a region search libex does not answer. `off` keeps the service to libex alone. | -| `SYNC_AUDIBLE_SEARCH_BUDGET` | `1200` | Region searches a cycle may ask Audible's catalogue [when libex's search does not answer](#when-libexs-search-does-not-answer), apart from the 300 record and series requests. `0` asks none. Only with `SYNC_AUDIBLE_FALLBACK` on. | +| `SYNC_AUDIBLE_SEARCH_BUDGET` | `1200` | Region searches a cycle may ask Audible's catalogue [when libex's search does not answer](#when-libexs-search-does-not-answer), apart from the 300 record and series requests. `0` asks none, which turns the search fallback off. Only with `SYNC_AUDIBLE_FALLBACK` on. | | `SYNC_TOOLS_DIR` | `/app/bin` | Where `metaimport`, `metafmt` and `metacheck` live. Empty means resolve on `PATH`. | | `SYNC_INTAKE_URL` | - | metaserve's address, for [the reported-ASIN lane](#the-reported-asin-lane). The lane runs only with this and the token both set. https, or plain http to a loopback host only. | | `SYNC_INTAKE_TOKEN` | - | The Bearer token of the intake queue's `GET` (metaserve's `METASERVE_INTAKE_TOKEN`). | From ced271da33cbf6ff1429149fa989b5d41a478da4 Mon Sep 17 00:00:00 2001 From: KodeStar Date: Sat, 14 Nov 2026 12:52:53 +0000 Subject: [PATCH 7/9] Say the held DM carries the latest copy the bot has read When the feed cannot be read on the poll where an entry's hold ends, the bot sends the copy it last read and a later change follows as a second DM, so the page no longer promises one DM with the latest version. The description now uses the page's own word for the wait. --- docs-developers/meta/discord-bot.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs-developers/meta/discord-bot.md b/docs-developers/meta/discord-bot.md index 874a3727..16b777e5 100644 --- a/docs-developers/meta/discord-bot.md +++ b/docs-developers/meta/discord-bot.md @@ -1,6 +1,6 @@ --- title: Discord notification bot -description: Deploy the private AudioSilo watchlist bot and understand its held feed notifications and delayed community announcements. +description: Deploy the private AudioSilo watchlist bot and understand when its feed notifications and community announcements are sent. --- ## Purpose @@ -18,10 +18,10 @@ fetch is a silent baseline; reordering and entries aging out do not notify. A new item ID waits `NOTIFY_HOLD` (6 hours by default) before its DM. [meta-sync](./sync-bot.md#every-stores-copy-of-a-book) often adds a book's other stores in a later cycle, so the wait lets the linked page fill in first. -If the entry changes while it waits, the one DM carries the latest version, and -the wait still counts from when the ID first appeared. A change to an entry -that is not waiting, such as a date correction to one already sent or in the -baseline, goes out at once. Polling continues while entries wait, and `status` +If the entry changes while it waits, its DM carries the latest version the bot +has read, and the wait still counts from when the ID first appeared. A change +to an entry that is not waiting, such as a date correction to one already sent +or in the baseline, goes out at once. Polling continues while entries wait, and `status` counts them among its queued notifications. `NOTIFY_HOLD=0` sends new entries at once and releases anything still waiting. From fc32b61c084942ddb3fad5b06258f9f685bfdc82 Mon Sep 17 00:00:00 2001 From: KodeStar Date: Sat, 14 Nov 2026 13:00:54 +0000 Subject: [PATCH 8/9] Give a store one 60-second inline try a cycle in the dead-store rule meta-sync now asks a store that went unanswered three times in a row one inline search a cycle, waiting up to 60 seconds. An answer keeps the store alive with no further inline try that cycle, a timeout or server fault makes it dead and sends its searches to Audible, and a rejection, a 429 or libex saying it cannot reach Audible decides nothing (the last holds the store). The page said a slow store got the full timeout once in every four searches. Part of the Audible search fallback work. --- docs-developers/meta/sync-bot.md | 28 ++++++++++++++++++---------- 1 file changed, 18 insertions(+), 10 deletions(-) diff --git a/docs-developers/meta/sync-bot.md b/docs-developers/meta/sync-bot.md index 56ee7890..66facd4d 100644 --- a/docs-developers/meta/sync-bot.md +++ b/docs-developers/meta/sync-bot.md @@ -337,18 +337,26 @@ for as long as libex's `Retry-After` asks, at most an hour. that store, never a store without the book, so the follow-up asks it again later. - **When a store counts as dead.** After three searches of one store in a - row that got no answer, the service gives the store's next search libex's - full timeout instead of 5 seconds. A store libex has not seen yet is slow on - a healthy day, not dead, and this wait tells the two apart. If that search - gets no answer either, the store is dead for the rest of the cycle. The - service then asks libex nothing more about it, and sends its searches - straight to Audible. Any answer, copies or a 404, starts the count over, so - a store that stays slow gets the full timeout once in every four searches. - A store on hold never becomes dead. + row that got no answer, the store's next search is its one inline try of the + cycle, which waits up to 60 seconds instead of 5. A store libex has not seen + yet is slow on a healthy day, not dead, and this wait tells the two apart. + - If the try answers (copies or a 404), the store is slow, not dead. For the + rest of the cycle its searches go on as before, with no further inline + try. + - If it times out or meets a server fault, the store is dead for the rest of + the cycle. The service then asks libex nothing more about it, and sends its + searches straight to Audible. + - If libex rejects it, rate-limits it, or says it cannot reach Audible, the + try decides nothing, and the store's next search is its inline try again. + libex saying it cannot reach Audible puts the store on hold, so its + searches go to Audible while the hold lasts. + + Before the inline try, any answer starts the count over. A 429 never counts + toward it, and a store on hold never becomes dead. - **Only while Audible can be asked.** Once the Audible breaker is open, a search left to Audible alone could only fail. From then on held stores fail - at once, no store is dead (libex is asked about a dead store again), and the - service behaves as with the fallback off. + at once, no store gets an inline try, no store is dead (libex is asked about + a dead store again), and the service behaves as with the fallback off. A cycle that made region searches and got not one answer, from libex or from Audible, is degraded. The cycle log and the pull request body say so, and From b8b9251ac4dddeb7843e4afe3e76b911b952b4a9 Mon Sep 17 00:00:00 2001 From: KodeStar Date: Sat, 14 Nov 2026 13:41:43 +0000 Subject: [PATCH 9/9] Make a store dead when libex cannot reach Audible on its inline try meta-sync now treats libex's 503 upstream_unavailable on a store's inline try as a dead store for the cycle, since a hold alone lapses after 5 minutes and would bring another inline wait through a long outage. Only a refusal or a 429 gives the store its inline try again, and a hold met as the try comes due sends nothing and settles nothing. Part of the Audible search fallback work. --- docs-developers/meta/sync-bot.md | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/docs-developers/meta/sync-bot.md b/docs-developers/meta/sync-bot.md index 66facd4d..8dd4f64d 100644 --- a/docs-developers/meta/sync-bot.md +++ b/docs-developers/meta/sync-bot.md @@ -343,16 +343,18 @@ for as long as libex's `Retry-After` asks, at most an hour. - If the try answers (copies or a 404), the store is slow, not dead. For the rest of the cycle its searches go on as before, with no further inline try. - - If it times out or meets a server fault, the store is dead for the rest of - the cycle. The service then asks libex nothing more about it, and sends its - searches straight to Audible. - - If libex rejects it, rate-limits it, or says it cannot reach Audible, the - try decides nothing, and the store's next search is its inline try again. - libex saying it cannot reach Audible puts the store on hold, so its - searches go to Audible while the hold lasts. - - Before the inline try, any answer starts the count over. A 429 never counts - toward it, and a store on hold never becomes dead. + - If it times out, meets a server fault, or libex says it cannot reach + Audible, the store is dead for the rest of the cycle. The service then asks + libex nothing more about it, and sends its searches straight to Audible. + - If libex rejects it or rate-limits it, the try decides nothing, and the + store's next search is its inline try again. + - If the store is on hold when the try comes due, the service sends libex + nothing and the try decides nothing. The search goes to Audible like any + held store's, and the try waits for the hold to end. + + So a store gets one inline try a cycle, unless libex rejects or rate-limits + it. Before the inline try, any answer starts the count over, and a 429 never + counts toward it. - **Only while Audible can be asked.** Once the Audible breaker is open, a search left to Audible alone could only fail. From then on held stores fail at once, no store gets an inline try, no store is dead (libex is asked about