Skip to content

Latest commit

 

History

History
2017 lines (1671 loc) · 108 KB

known-issues.md

File metadata and controls

2017 lines (1671 loc) · 108 KB

Known Issues and Gotchas

This file is automatically maintained by the reflexion system. When the AI discovers issues, workarounds, or corrections while building applications, they are logged here.

API Gotchas

Trailing Slash Required (v2 only)

All v2 endpoint URLs MUST end with /. Without it, the server redirects (301), causing two API calls to be counted against your rate limit.

v3 Beta Endpoints Must NOT Have Trailing Slashes (discovered 2026-04-02)

Unlike v2, the v3 beta endpoints (/api/v3/beta/...) return 404 Resource not found if the URL ends with a trailing slash. Omit the slash entirely. For example, use /api/v3/beta/modem_software_versions not /api/v3/beta/modem_software_versions/. Non-beta v3 endpoints (e.g. /api/v3/subscriptions/) still accept trailing slashes.

Configuration Manager ID ≠ Router ID

The configuration_managers endpoint has its own id field that may differ from the router's id. Always use get_configuration_manager_id(router_id) to look up the correct config manager ID before making PUT/PATCH calls.

PATCH CAN Remove Config Items — via the Removals List (corrected 2026-08-14)

This entry previously said the opposite. It claimed "PATCH only adds or updates. To remove config items, you must use PUT with the removals list," and a 2026-08-13 caveat built further on that mistake. Both were wrong.

PATCH accepts the removals list, and NCM's own web UI uses it. Removing a single address from a host identity produces this PATCH body — empty updates dict, one removal path:

[
    {},
    [["identities", "ip", "0897fa24-b4a4-4aa2-9cab-0bed9507c233", "members", 2]]
]

Adding or changing addresses in the same identity produces an updates-only PATCH:

[
    {
        "identities": {
            "ip": {
                "0897fa24-b4a4-4aa2-9cab-0bed9507c233": {
                    "members": {"2": {"address": "4.5.6.7"}, "3": {"address": "5.6.7.8"}},
                    "_id_": "0897fa24-b4a4-4aa2-9cab-0bed9507c233"
                }
            }
        }
    },
    []
]

Both slots can be populated in one PATCH: set values via updates and prune via removals in a single call.

Array elements are addressed by integer index in removal paths, not by string key. Note the asymmetry against updates, where the same array position is a string key:

  • update: {"members": {"2": {...}}} — string "2"
  • removal: ["identities", "ip", "<uuid>", "members", 2] — integer 2

Practical consequences:

  • You do not need PUT to delete one branch of a group config. Prefer PATCH with removals — PUT resets unmentioned fields to defaults, which at /groups/{id}/ scope wipes every other group-level setting.
  • UUID-keyed collections (identities.*, lan, vpn.tunnels, security.zfw.zones, ...) can be mirrored exactly between groups. Remove a whole entry with a path ending at its UUID: ["identities", "ip", "<uuid>"].
  • When removing several elements from the same array, emit the indices highest-first. Whether NCM resolves removal paths against the pre-change config or applies them sequentially with reindexing is unconfirmed, and descending order is correct under either interpretation.

Source: NCM UI request traffic reported by a user on 2026-08-14. The formats above are what the UI sends; they have not been independently replayed against the API from this repo.

Arrays in PATCH Replace Entirely

When using arrays (not objects) in a PATCH body, the entire array is replaced. Use objects with string keys for partial array updates.

id Must Be Included Twice

When using a UUID _id_ as the key for an array element, you must also include the _id_ field inside the object body, or you get a validation error.

Passwords Return "*"

Password fields always return "*" on GET. You cannot read existing passwords. Set new passwords by sending cleartext — they are encrypted automatically.

Config Rollback After 15 Minutes

If a device applies a config but can't reach NCM within 15 minutes, it rolls back and suspends sync. Check the "Configuration Rejected" alert for details.

v3 Content-Type

API v3 requires both Content-Type: application/vnd.api+json AND Accept: application/vnd.api+json. Missing either causes errors.

Deprecated: overlay_network_bindings

This endpoint and the routers.overlay_network_binding field were deprecated 09/30/2024. Remove all references.

Deprecated: net_devices.is_upgrade_available

This field still appears in responses but returns inaccurate data since 12/31/2023.


SDK Gotchas

NcmClient Auto-Detection

NcmClient() returns different class instances based on what keys you provide:

  • Only v2 keys → NcmClientv2
  • Only v3 token → NcmClientv3
  • Both → NcmClientv2v3

Environment Variable Loading

The SDK checks CP_BASE_URL for v2 and CP_BASE_URL_V3 for v3 base URLs.

Module-Level SDK Delegation Hangs with v3-Only Credentials (discovered 2026-04-07)

Using the SDK's module-level method delegation (import ncm; ncm.get_routers(...)) auto-initializes a singleton via get_ncm_instance(). If only a v3 token is set (no v2 API keys), the singleton is a NcmClientv3. Calling a v2-only method like get_routers() on it will not raise an error — instead, the v2 __get_json pagination loop sends requests to the v2 base URL without proper auth headers. The SDK's retry adapter (5 retries, exponential backoff on 408/503/504) causes the script to appear to hang indefinitely with no output.

Workaround: in v3-only scripts, do NOT use module-level delegation for v2 methods. Either instantiate NcmClientv3 directly and use v3 equivalents (e.g. get_asset_endpoints() instead of get_routers()), or ensure all four v2 API keys are set in the environment.

SDK _return_handler Error Behavior (discovered 2026-04-07, corrected 2026-08-13)

Correction: this entry previously stated that _return_handler returns error strings like "ERROR: 400: {...}" instead of raising. That is no longer true — the SDK now raises requests.exceptions.HTTPError for 400, 401, 404 and 500. Verified by calling _return_handler directly across status codes:

Status Behavior
200 returns "<obj_type> operation successful."
201, 204 returns the response text
400, 401, 404, 500 raises requests.exceptions.HTTPError as "<code>: <body>"
anything else (403, 409, 429, 418, ...) logs, then returns None

So try/except around SDK calls does catch those four error codes. Two traps remain:

  1. Unhandled status codes return None, silently. 403, 409 and 429 all fall through the else branch — notably 429 (rate limit) and 409 (which v3 uses for both rate limiting and validation errors). Code that assumes a truthy return value or a list will misread these as success.
  2. v2's __get_json still swallows errors independently of _return_handler. It checks if not (200 <= status < 300): break before calling the handler, so a mid-pagination failure exits the loop and returns partial (possibly empty) results with no exception and no indication. An unexpectedly empty or short list from a v2 GET may be a silent error.

Workaround: wrap SDK calls in try/except requests.exceptions.HTTPError (or bare except Exception) to catch 4xx/5xx on writes, and additionally treat None and suspiciously empty list results as failures.

SDK regrade() Missing JSON:API Atomic Extension Header (discovered 2026-04-07)

The regrade() method sends an atomic:operations payload but does not set the required JSON:API atomic extension Content-Type header. Compare with unlicense_devices(), which correctly sets:

Content-Type: application/vnd.api+json;ext="https://jsonapi.org/ext/atomic"
Accept: application/vnd.api+json;ext="https://jsonapi.org/ext/atomic"

The missing header can cause the API to reject the request with a 400 error (e.g. "mac_address must be specified") even when the field is present in the payload, because the server doesn't parse the atomic operations format without the extension header.

SDK regrade() MAC Normalization Only Handles Colons (discovered 2026-04-07)

The regrade() method normalizes MAC addresses by stripping colons only when the input is exactly 17 characters (len(smac) == 17). This misses:

  • Dash-separated MACs (00-30-44-1A-2B-3C, 17 chars) — colons are stripped (finding none), dashes pass through
  • Dot-separated Cisco format (0030.441A.2B3C, 14 chars) — passes through with dots intact
  • Lowercase MACs — passed through as-is (API may expect uppercase)

Workaround: always normalize MACs to bare uppercase hex before passing to regrade(): mac.upper().replace(':', '').replace('-', '').replace('.', '')

v3 Regrades Endpoint Rejects Duplicate MACs in a Single Batch (discovered 2026-04-09)

The POST /asset_endpoints/regrades atomic operations endpoint rejects the entire batch with a 400 Bad Request if the same mac_address value appears more than once across operations in a single request. The error message is: "mac_address values must only occur once". This applies per-request, not globally — the same MAC can appear in separate requests. Always deduplicate MAC addresses within each batch before sending.

v3 Regrades Endpoint Requires Exactly 12 Hex Digits for MAC (discovered 2026-04-09)

The POST /asset_endpoints/regrades endpoint strictly validates that mac_address is exactly 12 hexadecimal characters (uppercase or lowercase, no separators). Any other format — including MACs with colons, dashes, dots, or fewer/more than 12 characters — returns 400 Bad Request with: "mac_address must be 12 digit hexadecimal with optional colons". Despite the error message mentioning "optional colons", bare 12-digit hex is the safest format. Validate with ^[0-9A-Fa-f]{12}$ before sending.


Discovered Issues Log

SDK vs Session Utility Credential Key Names (discovered 2026-03-31)

The NCM SDK and the scripts/utils/session.py utility use different key naming conventions for credentials. Mixing them up causes silent auth failures.

  • SDK (NcmClient): {'X-CP-API-ID': '...', 'X-CP-API-KEY': '...', 'X-ECM-API-ID': '...', 'X-ECM-API-KEY': '...'}
  • Session utility (APISession): cp_api_id='...', cp_api_key='...', ecm_api_id='...', ecm_api_key='...'
  • credentials.py: uses snake_case keys matching the session utility, NOT the SDK

If using get_credentials() with the SDK, you must remap the keys to HTTP header names.

v3 Bearer Token Environment Variable Name Is Standardized as NCM_API_TOKEN (discovered 2026-06-06)

The standard env var name for the v3 bearer token in this project is NCM_API_TOKEN. All scripts have been refactored to use this name. Historical names that may appear in older code or external references:

  • TOKEN — legacy name (removed from scripts)
  • V3_BEARER_TOKEN — Inventory Dashboard legacy (migrated to NCM_API_TOKEN)
  • CP_API_TOKEN — never used by any actual script

When building new scripts/dashboards that need v3, use NCM_API_TOKEN.

__in Filter 100-Value Limit (discovered 2026-03-31)

The NCM API limits __in filter parameters to 100 comma-separated values per request. The SDK auto-chunks these transparently, but if you're making raw API calls with requests or the session utility, you must chunk manually or the API will error.

Venv pip Bad Interpreter After Repo Move (discovered 2026-03-31)

If the repo is cloned or moved to a different path than where the .venv was originally created, the pip (and other scripts in .venv/bin/) will have a stale shebang pointing to the old Python path. Running .venv/bin/pip install ... fails with bad interpreter: no such file or directory. Workaround: use .venv/bin/python -m pip install ... instead, or recreate the venv.

Flask Not Installed Despite Being in requirements.txt (discovered 2026-03-31)

The requirements.txt lists flask but it may not be installed in the venv if dependencies were never fully installed. Always run .venv/bin/python -m pip install -r requirements.txt after cloning or setting up the project for the first time.

Recreating Venv Wipes Injected API Credentials (discovered 2026-03-31)

Running python3 -m venv .venv --clear to fix a broken venv also deletes the API credentials that setup_env.py injected into .venv/bin/activate. After recreating the venv, you must re-run setup_env.py and then source .venv/bin/activate to restore the environment variables. This affects all scripts that rely on check_env().

SDK fields Parameter Not Universally Supported (discovered 2026-03-31)

The NCM API v2 supports ?fields= on most endpoints for partial responses, but the Python SDK only allows fields on methods where it's explicitly in the allowed_params list. Before using fields with any SDK method, check its allowed_params in the source. Methods known to support fields: get_routers(), get_groups() (added 2026-03-31). Methods known to NOT support fields: get_products(), get_firmwares().

v2 Router Relational Fields Are URLs, Not Values (discovered 2026-04-01)

When fetching routers without the fields parameter, relational fields like group, product, actual_firmware, and target_firmware return full API URLs (e.g. https://www.cradlepointecm.com/api/v2/groups/12345/), not names or IDs. To get human-readable values, either use expand=group on the request, or build a lookup dict from a separate get_groups() call. The full_product_name field does return the model name directly (e.g. "AER1600").

Prefer expand=group Over Separate /groups/ Fetch (discovered 2026-04-09)

On large accounts the /groups/ endpoint can return thousands of groups (4000+), requiring many paginated requests that take minutes to complete. Using expand=group on the /routers/ call is dramatically faster because the API inlines the group object (with name, id, etc.) directly into each router response, eliminating the separate fetch entirely. The same applies to expand=account. Always prefer expand over a separate lookup fetch when you only need the related resource's name or ID.

When the group list is the product (measured 2026-09-28). expand=group does not help if you need a group picker rather than group names hung off routers. On a test account with 5,889 groups, get_groups(limit='all') took 66 seconds — sequential pages of 500. Per-group calls afterwards were fast by comparison (net_devices for one group 0.5s, v3 modem_software_versions 1.4s), so the group enumeration is the whole cost.

A UI that blocks on this looks hung for over a minute on first load. Options, none of which the API solves for you:

  • Cache the list to disk and serve it instantly on later loads, refreshing in the background (web_apps/cellular_health_dashboard does this for its dataset via a snapshot_cache.json).
  • Narrow with fields='id,name' to cut payload size — this does not reduce the page count, so expect a modest gain at best.
  • Avoid enumerating at all where possible: if the user already knows the group, take an ID directly, or reach groups via the routers you are already fetching.

Do not assume a group list is cheap because it was fast on a small account; the cost scales with the tenant, not with your code.

v3 Subscriptions Have No status Field (discovered 2026-04-01)

The v3 /subscriptions/ endpoint returns attributes.start_time and attributes.end_time but no explicit status field. To determine if a subscription is active or expired, compare end_time against the current UTC time. The subscription type is in relationships.subscription_type.data.id.

SDK fields Parameter Returns Empty in Flask Context (discovered 2026-04-01)

When using the NCM SDK's get_routers(fields='...') inside a Flask application, it returns an empty list, even though the same call works correctly in a standalone Python script. The root cause is unclear but may relate to how Flask's request context interacts with the SDK's session/urllib. Workaround: omit the fields parameter and fetch full objects instead.

v2 and v3 MAC Address Formats Differ (discovered 2026-04-01)

The v2 /routers/ endpoint returns MAC addresses in the mac field using colon-separated format (e.g. 00:30:44:1A:2B:3C), while the v3 /asset_endpoints endpoint returns them in the mac_address field using bare uppercase hex (e.g. 0030441A2B3C). When joining data across v2 and v3, you must normalize both to the same format (e.g. strip colons/dashes and upper-case) before matching.

v3 Beta Endpoints Use /beta/ Prefix (discovered 2026-04-01, updated 2026-04-02, corrected 2026-09-28)

Several v3 endpoints are served under /api/v3/beta/ rather than /api/v3/. Requests without the /beta/ prefix may 404. Known beta endpoints include exchange_sites. Because these are beta endpoints, their contracts may change without notice. Always check the OpenAPI spec or release notes for the correct path prefix.

Correction (2026-09-28): this entry previously also listed modem_software_versions and modem_upgrades. Those two are not beta — the official Swagger spec documents them at /api/v3/modem_software_versions and /api/v3/modem_upgrades. See "Modem Upgrade Endpoints Are NOT Beta" below. The broader lesson holds: verify the prefix per endpoint against the spec instead of generalizing from a sibling.

net_devices is_asset Filter (discovered 2026-04-01)

The v2 /net_devices/ endpoint supports an is_asset boolean filter. Setting is_asset=true returns only the primary modem interfaces (the physical cellular modems), filtering out virtual/logical interfaces. This is useful when you only need modem-level details (IMEI, ICCID, carrier) and want to avoid processing hundreds of non-modem net_device records.

net_devices Supports expand=router and expand=account (discovered 2026-06-06)

The v2 /net_devices/ endpoint supports expand=router and expand=account. When using expand=router, the router field becomes an inline object with the full router record (id, name, state, mac, group URL, etc.) instead of a URL. This eliminates the need for a separate /routers/ fetch when building per-modem views. Note: if the net_device has no associated router, the field remains null even with expand.

net_device_health Returns Records for All Net Device Types (discovered 2026-06-05)

The /net_device_health/ endpoint returns health records for net_devices of ALL modes (wan, lan, mdm), not just WAN or modem interfaces. When joining health data to net_devices, do NOT filter by mode='wan' or you will silently miss most records. Fetch all net_devices (no mode filter) and join by net_device ID to get complete results.

net_device_health Only Contains Score, Not Signal Metrics (discovered 2026-06-06)

The /net_device_health/ endpoint returns only:

  • cellular_health_category — string: "poor", "fair", "good", or "excellent"
  • cellular_health_score — integer: 0–100
  • net_device — URL reference to the net_device
  • id, resource_url

It does NOT contain signal metrics (RSSI, RSRP, RSRQ, SINR, etc.). For actual signal data, use /net_device_metrics/ filtered by the same net_device IDs. The typical pattern for a cellular health dashboard is:

  1. GET /net_device_health/ → health scores + net_device URLs
  2. GET /net_device_metrics/?net_device__in=id1,id2,... → signal metrics
  3. GET /net_devices/?id__in=id1,id2,... → carrier, model, router ref
  4. GET /routers/?expand=group → device names, groups, state

net_device_health Has No Timestamp Filter — Always a Full Pull (discovered 2026-08-27)

Confirmed against docs/api-v2-full-reference.md and the SDK's allowed_params in ncm/ncm/ncm.py: /net_device_health/ supports only net_device, net_device__in, id__gt/gte/lt/lte, limit, offset. There is no update_ts__gt/lt, created_at__gt/lt, or any other timestamp-based filter — unlike /net_device_metrics/, which does support update_ts__gt/lt.

This means an incremental/delta refresh (only fetch health records that changed since the last poll) is impossible for this endpoint; every refresh must fully re-paginate all records with limit='all'. If you want incremental refreshing for a cellular health dashboard, it can only apply to /net_device_metrics/ (and only if you also accept that health scores/categories stay on a full-pull cadence, since there's no way to ask "what health records changed").

net_devices.connection_state vs routers.state — Different Things (discovered 2026-08-27)

These are easy to conflate when building an "online/offline" indicator for a cellular interface, but they answer different questions:

  • routers.state — whether the parent router/device is online, offline, or initialized with NCM overall.
  • net_devices.connection_state — whether THIS specific network interface (e.g. one modem) is actually connected. Observed values on a live account: connected, disconnected, connecting, standby, standby_connecting, unplugged.

A router can be online while one of its modem interfaces is disconnected, connecting, or in standby — the router itself has connectivity (e.g. via another WAN) even though a given cellular interface does not. Using router.state (or falling back to it) to decide whether a modem interface is "online" overcounts: on one sampled account, 40 of 472 interfaces were router.state=online while connection_state=disconnected, and another 10 were online while connecting/standby/standby_connecting.

For a per-interface online/offline indicator, use only net_devices.connection_state and treat it as binary: connected is online, every other value is offline. Resist the temptation to promote unplugged (or connecting/standby) to a third badge — a device in any of those states is not passing traffic, and a third bucket means summary counts no longer sum to the total, which reads as a bug.

Cellular Health Data Is Per-Modem, Not Per-Router — Router Name Is Not a Key (discovered 2026-08-27)

net_device_health, net_device_metrics and net_devices are all keyed by net_device (one record per network interface), not by router. A router with two modems yields two records, and joining them to routers via expand=router duplicates the router name across those rows.

Consequences when building per-modem views or reports:

  • Do not treat the router name (or router ID) as a unique row key. Sorting or grouping by it alone produces what look like duplicate entries. The unique key is the net_device ID; (router_name, interface_name) is the useful display key.
  • Counts are interface counts, not device counts. "472 records" on an account is 472 modem interfaces across fewer physical routers. Labeling such a total "Devices" in a UI misleads users into thinking they have more routers than they do — name it modems/interfaces.
  • Per-router rollups need explicit aggregation. Any "how many routers are unhealthy" question requires grouping by router ID first and deciding how to combine multiple modems' scores; the raw records cannot answer it directly.

The interface name (net_devices.name) looks like mdm-211c287d, and net_devices.model is the modem model (e.g. Internal 5GE-AM (SIM1)), not the router model — another place where "device" is ambiguous between the two levels.

net_devices Do Not Contain Signal Fields (discovered 2026-06-06)

The /net_devices/ endpoint does NOT return signal strength fields (rssi, rsrp, rsrq, sinr, signal_percent). These fields only exist on the /net_device_metrics/ endpoint. The /net_devices/ response includes modem hardware info: carrier, model, mfg_model, mfg_product, connection_state, imei, iccid, apn, modem_fw, etc.

net_devices.uptime Is Seconds-as-Float, Often Null (discovered 2026-11-02)

The /net_devices/ field table in api-v2-full-reference.md shows uptime with a null sample, which reads as if the field is unusable. It is not: on a live account it holds a float of seconds — the modem interface's connection uptime (e.g. 8912.47 ≈ 2h 28m, 747706.06 ≈ 8.6 days). It is null for many records (modems that have not reported one), so any consumer must handle None. It is a per-net_device (per-modem) value, not a router-level uptime. To display it, floor to an int and format as Xd Yh Zm; render null as a dash. No extra API call is needed — it rides along on the /net_devices/ record a cellular health join already fetches.

net_devices router Field Can Be Null (discovered 2026-06-06)

The router field on /net_devices/ can be null for orphaned or unassigned modems. Code that assumes router is always a URL will fail. Always check for null before extracting the router ID. For display purposes, fall back to hostname or name from the net_device record.

v2 API IDs Are Returned as Strings (discovered 2026-06-06)

Despite being numeric, v2 API responses return id fields as strings (e.g. "67468693" not 67468693). When building lookup dictionaries for joining across endpoints, always normalize IDs to strings. Using int() for lookups will cause key mismatches if the dict was keyed with the raw string from the API response.

net_device_metrics Response Schema (discovered 2026-06-06)

The /net_device_metrics/ endpoint returns:

  • Signal: rsrp, rsrq, rssi, rssnr, sinr, dbm, cinr, ecio, signal_strength (integer 0–100, NOT signal_percent)
  • Cell info: cell_id, mcc, mnc, tac, lac, service_type (e.g. "LTE", "WiFi", "Ethernet", "Not Available")
  • Usage: bytes_in, bytes_out
  • Meta: update_ts, bmask_applied, net_device (URL), id, resource_url

Note: id matches the net_device ID (same value), making joins trivial. Non-cellular interfaces (WiFi, Ethernet) have null signal fields.

expand=router on net_devices Does NOT Expand Nested Relations (discovered 2026-06-06)

When using expand=router on /net_devices/, the inline router object's relational fields (like group, account, product, actual_firmware) are still URLs, not expanded objects. Only one level of expansion is supported. To resolve group names when using this pattern, you must still call /groups/ separately and build a lookup by URL or ID.

v3 API Returns 409 Conflict as a Rate Limit (discovered 2026-04-01)

When making concurrent requests to v3 endpoints (e.g. /asset_endpoints), the API may return 409 Conflict with the message "Conflict with internal rules. i.e. invalid app-key." instead of the expected 429 Too Many Requests. This is effectively a rate limit response. Treat 409 as retryable with exponential backoff, the same as 429. The NCM API enforces a limit of approximately 500 calls per minute across all endpoints.

v2 API Transient 500 "shard-router" Errors (discovered 2026-04-01)

During heavy pagination of v2 endpoints (especially /net_devices/ and /routers/ on large accounts), the API may return 500 Internal Server Error with a body like "cp internal error shard-router-3". This is a transient server-side error, not a client issue. Retry with backoff. Include 500, 502, 503, and 504 in your retryable status code set.

httpx params={} Strips Query Parameters from Pagination URLs (discovered 2026-04-01)

When using httpx (or requests) for pagination, passing params={} (empty dict) to client.get(url, params={}) strips any existing query parameters from the URL. If the meta.next URL is https://...?limit=500&offset=500, passing params={} turns it into https://.../routers/ — causing an infinite loop that re-fetches page 1 forever. Use params=None instead of params={} on subsequent pagination requests where the query parameters are already embedded in the next URL.

v3 asset_endpoint Subscription IDs Are Assignment-Level, Not Parent (discovered 2026-04-01)

v2 and v3 Are Fully Separate API Systems — No Cross-Referencing (discovered 2026-04-28)

The v2 and v3 APIs are architecturally independent interfaces to the same underlying data. They do NOT cross-reference each other:

  • Auth: v2 uses 4 API key headers; v3 uses a single Bearer token.
  • IDs: v2 uses integers; v3 uses strings. They are different ID spaces.
  • Spec: v3 follows JSON:API; v2 does not.
  • Relationships: v2 embeds full URLs to other v2 resources. v3 uses JSON:API relationships blocks with type/id references to other v3 resources only.
  • Expand: ?expand= is v2-only. v3 uses JSON:API ?include= (where supported) or you follow the relationship type/id to the v3 endpoint.

The key implication: v3 relationship references can only be resolved through v3 endpoints. If a related resource doesn't have a v3 endpoint yet, that relationship either won't appear in the v3 response or the data will be flattened into attributes. You cannot use a v2 endpoint to resolve a v3 relationship reference (or vice versa). When migrating scripts from v2 to v3, all related resource lookups must also be migrated to their v3 equivalents.

v2 Router IDs and v3 Asset Endpoint IDs Are Different ID Spaces (discovered 2026-04-07)

The v2 /routers/ endpoint assigns numeric IDs (e.g. 1234567) to devices. The v3 /asset_endpoints endpoint has its own id field that does NOT correspond to the v2 router ID. Passing a v2 router ID to get_asset_endpoints(id=...) (which becomes filter[id]=1234567) will return the wrong device or no results. To cross-reference between v2 and v3, use a shared natural key like mac_address or serial_number instead of either system's internal ID. The subscription_ids on v3 /asset_endpoints are per-device assignment IDs (e.g. 4c03Hj1hALlWWvK), NOT the parent subscription IDs returned by an unfiltered GET /subscriptions (e.g. 55050000002xYV9). These two ID spaces do not overlap. To resolve assignment IDs to full subscription details (name, start/end dates), use GET /subscriptions?filter[id]=<assignment_id>. Multiple IDs can be comma-separated: filter[id]=id1,id2,id3.

v3 /subscriptions/{id} Returns 404 for Assignment IDs (discovered 2026-04-01)

Looking up an assignment-level subscription ID by path (GET /subscriptions/4c03Hj1hALlWWvK) returns 404, even though the ID is valid. Use the query filter instead: GET /subscriptions?filter[id]=4c03Hj1hALlWWvK. This is a JSON:API quirk — the filter endpoint resolves IDs that the direct path lookup does not.

check_env() Prints v2 Errors in v3-Only Scripts (discovered 2026-04-02)

check_env() from utils.env_check always validates v2 API keys (CP_API_ID, CP_API_KEY, ECM_API_ID, ECM_API_KEY). In v3-only scripts that don't need v2 keys, it prints a misleading "Missing required environment variables" message to stderr before calling sys.exit(1). Workaround: suppress stderr and catch the SystemExit:

import io, sys
from utils.env_check import check_env
try:
    _stderr = sys.stderr
    sys.stderr = io.StringIO()
    check_env()
except SystemExit:
    pass
finally:
    sys.stderr = _stderr

Then validate v3-specific vars (NCM_API_TOKEN, etc.) separately.

modem_upgrades POST Returns 200, Not 201 (discovered 2026-04-02)

The OpenAPI spec for POST /api/v3/beta/modem_upgrades documents a 201 Created response, but the API actually returns 200 OK with the created resource in the body. Code that checks for status_code == 201 to confirm creation will incorrectly treat a successful POST as a failure. Check for both 200 and 201.

modem_upgrades Request Type Must Be Collection Name (discovered 2026-04-02)

The OpenAPI spec says the data.type field in POST/PUT request bodies for /api/v3/beta/modem_upgrades should be "modem_upgrade_parent". The API rejects this with a 409 Conflict: "The resource object's type (modem_upgrade_parent) is not the type that constitute the collection represented by the endpoint (modem_upgrades)." Use "modem_upgrades" in the request body instead. Note: the response still returns "modem_upgrade_parent" as the type — the asymmetry is intentional.

v3 409 Conflict Is Overloaded: Rate Limit vs Validation Error (discovered 2026-04-02)

The existing documentation notes that v3 returns 409 Conflict as a rate limit. However, 409 is also used for JSON:API validation errors (e.g. wrong data.type). To distinguish: if the 409 response body contains an errors array, it is a real validation error and should NOT be retried. If it does not contain errors, treat it as a rate limit and retry with backoff.

v2 Time-Series Endpoints Return 409 Without Required Filter (discovered 2026-06-06)

Several v2 time-series endpoints return 409 Conflict if called without a mandatory router or net_device filter. This is NOT a rate limit — it's a validation error requiring you to scope the query to specific device(s).

Endpoint Required Filter
historical_locations router
net_device_signal_samples net_device or net_device__in
net_device_usage_samples net_device or net_device__in
router_logs router
router_state_samples router or router__in
router_stream_usage_samples router or router__in

These endpoints use TimeUUID-based pagination and are optimized for per-device queries. Always provide the filter or you'll get a 409.

Cradlepoint Config .bin Files Use Varying Compression (discovered 2026-05-13)

Cradlepoint router configuration .bin files are compressed JSON, but the compression format varies. Some use standard zlib (wbits=15), others use raw deflate (no header, wbits=-15), and others use gzip (wbits=31). When decoding .bin files, try multiple zlib.decompress(data, wbits) values in sequence: 15, -15, 31, 47 (auto-detect). The first one that succeeds without a zlib.error is the correct format. The decompressed content is always UTF-8 encoded JSON.

Cradlepoint .bin Config Files Contain Both "config" and "state" Keys (discovered 2026-05-13)

When decompressing a Cradlepoint .bin configuration file, the resulting JSON is an array structured as [{"config": {...}, "state": {...}, "fw_info": {...}}, [[...], ...]]. The first element is an object containing "config" (the actual configuration), "state" (runtime state data), and "fw_info" (firmware version). The second element is an array of path references (e.g. ["config", "lan", 0, "devices", 2]) that appear to be deletion/override markers for the config tree.

When working with configs for templating or pushing to devices:

  • Reading: Extract only parsed[0]["config"]
  • Writing: Wrap config as [{"config": <config_obj>}, []] then zlib compress

The .bin files use standard zlib compression (magic bytes 78 9c, wbits=15). Config top-level keys include: certmgmt, container, ecm, identities, lan, routing, security, stats, system, wan, wlan.

Cradlepoint Config JSON Contains Raw Control Characters (discovered 2026-05-14)

Cradlepoint configuration JSON (extracted from .bin files or via API) contains string values with raw newline characters (e.g. embedded YAML in container project configs, PEM certificates). Python's json.loads() in strict mode rejects these as "Invalid control character". Use json.loads(content, strict=False) when parsing Cradlepoint config JSON to allow control characters in strings.

NCM SDK SSL Verification Fails Behind Corporate Proxies (discovered 2026-06-08)

The NCM SDK uses requests.Session() with default SSL verification. Behind corporate proxies that use TLS interception (MITM with custom CA certificates), all API calls fail with SSLCertVerificationError: CERTIFICATE_VERIFY_FAILED. The SDK does not expose a verify parameter, so callers must patch the session directly after constructing the client:

client = ncm.NcmClient(api_keys=api_keys)
client.session.verify = False  # Disable SSL verification

To also suppress the InsecureRequestWarning that urllib3 emits on every request when verify is disabled:

import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

Alternative (proper fix): set the REQUESTS_CA_BUNDLE env var to the path of the proxy's CA certificate bundle:

export REQUESTS_CA_BUNDLE=/path/to/corporate-ca-bundle.crt

This affects ALL scripts and dashboards using the SDK, not just specific apps.

v2 device_apps Returns IDs as Strings, Not Integers (discovered 2026-05-14)

The /device_apps/ endpoint returns the id field as a string (e.g. "83") rather than an integer. This differs from most other v2 endpoints (like /routers/ and /groups/) which return integer IDs. When building lookup dictionaries keyed by app ID — for example, to match device_app_versions back to their parent app — use string keys or cast consistently. Attempting int() conversion on the URL-extracted ID and comparing against the raw response ID will fail silently (lookup miss) if one side is a string and the other is an int.

v2 resource_url Base Hostnames Vary Across Endpoints (discovered 2026-05-14)

The resource_url and relational URL fields in v2 API responses can use different base hostnames depending on the account's regional shard (e.g. https://www.us0.cradlepointecm.com/api/v2/accounts/123/ vs https://www.cradlepointecm.com/api/v2/accounts/123/). This means matching a group's account URL against an account's resource_url by exact string comparison will fail if the hostnames differ. Always extract the numeric ID from the URL path (the last path segment before the trailing slash) and match on that instead of comparing full URLs.

expand=account IS Supported on /groups/ Endpoint (discovered 2026-05-14)

The v2 /groups/ endpoint DOES support expand=account. When used, the account field becomes an inline object with id, name, is_disabled, resource_uri, and account (parent account URL). This eliminates the need for a separate /accounts/ fetch to resolve group account names. Known endpoints that support expand: /routers/ (group, account), /groups/ (account).

Long-Running Servers Need source .venv/bin/activate, Not Just .venv/bin/python (discovered 2026-08-04)

setup_env.py injects real credential values (X_CP_API_ID, X_CP_API_KEY, X_ECM_API_ID, X_ECM_API_KEY) as export statements directly into .venv/bin/activate. If .env is missing or incomplete (e.g. deleted, partial setup), those exported values in activate are the only place the credentials exist.

Calling .venv/bin/python script.py directly executes the interpreter without sourcing activate, so those exported vars are never set in the process environment — even though the venv itself (sys.prefix) is correctly active. For one-shot scripts that call check_env() at startup, this fails loudly and is easy to catch. For long-running servers (FastAPI/Flask dashboards under web_apps/) that read os.environ lazily per-request (e.g. via _build_client() on each API call) instead of validating at startup, the server boots fine and returns 200 on static routes, but every API call silently fails with a missing/empty credential error — there is no startup-time signal that credentials are missing.

Workaround: always launch long-running dev servers with source .venv/bin/activate && python path/to/serve.py, not .venv/bin/python path/to/serve.py, whenever credentials may only be present in the activate script rather than .env. This is safe to run in a single non-interactive shell command (no prompts), unlike setup_env.py with no flags.

v2 /alerts/ Endpoint Does Not Support order_by (discovered 2026-06-26)

The /alerts/ endpoint returns 409 Conflict with "Invalid ordering field specified: created_at" when using order_by=-created_at or any order_by parameter. The endpoint does not support ordering — results must be sorted client-side. The endpoint does support created_at__gt for time filtering, but requires full ISO 8601 format with microseconds and timezone offset (e.g. 2026-06-26T15:00:37.703000+00:00). The alert type field for custom alerts is custom_alert (not custom).

SDK put_group_configuration() Raises AttributeError After a Successful PUT (discovered 2026-08-13)

NcmClientv2.put_group_configuration() calls self.__return_handler(...) with two leading underscores. Inside the class body that name-mangles to self._NcmClientv2__return_handler, which does not exist — the real method is _return_handler (defined on BaseNcmClient). Every other config method (patch_group_configuration, put_configuration_managers, patch_configuration_managers) uses the correct single-underscore name.

Verified by stubbing session.put and calling the method:

patch_group_configuration -> 'Configuration Manager operation successful.'
put_group_configuration   -> AttributeError: 'NcmClientv2' object has no
                             attribute '_NcmClientv2__return_handler'

The critical detail: the HTTP PUT is sent before the exception is raised. The group config is already modified in NCM by the time the caller sees the error, so treating the AttributeError as "the write failed" and retrying will apply the PUT twice.

Workaround: use patch_group_configuration() when a merge is acceptable. When a true PUT is required (the only way to honor the removals list and reset unmentioned fields), issue it directly and skip the wrapper:

resp = client.session.put(
    f'{client.base_url}/groups/{group_id}/',
    data=json.dumps({'configuration': [updates, removals]})
)
resp.raise_for_status()

Credentials Can Drift Between .env and Individual Activate Scripts (discovered 2026-08-14)

setup_env.py stores credentials in two places: .env, which scripts read directly, and the venv activate scripts, which only an activated shell picks up. These can fall out of sync, and each activate script is independent — a venv can have some populated and others not.

Confirmed instance: a venv whose credentials were last written on Jun 24 had all four variables in activate, activate.fish and activate.csh, but zero in Activate.ps1 (no marker comments at all, file untouched since venv creation), and no .env at all. Activate.ps1 injection was added to setup_env.py on 2026-08-04 (8be9817), after that credential run — so any venv whose credentials predate a change to the injection code will be missing whatever was added since.

Why it bites: nothing fails at setup time. A bash/zsh user sees a working environment indefinitely. The failure lands on whoever uses a different shell — a PowerShell user gets an activated venv with no credentials and no warning — or on anyone running .venv/bin/python directly when .env is absent, which resolves zero of four keys.

Diagnose with setup_env.py --check, which reports .env presence and per-script credential status:

[warn] .env: not found - only activated shells will have credentials
[warn] activate scripts without all credentials: Activate.ps1 (none)

Fix by re-running setup_env.py --credentials-only (prompts, hidden input), which rewrites .env and every activate script with the current code. From a shell that already has the credentials exported, setup_env.py --skip-credentials achieves the same thing non-interactively by reusing what is in os.environ.

update_activate_scripts() now reports every script by name with its outcome, reads the values back to confirm they landed, and warns when a script that exists did not receive them, so this drift is visible at write time rather than silent. Absence of activate.bat on macOS/Linux is expected — python -m venv only creates it on Windows.

Web App Settings Modals Are a Third Credential Store (discovered 2026-08-14, tracking resolved 2026-08-14)

Credentials live in more places than .env and the venv activate scripts. Every web app with a Settings modal writes what you type into a per-app JSON file beside its serve.py, in plaintext:

  • profiles.json — named credential profiles (alert_dashboard, cellular_health_dashboard, geo_ip_blocker, inventory_dashboard)
  • config.json — active credentials plus saved job state (host_identity_copier)

Consequences worth knowing:

  • These files survive rm -rf .venv, so an app can keep authenticating after the venv-injected credentials are gone. Convenient, and easy to forget when you think you have removed every copy of a key.
  • They are a plaintext credential store in the working tree. Apps that write them should chmod 0600 on POSIX; not all do.

.gitignore does not retroactively untrack — the mechanism, kept because it will recur. web_apps/*/config.json and web_apps/*/profiles.json were added to .gitignore on 2026-08-13. That stops new files being committed but has no effect on files already in the index, so web_apps/cellular_health_dashboard/profiles.json and web_apps/inventory_dashboard/profiles.json stayed tracked and unprotected. Fixing that requires git rm --cached <path>; after that the pattern takes over.

Resolved 2026-08-14. No profiles.json is tracked, present in HEAD, or on disk anywhere in the repo. The two above were removed in commit 364fbff. A third, web_apps/geo_ip_blocker/profiles.json, appeared later the same day holding four real key values; it was never tracked (the pattern worked, because it was new) and was deleted after confirming all four values were byte-identical to .env, so the app keeps working from environment variables.

Do not read the resolution as "this cannot happen again." Any app with a Settings modal recreates its profiles.json the moment someone saves a profile. Deleting the file is not the protection — the .gitignore pattern is, and it only works for files that are not already tracked. Before committing, git status is the check that matters.

Committed history is clean — do not rotate on this basis. An earlier version of this entry claimed the values were already in committed history and the keys should be rotated. That was wrong, and it is an expensive thing to get wrong. Verified 2026-08-14:

  • only one commit (426d2fa) ever touched either path, and the blob is {} in both
  • git log --all --full-history -p -- <both paths> adds no credential-valued lines
  • every one of the 645 blobs across all 262 commits was searched for the four exact secret strings from the working-tree file: zero matches

The live keys existed only in the working tree, never in a commit. The risk was prospective — a git add -A or git commit -a would have committed them — not historical. No rotation was needed, and none was done.

Diagnostic subtlety: git check-ignore lies about tracked files. It skips paths present in the index and reports them as not-ignored, which reads as "your pattern is broken" when the pattern is fine. Separate the two questions:

git check-ignore -v --no-index <path>   # is the PATTERN right?      -> yes, matches line 13
git ls-files -c <path>                  # is the file TRACKED?       -> yes, that's the problem

Checking only the first form is what makes a tracked credential file look protected. Also note git check-ignore needs git on PATH; if it is missing, the command fails and a naive if git check-ignore -q "$f" shell test silently reports every path as unignored.

When adding a Settings modal to a new app: write to a gitignored filename, set 0600 on POSIX, mask secrets on read-back (return "***" rather than the value), and never echo values into logs or responses.

Bare setup_env.py Hangs From an Agent Tool Call — Tool Calls Get a TTY (discovered 2026-08-14)

Interactivity in setup_env.py is gated on one line:

interactive = sys.stdin is not None and sys.stdin.isatty() and not args.skip_credentials

Measured from inside an agent tool call in this repo:

stdin isatty: True
stdout isatty: True

So the bare form takes the interactive branch and blocks in getpass. Nobody is on the other end of that tty — command output is only returned to the agent when the command exits, so the user never sees the prompt and cannot answer it. The run hangs until the tool timeout. It does not skip gracefully.

The misleading evidence that makes this easy to get backwards. Running setup_env.py --skip-credentials prints:

[warn] Skipping credentials (non-interactive mode)

That message proves nothing about isatty. The flag sets interactive to False by itself through the and not args.skip_credentials clause, so the non-interactive branch is taken whether or not a terminal is present. Do not read it as evidence that tool calls lack a tty. They have one.

The module docstring's "safe to run from automation and from Kiro hooks" is correct for hooks but should not be extended to agent tool calls.

Hook command actions are the opposite case: they receive the session JSON piped on stdin, so stdin is a pipe, isatty() is False, and the non-interactive branch runs. A hook therefore will not hang — but it also collects nothing, beyond reusing whatever os.environ already holds. (Reasoned from the hook contract; the tty result above is measured, this is not.)

Consequence worth stating plainly: credentials cannot be collected by automation at all. Not from a tool call, not from a manual hook. The working paths are:

  • a human terminal, including the IDE's integrated terminal: setup_env.py --credentials-only (input hidden by getpass)
  • a shell that already exports the four variables, then setup_env.py --skip-credentials, which reuses os.environ non-interactively
  • a web app's Settings modal — subject to the plaintext-storage caveats in the entry above

This is a useful boundary rather than only a limitation: any channel an agent could prompt through would place API keys in the agent's context and in the session transcript. The isatty gate is what keeps them out.

Repo Layout Claims in Steering Are Stale — Wrong Entry Points, Dead fileMatch Patterns (discovered 2026-08-14)

Several always-loaded and fileMatch steering claims describe a layout the repo no longer has. These matter more than ordinary doc rot, because agents act on them without checking.

1. Not every web app has a serve.py. project-setup.md and AGENTS.md both stated "Each has a serve.py and a fixed port." That is false for five of the twelve apps in web_apps/:

Entry point Apps
serve.py inventory_dashboard, cellular_health_dashboard, alert_dashboard, geo_ip_blocker, host_identity_copier, assign_sdk, web_app_template
config_builder.py config_builder
script_manager.py script_manager
ncm_api_key_encryptor.py ncm_api_key_encryptor
router_lookup.py netcloud_router_lookup
app.py cisco_to_cradlepoint_zfw_converter

web_apps/README.md compounded it with a generic python3 web_apps/<app_name>/serve.py. Run ls web_apps/<name>/ before launching. Both files now carry the table.

2. ncm-api-development.md never loaded for web app code. Its pattern was:

fileMatchPattern: "{scripts/**/*.py,ncm/**/*.py,ncm2/**/*.py,dashboards/**/*.py}"

dashboards/ does not exist, and web_apps/**/*.py was absent — so the endpoint routing table, the trailing-slash rule, the pagination and deprecation rules did not auto-load when editing web app Python, which is where most of this repo's API calls live. AGENTS.md meanwhile advertised the section as applying to web_apps/, so the mirror promised coverage the Kiro config did not deliver. Pattern corrected to {scripts/**/*.py,ncm/**/*.py,ncm2/**/*.py,web_apps/**/*.py}.

Worth generalizing: a fileMatchPattern naming a directory that does not exist fails silently. Nothing warns you; the steering simply never activates. When adding or renaming a top-level directory, grep .kiro/steering/*.md front matter for the old name.

3. dashboards/cellular_health/ does not exist. web-ui-standards.md pointed its dashboard reference implementation at that path; the real one is web_apps/cellular_health_dashboard/. Its own fileMatchPattern also carried a dead **/dashboards/** clause. Both fixed.

4. scripts/script_manager/csv_files/ does not exist. code-standards.md and AGENTS.md both instructed storing CSV exports there. The real path is web_apps/script_manager/csv_files/. Both fixed. The same stale path was also sitting in .gitignore, protecting nothing while the real csv_files/ went unignored; removed 2026-08-14.

5. ncm2/ is untracked. It is referenced by ncm-api-development.md's fileMatchPattern and by AGENTS.md, but git ls-files ncm2/ returns zero files while ncm/ returns eleven. It exists only in this working tree, so a fresh clone will not have it. Do not import from ncm2 in committed code, and do not cite it in docs as though contributors have it. Use ncm/ — that is the packaged SDK.

6. web_apps/mac_whitelist_copier/ is an empty directory. Zero files, so it is not an app and git does not see it at all (git does not track empty directories). Excluded from the README tables.

7. Entry-point docstrings drift too. web_apps/alert_dashboard/serve.py's docstring told you to run web_apps/custom_alert_dashboard/serve.py and open port 8060; the directory is alert_dashboard and PORT = 8065. Fixed 2026-08-14, and its usage line now uses the activate form rather than .venv/bin/python, since a long-running server launched the bare way boots fine and then fails every API call. The other eleven entry points were swept the same day — docstring path and port now match source in all twelve, so treat a mismatch as a regression rather than the norm.

api-v2-full-reference.md Query-Param Tables Can Be Incomplete — Cross-Check ncm.py (discovered 2026-08-28)

The generated tables in docs/api-v2-full-reference.md are not guaranteed to list every filter an endpoint accepts. Read them as a floor, not a ceiling.

Confirmed instance: locations. Its table listed only id, id__in, limit and offset — no router or router__in. Meanwhile the shipped client and two shipped scripts depend on a router filter:

  • ncm/ncm/ncm.py → get_locations() declares allowed_params = ['id', 'id__in', 'router', 'router__in', 'limit', 'offset']
  • scripts/export_locations.py and web_apps/script_manager/scripts/Export Locations.py both batch with n2.get_locations(router__in=batch)

Every other router-scoped endpoint in the same reference (configuration_managers, device_app_states, net_devices, router_alerts, router_state_samples, router_stream_usage_samples) does document router__in, so locations was the outlier rather than the rule. The two rows have been added to the reference with a note.

Why it costs you: a reader who trusts the table concludes there is no server-side router filter for locations, and fetches the whole collection to filter client-side. On a large account that is many paginated requests instead of a batched query.

Verification status. The discrepancy between the doc and the shipped code is directly observed. Whether router__in behaves as expected against a live account is UNVERIFIED — no request was made to /api/v2/locations/ when this was written. Also UNVERIFIED: whether any endpoint other than locations has an incomplete table. One instance was found; the rest were not audited.

Practical check before concluding a filter does not exist: grep the matching method's allowed_params in ncm/ncm/ncm.py, and grep scripts/ and web_apps/ for existing callers. Working code that predates the generated doc is the stronger evidence.

v2 List Endpoints Return Duplicate Records — /firmwares/, /products/ (discovered 2026-09-25)

Paginating the full /api/v2/firmwares/ collection yields substantially more rows than there are distinct firmware records. Measured against a live account with limit=500: 20,000+ rows returned for 17,677 unique id values. Duplicates also show up inside a single small page — ?limit=3&version=7.24.40 returned firmware id 9395 twice in three rows.

Always dedupe by id when accumulating pages:

records = {}
for page in pages:
    for item in page:
        records[str(item["id"])] = item   # last write wins; records are identical
firmwares = list(records.values())

Why it costs you: a naive count = len(all_rows) overstates the catalog by ~15%, and any per-record work (HEAD requests, hashing, table rows) is done several times over. A UI that renders the raw rows shows visibly duplicated entries.

It is not only /firmwares/, and not only across pages. /products/ does the same thing within a single response: one ?limit=500 request returned 154 rows for 102 unique ids on one page. So "duplicates come from pagination" is the wrong mental model — treat any v2 list response as a multiset and dedupe by id as a habit, not just on large collections.

Verification status. Directly observed against a live account on 2026-09-25 for /firmwares/ (full pagination and a single filtered page) and /products/ (single page). Whether the duplication is stable (same ids duplicated on every pull) is UNVERIFIED — dedupe unconditionally rather than relying on a fixed pattern. Only these two endpoints were checked; others may or may not behave the same way.

The Same Model + NCOS Version Has Many Distinct Firmware Builds (discovered 2026-09-25)

model + version does not identify a firmware image. NCM publishes multiple distinct builds under one version string, each with its own id, built_at, and hash. Measured on a live account: 3,426 of 5,389 model/version pairs — about two thirds — had more than one distinct hash, some with six builds.

IBR600C  7.26.100  built 2026-09-16T00:34:43Z  hash d53a0427…
IBR600C  7.26.100  built 2026-09-21T17:37:17Z  hash 85b622ca…

Why it costs you: any code that keys firmware by f"{model}-{version}" — cache keys, download filenames, dedupe sets, "do I already have this?" checks — silently collapses several different images into one. Downloading "all builds of 7.26.100" to IBR600C-7.26.100.bin writes one file and silently drops the rest.

built_at does not break the tie either. Distinct images regularly share an identical build timestamp, so model + version + built_at is still not unique: 2,166 of 15,505 (model, version, built_at) triples mapped to more than one hash — 14% — with up to four images on a single triple.

AER2200  7.1.20  built 2019-08-27T15:05:37Z  id 1654  hash c1a467895797…
AER2200  7.1.20  built 2019-08-27T15:05:37Z  id 1661  hash befd3dafac2b…

The practical consequence: "take the newest build" is not always a decision. For 681 of 5,389 model/version pairs, the newest built_at still maps to several different images, so a max(built_at) pick silently chooses arbitrarily between them. hash (or id) is the only field that reliably separates them.

Use id as the identity, and where a human-readable name is needed, disambiguate with built_at and a short hash prefix — the hash is the part doing the real work:

groups = {}
for fw in firmwares:
    groups.setdefault((fw["model"], fw["version"]), set()).add(fw["hash"])

def filename(fw):
    base = f"{fw['model']}-{fw['version']}"
    if len(groups[(fw["model"], fw["version"])]) > 1:
        return f"{base}-{fw['built_at'][:10].replace('-', '')}-{fw['hash'][:6]}.bin"
    return f"{base}.bin"

Also note built_at is UTC. If you bake the build date into a filename, render the date in UTC in any accompanying UI too, or a user in a negative offset sees "Sep 15" next to a file named …20260916….

If a UI shows these rows, show the hash (first 7 chars is enough). Otherwise rows with the same model, version, and build date look like duplicated data and users reasonably report it as a bug.

What the pairs actually are: multiimage builds. The distinguishing information is in the url filename, not in any metadata field. NCM publishes most builds twice — a standard image and a multiimage one — sharing the model, version, and build timestamp:

/E3000-multiimage-2026-09-16T00%3A39%3A53.bin    hash 83e64d1b…
/E3000-2026-09-16T00%3A39%3A53.bin               hash 31e827b0…

Both point at product 56 (E3000), same version, same built_at. This accounts for the collisions exactly: 1,963 of 17,677 records carry the multiimage tag, and 2,166 (model, version, built_at) triples have both a standard and a multiimage image.

So parse the variant out of url and treat it as part of the identity:

def image_variant(url):
    return "multiimage" if "-multiimage-" in urllib.parse.unquote(url).lower() else ""

Two consequences:

  • Surface the variant in any UI. Without it the rows look like duplicated data.
  • If you parse the model from url as a fallback, strip the variant tag first, or you invent a phantom model named E3000-MULTIIMAGE.

Verification status. Directly measured across a full 17,677-record catalog on 2026-09-25, including the triple collision counts and the multiimage correspondence. What multiimage means functionally (dual-boot slots, combined hardware variants, something else) is UNVERIFIED — the API exposes no field describing it, and it was not tested against a device. Check NCM release notes before assuming which variant a deployment needs; don't automate that choice on the strength of the API alone.

Firmware url Is Not Always a Path, and Not Always .bin (discovered 2026-09-25)

Code that does CDN_BASE + firmware["url"] and names the result .bin is wrong for a minority of records. Across a 17,677-record catalog:

url shape Count Example
Path, standard image 15,231 /170900-022-2025-08-21T14:26:57.bin
Path, multiimage variant 1,955 /AER2200-multiimage-2019-08-27T15:05:37.bin
Path, not .bin 450 /cr4250-core-image-cr4250-upgrade-1.0-20190827161513.vmp
Fully-qualified URL 41 https://d251cfg5d9gyuq.cloudfront.net/CBA250-2012-04-16T11%3A00%3A57.bin

Three separate traps:

  1. 41 records are absolute URLs. Prepending the CDN base produces https://…cloudfront.nethttps://…, which fails. Worse, a guard like if not url.startswith("/"): continue skips them silently. Normalize instead:

    def image_url(url):
        if url.lower().startswith(("http://", "https://")):
            return url
        return FIRMWARE_CDN + url
  2. A second CDN host appears, and it is dead. Of those 41, 22 point at d30wehszhs5ixg.cloudfront.net, which does not resolve (DNS NameResolutionError, tested 2026-09-25) — so those images are unobtainable regardless of client code. The other 19 use the usual host and download fine. Report such failures rather than hiding them; a user needs to know the image is gone, not watch a download silently not happen.

  3. 450 records are .vmp, not .bin (CR4250). Derive the extension from the URL rather than hardcoding it, or files land on disk misnamed.

Also worth knowing: individual CDN objects can return 403 even when sibling objects for the same model and version return 200 (observed on one CR4250 .vmp). Treat a per-image HTTP failure as normal and surface it per row.

Verification status. Counts measured directly against a live catalog on 2026-09-25. The dead host, the working absolute-URL downloads (SHA-1 verified), and the 403 were all confirmed with live requests. Whether the dead host affects other accounts, and whether d30wehszhs5ixg was ever reachable, is UNVERIFIED.

/firmwares/ Has No Product or Model Filter (discovered 2026-09-25)

allowed_params for /firmwares/ is id, id__in, version, version__in, limit, offset — there is no product or product__in. Listing the firmware available for one model therefore requires pulling the whole collection and filtering client-side; filtering by version server-side is the only narrowing available.

The model name is also not on the firmware record. product is a URL (.../api/v2/products/63/), so resolving names means a separate /products/ pull (one page, fast) and a join on the extracted id. That join is authoritative when it resolves — but you need both a fallback and a dedupe step:

  • /products/ repeats records too, and not as a pagination artifact: a single ?limit=500 response returned 154 rows for 102 unique ids. Key your lookup dict by id (idempotent) rather than appending to a list and trusting the length. Unique ids and unique names both came to 102, so no two distinct products shared a name.
  • Not every referenced product exists in /products/. Across a full catalog, 12 of the 113 product ids referenced by firmware records were absent, covering 274 records (1.6%) — ids 103, 99, 78, 77, 55 and others, presumably retired products. Those firmwares get no name from the join, so a fallback is genuinely required; just don't reach for it first.

Parsing the model out of url looks like an equivalent shortcut, since most paths are shaped /{MODEL}-{built_at}.bin (e.g. /IBR600C-2026-09-16T00%3A34%3A43.bin). It is not. Measured against a 17,677-record catalog, a regex anchored on the ISO build timestamp:

  • fails entirely for 491 records (2.8%) — the 450 CR4250 .vmp paths use a compact timestamp (…-20190827161513.vmp) and the 41 absolute URLs don't match the leading /{MODEL}- shape at all;
  • returns a different, legacy name for 9 records — /NCSG-KVM-… and /NCSG-… for products actually named NCXSG-KVM and NCXSG-Azure. The URL keeps the old NCSG branding, so a URL-derived catalog splits one model across two names;
  • needs the variant stripped first, or multiimage builds produce a phantom MODEL-MULTIIMAGE model (see the multiimage entry above).

So: join on product first, fall back to URL parsing only for the ~1.6% of records whose product id is missing, and strip the variant tag when you do.

Verification status. Parameter list confirmed against docs/api-v2-full-reference.md, get_firmwares() in ncm/ncm/ncm.py, and a live account on 2026-09-25. The parse-failure and name-mismatch counts were measured directly against the same catalog. Whether an undocumented product filter is silently accepted was not tested.

Modem Upgrade Endpoints Are NOT Beta — Correcting This File (corrected 2026-09-28)

The /beta/ prefix entry above is wrong for these two endpoints. The official OpenAPI spec at https://developer.cradlepoint.com/swagger/spec/Modem_Software_Upgrades.json documents them without the beta segment:

GET  /api/v3/modem_software_versions
GET  /api/v3/modem_upgrades
POST /api/v3/modem_upgrades
PUT  /api/v3/modem_upgrades/{id}

Three sources in this repo say otherwise and should not be trusted on this point: the Postman collection (whose folder is even titled "Modem Software Upgrades (BETA)"), scripts/test_modem_management_api.py, and the /beta/ prefix entry earlier in this file. The ncm SDK's live methods (ncm/ncm/ncm.py L4515-4790) use the non-beta path and are correct; its commented-out older methods (L5787+) use the beta path and are stale.

The likely history: the endpoints were beta when the Postman collection and the test script were written, and were promoted without those artifacts being updated. Treat the Swagger spec as authoritative and the /beta/ prefix as a fallback for older deployments. web_apps/modem_upgrades requests the non-beta path first and falls back to beta on a 404.

modem_upgrades Uses JSON:API filter[...], Not Postman's group/job_type (corrected 2026-09-28)

The Postman collection documents the list parameters as plain group and job_type. The Swagger spec documents them as JSON:API filters, and marks the first two required:

Parameter Required Notes
filter[group] Yes integer group ID
filter[type] Yes modem_upgrade_parent | modem_upgrade_child
filter[modem_upgrade_parent] Only when filter[type]=modem_upgrade_child parent job ID

scripts/test_modem_management_api.py gets this right. Postman's job_type spelling appears nowhere in the spec — do not use it.

Note that filter[type] being required means there is no "just list everything for this group" call; you must decide up front whether you want jobs or per-modem activities.

modem_upgrades Paging Is Cursor-Based and page[size] Caps at 50 (discovered 2026-09-28)

Both modem endpoints page with page[size] (default 20, maximum 50) plus page[after] / page[before] cursors, and return links.first/last/next/prev. Asking for more than 50 is a 400 Bad Request, which is easy to trip if you carry over the v2 habit of limit=500. Follow links.next until it is null rather than computing offsets.

Both endpoints also return 204 No Content — not an empty 200 — when nothing matches ("Could not find available modem packages" / "No jobs or activities found"). Code that assumes a JSON body on any 2xx will throw on the empty response; treat 204 as an empty result set.

modem_software_versions Field Is package_version, Not version (discovered 2026-09-28)

The available-firmware field is package_version. The SDK's disabled older code and several plausible guesses use available_version or version, which are not in the spec. A reader looking for the wrong key silently gets a blank version column rather than an error.

Full attribute set for modem_software_versions:

Attribute Meaning
carrier carrier name, e.g. verizon, att, sprint
modem_type_name modem module type, e.g. LP4
package_version the modem firmware package version
min_router_version minimum NCOS version the router must run to accept it

min_router_version is worth surfacing in any upgrade UI: a modem package can be available to a group while some routers in that group are on too old an NCOS build to take it.

carrier + modem_type_name from a record here are exactly the two required attributes for POST /modem_upgrades, so the versions endpoint is the intended way to populate an upgrade request.

Modem Upgrade Jobs Cannot Target Individual Routers (discovered 2026-09-28)

Confirmed against the Swagger spec: POST /modem_upgrades requires data.relationships.group and accepts no other target. There is no router, net_device, or ID-list attribute anywhere in ModemUpgradeRequest, so a job necessarily applies to every matching modem in the group.

This is easy to get wrong when building tooling: a UI that lets an operator select specific routers and then submits a group-scoped job will upgrade far more devices than the operator chose. Either scope the UI to groups, or say plainly that a device-level selection is a display filter only. To genuinely narrow the blast radius, the devices have to be moved into their own group in NCM first.

The only in-request narrowing levers are connection_states (e.g. restrict to ["connected"]) and overwrite (whether to re-push to modems already on the target version).

Modem Upgrade Progress Lives in Child Records, Not the Parent Job (discovered 2026-09-28)

modem_upgrades is a parent/child collection. filter[type]=modem_upgrade_parent returns jobs; filter[type]=modem_upgrade_child (optionally narrowed with filter[modem_upgrade_parent]={id}) returns one record per modem, and those children are where per-device status lives.

Worse, there is no documented GET /modem_upgrades/{id}. The spec defines only the filtered list and PUT /modem_upgrades/{id}; the SDK's get_modem_upgrades( modem_upgrade_id=...) builds an item-read path that appears nowhere in the spec. To read one job's current state portably, list the group's parents and match on ID, or attempt the item read and fall back to the list on a 404.

Consequences for a monitor:

  1. Fetch children on every poll, not just the parent.
  2. Do not rely solely on the parent's status. Parent counters (modem_count, success_count, failed_count) are documented but not always populated. Treating "every child is terminal" as an additional completion signal covers the gap — and a wall-clock timeout is still required, since a job can stall indefinitely on a modem that never reports back.

The status enum is documented, and none of the values are the ones you would guess:

Status Terminal? Meaning
created no job accepted
queued no waiting to start
initializing no starting up
running no in flight
finished yes completed
error yes failed
canceled yes cancelled (one l)

Note finished rather than complete/completed, error rather than failed, and the single-l American canceled. Code that matches on completed or cancelled will never fire. Since this is a young endpoint family, matching the documented values exactly and falling back to substring classification for anything unrecognized is the durable approach.

Modem Upgrade Child Records Are Under-Documented — Expect Only net_device IDs (discovered 2026-09-28)

The Swagger spec reuses a single schema (ModemUpgradeResource) for both parent jobs and child activities, and that schema documents only a group relationship and job-level attributes. So the spec does not describe how a child identifies the modem it belongs to, even though that is the whole point of a child record.

In practice children reference a modem by net_device ID, and the SDK's disabled older get_group_modem_upgrade_device_summary lists router_name, net_device_name and current_version as available fields — suggesting the API returns more than the spec admits. Because the shape is not pinned down, extract these fields defensively (probe relationships.net_device, then attributes.net_device_id, then attributes.net_device) rather than assuming one location, and keep the raw attributes dict so unexpected fields are still visible.

A bare net_device ID tells an operator nothing, so an upgrade report built from v3 data alone reads as a list of opaque integers.

Resolve them through v2: GET /net_devices/?id__in=<ids>&expand=router returns the inline router object (name, state, model) plus the modem's version/ver_pkg, which is the "firmware before the upgrade" column a report needs. Chunk the IDs at 100 per call and cache the results — the mapping does not change during a job.

Note the two APIs must both be authorized for this: v3 Bearer token for the upgrade records, v2 header keys for the name lookup. An app that resolves names needs both credential sets, which is unusual — most tools in this repo need only one.

A modem_upgrades Preview Returns a COUNT, Not a Modem List (discovered 2026-09-28, verified live)

This is the most consequential thing to know before building anything on operation: "preview", and it is not in the spec.

A preview job never creates child activity records. Verified against a live account: after POST /modem_upgrades with operation: "preview", the child query

GET /api/v3/modem_upgrades?filter[group]=636741
    &filter[type]=modem_upgrade_child&filter[modem_upgrade_parent]=67683780

answers 200 with {"links":{...},"data":[],"meta":{...}} — an empty array, and it stays empty indefinitely (polled well past 100 seconds on a job NCM had already finished evaluating). The entire result of a dry run is the parent's modem_count.

Sequence observed:

  1. POST returns immediately: status: "created", modem_count: null.
  2. Within about a second NCM evaluates the group, sets modem_count, and settles status at "preview".
  3. No children, ever.

So a UI that promises "here are the modems this would touch" cannot get that from the API. Options: show the count as authoritative and derive a candidate list locally from v2 net_devices (clearly labelled as derived — see below), or show only the count.

Do not build a long poll waiting for preview children. It will spin until its timeout on every run. Poll only until modem_count stops being null.

A Preview's modem_count Will Not Match a Naive Local Match (discovered 2026-09-28)

If you do derive a candidate list locally, expect it to disagree with NCM. On a group of 4 online E320s with 8 5GB modems, all on older TMO firmware and none on the target generic package, NCM reported 3 eligible while matching on modem type + "version differs" yielded 8.

The eligibility rule is not documented and clearly involves more than type and version — plausibly min_router_version against each router's NCOS build, SIM or carrier state, or per-router constraints. Do not present a locally derived list as the scope of the job. Treat modem_count as authoritative and label anything you compute yourself as indicative.

Related: a group whose router is offline reported modem_count: 0 even with overwrite: true, so offline devices appear to be excluded outright.

status: "preview" Is Missing From the Documented Enum (discovered 2026-09-28)

The spec's status enum is created, queued, initializing, running, error, finished, canceled. A live preview job settles at status: "preview", which is not in that list.

Two lessons: the enum is incomplete, so match the documented values but always have a fallback branch for unrecognized ones; and "preview" is a settled state rather than an in-flight one, even though nothing about the string says so. Code that waits for a terminal status on a preview job will wait forever.

GET /modem_upgrades/{id} Returns 403, Not 404 (discovered 2026-09-28)

The spec defines no item-read endpoint, and the live API refuses it with 403 Forbidden rather than 404 Not Found. The ncm SDK's get_modem_upgrades(modem_upgrade_id=...) builds exactly this URL, so that call path cannot work.

Two traps follow:

  • Do not treat 403 as "bad token" on v3. The obvious handler — map 401/403 to "check your credentials" — produces a completely misleading error here, since the same token succeeds on the list and on POST. On these endpoints 403 means "this operation is not available", and the spec lists 403 Forbidden - Request forbidden as a normal response for every documented operation.
  • A fallback keyed on 404 will not fire. Anything falling back from the item read to the filtered list must trigger on 403 as well.

Read a single job with filter[group] + filter[type]=modem_upgrade_parent and match the ID client-side.

filter[modem_upgrade_parent] Is Enforced, With a Good Error (discovered 2026-09-28)

Requesting children without a parent gives a genuinely helpful 400:

{"errors":[{"detail":"filter[modem_upgrade_parent] is required when filter[type]=modem_upgrade_child",
            "status":"400","source":{"pointer":"/data"},"code":"invalid"}]}

Worth knowing because it is easy to destroy this message. If your client retries a 400 using a different parameter spelling — e.g. falling back from filter[type] to Postman's job_type — the retry replaces the clear message above with a baffling one about a field you never sent:

HTTP 400: This field may not be null. (/data/attributes/job_type)

Keep parameter-style fallbacks narrow: only retry when the error actually names an unknown or invalid parameter, and never when it names a missing required one. A broad "any 400 means try the other spelling" rule turns good diagnostics into bad ones. (The job_type pointer does reveal that the server's internal serializer field is named job_type, which is presumably where Postman's spelling came from — but it is not accepted as a query parameter.)

When implementing that rule, do not use "filter" as the discriminator. It is the obvious choice and it is worthless: the word appears in both the error you want to retry (invalid filter: group) and the one you must not (filter[...] is required), so matching on it reintroduces the exact bug. Two things that do work: match only unambiguous unknown-parameter wording (unknown, unrecognized, unsupported, not a valid), and explicitly bail out when the message contains required. Also only consider a fallback at all while the correct spelling is still undetermined — once a working form is known, a 400 is real and should surface as-is.

Empty v3 Collections Come Back as 200 With data: [], Not Always 204 (discovered 2026-09-28)

The spec documents 204 No Content for "no jobs or activities found" on modem_upgrades. In practice an empty child query returned 200 with "data": [] plus a populated links/meta envelope. Handle both: check for a 2xx, then treat a missing or empty data as an empty result rather than keying off the status code. Responses also carry meta.pagination.before/after alongside links, which the spec does not mention.

connection_states Accepts Fewer States Than Modems Actually Report (discovered 2026-09-28)

The connection_states filter on POST /modem_upgrades restricts which modems a job targets, by each modem's own cellular connection state. Its enum has three values:

connected | disconnected | standby

But the v2 net_devices.connection_state field those modems report has six observed values (see the net_devices.connection_state vs routers.state entry above): connected, disconnected, connecting, standby, standby_connecting, unplugged.

So connecting, standby_connecting and unplugged cannot be expressed in the filter, and how NCM treats modems in those states is unknown — whether they are excluded outright, folded into disconnected, or ignored by the filter entirely. This was not tested: every preview run during this work sent connection_states: null. Also untested is whether the filter is evaluated once at job creation or re-checked per modem as the job progresses, which matters because a modem's state can change mid-job.

Two practical consequences:

  • A UI offering exactly the three enum values is correct per the spec but cannot express "target the modems that are currently connecting". Don't silently map a sixth-value state onto one of the three — you would be guessing at NCM's rule.
  • Leaving the filter null (the default) is the safer choice when you want NCM's own eligibility rules to decide, which is also the only behavior verified here.

Beware the related trap: a router can be online while every one of its modems is disconnected. On a live group of 4 online E320s, all 8 modems reported disconnected, so a ["connected"] filter there would plausibly have targeted nothing despite the routers looking healthy. Filter on modem state only when you actually mean modem state.

v3 Endpoint Reference Had Drifted From the Verified Path Conventions (discovered 2026-09-28)

The docs/api-v3-endpoints.md reference had fallen out of sync with what the rest of this repo already knew (the "v3 Beta Endpoints" and regrades entries above, and the official Swagger specs at developer.cradlepoint.com/swagger/spec/{name}.json). Cross-checking the specs surfaced four errors in that reference, now corrected:

  • users was listed as a v3 CRUD endpoint. It is a v2 endpoint (swagger/spec/users.json → basePath https://www.cradlepointecm.com/api/v2, four X- header keys, GET only, filters id/username/email). Moved to api-v2-endpoints.md.
  • regrades was listed at /api/v3/regrades/. The real path is /api/v3/asset_endpoints/regrades (and /{id}). GET requires filter[action] (UPGRADE/DOWNGRADE/LICENSE/UNLICENSE) and filter[status] (RUNNING/COMPLETED/FAILED); POST is JSON:API atomic, required fields action + mac_address. (The regrades path was already correct in the SDK-gotcha entries above — only the reference doc was wrong.)
  • PCN and NCX endpoints were listed at bare /api/v3/... with trailing slashes. Both families are under /api/v3/beta/... with no trailing slash, per their specs (private_cellular_networks, private_cellular_radios, exchange_sites, exchange_resources). This matches the "v3 Beta Endpoints Must NOT Have Trailing Slashes" entry above.
  • subscriptions was listed with a trailing slash and an item-read /{id} path. The spec defines only GET /api/v3/subscriptions (no trailing slash, no /{id}, GET only), page[size] max 50, filter[start_time]/filter[end_time].

Lesson: the Swagger spec JSON is directly fetchable and authoritative. When the endpoint reference and the discovered-issues log disagree, the log (and the spec) won — the reference was the stale artifact.

The Removals List Means Different Things at Group and Device Scope (discovered 2026-10-05)

The second element of a config diff ([updates, removals]) is described in api-configuration.md as "paths to remove from defaults". That phrasing hides an asymmetry that reverses the correct approach depending on which endpoint you are writing to.

The config diff is not a command that gets executed and discarded; it is persisted state that NCM layers on top of what the device would otherwise inherit (Target = Group config + Individual config). So a removals entry is a standing instruction to suppress a path, not a one-time delete.

At group scope (/groups/{id}/), a removals path prunes something the group would otherwise contribute — the intuitive reading, and why common-patterns.md correctly recommends "prefer PATCH with removals over PUT" for mirroring collections between groups.

At device scope (/configuration_managers/{id}/), the stored diff is the device's override layer. A removals path there suppresses the path from what the group supplies. Consequences:

  • To make a device stop overriding the group and start inheriting it, delete the key from updates (element 0). Adding it to removals does the opposite: it persists a new device-level override that blocks the group's value entirely.
  • A security-rooted path sitting in a device's removals list is itself blocking inheritance of the group's firewall config, so restoring inheritance means clearing those entries too — not just the updates branch.
  • PATCH cannot do this. PATCH merges updates, so no PATCH body removes a key from it. The operation is read-modify-write with PUT, sending the device's complete diff minus the one branch. PUT's "resets unmentioned fields to defaults" hazard does not apply, because you are sending the whole document, not a fragment — and the documented device-config reset is itself a PUT of [{}, []] (api-configuration.md).

The "PATCH CAN Remove Config Items" entry above remains accurate: PATCH does accept removals, and that is the right tool for pruning a group's own config. It is the wrong tool for restoring group inheritance on a device.

Verified end-to-end against 400 live devices while building web_apps/revert_device_firewall/ (74 carried device-level security config). The read, diff and payload-construction paths were exercised against production data; the PUT itself was verified against a stubbed session rather than live devices.

Related trap: root-level firewall and root-level security are different things. Zone firewall config (zones, filter policies, forwardings), IPS and app sets live under security; firewall holds port forwarding and similar. Many devices carry both. Removing "the firewall config" by name alone will take out the wrong branch.

Not Every Router Has a configuration_managers Record (discovered 2026-10-05)

A batched get_configuration_managers(router__in=…) can return fewer records than router IDs you asked for. Observed against a live account while scanning 400 routers for web_apps/revert_device_firewall/: one device (a CR4250) had no configuration manager record at all, while the other 399 did.

Complements the "Configuration Manager ID ≠ Router ID" entry above: that one says don't assume the IDs match, this one says don't assume a record exists.

Why it bites: the natural way to consume a batched read is to zip the response against the request list, or to index into it positionally. Both silently misalign when a device is missing — every device after the gap gets attributed to the wrong router, which in a config-writing script means pushing one device's config onto another's. A length check does not save you either, since the response is also capped by pagination.

Map explicitly on the router field of each returned record and treat absence as a skip, not an error:

records = client.get_configuration_managers(
    router__in=','.join(router_ids),
    fields='id,router,configuration,suspended,synched',
    limit='all',
)
by_router = {}
for rec in records or []:
    rid = str(rec.get('router', '')).rstrip('/').split('/')[-1]   # it is a URL
    if rid:
        by_router[rid] = rec

missing = [r for r in router_ids if r not in by_router]   # report, do not crash
for rid in router_ids:
    rec = by_router.get(rid)
    if rec is None:
        continue        # nothing to write for this device

Note that the router field comes back as a resource URL (https://.../api/v2/routers/123456/), not a bare ID, so it needs parsing before it can be used as a dict key. The root cause of the missing record was not investigated — the practical point is that the gap exists in real accounts and must be handled.

Web apps must not mount a sibling app's static folder (fixed 2026-10-06)

Seven apps used to serve their logo and favicon from a sibling directory, web_apps/script_manager/static, via .parent.parent / "script_manager" / "static". Copy any one of those app folders on its own and the path no longer resolves, so /static/logo.png returns 404 and the header renders a broken-image icon.

The failure is quiet in the worst way: most of those apps guarded the mount with if STATIC_DIR.is_dir(), so the server starts normally and every API route works. Nothing in the logs indicates a problem.

The rule is that each app is self-contained — no app reads files from another at runtime. Each app owns its static/ folder:

STATIC_DIR = Path(__file__).resolve().parent / "static"      # own folder
if STATIC_DIR.is_dir():
    app.mount("/static", StaticFiles(directory=str(STATIC_DIR)), name="static")

Duplicating a 6 KB logo per app is the right trade. Resist the urge to "fix" a missing shared asset with onerror="this.style.display='none'" on the <img> — that hides the symptom and leaves the app unbrandable when shipped.

All seven were converted on 2026-10-06 (alert_dashboard, cellular_health_dashboard, geo_ip_blocker, host_identity_copier, inventory_dashboard, ncos_downloader, revert_device_firewall), each with its own static/logo.png and logo_dark.png. config_builder and netcloud_router_lookup already had their own.

Still outstanding: geo_ip_blocker and host_identity_copier also reach to the repo root with sys.path.insert(0, str(PROJECT_ROOT / 'ncm')) to import the vendored SDK, and load .env from PROJECT_ROOT. That is a repo dependency rather than a cross-app one, but it breaks the same way in a standalone copy. Every other app just does import ncm against the pip-installed package.

Two related points for standalone distribution:

  • Per-app requirements.txt. The root requirements.txt is the union across all apps, so it installs packages a single app does not need. A per-app file listing only what that app imports keeps a shipped copy small. Derive it from the actual imports rather than copying the root file.
  • Python floor is 3.10, not 3.9. The repo standard says 3.9+, but current fastapi, uvicorn, starlette, python-dotenv and urllib3 all publish Requires-Python >=3.10. Any FastAPI app in this repo therefore needs 3.10+ on a fresh install; pip will refuse on 3.9. Verify with python -c "import importlib.metadata as m; print(m.metadata('fastapi')['Requires-Python'])".

Backgrounding a web app silently loses its stdout startup banner (discovered 2026-10-06)

Every web_apps/ entry point prints its URL with plain print() before calling uvicorn.run(). Start one in the background with output redirected — which is the prescribed way to launch these — and that banner never reaches the log:

$ .venv/bin/python serve.py > run.log 2>&1 &
$ cat run.log
INFO:     Started server process [34287]
INFO:     Uvicorn running on http://127.0.0.1:8090 (Press CTRL+C to quit)

The "Open http://localhost:8090 in your browser" line is missing. Cause: when stdout is not a tty it is block-buffered (4 KB), so a ~150 byte banner sits in the buffer indefinitely. Uvicorn's lines appear because they go through logging to stderr, which is not block-buffered. Kill the process and the buffered stdout is discarded entirely, so the banner is never written at all.

Measured on revert_device_firewall: grep -c "Open http" gives 0 under default buffering and 1 with -u. Both forms fix it:

.venv/bin/python -u serve.py > run.log 2>&1 &      # or
PYTHONUNBUFFERED=1 .venv/bin/python serve.py > run.log 2>&1 &

The trap is the inference, not the missing text: absence of the banner is not evidence the app failed to start. The server in the example above was serving fine. Confirm startup from uvicorn's own stderr line, or by hitting the port (curl -s -o /dev/null -w '%{http_code}' http://localhost:<port>/), rather than by grepping for the app's own print output. Note also that a print() diagnostic added while debugging a backgrounded app may never appear for the same reason — use flush=True, stderr, or -u.

Unquoted --include=*.py makes a zsh grep report a false clean result (discovered 2026-10-06)

zsh (the default macOS shell) applies filename globbing to --include=*.py. Nothing in the working directory matches that literal pattern, so with zsh's default nomatch option the entire command aborts before grep runs:

$ grep -rn "pattern" --include=*.py . || echo "NONE - clean"
zsh: no matches found: --include=*.py
NONE - clean          # <-- grep never ran; this "clean" is meaningless

The || echo fallback converts a shell error into what reads like a verified negative result. This happened during a real cross-app dependency audit in this repo and briefly produced a false all-clear.

Quote the pattern, or prefer the dedicated grep tool:

grep -rn "pattern" --include='*.py' .

More generally, when a command's absence of output is the evidence for a claim, confirm the command actually executed — check the exit code or run it once in a form you know produces hits.

A cached NodeList goes stale on every selection click, so loops select one row (discovered 2026-10-07)

The picker tables in these apps redraw with container.innerHTML = … inside render(), and render() runs on each selection click. Every existing element reference is discarded, so a loop over a NodeList captured beforehand clicks detached nodes after the first iteration:

const rows = document.querySelectorAll('#listContainer tr[data-id]');
for (let i = 0; i < 12; i++) rows[i].click();      // selects exactly 1

The clicks land on orphaned nodes and are swallowed by the document-level delegation, with no error. The UI reported "1 router selected" after twelve clicks, which reads as a broken selection handler rather than a stale-reference problem in the harness.

Re-query inside the loop, and make the step idempotent so a re-run does not toggle selections back off:

for (let i = 0; i < 12; i++) {
  const rows = document.querySelectorAll('#listContainer tr[data-id]');
  if (!rows[i].classList.contains('selected')) rows[i].click();
}

Applies to any browser-driven check of an app that re-renders a list on interaction — so all of the pickers and dashboards here. The same trap makes a verification step pass when the count happens not to be asserted, so assert the resulting count (#selectionSummary, the action button's label) rather than assuming the clicks took.

Related: verify the frontend against a stub backend when the live fetch is the bottleneck. On a ~10.8k-router tenant, /api/inventory had not returned after several minutes, which blocks any UI check that needs populated rows. A throwaway http.server that serves the app's real index.html with synthetic /api/* JSON exercises the actual frontend code paths in seconds, with no risk of touching fleet config — useful for destructive apps specifically, since a stub cannot write. The app's own internals are typically wrapped in an IIFE and unreachable from page.evaluate, so driving the real buttons against a stub is the only way to test the logic rather than re-implementing it. Keep the stub outside the app directory and delete it afterward; apps must stay self-contained. (The inventory stall itself is unexplained and not confirmed to be a bug.)

A root AGENTS.md is loaded into every Kiro request, duplicating inclusion: always steering (discovered 2026-10-08)

Kiro auto-loads a repo-root AGENTS.md as a workspace rule, injected in full on every request — the same standing cost as a steering file with inclusion: always, but with no front matter to scope it. Confirmed directly: creating AGENTS.md caused it to appear as ## Included Rules (AGENTS.md) [Workspace] on the very next turn, alongside project-setup.md. A root claude.md was not injected in the same session.

So AGENTS.md is the wrong place for anything already covered by always-on steering. This repo's copy had mirrored all six steering files in full, which both doubled the always-on budget and drifted — it still described users as a v3 CRUD endpoint after docs/api-v3-endpoints.md had been corrected to v2 GET-only, and listed 12 web apps against 16 on disk. Prior entries in this file record four earlier rounds of the same drift (stale CSV path, serve.py assumption, fileMatchPattern, untracked ncm2/).

The file has been deleted. .kiro/steering/*.md is the single source of truth; the cross-agent entry point is claude.md, which holds the always-applicable essentials plus an index of which steering file to read when. If you need to serve agents that read neither, add a pointer file — do not re-mirror the rules.

Generalizes to two things worth separating when auditing agent instructions: where a file is loaded (always-on versus on demand) and whether its content is a copy. A full mirror is a maintenance liability on its own, and an always-on mirror is also a per-request tax. A pointer costs one read only on the turns that need it.

Starlette's TemplateResponse now requires request as the first positional arg; the old (name, {"request": ...}) form fails with TypeError: unhashable type: 'dict' (discovered 2026-10-15)

web_apps/ncm_api_key_encryptor/ returned a 500 on every page load under Starlette 1.6.0. The handler called the legacy signature:

templates.TemplateResponse("index.html", {"request": request, ...})   # old, removed

Modern Starlette (the deprecation landed in 0.29 and the old form is now gone) expects the Request first:

templates.TemplateResponse(request, "index.html", {...})              # correct

With the old call, Starlette reads the context dict as the template name and uses it as a cache key — a dict is unhashable, so you get TypeError: unhashable type: 'dict' from deep inside jinja2/utils.py (self._mapping[key]), not from your code. The traceback points at the TemplateResponse line but names no obvious cause, which makes it easy to misread as a Jinja/template-cache bug rather than a signature change.

Two practical notes:

  • Drop the now-redundant "request": request key from the context — Starlette injects the request into the template context automatically under the new signature.
  • This is not app-specific. Any web app in this repo still on the two-arg form breaks identically the moment Starlette is ≥0.29. When touching an app that renders Jinja templates, grep it for TemplateResponse( and migrate any (name, {...}) calls. A clean startup banner proves nothing here — the failure is per-request, so confirm by actually hitting the route and asserting a 200.

Root cause of the above is an unpinned, transitive Starlette — "worked then broke with no code change" (2026-10-15)

The TemplateResponse break above surfaced with no change to the app (last commit 2026-06-08) and no recent dependency install — Starlette 1.6.0 had been in the venv since 2026-08-14. The explanation is that Starlette is never pinned, and never even listed: both requirements.txt and web_apps/ncm_api_key_encryptor/requirements.txt declare only fastapi with no version, and Starlette arrives transitively through FastAPI. So the installed Starlette is "whatever PyPI served the day this venv was last built," not a deliberate choice. Rebuild the venv on a later day and the version silently moves — which is exactly how an app can "work a couple weeks ago" and 500 now with an unchanged tree.

Two generally-applicable takeaways:

  • When diagnosing a "worked before, broke now, code unchanged" failure, trust package mtimes and dist-info over memory of when it last ran. The dates here contradicted the "couple of weeks ago" recollection (packages were two months old), and the app's startup banner prints fine regardless because the fault is per-request — so "I saw it running" is not evidence the route worked.
  • A floating transitive dependency is the real defect. Pinning fastapi alone does not fix it — Starlette is a transitive dependency, and fastapi==0.141.1 still resolves a range of Starlette versions, so a rebuild can move it. You must add an explicit starlette==<version> line alongside the fastapi pin. The fix for this repo pinned fastapi==0.141.1, starlette==1.6.0, uvicorn==0.52.3 in every FastAPI app. Note the repo convention that each web app owns a requirements.txt of only what it imports — so the explicit Starlette pin has to be added per app, not just at the root (ncos_downloader and revert_device_firewall already pinned fastapi/uvicorn but were still exposed until Starlette was pinned too). Verify by dry-running pip install -r <file> against the known-good venv: a clean no-op resolve confirms the pins match the verified set rather than silently upgrading anything.

SDK v3 List Helpers Return Only the First 50 Records Unless limit=0 (discovered 2026-10-31)

NcmClientv3.__get_json sets limit = 50 whenever the caller passes no limit, then stops paginating at 50 results. So a bare call like get_remote_connect_profiles() or get_lan_manager_options() silently returns exactly 50 records with no warning, even though links.next is present. This was verified live on those two endpoints: 50 returned by default, against 573 and 613 with limit=0. The same code path backs every v3 list helper that routes through __get_json.

Pass limit=0 to get everything. __get_json treats 0 as unlimited and caps page[size] at 50. A default of exactly 50 results is the telltale sign of this cap.

Remote Connect Profile Names Are Not Unique per Router (discovered 2026-10-31)

/api/v3/remote_connect_profiles accepts more than one profile with the same name on the same router. A live account had 6 router/name pairs with more than one profile each. Bulk "push this profile" code must look up existing profiles by router + name and PUT over them, deleting any extra same-name copies. A plain POST creates a duplicate.

Two related details:

  • filter[name] is an exact, case-sensitive match (Foo doesn't match foo). Leading whitespace in the filter value was trimmed in one test, so a name with a leading space still matched. Do case-insensitive or substring searches client-side.
  • Both remote_connect_profiles and lan_manager_options cap page[size] at 50. Asking for more returns 400 with "You requested a size of 100, but 50 is the maximum."

v2 /routers/{id}/ Returns 403 for Missing Routers, So You Can't Tell "Deleted" From "No Access" (discovered 2026-10-31)

GET /api/v2/routers/{id}/ returns 403 Forbidden for a router ID that doesn't exist, not 404. Router ID 1 returned 403 in a test, and so did all 208 IDs in the orphan check below. A router in an account your keys can't see also returns 403. So a 403 here doesn't tell you whether the router was deleted or is out of scope. Don't code a "router deleted" check that waits for a 404. Compare against the full /routers/ list instead, and use the record's account to judge scope.

v3 Remote Connect Records Outlive Their Routers (discovered 2026-10-31)

remote_connect_profiles and lan_manager_options appear to keep their records after a router is unregistered or deleted. In one account, 281 of 573 profiles (on 208 routers) and 322 of 613 lan_manager_options records pointed at router IDs that weren't in the v2 /routers/ list. Every one of those profiles belonged to an account the same v2 keys could see, so out-of-scope accounts don't explain the gap. Their creation dates go back to 2017.

Not fully proven. A router moved into an account the keys can't see would look the same, because the profile keeps the account it was created under, and the v2 detail read returns 403 either way (previous entry). Spot-check a few IDs in the NCM UI before bulk-deleting.

Practical consequences:

  • Join v3 profile/option records to v2 routers by the router relationship ID (it's the v2 router ID), and expect unmatched IDs. Don't assume every record resolves to a named router.
  • Profiles can be removed with DELETE /remote_connect_profiles/{id}. lan_manager_options has no DELETE. A live DELETE /lan_manager_options/{id} returned 404 {"message": "no Route matched with those values"}, which is a gateway route miss, not "record not found". Once a router has an options record it can only be set to enabled: false, never removed.

Listing Every v2 Group Can Take a Minute; Fetch Only the Groups Routers Use (discovered 2026-10-31)

An account can hold thousands of groups, most of them empty. In one account, get_groups(limit='all', fields='id,name,account') returned 6,018 groups in 61 s, while only 759 of those groups contained any routers. The fix:

  1. List routers first (1,763 routers took about 5 s).
  2. Collect their distinct group IDs.
  3. Fetch just those groups with get_groups(id__in=",".join(ids), ...).

The SDK chunks __in into batches of 100. The routers and groups fetch together dropped to about 12 s, and no router was missing its group. An empty group has no routers to target anyway, so a picker that targets routers loses nothing.

SDK Remote Connect Write Helpers Mishandle 200, 204 and 403/409 (found by reading the SDK, 2026-10-31)

In ncm/ncm/ncm.py, the v3 create_, update_ and delete_remote_connect_profile helpers, and the create_/update_lan_manager_option helpers, all return self._return_handler(ncm.status_code, ncm.json(), call_type). That has three consequences:

  • DELETE: the endpoint returns 204 No Content, so ncm.json() raises requests.exceptions.JSONDecodeError before the handler runs. A successful delete therefore surfaces as an exception.
  • PUT: a 200 returns the string "<call_type> operation successful.", not the updated record.
  • 403 and 409: these return None silently (see the _return_handler entry above).

This was found by reading the code, not by running these helpers live. To get the 409 rate-limit handling and still see the real status and body, call the client's _v3_request(verb, url, json=...) directly and check status_code yourself. web_apps/lan_manager_profile_manager/serve.py does this.

SSH Remote Connect Profiles Can Exist Without a username, but the API Can't Create One (discovered 2026-11-01)

In one account, 54 of 114 remote_connect_ssh_profile records had "username": null. That's a real null, not an empty string. They were created from 2017-09 through 2026-10. The v3 API itself still enforces the documented rule. In a live test on one router, each of these returned 400 "username is required for ssh_profile" (pointer /data):

  • POST with username omitted
  • POST with "username": null
  • PUT setting username to null on an existing SSH profile (the record kept its old value)

So these profiles come from somewhere other than the public v3 API, most likely the NCM UI. Consequences:

  • Read and compare them normally. A null SSH username is a valid stored state.
  • They can't be copied or re-pushed as-is through the API. A copy needs a username supplied.
  • Any PUT to one of them must include a username, because PUT requires the full attribute set. You can't "just rename" such a profile without giving it one.
  • Not tested: whether "" is accepted.