This file is automatically maintained by the reflexion system. When the AI discovers issues, workarounds, or corrections while building applications, they are logged here.
All v2 endpoint URLs MUST end with /. Without it, the server redirects (301),
causing two API calls to be counted against your rate limit.
Unlike v2, the v3 beta endpoints (/api/v3/beta/...) return 404 Resource not found if the URL ends with a trailing slash. Omit the slash entirely.
For example, use /api/v3/beta/modem_software_versions not
/api/v3/beta/modem_software_versions/. Non-beta v3 endpoints (e.g.
/api/v3/subscriptions/) still accept trailing slashes.
The configuration_managers endpoint has its own id field that may differ from the
router's id. Always use get_configuration_manager_id(router_id) to look up the
correct config manager ID before making PUT/PATCH calls.
This entry previously said the opposite. It claimed "PATCH only adds or updates. To remove config items, you must use PUT with the removals list," and a 2026-08-13 caveat built further on that mistake. Both were wrong.
PATCH accepts the removals list, and NCM's own web UI uses it. Removing a single address from a host identity produces this PATCH body — empty updates dict, one removal path:
[
{},
[["identities", "ip", "0897fa24-b4a4-4aa2-9cab-0bed9507c233", "members", 2]]
]Adding or changing addresses in the same identity produces an updates-only PATCH:
[
{
"identities": {
"ip": {
"0897fa24-b4a4-4aa2-9cab-0bed9507c233": {
"members": {"2": {"address": "4.5.6.7"}, "3": {"address": "5.6.7.8"}},
"_id_": "0897fa24-b4a4-4aa2-9cab-0bed9507c233"
}
}
}
},
[]
]Both slots can be populated in one PATCH: set values via updates and prune via removals in a single call.
Array elements are addressed by integer index in removal paths, not by string key. Note the asymmetry against updates, where the same array position is a string key:
- update:
{"members": {"2": {...}}}— string"2" - removal:
["identities", "ip", "<uuid>", "members", 2]— integer2
Practical consequences:
- You do not need PUT to delete one branch of a group config. Prefer PATCH
with removals — PUT resets unmentioned fields to defaults, which at
/groups/{id}/scope wipes every other group-level setting. - UUID-keyed collections (
identities.*,lan,vpn.tunnels,security.zfw.zones, ...) can be mirrored exactly between groups. Remove a whole entry with a path ending at its UUID:["identities", "ip", "<uuid>"]. - When removing several elements from the same array, emit the indices highest-first. Whether NCM resolves removal paths against the pre-change config or applies them sequentially with reindexing is unconfirmed, and descending order is correct under either interpretation.
Source: NCM UI request traffic reported by a user on 2026-08-14. The formats above are what the UI sends; they have not been independently replayed against the API from this repo.
When using arrays (not objects) in a PATCH body, the entire array is replaced. Use objects with string keys for partial array updates.
When using a UUID _id_ as the key for an array element, you must also include
the _id_ field inside the object body, or you get a validation error.
Password fields always return "*" on GET. You cannot read existing passwords.
Set new passwords by sending cleartext — they are encrypted automatically.
If a device applies a config but can't reach NCM within 15 minutes, it rolls back and suspends sync. Check the "Configuration Rejected" alert for details.
API v3 requires both Content-Type: application/vnd.api+json AND
Accept: application/vnd.api+json. Missing either causes errors.
This endpoint and the routers.overlay_network_binding field were deprecated 09/30/2024.
Remove all references.
This field still appears in responses but returns inaccurate data since 12/31/2023.
NcmClient() returns different class instances based on what keys you provide:
- Only v2 keys →
NcmClientv2 - Only v3 token →
NcmClientv3 - Both →
NcmClientv2v3
The SDK checks CP_BASE_URL for v2 and CP_BASE_URL_V3 for v3 base URLs.
Using the SDK's module-level method delegation (import ncm; ncm.get_routers(...))
auto-initializes a singleton via get_ncm_instance(). If only a v3 token is set
(no v2 API keys), the singleton is a NcmClientv3. Calling a v2-only method like
get_routers() on it will not raise an error — instead, the v2 __get_json
pagination loop sends requests to the v2 base URL without proper auth headers.
The SDK's retry adapter (5 retries, exponential backoff on 408/503/504) causes
the script to appear to hang indefinitely with no output.
Workaround: in v3-only scripts, do NOT use module-level delegation for v2 methods.
Either instantiate NcmClientv3 directly and use v3 equivalents (e.g.
get_asset_endpoints() instead of get_routers()), or ensure all four v2 API
keys are set in the environment.
Correction: this entry previously stated that _return_handler returns error
strings like "ERROR: 400: {...}" instead of raising. That is no longer true —
the SDK now raises requests.exceptions.HTTPError for 400, 401, 404 and 500.
Verified by calling _return_handler directly across status codes:
| Status | Behavior |
|---|---|
| 200 | returns "<obj_type> operation successful." |
| 201, 204 | returns the response text |
| 400, 401, 404, 500 | raises requests.exceptions.HTTPError as "<code>: <body>" |
| anything else (403, 409, 429, 418, ...) | logs, then returns None |
So try/except around SDK calls does catch those four error codes. Two
traps remain:
- Unhandled status codes return
None, silently. 403, 409 and 429 all fall through theelsebranch — notably 429 (rate limit) and 409 (which v3 uses for both rate limiting and validation errors). Code that assumes a truthy return value or a list will misread these as success. - v2's
__get_jsonstill swallows errors independently of_return_handler. It checksif not (200 <= status < 300): breakbefore calling the handler, so a mid-pagination failure exits the loop and returns partial (possibly empty) results with no exception and no indication. An unexpectedly empty or short list from a v2 GET may be a silent error.
Workaround: wrap SDK calls in try/except requests.exceptions.HTTPError (or bare
except Exception) to catch 4xx/5xx on writes, and additionally treat None and
suspiciously empty list results as failures.
The regrade() method sends an atomic:operations payload but does not set
the required JSON:API atomic extension Content-Type header. Compare with
unlicense_devices(), which correctly sets:
Content-Type: application/vnd.api+json;ext="https://jsonapi.org/ext/atomic"
Accept: application/vnd.api+json;ext="https://jsonapi.org/ext/atomic"
The missing header can cause the API to reject the request with a 400 error
(e.g. "mac_address must be specified") even when the field is present in
the payload, because the server doesn't parse the atomic operations format
without the extension header.
The regrade() method normalizes MAC addresses by stripping colons only when
the input is exactly 17 characters (len(smac) == 17). This misses:
- Dash-separated MACs (
00-30-44-1A-2B-3C, 17 chars) — colons are stripped (finding none), dashes pass through - Dot-separated Cisco format (
0030.441A.2B3C, 14 chars) — passes through with dots intact - Lowercase MACs — passed through as-is (API may expect uppercase)
Workaround: always normalize MACs to bare uppercase hex before passing to
regrade(): mac.upper().replace(':', '').replace('-', '').replace('.', '')
The POST /asset_endpoints/regrades atomic operations endpoint rejects the
entire batch with a 400 Bad Request if the same mac_address value appears
more than once across operations in a single request. The error message is:
"mac_address values must only occur once". This applies per-request, not
globally — the same MAC can appear in separate requests. Always deduplicate
MAC addresses within each batch before sending.
The POST /asset_endpoints/regrades endpoint strictly validates that
mac_address is exactly 12 hexadecimal characters (uppercase or lowercase,
no separators). Any other format — including MACs with colons, dashes, dots,
or fewer/more than 12 characters — returns 400 Bad Request with:
"mac_address must be 12 digit hexadecimal with optional colons". Despite
the error message mentioning "optional colons", bare 12-digit hex is the
safest format. Validate with ^[0-9A-Fa-f]{12}$ before sending.
The NCM SDK and the scripts/utils/session.py utility use different key naming
conventions for credentials. Mixing them up causes silent auth failures.
- SDK (
NcmClient):{'X-CP-API-ID': '...', 'X-CP-API-KEY': '...', 'X-ECM-API-ID': '...', 'X-ECM-API-KEY': '...'} - Session utility (
APISession):cp_api_id='...', cp_api_key='...', ecm_api_id='...', ecm_api_key='...' credentials.py: uses snake_case keys matching the session utility, NOT the SDK
If using get_credentials() with the SDK, you must remap the keys to HTTP header names.
The standard env var name for the v3 bearer token in this project is NCM_API_TOKEN.
All scripts have been refactored to use this name. Historical names that may
appear in older code or external references:
TOKEN— legacy name (removed from scripts)V3_BEARER_TOKEN— Inventory Dashboard legacy (migrated to NCM_API_TOKEN)CP_API_TOKEN— never used by any actual script
When building new scripts/dashboards that need v3, use NCM_API_TOKEN.
The NCM API limits __in filter parameters to 100 comma-separated values per request.
The SDK auto-chunks these transparently, but if you're making raw API calls with
requests or the session utility, you must chunk manually or the API will error.
If the repo is cloned or moved to a different path than where the .venv was
originally created, the pip (and other scripts in .venv/bin/) will have a
stale shebang pointing to the old Python path. Running .venv/bin/pip install ...
fails with bad interpreter: no such file or directory.
Workaround: use .venv/bin/python -m pip install ... instead, or recreate the venv.
The requirements.txt lists flask but it may not be installed in the venv if
dependencies were never fully installed. Always run
.venv/bin/python -m pip install -r requirements.txt after cloning or setting up
the project for the first time.
Running python3 -m venv .venv --clear to fix a broken venv also deletes the
API credentials that setup_env.py injected into .venv/bin/activate. After
recreating the venv, you must re-run setup_env.py and then source .venv/bin/activate
to restore the environment variables. This affects all scripts that rely on check_env().
The NCM API v2 supports ?fields= on most endpoints for partial responses, but
the Python SDK only allows fields on methods where it's explicitly in the
allowed_params list. Before using fields with any SDK method, check its
allowed_params in the source.
Methods known to support fields: get_routers(), get_groups() (added 2026-03-31).
Methods known to NOT support fields: get_products(), get_firmwares().
When fetching routers without the fields parameter, relational fields like group,
product, actual_firmware, and target_firmware return full API URLs
(e.g. https://www.cradlepointecm.com/api/v2/groups/12345/), not names or IDs.
To get human-readable values, either use expand=group on the request, or build
a lookup dict from a separate get_groups() call. The full_product_name field
does return the model name directly (e.g. "AER1600").
On large accounts the /groups/ endpoint can return thousands of groups (4000+),
requiring many paginated requests that take minutes to complete. Using
expand=group on the /routers/ call is dramatically faster because the API
inlines the group object (with name, id, etc.) directly into each router
response, eliminating the separate fetch entirely. The same applies to
expand=account. Always prefer expand over a separate lookup fetch when you
only need the related resource's name or ID.
When the group list is the product (measured 2026-09-28). expand=group does
not help if you need a group picker rather than group names hung off routers. On a
test account with 5,889 groups, get_groups(limit='all') took 66 seconds —
sequential pages of 500. Per-group calls afterwards were fast by comparison
(net_devices for one group 0.5s, v3 modem_software_versions 1.4s), so the group
enumeration is the whole cost.
A UI that blocks on this looks hung for over a minute on first load. Options, none of which the API solves for you:
- Cache the list to disk and serve it instantly on later loads, refreshing in the
background (
web_apps/cellular_health_dashboarddoes this for its dataset via asnapshot_cache.json). - Narrow with
fields='id,name'to cut payload size — this does not reduce the page count, so expect a modest gain at best. - Avoid enumerating at all where possible: if the user already knows the group, take an ID directly, or reach groups via the routers you are already fetching.
Do not assume a group list is cheap because it was fast on a small account; the cost scales with the tenant, not with your code.
The v3 /subscriptions/ endpoint returns attributes.start_time and
attributes.end_time but no explicit status field. To determine if a
subscription is active or expired, compare end_time against the current UTC time.
The subscription type is in relationships.subscription_type.data.id.
When using the NCM SDK's get_routers(fields='...') inside a Flask application,
it returns an empty list, even though the same call works correctly in a standalone
Python script. The root cause is unclear but may relate to how Flask's request
context interacts with the SDK's session/urllib. Workaround: omit the fields
parameter and fetch full objects instead.
The v2 /routers/ endpoint returns MAC addresses in the mac field using
colon-separated format (e.g. 00:30:44:1A:2B:3C), while the v3
/asset_endpoints endpoint returns them in the mac_address field using
bare uppercase hex (e.g. 0030441A2B3C). When joining data across v2 and v3,
you must normalize both to the same format (e.g. strip colons/dashes and
upper-case) before matching.
v3 Beta Endpoints Use /beta/ Prefix (discovered 2026-04-01, updated 2026-04-02, corrected 2026-09-28)
Several v3 endpoints are served under /api/v3/beta/ rather than /api/v3/.
Requests without the /beta/ prefix may 404. Known beta endpoints include
exchange_sites. Because these are beta endpoints, their contracts may change
without notice. Always check the OpenAPI spec or release notes for the correct
path prefix.
Correction (2026-09-28): this entry previously also listed
modem_software_versions and modem_upgrades. Those two are not beta — the
official Swagger spec documents them at /api/v3/modem_software_versions and
/api/v3/modem_upgrades. See "Modem Upgrade Endpoints Are NOT Beta" below. The
broader lesson holds: verify the prefix per endpoint against the spec instead of
generalizing from a sibling.
The v2 /net_devices/ endpoint supports an is_asset boolean
filter. Setting is_asset=true returns only the primary modem interfaces
(the physical cellular modems), filtering out virtual/logical interfaces.
This is useful when you only need modem-level details (IMEI, ICCID, carrier)
and want to avoid processing hundreds of non-modem net_device records.
The v2 /net_devices/ endpoint supports expand=router and expand=account.
When using expand=router, the router field becomes an inline object with
the full router record (id, name, state, mac, group URL, etc.) instead of a
URL. This eliminates the need for a separate /routers/ fetch when building
per-modem views. Note: if the net_device has no associated router, the field
remains null even with expand.
The /net_device_health/ endpoint returns health records for net_devices of
ALL modes (wan, lan, mdm), not just WAN or modem interfaces. When joining
health data to net_devices, do NOT filter by mode='wan' or you will silently
miss most records. Fetch all net_devices (no mode filter) and join by
net_device ID to get complete results.
The /net_device_health/ endpoint returns only:
cellular_health_category— string: "poor", "fair", "good", or "excellent"cellular_health_score— integer: 0–100net_device— URL reference to the net_deviceid,resource_url
It does NOT contain signal metrics (RSSI, RSRP, RSRQ, SINR, etc.). For actual
signal data, use /net_device_metrics/ filtered by the same net_device IDs.
The typical pattern for a cellular health dashboard is:
GET /net_device_health/→ health scores + net_device URLsGET /net_device_metrics/?net_device__in=id1,id2,...→ signal metricsGET /net_devices/?id__in=id1,id2,...→ carrier, model, router refGET /routers/?expand=group→ device names, groups, state
Confirmed against docs/api-v2-full-reference.md and the SDK's allowed_params
in ncm/ncm/ncm.py: /net_device_health/ supports only net_device,
net_device__in, id__gt/gte/lt/lte, limit, offset. There is no
update_ts__gt/lt, created_at__gt/lt, or any other timestamp-based filter —
unlike /net_device_metrics/, which does support update_ts__gt/lt.
This means an incremental/delta refresh (only fetch health records that changed
since the last poll) is impossible for this endpoint; every refresh must fully
re-paginate all records with limit='all'. If you want incremental refreshing
for a cellular health dashboard, it can only apply to /net_device_metrics/
(and only if you also accept that health scores/categories stay on a full-pull
cadence, since there's no way to ask "what health records changed").
These are easy to conflate when building an "online/offline" indicator for a cellular interface, but they answer different questions:
routers.state— whether the parent router/device is online, offline, or initialized with NCM overall.net_devices.connection_state— whether THIS specific network interface (e.g. one modem) is actually connected. Observed values on a live account:connected,disconnected,connecting,standby,standby_connecting,unplugged.
A router can be online while one of its modem interfaces is
disconnected, connecting, or in standby — the router itself has
connectivity (e.g. via another WAN) even though a given cellular interface does
not. Using router.state (or falling back to it) to decide whether a modem
interface is "online" overcounts: on one sampled account, 40 of 472
interfaces were router.state=online while connection_state=disconnected,
and another 10 were online while connecting/standby/standby_connecting.
For a per-interface online/offline indicator, use only
net_devices.connection_state and treat it as binary: connected is online,
every other value is offline. Resist the temptation to promote unplugged (or
connecting/standby) to a third badge — a device in any of those states is not
passing traffic, and a third bucket means summary counts no longer sum to the
total, which reads as a bug.
Cellular Health Data Is Per-Modem, Not Per-Router — Router Name Is Not a Key (discovered 2026-08-27)
net_device_health, net_device_metrics and net_devices are all keyed by
net_device (one record per network interface), not by router. A router with two
modems yields two records, and joining them to routers via expand=router
duplicates the router name across those rows.
Consequences when building per-modem views or reports:
- Do not treat the router name (or router ID) as a unique row key. Sorting or
grouping by it alone produces what look like duplicate entries. The unique key
is the net_device ID;
(router_name, interface_name)is the useful display key. - Counts are interface counts, not device counts. "472 records" on an account is 472 modem interfaces across fewer physical routers. Labeling such a total "Devices" in a UI misleads users into thinking they have more routers than they do — name it modems/interfaces.
- Per-router rollups need explicit aggregation. Any "how many routers are unhealthy" question requires grouping by router ID first and deciding how to combine multiple modems' scores; the raw records cannot answer it directly.
The interface name (net_devices.name) looks like mdm-211c287d, and
net_devices.model is the modem model (e.g. Internal 5GE-AM (SIM1)), not the
router model — another place where "device" is ambiguous between the two levels.
The /net_devices/ endpoint does NOT return signal strength fields (rssi,
rsrp, rsrq, sinr, signal_percent). These fields only exist on the
/net_device_metrics/ endpoint. The /net_devices/ response includes modem
hardware info: carrier, model, mfg_model, mfg_product, connection_state,
imei, iccid, apn, modem_fw, etc.
The /net_devices/ field table in api-v2-full-reference.md shows uptime with
a null sample, which reads as if the field is unusable. It is not: on a live
account it holds a float of seconds — the modem interface's connection uptime
(e.g. 8912.47 ≈ 2h 28m, 747706.06 ≈ 8.6 days). It is null for many records
(modems that have not reported one), so any consumer must handle None. It is a
per-net_device (per-modem) value, not a router-level uptime. To display it, floor
to an int and format as Xd Yh Zm; render null as a dash. No extra API call is
needed — it rides along on the /net_devices/ record a cellular health join
already fetches.
The router field on /net_devices/ can be null for orphaned or
unassigned modems. Code that assumes router is always a URL will fail.
Always check for null before extracting the router ID. For display purposes,
fall back to hostname or name from the net_device record.
Despite being numeric, v2 API responses return id fields as strings
(e.g. "67468693" not 67468693). When building lookup dictionaries for
joining across endpoints, always normalize IDs to strings. Using int() for
lookups will cause key mismatches if the dict was keyed with the raw string
from the API response.
The /net_device_metrics/ endpoint returns:
- Signal:
rsrp,rsrq,rssi,rssnr,sinr,dbm,cinr,ecio,signal_strength(integer 0–100, NOTsignal_percent) - Cell info:
cell_id,mcc,mnc,tac,lac,service_type(e.g. "LTE", "WiFi", "Ethernet", "Not Available") - Usage:
bytes_in,bytes_out - Meta:
update_ts,bmask_applied,net_device(URL),id,resource_url
Note: id matches the net_device ID (same value), making joins trivial.
Non-cellular interfaces (WiFi, Ethernet) have null signal fields.
When using expand=router on /net_devices/, the inline router object's
relational fields (like group, account, product, actual_firmware) are
still URLs, not expanded objects. Only one level of expansion is supported.
To resolve group names when using this pattern, you must still call
/groups/ separately and build a lookup by URL or ID.
When making concurrent requests to v3 endpoints (e.g. /asset_endpoints),
the API may return 409 Conflict with the message "Conflict with internal
rules. i.e. invalid app-key." instead of the expected 429 Too Many Requests.
This is effectively a rate limit response. Treat 409 as retryable with
exponential backoff, the same as 429. The NCM API enforces a limit of
approximately 500 calls per minute across all endpoints.
During heavy pagination of v2 endpoints (especially /net_devices/ and
/routers/ on large accounts), the API may return 500 Internal Server Error
with a body like "cp internal error shard-router-3". This is a transient
server-side error, not a client issue. Retry with backoff. Include 500, 502,
503, and 504 in your retryable status code set.
When using httpx (or requests) for pagination, passing params={} (empty
dict) to client.get(url, params={}) strips any existing query parameters
from the URL. If the meta.next URL is
https://...?limit=500&offset=500, passing params={} turns it into
https://.../routers/ — causing an infinite loop that re-fetches page 1
forever. Use params=None instead of params={} on subsequent pagination
requests where the query parameters are already embedded in the next URL.
The v2 and v3 APIs are architecturally independent interfaces to the same underlying data. They do NOT cross-reference each other:
- Auth: v2 uses 4 API key headers; v3 uses a single Bearer token.
- IDs: v2 uses integers; v3 uses strings. They are different ID spaces.
- Spec: v3 follows JSON:API; v2 does not.
- Relationships: v2 embeds full URLs to other v2 resources. v3 uses
JSON:API
relationshipsblocks with type/id references to other v3 resources only. - Expand:
?expand=is v2-only. v3 uses JSON:API?include=(where supported) or you follow the relationship type/id to the v3 endpoint.
The key implication: v3 relationship references can only be resolved through v3 endpoints. If a related resource doesn't have a v3 endpoint yet, that relationship either won't appear in the v3 response or the data will be flattened into attributes. You cannot use a v2 endpoint to resolve a v3 relationship reference (or vice versa). When migrating scripts from v2 to v3, all related resource lookups must also be migrated to their v3 equivalents.
The v2 /routers/ endpoint assigns numeric IDs (e.g. 1234567) to devices.
The v3 /asset_endpoints endpoint has its own id field that does NOT
correspond to the v2 router ID. Passing a v2 router ID to
get_asset_endpoints(id=...) (which becomes filter[id]=1234567) will
return the wrong device or no results. To cross-reference between v2 and v3,
use a shared natural key like mac_address or serial_number instead of
either system's internal ID.
The subscription_ids on v3 /asset_endpoints are per-device assignment IDs
(e.g. 4c03Hj1hALlWWvK), NOT the parent subscription IDs returned by an
unfiltered GET /subscriptions (e.g. 55050000002xYV9). These two ID spaces
do not overlap. To resolve assignment IDs to full subscription details (name,
start/end dates), use GET /subscriptions?filter[id]=<assignment_id>.
Multiple IDs can be comma-separated: filter[id]=id1,id2,id3.
Looking up an assignment-level subscription ID by path
(GET /subscriptions/4c03Hj1hALlWWvK) returns 404, even though the ID is
valid. Use the query filter instead: GET /subscriptions?filter[id]=4c03Hj1hALlWWvK.
This is a JSON:API quirk — the filter endpoint resolves IDs that the direct
path lookup does not.
check_env() from utils.env_check always validates v2 API keys
(CP_API_ID, CP_API_KEY, ECM_API_ID, ECM_API_KEY). In v3-only
scripts that don't need v2 keys, it prints a misleading "Missing required
environment variables" message to stderr before calling sys.exit(1).
Workaround: suppress stderr and catch the SystemExit:
import io, sys
from utils.env_check import check_env
try:
_stderr = sys.stderr
sys.stderr = io.StringIO()
check_env()
except SystemExit:
pass
finally:
sys.stderr = _stderrThen validate v3-specific vars (NCM_API_TOKEN, etc.) separately.
The OpenAPI spec for POST /api/v3/beta/modem_upgrades documents a 201 Created response, but the API actually returns 200 OK with the created
resource in the body. Code that checks for status_code == 201 to confirm
creation will incorrectly treat a successful POST as a failure. Check for
both 200 and 201.
The OpenAPI spec says the data.type field in POST/PUT request bodies for
/api/v3/beta/modem_upgrades should be "modem_upgrade_parent". The API
rejects this with a 409 Conflict: "The resource object's type
(modem_upgrade_parent) is not the type that constitute the collection
represented by the endpoint (modem_upgrades)." Use "modem_upgrades" in
the request body instead. Note: the response still returns
"modem_upgrade_parent" as the type — the asymmetry is intentional.
The existing documentation notes that v3 returns 409 Conflict as a rate
limit. However, 409 is also used for JSON:API validation errors (e.g. wrong
data.type). To distinguish: if the 409 response body contains an errors
array, it is a real validation error and should NOT be retried. If it does
not contain errors, treat it as a rate limit and retry with backoff.
Several v2 time-series endpoints return 409 Conflict if called without a
mandatory router or net_device filter. This is NOT a rate limit — it's a
validation error requiring you to scope the query to specific device(s).
| Endpoint | Required Filter |
|---|---|
historical_locations |
router |
net_device_signal_samples |
net_device or net_device__in |
net_device_usage_samples |
net_device or net_device__in |
router_logs |
router |
router_state_samples |
router or router__in |
router_stream_usage_samples |
router or router__in |
These endpoints use TimeUUID-based pagination and are optimized for per-device queries. Always provide the filter or you'll get a 409.
Cradlepoint router configuration .bin files are compressed JSON, but the
compression format varies. Some use standard zlib (wbits=15), others use raw
deflate (no header, wbits=-15), and others use gzip (wbits=31). When decoding
.bin files, try multiple zlib.decompress(data, wbits) values in sequence:
15, -15, 31, 47 (auto-detect). The first one that succeeds without a
zlib.error is the correct format. The decompressed content is always UTF-8
encoded JSON.
When decompressing a Cradlepoint .bin configuration file, the resulting JSON
is an array structured as [{"config": {...}, "state": {...}, "fw_info": {...}}, [[...], ...]].
The first element is an object containing "config" (the actual configuration),
"state" (runtime state data), and "fw_info" (firmware version). The second
element is an array of path references (e.g. ["config", "lan", 0, "devices", 2])
that appear to be deletion/override markers for the config tree.
When working with configs for templating or pushing to devices:
- Reading: Extract only
parsed[0]["config"] - Writing: Wrap config as
[{"config": <config_obj>}, []]then zlib compress
The .bin files use standard zlib compression (magic bytes 78 9c, wbits=15).
Config top-level keys include: certmgmt, container, ecm, identities, lan,
routing, security, stats, system, wan, wlan.
Cradlepoint configuration JSON (extracted from .bin files or via API) contains
string values with raw newline characters (e.g. embedded YAML in container
project configs, PEM certificates). Python's json.loads() in strict mode
rejects these as "Invalid control character". Use json.loads(content, strict=False)
when parsing Cradlepoint config JSON to allow control characters in strings.
The NCM SDK uses requests.Session() with default SSL verification. Behind
corporate proxies that use TLS interception (MITM with custom CA certificates),
all API calls fail with SSLCertVerificationError: CERTIFICATE_VERIFY_FAILED.
The SDK does not expose a verify parameter, so callers must patch the session
directly after constructing the client:
client = ncm.NcmClient(api_keys=api_keys)
client.session.verify = False # Disable SSL verificationTo also suppress the InsecureRequestWarning that urllib3 emits on every
request when verify is disabled:
import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)Alternative (proper fix): set the REQUESTS_CA_BUNDLE env var to the path of
the proxy's CA certificate bundle:
export REQUESTS_CA_BUNDLE=/path/to/corporate-ca-bundle.crtThis affects ALL scripts and dashboards using the SDK, not just specific apps.
The /device_apps/ endpoint returns the id field as a string (e.g. "83")
rather than an integer. This differs from most other v2 endpoints (like
/routers/ and /groups/) which return integer IDs. When building lookup
dictionaries keyed by app ID — for example, to match device_app_versions
back to their parent app — use string keys or cast consistently. Attempting
int() conversion on the URL-extracted ID and comparing against the raw
response ID will fail silently (lookup miss) if one side is a string and the
other is an int.
The resource_url and relational URL fields in v2 API responses can use
different base hostnames depending on the account's regional shard (e.g.
https://www.us0.cradlepointecm.com/api/v2/accounts/123/ vs
https://www.cradlepointecm.com/api/v2/accounts/123/). This means matching
a group's account URL against an account's resource_url by exact string
comparison will fail if the hostnames differ. Always extract the numeric ID
from the URL path (the last path segment before the trailing slash) and match
on that instead of comparing full URLs.
The v2 /groups/ endpoint DOES support expand=account. When used, the
account field becomes an inline object with id, name, is_disabled,
resource_uri, and account (parent account URL). This eliminates the need
for a separate /accounts/ fetch to resolve group account names. Known
endpoints that support expand: /routers/ (group, account), /groups/
(account).
Long-Running Servers Need source .venv/bin/activate, Not Just .venv/bin/python (discovered 2026-08-04)
setup_env.py injects real credential values (X_CP_API_ID, X_CP_API_KEY,
X_ECM_API_ID, X_ECM_API_KEY) as export statements directly into
.venv/bin/activate. If .env is missing or incomplete (e.g. deleted,
partial setup), those exported values in activate are the only place the
credentials exist.
Calling .venv/bin/python script.py directly executes the interpreter
without sourcing activate, so those exported vars are never set in the
process environment — even though the venv itself (sys.prefix) is correctly
active. For one-shot scripts that call check_env() at startup, this fails
loudly and is easy to catch. For long-running servers (FastAPI/Flask
dashboards under web_apps/) that read os.environ lazily per-request
(e.g. via _build_client() on each API call) instead of validating at
startup, the server boots fine and returns 200 on static routes, but every
API call silently fails with a missing/empty credential error — there is no
startup-time signal that credentials are missing.
Workaround: always launch long-running dev servers with
source .venv/bin/activate && python path/to/serve.py, not
.venv/bin/python path/to/serve.py, whenever credentials may only be
present in the activate script rather than .env. This is safe to run in a
single non-interactive shell command (no prompts), unlike setup_env.py
with no flags.
The /alerts/ endpoint returns 409 Conflict with "Invalid ordering field specified: created_at" when using order_by=-created_at or any order_by
parameter. The endpoint does not support ordering — results must be sorted
client-side. The endpoint does support created_at__gt for time filtering,
but requires full ISO 8601 format with microseconds and timezone offset
(e.g. 2026-06-26T15:00:37.703000+00:00). The alert type field for
custom alerts is custom_alert (not custom).
NcmClientv2.put_group_configuration() calls self.__return_handler(...) with
two leading underscores. Inside the class body that name-mangles to
self._NcmClientv2__return_handler, which does not exist — the real method is
_return_handler (defined on BaseNcmClient). Every other config method
(patch_group_configuration, put_configuration_managers,
patch_configuration_managers) uses the correct single-underscore name.
Verified by stubbing session.put and calling the method:
patch_group_configuration -> 'Configuration Manager operation successful.'
put_group_configuration -> AttributeError: 'NcmClientv2' object has no
attribute '_NcmClientv2__return_handler'
The critical detail: the HTTP PUT is sent before the exception is raised.
The group config is already modified in NCM by the time the caller sees the
error, so treating the AttributeError as "the write failed" and retrying will
apply the PUT twice.
Workaround: use patch_group_configuration() when a merge is acceptable. When a
true PUT is required (the only way to honor the removals list and reset
unmentioned fields), issue it directly and skip the wrapper:
resp = client.session.put(
f'{client.base_url}/groups/{group_id}/',
data=json.dumps({'configuration': [updates, removals]})
)
resp.raise_for_status()setup_env.py stores credentials in two places: .env, which scripts read
directly, and the venv activate scripts, which only an activated shell picks up.
These can fall out of sync, and each activate script is independent — a venv can
have some populated and others not.
Confirmed instance: a venv whose credentials were last written on Jun 24 had all
four variables in activate, activate.fish and activate.csh, but zero in
Activate.ps1 (no marker comments at all, file untouched since venv creation),
and no .env at all. Activate.ps1 injection was added to setup_env.py on
2026-08-04 (8be9817), after that credential run — so any venv whose credentials
predate a change to the injection code will be missing whatever was added since.
Why it bites: nothing fails at setup time. A bash/zsh user sees a working
environment indefinitely. The failure lands on whoever uses a different shell — a
PowerShell user gets an activated venv with no credentials and no warning — or on
anyone running .venv/bin/python directly when .env is absent, which resolves
zero of four keys.
Diagnose with setup_env.py --check, which reports .env presence and per-script
credential status:
[warn] .env: not found - only activated shells will have credentials
[warn] activate scripts without all credentials: Activate.ps1 (none)
Fix by re-running setup_env.py --credentials-only (prompts, hidden input), which
rewrites .env and every activate script with the current code. From a shell that
already has the credentials exported, setup_env.py --skip-credentials achieves
the same thing non-interactively by reusing what is in os.environ.
update_activate_scripts() now reports every script by name with its outcome,
reads the values back to confirm they landed, and warns when a script that exists
did not receive them, so this drift is visible at write time rather than silent.
Absence of activate.bat on macOS/Linux is expected — python -m venv only
creates it on Windows.
Web App Settings Modals Are a Third Credential Store (discovered 2026-08-14, tracking resolved 2026-08-14)
Credentials live in more places than .env and the venv activate scripts. Every
web app with a Settings modal writes what you type into a per-app JSON file beside
its serve.py, in plaintext:
profiles.json— named credential profiles (alert_dashboard,cellular_health_dashboard,geo_ip_blocker,inventory_dashboard)config.json— active credentials plus saved job state (host_identity_copier)
Consequences worth knowing:
- These files survive
rm -rf .venv, so an app can keep authenticating after the venv-injected credentials are gone. Convenient, and easy to forget when you think you have removed every copy of a key. - They are a plaintext credential store in the working tree. Apps that write them
should
chmod 0600on POSIX; not all do.
.gitignore does not retroactively untrack — the mechanism, kept because it will
recur. web_apps/*/config.json and web_apps/*/profiles.json were added to
.gitignore on 2026-08-13. That stops new files being committed but has no effect
on files already in the index, so
web_apps/cellular_health_dashboard/profiles.json and
web_apps/inventory_dashboard/profiles.json stayed tracked and unprotected. Fixing
that requires git rm --cached <path>; after that the pattern takes over.
Resolved 2026-08-14. No profiles.json is tracked, present in HEAD, or on disk
anywhere in the repo. The two above were removed in commit 364fbff. A third,
web_apps/geo_ip_blocker/profiles.json, appeared later the same day holding four
real key values; it was never tracked (the pattern worked, because it was new) and
was deleted after confirming all four values were byte-identical to .env, so the
app keeps working from environment variables.
Do not read the resolution as "this cannot happen again." Any app with a Settings
modal recreates its profiles.json the moment someone saves a profile. Deleting the
file is not the protection — the .gitignore pattern is, and it only works for files
that are not already tracked. Before committing, git status is the check that
matters.
Committed history is clean — do not rotate on this basis. An earlier version of this entry claimed the values were already in committed history and the keys should be rotated. That was wrong, and it is an expensive thing to get wrong. Verified 2026-08-14:
- only one commit (
426d2fa) ever touched either path, and the blob is{}in both git log --all --full-history -p -- <both paths>adds no credential-valued lines- every one of the 645 blobs across all 262 commits was searched for the four exact secret strings from the working-tree file: zero matches
The live keys existed only in the working tree, never in a commit. The risk was
prospective — a git add -A or git commit -a would have committed them — not
historical. No rotation was needed, and none was done.
Diagnostic subtlety: git check-ignore lies about tracked files. It skips paths
present in the index and reports them as not-ignored, which reads as "your pattern
is broken" when the pattern is fine. Separate the two questions:
git check-ignore -v --no-index <path> # is the PATTERN right? -> yes, matches line 13
git ls-files -c <path> # is the file TRACKED? -> yes, that's the problemChecking only the first form is what makes a tracked credential file look protected.
Also note git check-ignore needs git on PATH; if it is missing, the command
fails and a naive if git check-ignore -q "$f" shell test silently reports every
path as unignored.
When adding a Settings modal to a new app: write to a gitignored filename, set
0600 on POSIX, mask secrets on read-back (return "***" rather than the value),
and never echo values into logs or responses.
Interactivity in setup_env.py is gated on one line:
interactive = sys.stdin is not None and sys.stdin.isatty() and not args.skip_credentialsMeasured from inside an agent tool call in this repo:
stdin isatty: True
stdout isatty: True
So the bare form takes the interactive branch and blocks in getpass. Nobody is
on the other end of that tty — command output is only returned to the agent when
the command exits, so the user never sees the prompt and cannot answer it. The
run hangs until the tool timeout. It does not skip gracefully.
The misleading evidence that makes this easy to get backwards. Running
setup_env.py --skip-credentials prints:
[warn] Skipping credentials (non-interactive mode)
That message proves nothing about isatty. The flag sets interactive to False
by itself through the and not args.skip_credentials clause, so the
non-interactive branch is taken whether or not a terminal is present. Do not
read it as evidence that tool calls lack a tty. They have one.
The module docstring's "safe to run from automation and from Kiro hooks" is correct for hooks but should not be extended to agent tool calls.
Hook command actions are the opposite case: they receive the session JSON piped
on stdin, so stdin is a pipe, isatty() is False, and the non-interactive branch
runs. A hook therefore will not hang — but it also collects nothing, beyond
reusing whatever os.environ already holds. (Reasoned from the hook contract;
the tty result above is measured, this is not.)
Consequence worth stating plainly: credentials cannot be collected by automation at all. Not from a tool call, not from a manual hook. The working paths are:
- a human terminal, including the IDE's integrated terminal:
setup_env.py --credentials-only(input hidden bygetpass) - a shell that already exports the four variables, then
setup_env.py --skip-credentials, which reusesos.environnon-interactively - a web app's Settings modal — subject to the plaintext-storage caveats in the entry above
This is a useful boundary rather than only a limitation: any channel an agent
could prompt through would place API keys in the agent's context and in the
session transcript. The isatty gate is what keeps them out.
Repo Layout Claims in Steering Are Stale — Wrong Entry Points, Dead fileMatch Patterns (discovered 2026-08-14)
Several always-loaded and fileMatch steering claims describe a layout the repo no longer has. These matter more than ordinary doc rot, because agents act on them without checking.
1. Not every web app has a serve.py. project-setup.md and AGENTS.md both
stated "Each has a serve.py and a fixed port." That is false for five of the
twelve apps in web_apps/:
| Entry point | Apps |
|---|---|
serve.py |
inventory_dashboard, cellular_health_dashboard, alert_dashboard, geo_ip_blocker, host_identity_copier, assign_sdk, web_app_template |
config_builder.py |
config_builder |
script_manager.py |
script_manager |
ncm_api_key_encryptor.py |
ncm_api_key_encryptor |
router_lookup.py |
netcloud_router_lookup |
app.py |
cisco_to_cradlepoint_zfw_converter |
web_apps/README.md compounded it with a generic
python3 web_apps/<app_name>/serve.py. Run ls web_apps/<name>/ before launching.
Both files now carry the table.
2. ncm-api-development.md never loaded for web app code. Its pattern was:
fileMatchPattern: "{scripts/**/*.py,ncm/**/*.py,ncm2/**/*.py,dashboards/**/*.py}"
dashboards/ does not exist, and web_apps/**/*.py was absent — so the endpoint
routing table, the trailing-slash rule, the pagination and deprecation rules did not
auto-load when editing web app Python, which is where most of this repo's API calls
live. AGENTS.md meanwhile advertised the section as applying to web_apps/, so the
mirror promised coverage the Kiro config did not deliver. Pattern corrected to
{scripts/**/*.py,ncm/**/*.py,ncm2/**/*.py,web_apps/**/*.py}.
Worth generalizing: a fileMatchPattern naming a directory that does not exist fails
silently. Nothing warns you; the steering simply never activates. When adding or
renaming a top-level directory, grep .kiro/steering/*.md front matter for the old
name.
3. dashboards/cellular_health/ does not exist. web-ui-standards.md pointed
its dashboard reference implementation at that path; the real one is
web_apps/cellular_health_dashboard/. Its own fileMatchPattern also carried a dead
**/dashboards/** clause. Both fixed.
4. scripts/script_manager/csv_files/ does not exist. code-standards.md and
AGENTS.md both instructed storing CSV exports there. The real path is
web_apps/script_manager/csv_files/. Both fixed. The same stale path was also sitting
in .gitignore, protecting nothing while the real csv_files/ went unignored; removed
2026-08-14.
5. ncm2/ is untracked. It is referenced by ncm-api-development.md's
fileMatchPattern and by AGENTS.md, but git ls-files ncm2/ returns zero files
while ncm/ returns eleven. It exists only in this working tree, so a fresh clone
will not have it. Do not import from ncm2 in committed code, and do not cite it
in docs as though contributors have it. Use ncm/ — that is the packaged SDK.
6. web_apps/mac_whitelist_copier/ is an empty directory. Zero files, so it is
not an app and git does not see it at all (git does not track empty directories).
Excluded from the README tables.
7. Entry-point docstrings drift too. web_apps/alert_dashboard/serve.py's
docstring told you to run web_apps/custom_alert_dashboard/serve.py and open port
8060; the directory is alert_dashboard and PORT = 8065. Fixed 2026-08-14, and its
usage line now uses the activate form rather than .venv/bin/python, since a
long-running server launched the bare way boots fine and then fails every API call.
The other eleven entry points were swept the same day — docstring path and port now
match source in all twelve, so treat a mismatch as a regression rather than the norm.
api-v2-full-reference.md Query-Param Tables Can Be Incomplete — Cross-Check ncm.py (discovered 2026-08-28)
The generated tables in docs/api-v2-full-reference.md are not guaranteed to list
every filter an endpoint accepts. Read them as a floor, not a ceiling.
Confirmed instance: locations. Its table listed only id, id__in, limit
and offset — no router or router__in. Meanwhile the shipped client and two
shipped scripts depend on a router filter:
ncm/ncm/ncm.py→get_locations()declaresallowed_params = ['id', 'id__in', 'router', 'router__in', 'limit', 'offset']scripts/export_locations.pyandweb_apps/script_manager/scripts/Export Locations.pyboth batch withn2.get_locations(router__in=batch)
Every other router-scoped endpoint in the same reference (configuration_managers,
device_app_states, net_devices, router_alerts, router_state_samples,
router_stream_usage_samples) does document router__in, so locations was the
outlier rather than the rule. The two rows have been added to the reference with a
note.
Why it costs you: a reader who trusts the table concludes there is no server-side router filter for locations, and fetches the whole collection to filter client-side. On a large account that is many paginated requests instead of a batched query.
Verification status. The discrepancy between the doc and the shipped code is
directly observed. Whether router__in behaves as expected against a live account is
UNVERIFIED — no request was made to /api/v2/locations/ when this was written.
Also UNVERIFIED: whether any endpoint other than locations has an incomplete
table. One instance was found; the rest were not audited.
Practical check before concluding a filter does not exist: grep the matching method's
allowed_params in ncm/ncm/ncm.py, and grep scripts/ and web_apps/ for existing
callers. Working code that predates the generated doc is the stronger evidence.
Paginating the full /api/v2/firmwares/ collection yields substantially more rows
than there are distinct firmware records. Measured against a live account with
limit=500: 20,000+ rows returned for 17,677 unique id values. Duplicates also
show up inside a single small page — ?limit=3&version=7.24.40 returned firmware id
9395 twice in three rows.
Always dedupe by id when accumulating pages:
records = {}
for page in pages:
for item in page:
records[str(item["id"])] = item # last write wins; records are identical
firmwares = list(records.values())Why it costs you: a naive count = len(all_rows) overstates the catalog by ~15%, and
any per-record work (HEAD requests, hashing, table rows) is done several times over.
A UI that renders the raw rows shows visibly duplicated entries.
It is not only /firmwares/, and not only across pages. /products/ does the same
thing within a single response: one ?limit=500 request returned 154 rows for 102
unique ids on one page. So "duplicates come from pagination" is the wrong mental model
— treat any v2 list response as a multiset and dedupe by id as a habit, not just on
large collections.
Verification status. Directly observed against a live account on 2026-09-25 for
/firmwares/ (full pagination and a single filtered page) and /products/ (single
page). Whether the duplication is stable (same ids duplicated on every pull) is
UNVERIFIED — dedupe unconditionally rather than relying on a fixed pattern. Only
these two endpoints were checked; others may or may not behave the same way.
model + version does not identify a firmware image. NCM publishes multiple
distinct builds under one version string, each with its own id, built_at, and
hash. Measured on a live account: 3,426 of 5,389 model/version pairs — about two
thirds — had more than one distinct hash, some with six builds.
IBR600C 7.26.100 built 2026-09-16T00:34:43Z hash d53a0427…
IBR600C 7.26.100 built 2026-09-21T17:37:17Z hash 85b622ca…
Why it costs you: any code that keys firmware by f"{model}-{version}" — cache keys,
download filenames, dedupe sets, "do I already have this?" checks — silently collapses
several different images into one. Downloading "all builds of 7.26.100" to
IBR600C-7.26.100.bin writes one file and silently drops the rest.
built_at does not break the tie either. Distinct images regularly share an
identical build timestamp, so model + version + built_at is still not unique:
2,166 of 15,505 (model, version, built_at) triples mapped to more than one hash
— 14% — with up to four images on a single triple.
AER2200 7.1.20 built 2019-08-27T15:05:37Z id 1654 hash c1a467895797…
AER2200 7.1.20 built 2019-08-27T15:05:37Z id 1661 hash befd3dafac2b…
The practical consequence: "take the newest build" is not always a decision. For
681 of 5,389 model/version pairs, the newest built_at still maps to several
different images, so a max(built_at) pick silently chooses arbitrarily between them.
hash (or id) is the only field that reliably separates them.
Use id as the identity, and where a human-readable name is needed, disambiguate with
built_at and a short hash prefix — the hash is the part doing the real work:
groups = {}
for fw in firmwares:
groups.setdefault((fw["model"], fw["version"]), set()).add(fw["hash"])
def filename(fw):
base = f"{fw['model']}-{fw['version']}"
if len(groups[(fw["model"], fw["version"])]) > 1:
return f"{base}-{fw['built_at'][:10].replace('-', '')}-{fw['hash'][:6]}.bin"
return f"{base}.bin"Also note built_at is UTC. If you bake the build date into a filename, render the
date in UTC in any accompanying UI too, or a user in a negative offset sees "Sep 15"
next to a file named …20260916….
If a UI shows these rows, show the hash (first 7 chars is enough). Otherwise rows with
the same model, version, and build date look like duplicated data and users reasonably
report it as a bug.
What the pairs actually are: multiimage builds. The distinguishing information is
in the url filename, not in any metadata field. NCM publishes most builds twice — a
standard image and a multiimage one — sharing the model, version, and build timestamp:
/E3000-multiimage-2026-09-16T00%3A39%3A53.bin hash 83e64d1b…
/E3000-2026-09-16T00%3A39%3A53.bin hash 31e827b0…
Both point at product 56 (E3000), same version, same built_at. This accounts for
the collisions exactly: 1,963 of 17,677 records carry the multiimage tag, and 2,166
(model, version, built_at) triples have both a standard and a multiimage image.
So parse the variant out of url and treat it as part of the identity:
def image_variant(url):
return "multiimage" if "-multiimage-" in urllib.parse.unquote(url).lower() else ""Two consequences:
- Surface the variant in any UI. Without it the rows look like duplicated data.
- If you parse the model from
urlas a fallback, strip the variant tag first, or you invent a phantom model namedE3000-MULTIIMAGE.
Verification status. Directly measured across a full 17,677-record catalog on
2026-09-25, including the triple collision counts and the multiimage correspondence.
What multiimage means functionally (dual-boot slots, combined hardware variants,
something else) is UNVERIFIED — the API exposes no field describing it, and it was
not tested against a device. Check NCM release notes before assuming which variant a
deployment needs; don't automate that choice on the strength of the API alone.
Code that does CDN_BASE + firmware["url"] and names the result .bin is wrong for a
minority of records. Across a 17,677-record catalog:
url shape |
Count | Example |
|---|---|---|
| Path, standard image | 15,231 | /170900-022-2025-08-21T14:26:57.bin |
| Path, multiimage variant | 1,955 | /AER2200-multiimage-2019-08-27T15:05:37.bin |
Path, not .bin |
450 | /cr4250-core-image-cr4250-upgrade-1.0-20190827161513.vmp |
| Fully-qualified URL | 41 | https://d251cfg5d9gyuq.cloudfront.net/CBA250-2012-04-16T11%3A00%3A57.bin |
Three separate traps:
-
41 records are absolute URLs. Prepending the CDN base produces
https://…cloudfront.nethttps://…, which fails. Worse, a guard likeif not url.startswith("/"): continueskips them silently. Normalize instead:def image_url(url): if url.lower().startswith(("http://", "https://")): return url return FIRMWARE_CDN + url
-
A second CDN host appears, and it is dead. Of those 41, 22 point at
d30wehszhs5ixg.cloudfront.net, which does not resolve (DNSNameResolutionError, tested 2026-09-25) — so those images are unobtainable regardless of client code. The other 19 use the usual host and download fine. Report such failures rather than hiding them; a user needs to know the image is gone, not watch a download silently not happen. -
450 records are
.vmp, not.bin(CR4250). Derive the extension from the URL rather than hardcoding it, or files land on disk misnamed.
Also worth knowing: individual CDN objects can return 403 even when sibling objects
for the same model and version return 200 (observed on one CR4250 .vmp). Treat a
per-image HTTP failure as normal and surface it per row.
Verification status. Counts measured directly against a live catalog on 2026-09-25.
The dead host, the working absolute-URL downloads (SHA-1 verified), and the 403 were all
confirmed with live requests. Whether the dead host affects other accounts, and whether
d30wehszhs5ixg was ever reachable, is UNVERIFIED.
allowed_params for /firmwares/ is id, id__in, version, version__in,
limit, offset — there is no product or product__in. Listing the firmware
available for one model therefore requires pulling the whole collection and filtering
client-side; filtering by version server-side is the only narrowing available.
The model name is also not on the firmware record. product is a URL
(.../api/v2/products/63/), so resolving names means a separate /products/ pull (one
page, fast) and a join on the extracted id. That join is authoritative when it
resolves — but you need both a fallback and a dedupe step:
/products/repeats records too, and not as a pagination artifact: a single?limit=500response returned 154 rows for 102 unique ids. Key your lookup dict byid(idempotent) rather than appending to a list and trusting the length. Unique ids and unique names both came to 102, so no two distinct products shared a name.- Not every referenced product exists in
/products/. Across a full catalog, 12 of the 113 product ids referenced by firmware records were absent, covering 274 records (1.6%) — ids103,99,78,77,55and others, presumably retired products. Those firmwares get no name from the join, so a fallback is genuinely required; just don't reach for it first.
Parsing the model out of url looks like an equivalent shortcut, since most paths are
shaped /{MODEL}-{built_at}.bin (e.g. /IBR600C-2026-09-16T00%3A34%3A43.bin). It is
not. Measured against a 17,677-record catalog, a regex anchored on the ISO build
timestamp:
- fails entirely for 491 records (2.8%) — the 450 CR4250
.vmppaths use a compact timestamp (…-20190827161513.vmp) and the 41 absolute URLs don't match the leading/{MODEL}-shape at all; - returns a different, legacy name for 9 records —
/NCSG-KVM-…and/NCSG-…for products actually namedNCXSG-KVMandNCXSG-Azure. The URL keeps the oldNCSGbranding, so a URL-derived catalog splits one model across two names; - needs the variant stripped first, or
multiimagebuilds produce a phantomMODEL-MULTIIMAGEmodel (see the multiimage entry above).
So: join on product first, fall back to URL parsing only for the ~1.6% of records
whose product id is missing, and strip the variant tag when you do.
Verification status. Parameter list confirmed against
docs/api-v2-full-reference.md, get_firmwares() in ncm/ncm/ncm.py, and a live
account on 2026-09-25. The parse-failure and name-mismatch counts were measured directly
against the same catalog. Whether an undocumented product filter is silently accepted
was not tested.
The /beta/ prefix entry above is wrong for these two endpoints. The official
OpenAPI spec at
https://developer.cradlepoint.com/swagger/spec/Modem_Software_Upgrades.json
documents them without the beta segment:
GET /api/v3/modem_software_versions
GET /api/v3/modem_upgrades
POST /api/v3/modem_upgrades
PUT /api/v3/modem_upgrades/{id}
Three sources in this repo say otherwise and should not be trusted on this point:
the Postman collection (whose folder is even titled "Modem Software Upgrades
(BETA)"), scripts/test_modem_management_api.py, and the /beta/ prefix entry
earlier in this file. The ncm SDK's live methods (ncm/ncm/ncm.py L4515-4790)
use the non-beta path and are correct; its commented-out older methods
(L5787+) use the beta path and are stale.
The likely history: the endpoints were beta when the Postman collection and the
test script were written, and were promoted without those artifacts being updated.
Treat the Swagger spec as authoritative and the /beta/ prefix as a fallback for
older deployments. web_apps/modem_upgrades requests the non-beta path first and
falls back to beta on a 404.
The Postman collection documents the list parameters as plain group and
job_type. The Swagger spec documents them as JSON:API filters, and marks the
first two required:
| Parameter | Required | Notes |
|---|---|---|
filter[group] |
Yes | integer group ID |
filter[type] |
Yes | modem_upgrade_parent | modem_upgrade_child |
filter[modem_upgrade_parent] |
Only when filter[type]=modem_upgrade_child |
parent job ID |
scripts/test_modem_management_api.py gets this right. Postman's job_type
spelling appears nowhere in the spec — do not use it.
Note that filter[type] being required means there is no "just list everything
for this group" call; you must decide up front whether you want jobs or per-modem
activities.
Both modem endpoints page with page[size] (default 20, maximum 50) plus
page[after] / page[before] cursors, and return links.first/last/next/prev.
Asking for more than 50 is a 400 Bad Request, which is easy to trip if you carry
over the v2 habit of limit=500. Follow links.next until it is null rather than
computing offsets.
Both endpoints also return 204 No Content — not an empty 200 — when nothing
matches ("Could not find available modem packages" / "No jobs or activities
found"). Code that assumes a JSON body on any 2xx will throw on the empty
response; treat 204 as an empty result set.
The available-firmware field is package_version. The SDK's disabled older code
and several plausible guesses use available_version or version, which are not
in the spec. A reader looking for the wrong key silently gets a blank version
column rather than an error.
Full attribute set for modem_software_versions:
| Attribute | Meaning |
|---|---|
carrier |
carrier name, e.g. verizon, att, sprint |
modem_type_name |
modem module type, e.g. LP4 |
package_version |
the modem firmware package version |
min_router_version |
minimum NCOS version the router must run to accept it |
min_router_version is worth surfacing in any upgrade UI: a modem package can be
available to a group while some routers in that group are on too old an NCOS build
to take it.
carrier + modem_type_name from a record here are exactly the two required
attributes for POST /modem_upgrades, so the versions endpoint is the intended way
to populate an upgrade request.
Confirmed against the Swagger spec: POST /modem_upgrades requires
data.relationships.group and accepts no other target. There is no router,
net_device, or ID-list attribute anywhere in ModemUpgradeRequest, so a job
necessarily applies to every matching modem in the group.
This is easy to get wrong when building tooling: a UI that lets an operator select specific routers and then submits a group-scoped job will upgrade far more devices than the operator chose. Either scope the UI to groups, or say plainly that a device-level selection is a display filter only. To genuinely narrow the blast radius, the devices have to be moved into their own group in NCM first.
The only in-request narrowing levers are connection_states (e.g. restrict to
["connected"]) and overwrite (whether to re-push to modems already on the
target version).
modem_upgrades is a parent/child collection. filter[type]=modem_upgrade_parent
returns jobs; filter[type]=modem_upgrade_child (optionally narrowed with
filter[modem_upgrade_parent]={id}) returns one record per modem, and those
children are where per-device status lives.
Worse, there is no documented GET /modem_upgrades/{id}. The spec defines only
the filtered list and PUT /modem_upgrades/{id}; the SDK's get_modem_upgrades( modem_upgrade_id=...) builds an item-read path that appears nowhere in the spec.
To read one job's current state portably, list the group's parents and match on ID,
or attempt the item read and fall back to the list on a 404.
Consequences for a monitor:
- Fetch children on every poll, not just the parent.
- Do not rely solely on the parent's
status. Parent counters (modem_count,success_count,failed_count) are documented but not always populated. Treating "every child is terminal" as an additional completion signal covers the gap — and a wall-clock timeout is still required, since a job can stall indefinitely on a modem that never reports back.
The status enum is documented, and none of the values are the ones you would
guess:
| Status | Terminal? | Meaning |
|---|---|---|
created |
no | job accepted |
queued |
no | waiting to start |
initializing |
no | starting up |
running |
no | in flight |
finished |
yes | completed |
error |
yes | failed |
canceled |
yes | cancelled (one l) |
Note finished rather than complete/completed, error rather than failed,
and the single-l American canceled. Code that matches on completed or
cancelled will never fire. Since this is a young endpoint family, matching the
documented values exactly and falling back to substring classification for
anything unrecognized is the durable approach.
Modem Upgrade Child Records Are Under-Documented — Expect Only net_device IDs (discovered 2026-09-28)
The Swagger spec reuses a single schema (ModemUpgradeResource) for both parent
jobs and child activities, and that schema documents only a group relationship
and job-level attributes. So the spec does not describe how a child identifies
the modem it belongs to, even though that is the whole point of a child record.
In practice children reference a modem by net_device ID, and the SDK's disabled
older get_group_modem_upgrade_device_summary lists router_name,
net_device_name and current_version as available fields — suggesting the API
returns more than the spec admits. Because the shape is not pinned down, extract
these fields defensively (probe relationships.net_device, then
attributes.net_device_id, then attributes.net_device) rather than assuming one
location, and keep the raw attributes dict so unexpected fields are still visible.
A bare net_device ID tells an operator nothing, so an upgrade report built from v3
data alone reads as a list of opaque integers.
Resolve them through v2: GET /net_devices/?id__in=<ids>&expand=router returns the
inline router object (name, state, model) plus the modem's version/ver_pkg,
which is the "firmware before the upgrade" column a report needs. Chunk the IDs at
100 per call and cache the results — the mapping does not change during a job.
Note the two APIs must both be authorized for this: v3 Bearer token for the upgrade records, v2 header keys for the name lookup. An app that resolves names needs both credential sets, which is unusual — most tools in this repo need only one.
This is the most consequential thing to know before building anything on
operation: "preview", and it is not in the spec.
A preview job never creates child activity records. Verified against a live
account: after POST /modem_upgrades with operation: "preview", the child query
GET /api/v3/modem_upgrades?filter[group]=636741
&filter[type]=modem_upgrade_child&filter[modem_upgrade_parent]=67683780
answers 200 with {"links":{...},"data":[],"meta":{...}} — an empty array, and it
stays empty indefinitely (polled well past 100 seconds on a job NCM had already
finished evaluating). The entire result of a dry run is the parent's
modem_count.
Sequence observed:
POSTreturns immediately:status: "created",modem_count: null.- Within about a second NCM evaluates the group, sets
modem_count, and settlesstatusat"preview". - No children, ever.
So a UI that promises "here are the modems this would touch" cannot get that from
the API. Options: show the count as authoritative and derive a candidate list
locally from v2 net_devices (clearly labelled as derived — see below), or show
only the count.
Do not build a long poll waiting for preview children. It will spin until its
timeout on every run. Poll only until modem_count stops being null.
If you do derive a candidate list locally, expect it to disagree with NCM. On a
group of 4 online E320s with 8 5GB modems, all on older TMO firmware and none on
the target generic package, NCM reported 3 eligible while matching on modem
type + "version differs" yielded 8.
The eligibility rule is not documented and clearly involves more than type and
version — plausibly min_router_version against each router's NCOS build, SIM or
carrier state, or per-router constraints. Do not present a locally derived list as
the scope of the job. Treat modem_count as authoritative and label anything you
compute yourself as indicative.
Related: a group whose router is offline reported modem_count: 0 even with
overwrite: true, so offline devices appear to be excluded outright.
The spec's status enum is created, queued, initializing, running,
error, finished, canceled. A live preview job settles at status: "preview", which is not in that list.
Two lessons: the enum is incomplete, so match the documented values but always
have a fallback branch for unrecognized ones; and "preview" is a settled state
rather than an in-flight one, even though nothing about the string says so. Code
that waits for a terminal status on a preview job will wait forever.
The spec defines no item-read endpoint, and the live API refuses it with 403 Forbidden rather than 404 Not Found. The ncm SDK's
get_modem_upgrades(modem_upgrade_id=...) builds exactly this URL, so that call
path cannot work.
Two traps follow:
- Do not treat 403 as "bad token" on v3. The obvious handler — map 401/403 to
"check your credentials" — produces a completely misleading error here, since the
same token succeeds on the list and on
POST. On these endpoints 403 means "this operation is not available", and the spec lists403 Forbidden - Request forbiddenas a normal response for every documented operation. - A fallback keyed on 404 will not fire. Anything falling back from the item read to the filtered list must trigger on 403 as well.
Read a single job with filter[group] + filter[type]=modem_upgrade_parent and
match the ID client-side.
Requesting children without a parent gives a genuinely helpful 400:
{"errors":[{"detail":"filter[modem_upgrade_parent] is required when filter[type]=modem_upgrade_child",
"status":"400","source":{"pointer":"/data"},"code":"invalid"}]}Worth knowing because it is easy to destroy this message. If your client retries a
400 using a different parameter spelling — e.g. falling back from filter[type]
to Postman's job_type — the retry replaces the clear message above with a
baffling one about a field you never sent:
HTTP 400: This field may not be null. (/data/attributes/job_type)
Keep parameter-style fallbacks narrow: only retry when the error actually names an
unknown or invalid parameter, and never when it names a missing required one.
A broad "any 400 means try the other spelling" rule turns good diagnostics into
bad ones. (The job_type pointer does reveal that the server's internal serializer
field is named job_type, which is presumably where Postman's spelling came from —
but it is not accepted as a query parameter.)
When implementing that rule, do not use "filter" as the discriminator. It is
the obvious choice and it is worthless: the word appears in both the error you want
to retry (invalid filter: group) and the one you must not (filter[...] is required), so matching on it reintroduces the exact bug. Two things that do work:
match only unambiguous unknown-parameter wording (unknown, unrecognized,
unsupported, not a valid), and explicitly bail out when the message contains
required. Also only consider a fallback at all while the correct spelling is still
undetermined — once a working form is known, a 400 is real and should surface as-is.
The spec documents 204 No Content for "no jobs or activities found" on
modem_upgrades. In practice an empty child query returned 200 with
"data": [] plus a populated links/meta envelope. Handle both: check for a
2xx, then treat a missing or empty data as an empty result rather than keying off
the status code. Responses also carry
meta.pagination.before/after alongside links, which the spec does not mention.
The connection_states filter on POST /modem_upgrades restricts which modems a
job targets, by each modem's own cellular connection state. Its enum has three
values:
connected | disconnected | standby
But the v2 net_devices.connection_state field those modems report has six
observed values (see the net_devices.connection_state vs routers.state entry
above): connected, disconnected, connecting, standby,
standby_connecting, unplugged.
So connecting, standby_connecting and unplugged cannot be expressed in the
filter, and how NCM treats modems in those states is unknown — whether they are
excluded outright, folded into disconnected, or ignored by the filter entirely.
This was not tested: every preview run during this work sent
connection_states: null. Also untested is whether the filter is evaluated once at
job creation or re-checked per modem as the job progresses, which matters because a
modem's state can change mid-job.
Two practical consequences:
- A UI offering exactly the three enum values is correct per the spec but cannot
express "target the modems that are currently
connecting". Don't silently map a sixth-value state onto one of the three — you would be guessing at NCM's rule. - Leaving the filter null (the default) is the safer choice when you want NCM's own eligibility rules to decide, which is also the only behavior verified here.
Beware the related trap: a router can be online while every one of its modems is
disconnected. On a live group of 4 online E320s, all 8 modems reported
disconnected, so a ["connected"] filter there would plausibly have targeted
nothing despite the routers looking healthy. Filter on modem state only when you
actually mean modem state.
The docs/api-v3-endpoints.md reference had fallen out of sync with what the rest
of this repo already knew (the "v3 Beta Endpoints" and regrades entries above, and
the official Swagger specs at developer.cradlepoint.com/swagger/spec/{name}.json).
Cross-checking the specs surfaced four errors in that reference, now corrected:
userswas listed as a v3 CRUD endpoint. It is a v2 endpoint (swagger/spec/users.json→basePath https://www.cradlepointecm.com/api/v2, fourX-header keys, GET only, filtersid/username/email). Moved toapi-v2-endpoints.md.regradeswas listed at/api/v3/regrades/. The real path is/api/v3/asset_endpoints/regrades(and/{id}). GET requiresfilter[action](UPGRADE/DOWNGRADE/LICENSE/UNLICENSE) andfilter[status](RUNNING/COMPLETED/FAILED); POST is JSON:API atomic, required fieldsaction+mac_address. (The regrades path was already correct in the SDK-gotcha entries above — only the reference doc was wrong.)- PCN and NCX endpoints were listed at bare
/api/v3/...with trailing slashes. Both families are under/api/v3/beta/...with no trailing slash, per their specs (private_cellular_networks,private_cellular_radios,exchange_sites,exchange_resources). This matches the "v3 Beta Endpoints Must NOT Have Trailing Slashes" entry above. subscriptionswas listed with a trailing slash and an item-read/{id}path. The spec defines onlyGET /api/v3/subscriptions(no trailing slash, no/{id}, GET only),page[size]max 50,filter[start_time]/filter[end_time].
Lesson: the Swagger spec JSON is directly fetchable and authoritative. When the endpoint reference and the discovered-issues log disagree, the log (and the spec) won — the reference was the stale artifact.
The second element of a config diff ([updates, removals]) is described in
api-configuration.md as "paths to remove from defaults". That phrasing hides an
asymmetry that reverses the correct approach depending on which endpoint you are
writing to.
The config diff is not a command that gets executed and discarded; it is persisted
state that NCM layers on top of what the device would otherwise inherit
(Target = Group config + Individual config). So a removals entry is a standing
instruction to suppress a path, not a one-time delete.
At group scope (/groups/{id}/), a removals path prunes something the group
would otherwise contribute — the intuitive reading, and why
common-patterns.md correctly recommends "prefer PATCH with removals over PUT" for
mirroring collections between groups.
At device scope (/configuration_managers/{id}/), the stored diff is the
device's override layer. A removals path there suppresses the path from what the
group supplies. Consequences:
- To make a device stop overriding the group and start inheriting it, delete the key
from
updates(element 0). Adding it toremovalsdoes the opposite: it persists a new device-level override that blocks the group's value entirely. - A
security-rooted path sitting in a device's removals list is itself blocking inheritance of the group's firewall config, so restoring inheritance means clearing those entries too — not just theupdatesbranch. - PATCH cannot do this. PATCH merges
updates, so no PATCH body removes a key from it. The operation is read-modify-write with PUT, sending the device's complete diff minus the one branch. PUT's "resets unmentioned fields to defaults" hazard does not apply, because you are sending the whole document, not a fragment — and the documented device-config reset is itself a PUT of[{}, []](api-configuration.md).
The "PATCH CAN Remove Config Items" entry above remains accurate: PATCH does accept removals, and that is the right tool for pruning a group's own config. It is the wrong tool for restoring group inheritance on a device.
Verified end-to-end against 400 live devices while building
web_apps/revert_device_firewall/ (74 carried device-level security config). The
read, diff and payload-construction paths were exercised against production data;
the PUT itself was verified against a stubbed session rather than live devices.
Related trap: root-level firewall and root-level security are different things.
Zone firewall config (zones, filter policies, forwardings), IPS and app sets live
under security; firewall holds port forwarding and similar. Many devices carry
both. Removing "the firewall config" by name alone will take out the wrong branch.
A batched get_configuration_managers(router__in=…) can return fewer records than
router IDs you asked for. Observed against a live account while scanning 400
routers for web_apps/revert_device_firewall/: one device (a CR4250) had no
configuration manager record at all, while the other 399 did.
Complements the "Configuration Manager ID ≠ Router ID" entry above: that one says don't assume the IDs match, this one says don't assume a record exists.
Why it bites: the natural way to consume a batched read is to zip the response against the request list, or to index into it positionally. Both silently misalign when a device is missing — every device after the gap gets attributed to the wrong router, which in a config-writing script means pushing one device's config onto another's. A length check does not save you either, since the response is also capped by pagination.
Map explicitly on the router field of each returned record and treat absence as a
skip, not an error:
records = client.get_configuration_managers(
router__in=','.join(router_ids),
fields='id,router,configuration,suspended,synched',
limit='all',
)
by_router = {}
for rec in records or []:
rid = str(rec.get('router', '')).rstrip('/').split('/')[-1] # it is a URL
if rid:
by_router[rid] = rec
missing = [r for r in router_ids if r not in by_router] # report, do not crash
for rid in router_ids:
rec = by_router.get(rid)
if rec is None:
continue # nothing to write for this deviceNote that the router field comes back as a resource URL
(https://.../api/v2/routers/123456/), not a bare ID, so it needs parsing before it
can be used as a dict key. The root cause of the missing record was not
investigated — the practical point is that the gap exists in real accounts and must
be handled.
Seven apps used to serve their logo and favicon from a sibling directory,
web_apps/script_manager/static, via .parent.parent / "script_manager" / "static".
Copy any one of those app folders on its own and the path no longer resolves, so
/static/logo.png returns 404 and the header renders a broken-image icon.
The failure is quiet in the worst way: most of those apps guarded the mount with
if STATIC_DIR.is_dir(), so the server starts normally and every API route works.
Nothing in the logs indicates a problem.
The rule is that each app is self-contained — no app reads files from another at
runtime. Each app owns its static/ folder:
STATIC_DIR = Path(__file__).resolve().parent / "static" # own folder
if STATIC_DIR.is_dir():
app.mount("/static", StaticFiles(directory=str(STATIC_DIR)), name="static")Duplicating a 6 KB logo per app is the right trade. Resist the urge to "fix" a
missing shared asset with onerror="this.style.display='none'" on the <img> —
that hides the symptom and leaves the app unbrandable when shipped.
All seven were converted on 2026-10-06 (alert_dashboard,
cellular_health_dashboard, geo_ip_blocker, host_identity_copier,
inventory_dashboard, ncos_downloader, revert_device_firewall), each with its
own static/logo.png and logo_dark.png. config_builder and
netcloud_router_lookup already had their own.
Still outstanding: geo_ip_blocker and host_identity_copier also reach to the repo
root with sys.path.insert(0, str(PROJECT_ROOT / 'ncm')) to import the vendored SDK,
and load .env from PROJECT_ROOT. That is a repo dependency rather than a
cross-app one, but it breaks the same way in a standalone copy. Every other app just
does import ncm against the pip-installed package.
Two related points for standalone distribution:
- Per-app
requirements.txt. The rootrequirements.txtis the union across all apps, so it installs packages a single app does not need. A per-app file listing only what that app imports keeps a shipped copy small. Derive it from the actual imports rather than copying the root file. - Python floor is 3.10, not 3.9. The repo standard says 3.9+, but current
fastapi,uvicorn,starlette,python-dotenvandurllib3all publishRequires-Python >=3.10. Any FastAPI app in this repo therefore needs 3.10+ on a fresh install; pip will refuse on 3.9. Verify withpython -c "import importlib.metadata as m; print(m.metadata('fastapi')['Requires-Python'])".
Every web_apps/ entry point prints its URL with plain print() before calling
uvicorn.run(). Start one in the background with output redirected — which is the
prescribed way to launch these — and that banner never reaches the log:
$ .venv/bin/python serve.py > run.log 2>&1 &
$ cat run.log
INFO: Started server process [34287]
INFO: Uvicorn running on http://127.0.0.1:8090 (Press CTRL+C to quit)
The "Open http://localhost:8090 in your browser" line is missing. Cause: when stdout
is not a tty it is block-buffered (4 KB), so a ~150 byte banner sits in the buffer
indefinitely. Uvicorn's lines appear because they go through logging to stderr,
which is not block-buffered. Kill the process and the buffered stdout is discarded
entirely, so the banner is never written at all.
Measured on revert_device_firewall: grep -c "Open http" gives 0 under default
buffering and 1 with -u. Both forms fix it:
.venv/bin/python -u serve.py > run.log 2>&1 & # or
PYTHONUNBUFFERED=1 .venv/bin/python serve.py > run.log 2>&1 &The trap is the inference, not the missing text: absence of the banner is not
evidence the app failed to start. The server in the example above was serving
fine. Confirm startup from uvicorn's own stderr line, or by hitting the port
(curl -s -o /dev/null -w '%{http_code}' http://localhost:<port>/), rather than by
grepping for the app's own print output. Note also that a print() diagnostic added
while debugging a backgrounded app may never appear for the same reason — use
flush=True, stderr, or -u.
zsh (the default macOS shell) applies filename globbing to --include=*.py. Nothing
in the working directory matches that literal pattern, so with zsh's default
nomatch option the entire command aborts before grep runs:
$ grep -rn "pattern" --include=*.py . || echo "NONE - clean"
zsh: no matches found: --include=*.py
NONE - clean # <-- grep never ran; this "clean" is meaningless
The || echo fallback converts a shell error into what reads like a verified
negative result. This happened during a real cross-app dependency audit in this repo
and briefly produced a false all-clear.
Quote the pattern, or prefer the dedicated grep tool:
grep -rn "pattern" --include='*.py' .More generally, when a command's absence of output is the evidence for a claim, confirm the command actually executed — check the exit code or run it once in a form you know produces hits.
A cached NodeList goes stale on every selection click, so loops select one row (discovered 2026-10-07)
The picker tables in these apps redraw with container.innerHTML = … inside render(),
and render() runs on each selection click. Every existing element reference is
discarded, so a loop over a NodeList captured beforehand clicks detached nodes after
the first iteration:
const rows = document.querySelectorAll('#listContainer tr[data-id]');
for (let i = 0; i < 12; i++) rows[i].click(); // selects exactly 1The clicks land on orphaned nodes and are swallowed by the document-level delegation, with no error. The UI reported "1 router selected" after twelve clicks, which reads as a broken selection handler rather than a stale-reference problem in the harness.
Re-query inside the loop, and make the step idempotent so a re-run does not toggle selections back off:
for (let i = 0; i < 12; i++) {
const rows = document.querySelectorAll('#listContainer tr[data-id]');
if (!rows[i].classList.contains('selected')) rows[i].click();
}Applies to any browser-driven check of an app that re-renders a list on interaction —
so all of the pickers and dashboards here. The same trap makes a verification step
pass when the count happens not to be asserted, so assert the resulting count
(#selectionSummary, the action button's label) rather than assuming the clicks took.
Related: verify the frontend against a stub backend when the live fetch is the
bottleneck. On a ~10.8k-router tenant, /api/inventory had not returned after
several minutes, which blocks any UI check that needs populated rows. A throwaway
http.server that serves the app's real index.html with synthetic /api/* JSON
exercises the actual frontend code paths in seconds, with no risk of touching fleet
config — useful for destructive apps specifically, since a stub cannot write. The
app's own internals are typically wrapped in an IIFE and unreachable from
page.evaluate, so driving the real buttons against a stub is the only way to test
the logic rather than re-implementing it. Keep the stub outside the app directory and
delete it afterward; apps must stay self-contained. (The inventory stall itself is
unexplained and not confirmed to be a bug.)
A root AGENTS.md is loaded into every Kiro request, duplicating inclusion: always steering (discovered 2026-10-08)
Kiro auto-loads a repo-root AGENTS.md as a workspace rule, injected in full on every
request — the same standing cost as a steering file with inclusion: always, but with
no front matter to scope it. Confirmed directly: creating AGENTS.md caused it to
appear as ## Included Rules (AGENTS.md) [Workspace] on the very next turn, alongside
project-setup.md. A root claude.md was not injected in the same session.
So AGENTS.md is the wrong place for anything already covered by always-on steering.
This repo's copy had mirrored all six steering files in full, which both doubled the
always-on budget and drifted — it still described users as a v3 CRUD endpoint after
docs/api-v3-endpoints.md had been corrected to v2 GET-only, and listed 12 web apps
against 16 on disk. Prior entries in this file record four earlier rounds of the same
drift (stale CSV path, serve.py assumption, fileMatchPattern, untracked ncm2/).
The file has been deleted. .kiro/steering/*.md is the single source of truth; the
cross-agent entry point is claude.md, which holds the always-applicable essentials
plus an index of which steering file to read when. If you need to serve agents that
read neither, add a pointer file — do not re-mirror the rules.
Generalizes to two things worth separating when auditing agent instructions: where a file is loaded (always-on versus on demand) and whether its content is a copy. A full mirror is a maintenance liability on its own, and an always-on mirror is also a per-request tax. A pointer costs one read only on the turns that need it.
Starlette's TemplateResponse now requires request as the first positional arg; the old (name, {"request": ...}) form fails with TypeError: unhashable type: 'dict' (discovered 2026-10-15)
web_apps/ncm_api_key_encryptor/ returned a 500 on every page load under Starlette
1.6.0. The handler called the legacy signature:
templates.TemplateResponse("index.html", {"request": request, ...}) # old, removedModern Starlette (the deprecation landed in 0.29 and the old form is now gone) expects
the Request first:
templates.TemplateResponse(request, "index.html", {...}) # correctWith the old call, Starlette reads the context dict as the template name and uses
it as a cache key — a dict is unhashable, so you get TypeError: unhashable type: 'dict' from deep inside jinja2/utils.py (self._mapping[key]), not from your code.
The traceback points at the TemplateResponse line but names no obvious cause, which
makes it easy to misread as a Jinja/template-cache bug rather than a signature change.
Two practical notes:
- Drop the now-redundant
"request": requestkey from the context — Starlette injects the request into the template context automatically under the new signature. - This is not app-specific. Any web app in this repo still on the two-arg form
breaks identically the moment Starlette is ≥0.29. When touching an app that renders
Jinja templates, grep it for
TemplateResponse(and migrate any(name, {...})calls. A clean startup banner proves nothing here — the failure is per-request, so confirm by actually hitting the route and asserting a 200.
Root cause of the above is an unpinned, transitive Starlette — "worked then broke with no code change" (2026-10-15)
The TemplateResponse break above surfaced with no change to the app (last commit
2026-06-08) and no recent dependency install — Starlette 1.6.0 had been in the venv
since 2026-08-14. The explanation is that Starlette is never pinned, and never even
listed: both requirements.txt and web_apps/ncm_api_key_encryptor/requirements.txt
declare only fastapi with no version, and Starlette arrives transitively through
FastAPI. So the installed Starlette is "whatever PyPI served the day this venv was
last built," not a deliberate choice. Rebuild the venv on a later day and the version
silently moves — which is exactly how an app can "work a couple weeks ago" and 500 now
with an unchanged tree.
Two generally-applicable takeaways:
- When diagnosing a "worked before, broke now, code unchanged" failure, trust package
mtimes and
dist-infoover memory of when it last ran. The dates here contradicted the "couple of weeks ago" recollection (packages were two months old), and the app's startup banner prints fine regardless because the fault is per-request — so "I saw it running" is not evidence the route worked. - A floating transitive dependency is the real defect. Pinning
fastapialone does not fix it — Starlette is a transitive dependency, andfastapi==0.141.1still resolves a range of Starlette versions, so a rebuild can move it. You must add an explicitstarlette==<version>line alongside thefastapipin. The fix for this repo pinnedfastapi==0.141.1,starlette==1.6.0,uvicorn==0.52.3in every FastAPI app. Note the repo convention that each web app owns arequirements.txtof only what it imports — so the explicit Starlette pin has to be added per app, not just at the root (ncos_downloaderandrevert_device_firewallalready pinnedfastapi/uvicornbut were still exposed until Starlette was pinned too). Verify by dry-runningpip install -r <file>against the known-good venv: a clean no-op resolve confirms the pins match the verified set rather than silently upgrading anything.
NcmClientv3.__get_json sets limit = 50 whenever the caller passes no limit,
then stops paginating at 50 results. So a bare call like
get_remote_connect_profiles() or get_lan_manager_options() silently returns
exactly 50 records with no warning, even though links.next is present.
This was verified live on those two endpoints: 50 returned by default, against
573 and 613 with limit=0. The same code path backs every v3 list helper that
routes through __get_json.
Pass limit=0 to get everything. __get_json treats 0 as unlimited and caps
page[size] at 50. A default of exactly 50 results is the telltale sign of this
cap.
/api/v3/remote_connect_profiles accepts more than one profile with the same
name on the same router. A live account had 6 router/name pairs with more than
one profile each. Bulk "push this profile" code must look up existing profiles by
router + name and PUT over them, deleting any extra same-name copies. A plain
POST creates a duplicate.
Two related details:
filter[name]is an exact, case-sensitive match (Foodoesn't matchfoo). Leading whitespace in the filter value was trimmed in one test, so a name with a leading space still matched. Do case-insensitive or substring searches client-side.- Both
remote_connect_profilesandlan_manager_optionscappage[size]at 50. Asking for more returns400with "You requested a size of 100, but 50 is the maximum."
v2 /routers/{id}/ Returns 403 for Missing Routers, So You Can't Tell "Deleted" From "No Access" (discovered 2026-10-31)
GET /api/v2/routers/{id}/ returns 403 Forbidden for a router ID that
doesn't exist, not 404. Router ID 1 returned 403 in a test, and so did all 208
IDs in the orphan check below. A router in an account your keys can't see also
returns 403. So a 403 here doesn't tell you whether the router was deleted or is
out of scope. Don't code a "router deleted" check that waits for a 404. Compare
against the full /routers/ list instead, and use the record's account to judge
scope.
remote_connect_profiles and lan_manager_options appear to keep their records
after a router is unregistered or deleted. In one account, 281 of 573 profiles
(on 208 routers) and 322 of 613 lan_manager_options records pointed at router
IDs that weren't in the v2 /routers/ list. Every one of those profiles belonged
to an account the same v2 keys could see, so out-of-scope accounts don't
explain the gap. Their creation dates go back to 2017.
Not fully proven. A router moved into an account the keys can't see would look the same, because the profile keeps the account it was created under, and the v2 detail read returns 403 either way (previous entry). Spot-check a few IDs in the NCM UI before bulk-deleting.
Practical consequences:
- Join v3 profile/option records to v2 routers by the
routerrelationship ID (it's the v2 router ID), and expect unmatched IDs. Don't assume every record resolves to a named router. - Profiles can be removed with
DELETE /remote_connect_profiles/{id}.lan_manager_optionshas no DELETE. A liveDELETE /lan_manager_options/{id}returned404 {"message": "no Route matched with those values"}, which is a gateway route miss, not "record not found". Once a router has an options record it can only be set toenabled: false, never removed.
An account can hold thousands of groups, most of them empty. In one account,
get_groups(limit='all', fields='id,name,account') returned 6,018 groups in
61 s, while only 759 of those groups contained any routers. The fix:
- List routers first (1,763 routers took about 5 s).
- Collect their distinct group IDs.
- Fetch just those groups with
get_groups(id__in=",".join(ids), ...).
The SDK chunks __in into batches of 100. The routers and groups fetch together
dropped to about 12 s, and no router was missing its group. An empty group
has no routers to target anyway, so a picker that targets routers loses nothing.
SDK Remote Connect Write Helpers Mishandle 200, 204 and 403/409 (found by reading the SDK, 2026-10-31)
In ncm/ncm/ncm.py, the v3 create_, update_ and delete_remote_connect_profile
helpers, and the create_/update_lan_manager_option helpers, all return
self._return_handler(ncm.status_code, ncm.json(), call_type). That has three
consequences:
- DELETE: the endpoint returns
204 No Content, soncm.json()raisesrequests.exceptions.JSONDecodeErrorbefore the handler runs. A successful delete therefore surfaces as an exception. - PUT: a
200returns the string"<call_type> operation successful.", not the updated record. - 403 and 409: these return
Nonesilently (see the_return_handlerentry above).
This was found by reading the code, not by running these helpers live. To get
the 409 rate-limit handling and still see the real status and body, call the
client's _v3_request(verb, url, json=...) directly and check status_code
yourself. web_apps/lan_manager_profile_manager/serve.py does this.
SSH Remote Connect Profiles Can Exist Without a username, but the API Can't Create One (discovered 2026-11-01)
In one account, 54 of 114 remote_connect_ssh_profile records had
"username": null. That's a real null, not an empty string. They were created
from 2017-09 through 2026-10. The v3 API itself still enforces the documented
rule. In a live test on one router, each of these returned
400 "username is required for ssh_profile" (pointer /data):
- POST with
usernameomitted - POST with
"username": null - PUT setting
usernametonullon an existing SSH profile (the record kept its old value)
So these profiles come from somewhere other than the public v3 API, most likely the NCM UI. Consequences:
- Read and compare them normally. A null SSH username is a valid stored state.
- They can't be copied or re-pushed as-is through the API. A copy needs a username supplied.
- Any PUT to one of them must include a username, because PUT requires the full attribute set. You can't "just rename" such a profile without giving it one.
- Not tested: whether
""is accepted.