From 5a1b08e4005add2590f828e4c5f1e682d6037a35 Mon Sep 17 00:00:00 2001 From: NarrativAI Agent Date: Tue, 8 Sep 2026 00:38:33 +0100 Subject: [PATCH] Store and serve the session-level data container The specification defines a session-level `data` container in 5.1.3, added on 13 August 2026. `SessionCreateRequest` was written on 6 August 2026 and enumerates named fields, none of them `data`; serde ignores unknown fields, so a conformant session document carrying `access_context` was accepted with a 201 and the container was discarded without a log line or a stored trace. Migration 0004 adds `session_data` to `sessions`. The container is bound on both session paths, stored as given, and materialised at the document root, so a round trip returns it unchanged. Nothing inside is normalised: consumers MUST tolerate unknown fields within the container and unknown `access_context` identifier schemes, so a scheme outside the core `ror`, `saml_entity_id` and `isni` is stored and served like any other. `access_context` is in core because COUNTER usage reporting asked for it: it names the institution whose entitlement the session used, and reporting usage by institution is already normal in scholarly publishing. Ingest checks its shape only - `identifiers` an array of objects each carrying a string `scheme` and `value`, which is what the schema requires and what the standard's two invalid fixtures exercise. The second change is what would have caught this on the day the specification moved. Both request types now capture top-level members the server does not define instead of letting serde drop them: ingest logs the names, migration 0004 stores them in `unrecognised_fields`, and materialisation returns them under `extensions.unrecognised_fields`. Recording rather than rejecting is what the consumer rules allow - a conforming consumer MUST tolerate unknown fields without error (5.7.4), so refusing the document is not open to us, and the session root is not an extension point (5.1.3), so the members are kept but never interpreted. `BulkSessionRequest::session_create` replaces the hand-written field copy in the bulk handler, so there is one mapping from the document format onto the stored session rather than a second list to forget to update. Tests: the standard's own `session-access-context.json` round-trips through ingest and materialisation with its container intact, its two invalid access-context fixtures are refused, an unknown identifier scheme survives the round trip, and the smoke script checks the same path over HTTP. The fixtures are copied byte-for-byte into `tests/spec/`. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Rid5BdMHf2W3umEWtiQj6F --- README.md | 17 +++ .../migrations/0004_session_data.sql | 20 +++ .../content-telemetry-core/src/conformance.rs | 138 ++++++++++++++++++ .../src/models/session.rs | 73 +++++++++ .../src/services/sessions.rs | 14 +- crates/content-telemetry-core/src/standard.rs | 74 ++++++++++ .../tests/integration.rs | 113 +++++++++++++- .../tests/spec/README.md | 18 +++ ...cess-context-identifier-missing-value.json | 18 +++ .../access-context-identifiers-not-array.json | 14 ++ .../tests/spec/session-access-context.json | 64 ++++++++ crates/content-telemetry-server/src/routes.rs | 84 ++++++----- scripts/smoke.sh | 21 +++ 13 files changed, 633 insertions(+), 35 deletions(-) create mode 100644 crates/content-telemetry-core/migrations/0004_session_data.sql create mode 100644 crates/content-telemetry-core/tests/spec/README.md create mode 100644 crates/content-telemetry-core/tests/spec/access-context-identifier-missing-value.json create mode 100644 crates/content-telemetry-core/tests/spec/access-context-identifiers-not-array.json create mode 100644 crates/content-telemetry-core/tests/spec/session-access-context.json diff --git a/README.md b/README.md index 9a82d77..c6acc87 100644 --- a/README.md +++ b/README.md @@ -146,6 +146,23 @@ instead of rejection — enum synonyms, fields withdrawn on privacy grounds, turn content above the declared privacy level — the server normalises and logs what it changed. +The session-level `data` container (section 5.1.3) is stored and served back +unchanged, `access_context` included. Only its shape is checked: +`identifiers` must be an array of objects each carrying a string `scheme` and +a string `value`. The scheme vocabulary is open, so a scheme outside the core +`ror`, `saml_entity_id` and `isni` is stored like any other, as are unknown +members alongside `access_context` and inside it. + +Top-level members of a session document that this server does not define are +recorded rather than dropped. The session root is not an extension point +(section 5.1.3), so nothing there is interpreted, but a consumer must tolerate +unknown fields without error (section 5.7.4), which rules out refusing the +document. Ingest logs the member names and stores them on the session, and +materialisation returns them under `extensions.unrecognised_fields`. A field +the specification adds that this server does not implement yet is therefore +visible in the log and in the document, rather than accepted into a success +response and discarded. + Materialising a session document keeps that honest in the other direction. Events that predate v1 or belong to an extension move under the document's `extensions` member rather than being dropped or rewritten, so the document diff --git a/crates/content-telemetry-core/migrations/0004_session_data.sql b/crates/content-telemetry-core/migrations/0004_session_data.sql new file mode 100644 index 0000000..3cf1793 --- /dev/null +++ b/crates/content-telemetry-core/migrations/0004_session_data.sql @@ -0,0 +1,20 @@ +-- The session-level data container, and the top-level members the +-- specification does not define. +-- +-- session_data (specification section 5.1.3): the session document's `data` +-- object, an extension container mirroring the event-level `data` field. +-- Core defines one member inside it, `access_context`, which records the +-- institution whose access rights the session used. Consumers MUST tolerate +-- unknown fields within the container and unknown identifier schemes within +-- access_context, so the column holds what the emitter sent, unnormalised, +-- and materialisation serves it back unchanged. +-- +-- unrecognised_fields (specification sections 5.1.3, 5.7.4): top-level +-- siblings of `events` that this server does not define. The session root is +-- not an extension point, so nothing here is interpreted, but a conforming +-- consumer MUST tolerate unknown fields without error - so they are recorded +-- instead of being dropped, and materialisation returns them under the +-- document's `extensions` member. + +ALTER TABLE sessions ADD COLUMN IF NOT EXISTS session_data JSONB; +ALTER TABLE sessions ADD COLUMN IF NOT EXISTS unrecognised_fields JSONB; diff --git a/crates/content-telemetry-core/src/conformance.rs b/crates/content-telemetry-core/src/conformance.rs index 5c94cd8..6ee7629 100644 --- a/crates/content-telemetry-core/src/conformance.rs +++ b/crates/content-telemetry-core/src/conformance.rs @@ -84,6 +84,17 @@ pub const WITHDRAWN_EVENT_TYPE_DISPLAYED: &str = "content_displayed"; /// v1 transition rule, not a general registry of withdrawn extension names. pub const WITHDRAWN_EVENT_DATA_FIELDS: &[&str] = &["ip_hash"]; +/// The one container core defines inside the session-level `data` object +/// (spec 5.1.3): the context from which the session's access rights derive, +/// an institution and never an individual. COUNTER usage reporting needs it, +/// which is why it is in core rather than in an extension. +pub const SESSION_ACCESS_CONTEXT: &str = "access_context"; + +/// Identifier schemes named in core (spec 5.1.3). Informative only: the +/// vocabulary is open, emitters MAY use others, and consumers MUST tolerate +/// unknown ones, so nothing validates against this list. +pub const CORE_ACCESS_CONTEXT_SCHEMES: &[&str] = &["ror", "saml_entity_id", "isni"]; + /// Conversation-turn privacy levels (spec 5.4). A closed enum: the schema /// rejects a turn whose `privacy_level` is outside this set. pub const PRIVACY_LEVELS: &[&str] = &["full", "summary", "intent", "minimal"]; @@ -380,6 +391,68 @@ pub fn v1_structural_violation( } } +/// The shape the schema gives the session-level `data` container's one core +/// member, `access_context` (spec 5.1.3): `identifiers` is an array, and +/// every entry is an object carrying a `scheme` and a `value`, both strings. +/// +/// Only the shape is checked. Scheme values are open — `ror`, +/// `saml_entity_id` and `isni` are the core ones and a consumer MUST +/// tolerate any other — and additional members inside `data`, inside +/// `access_context` and inside an identifier are tolerated unchanged, +/// because consumers MUST tolerate unknown fields within the container. +/// +/// Returns a description of the first violation, or None. +pub fn session_data_violation(data: Option<&Value>) -> Option { + let data = data?; + if data.is_null() { + return None; + } + let Some(data) = data.as_object() else { + return Some("session data must be an object".to_string()); + }; + + let access_context = data.get(SESSION_ACCESS_CONTEXT)?; + if access_context.is_null() { + return None; + } + let Some(access_context) = access_context.as_object() else { + return Some("data.access_context must be an object".to_string()); + }; + + let identifiers = access_context.get("identifiers")?; + let Some(identifiers) = identifiers.as_array() else { + return Some( + "data.access_context.identifiers must be an array of {scheme, value} objects" + .to_string(), + ); + }; + + for (index, identifier) in identifiers.iter().enumerate() { + let Some(identifier) = identifier.as_object() else { + return Some(format!( + "data.access_context.identifiers[{index}] must be an object" + )); + }; + for member in ["scheme", "value"] { + match identifier.get(member) { + Some(v) if v.is_string() => {} + Some(_) => { + return Some(format!( + "data.access_context.identifiers[{index}].{member} must be a string" + )); + } + None => { + return Some(format!( + "data.access_context.identifiers[{index}] requires {member}" + )); + } + } + } + } + + None +} + /// The v1 rule that `source_role` MUST be present on every /// `content_retrieved` event (spec 5.2.2, 5.7.5): without it a consumer /// cannot tell an agent-reported fetch from an origin- or edge-reported @@ -1146,4 +1219,69 @@ mod tests { }); assert!(strip_turn_privacy_violations(&mut turn).is_empty()); } + + #[test] + fn access_context_shape_is_checked_and_its_vocabulary_is_not() { + // The spec's own example (5.1.3), plus a scheme core does not name. + // Consumers MUST tolerate unknown schemes, so the unknown one is not + // a violation. + let spec_example = json!({ + "access_context": { + "identifiers": [ + { "scheme": "ror", "value": "https://ror.org/013meh722" }, + { "scheme": "saml_entity_id", "value": "https://idp.example.ac.uk/shibboleth" }, + { "scheme": "example_local", "value": "lib-4471" } + ] + } + }); + assert!(session_data_violation(Some(&spec_example)).is_none()); + + // Unknown fields inside the container, inside access_context and + // inside an identifier are all tolerated (5.1.3). + let extras = json!({ + "com.example.reporting_period": "2026-08", + "access_context": { + "asserted_by": "agent", + "identifiers": [ + { "scheme": "isni", "value": "0000000121032683", "note": "consortium seat" } + ] + } + }); + assert!(session_data_violation(Some(&extras)).is_none()); + + // A container with no access_context, and no container at all. + assert!(session_data_violation(Some(&json!({ "x": 1 }))).is_none()); + assert!(session_data_violation(None).is_none()); + assert!(session_data_violation(Some(&Value::Null)).is_none()); + } + + #[test] + fn malformed_access_context_is_a_violation() { + // Both cases are the standard's own invalid fixtures. + let missing_value = json!({ + "access_context": { "identifiers": [{ "scheme": "ror" }] } + }); + assert_eq!( + session_data_violation(Some(&missing_value)).as_deref(), + Some("data.access_context.identifiers[0] requires value") + ); + + let not_an_array = json!({ + "access_context": { "identifiers": "https://ror.org/013meh722" } + }); + assert!( + session_data_violation(Some(¬_an_array)) + .is_some_and(|v| v.contains("must be an array")) + ); + + let non_string_scheme = json!({ + "access_context": { "identifiers": [{ "scheme": 7, "value": "x" }] } + }); + assert_eq!( + session_data_violation(Some(&non_string_scheme)).as_deref(), + Some("data.access_context.identifiers[0].scheme must be a string") + ); + + assert!(session_data_violation(Some(&json!("not an object"))).is_some()); + } } diff --git a/crates/content-telemetry-core/src/models/session.rs b/crates/content-telemetry-core/src/models/session.rs index 362beb8..c00d744 100644 --- a/crates/content-telemetry-core/src/models/session.rs +++ b/crates/content-telemetry-core/src/models/session.rs @@ -26,6 +26,12 @@ pub struct SessionCreateRequest { pub conformance_level: Option, pub agent_id: Option, pub external_session_id: Option, + /// Session-level extension container (spec 5.1.3), carrying + /// `access_context` and any namespaced members alongside it. Stored and + /// served verbatim: consumers MUST tolerate unknown fields within it and + /// unknown identifier schemes inside `access_context`, so nothing here + /// is normalised or dropped. + pub data: Option, #[serde(default)] pub user_context: serde_json::Value, #[serde(default)] @@ -38,6 +44,13 @@ pub struct SessionCreateRequest { /// ingest. Defaults to NOW() when absent (the /sessions/start flow). pub started_at: Option>, pub ended_at: Option>, + /// Top-level members this server does not define. The session root is + /// not an extension point (spec 5.1.3), so nothing here is interpreted, + /// but a conforming consumer MUST tolerate unknown fields without error + /// (spec 5.7.4). Capturing them is what stops a member the specification + /// adds later from being accepted and silently discarded. + #[serde(flatten)] + pub unrecognised_fields: serde_json::Map, } #[derive(Debug, Clone, Deserialize)] @@ -90,6 +103,10 @@ pub struct BulkSessionRequest { pub prior_session_ids: Vec, pub started_at: Option>, pub ended_at: Option>, + /// Session-level extension container (spec 5.1.3), carrying + /// `access_context` and any namespaced members alongside it. Stored and + /// served verbatim. + pub data: Option, #[serde(default)] pub user_context: serde_json::Value, #[serde(default)] @@ -99,6 +116,56 @@ pub struct BulkSessionRequest { pub platform_id: Option, pub client_type: Option, pub client_info: Option, + /// Top-level members this server does not define (spec 5.1.3, 5.7.4). + /// Recorded rather than dropped; see `SessionCreateRequest`. + #[serde(flatten)] + pub unrecognised_fields: serde_json::Map, +} + +impl BulkSessionRequest { + /// The session this document opens, as the bulk ingest path creates it. + /// + /// One mapping from the document format onto the stored session, rather + /// than a hand-written copy in each handler: a member the format gains + /// is either carried here or visibly absent here, never dropped in a + /// place nobody looks. + /// + /// `conformance_level` passes through unnormalised — `create_session` + /// normalises it (spec 5.7) — and `ended_at` is withheld when the + /// document carries an outcome, because ending a session only updates a + /// row whose `ended_at` is still NULL and stamping it here would drop + /// the outcome. The caller hands that timestamp to `end_session`. + pub fn session_create(&self) -> SessionCreateRequest { + SessionCreateRequest { + initiator_type: self.initiator_type.clone(), + initiator: self.initiator.clone(), + parent_session_id: self.parent_session_id, + content_scope: self.content_scope.clone(), + manifest_ref: self.manifest_ref.clone(), + conformance_level: self.conformance_level.clone(), + agent_id: self.agent_id.clone(), + // The presented session id is the emitter's, not ours: it is + // stored as the external id under a server-minted primary key. + external_session_id: Some( + self.external_session_id + .clone() + .unwrap_or_else(|| self.session_id.to_string()), + ), + data: self.data.clone(), + user_context: self.user_context.clone(), + prior_session_ids: self.prior_session_ids.iter().map(Uuid::to_string).collect(), + platform_id: self.platform_id.clone(), + client_type: self.client_type.clone(), + client_info: self.client_info.clone(), + started_at: self.started_at, + ended_at: if self.outcome.is_some() { + None + } else { + self.ended_at + }, + unrecognised_fields: self.unrecognised_fields.clone(), + } + } } // --------------------------------------------------------------------------- @@ -119,6 +186,12 @@ pub struct SessionRow { pub agent_id: Option, pub external_session_id: Option, pub prior_session_ids: Option>, + /// The session-level `data` container as the emitter sent it + /// (spec 5.1.3). + pub session_data: Option, + /// Top-level members the specification does not define, kept so nothing + /// a conformant document carries is lost without trace (spec 5.7.4). + pub unrecognised_fields: Option, pub user_context: serde_json::Value, pub platform_id: Option, pub client_type: Option, diff --git a/crates/content-telemetry-core/src/services/sessions.rs b/crates/content-telemetry-core/src/services/sessions.rs index d1ced0c..2738952 100644 --- a/crates/content-telemetry-core/src/services/sessions.rs +++ b/crates/content-telemetry-core/src/services/sessions.rs @@ -35,16 +35,26 @@ pub async fn create_session( ); } + // The session-level `data` container and any top-level members this + // server does not define are stored as given (spec 5.1.3): nothing + // inside either is interpreted, normalised or dropped. + let unrecognised = if req.unrecognised_fields.is_empty() { + None + } else { + Some(serde_json::Value::Object(req.unrecognised_fields.clone())) + }; + sqlx::query_as::<_, SessionRow>( r"INSERT INTO sessions ( organization_id, parent_session_id, initiator_type, initiator, content_scope, manifest_ref, conformance_level, agent_id, external_session_id, prior_session_ids, + session_data, unrecognised_fields, user_context, platform_id, client_type, client_info, started_at, ended_at ) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, - COALESCE($15, NOW()), $16) + $15, $16, COALESCE($17, NOW()), $18) RETURNING *", ) .bind(owner_id) @@ -57,6 +67,8 @@ pub async fn create_session( .bind(&req.agent_id) .bind(&req.external_session_id) .bind(&prior) + .bind(&req.data) + .bind(&unrecognised) .bind(&req.user_context) .bind(&req.platform_id) .bind(&req.client_type) diff --git a/crates/content-telemetry-core/src/standard.rs b/crates/content-telemetry-core/src/standard.rs index b75271f..c6caf87 100644 --- a/crates/content-telemetry-core/src/standard.rs +++ b/crates/content-telemetry-core/src/standard.rs @@ -213,9 +213,28 @@ pub fn standard_document(swe: &SessionWithEvents) -> Value { ); doc.insert("started_at".to_string(), json!(session.started_at)); insert_if_some(&mut doc, "ended_at", session.ended_at.map(|v| json!(v))); + // The session-level data container (spec 5.1.3), served back exactly as + // it arrived. Consumers MUST tolerate unknown fields within it and + // unknown `access_context` identifier schemes, so nothing inside is + // normalised, defaulted or dropped: the round trip is lossless. + insert_if_some(&mut doc, "data", session.session_data.clone()); let mut extensions = Map::new(); + // Top-level members this server does not define. The session root is not + // an extension point (spec 5.1.3) and they are not interpreted, but a + // consumer MUST tolerate unknown fields without error (spec 5.7.4), so + // they are returned rather than dropped — a document cannot lose data + // here without the loss being visible. + if let Some(Value::Object(unrecognised)) = session.unrecognised_fields.as_ref() + && !unrecognised.is_empty() + { + extensions.insert( + "unrecognised_fields".to_string(), + Value::Object(unrecognised.clone()), + ); + } + // Normalise on read as well as ingest: rows written by a pre-rename // binary during the 0011 deploy window can still carry the legacy // 'attribution' value. @@ -324,6 +343,8 @@ mod tests { agent_id: Some("test-agent".to_string()), external_session_id: None, prior_session_ids: None, + session_data: None, + unrecognised_fields: None, user_context: json!({}), platform_id: None, client_type: None, @@ -455,6 +476,59 @@ mod tests { ); } + #[test] + fn session_data_materialises_at_the_document_root() { + // The container is served back byte-for-byte, unknown identifier + // schemes and namespaced neighbours included (spec 5.1.3). + let data = json!({ + "access_context": { + "identifiers": [ + { "scheme": "ror", "value": "https://ror.org/013meh722" }, + { "scheme": "example_local", "value": "lib-4471", "note": "consortium seat" } + ] + }, + "com.example.reporting_period": "2026-08" + }); + + let mut session = session_row(); + session.session_data = Some(data.clone()); + + let doc = standard_document(&SessionWithEvents { + session, + events: vec![], + }); + + assert_eq!(doc["data"], data); + assert_eq!( + doc["data"]["access_context"]["identifiers"][1]["scheme"], + "example_local" + ); + } + + #[test] + fn undefined_top_level_fields_come_back_under_extensions() { + // The session root is not an extension point (spec 5.1.3), but a + // consumer MUST tolerate unknown fields without error (spec 5.7.4). + // Recording them is what makes the loss visible: a member this + // server does not implement is returned rather than dropped. + let mut session = session_row(); + session.unrecognised_fields = Some(json!({ "access_summary": { "institutions": 3 } })); + + let doc = standard_document(&SessionWithEvents { + session, + events: vec![], + }); + + assert!( + doc.get("access_summary").is_none(), + "an undefined member must not be promoted to a standard field" + ); + assert_eq!( + doc["extensions"]["unrecognised_fields"]["access_summary"]["institutions"], + 3 + ); + } + #[test] fn stored_v0_rows_materialise_under_the_migration_rules() { // Rows written before versioned ingest lack the members v1 requires; diff --git a/crates/content-telemetry-core/tests/integration.rs b/crates/content-telemetry-core/tests/integration.rs index d4106d5..53849a7 100644 --- a/crates/content-telemetry-core/tests/integration.rs +++ b/crates/content-telemetry-core/tests/integration.rs @@ -4,9 +4,10 @@ use uuid::Uuid; use content_telemetry_core::models::event::{EdgeEventInput, TelemetryEventInput}; use content_telemetry_core::models::session::{ - SessionCreateRequest, SessionEndRequest, SessionOutcome, + BulkSessionRequest, SessionCreateRequest, SessionEndRequest, SessionOutcome, }; use content_telemetry_core::services::{click_tokens, events, queries, sessions}; +use content_telemetry_core::{conformance, standard}; // --------------------------------------------------------------------------- // Test constants @@ -769,3 +770,113 @@ async fn publisher_summary_breaks_down_status_codes(pool: PgPool) { "malformed and statusless events fold into the NULL bucket" ); } + +// =================================================================== +// Session data container (specification section 5.1.3) +// =================================================================== + +/// The standard's own fixtures, copied byte-for-byte; see `spec/README.md`. +const SPEC_SESSION_ACCESS_CONTEXT: &str = include_str!("spec/session-access-context.json"); +const SPEC_IDENTIFIER_MISSING_VALUE: &str = + include_str!("spec/access-context-identifier-missing-value.json"); +const SPEC_IDENTIFIERS_NOT_ARRAY: &str = + include_str!("spec/access-context-identifiers-not-array.json"); + +fn spec_document(source: &str) -> serde_json::Value { + serde_json::from_str(source).expect("spec fixture is valid JSON") +} + +#[sqlx::test(migrations = "./migrations", fixtures("setup"))] +async fn spec_access_context_survives_ingest_and_retrieval(pool: PgPool) { + let org = agent_org_id(); + let document = spec_document(SPEC_SESSION_ACCESS_CONTEXT); + + let request: BulkSessionRequest = + serde_json::from_value(document.clone()).expect("the spec fixture deserialises"); + assert!( + conformance::session_data_violation(request.data.as_ref()).is_none(), + "the standard's own valid fixture must pass the access_context check" + ); + + let session = sessions::create_session(&pool, org, &request.session_create()) + .await + .unwrap(); + events::create_events(&pool, session.id, org, &request.events) + .await + .unwrap(); + + let stored = sessions::get_session_with_events(&pool, session.id) + .await + .unwrap() + .unwrap(); + let materialised = standard::standard_document(&stored); + + // The round trip is lossless: the container comes back as sent, both + // core schemes included, with nothing normalised or reordered. + assert_eq!(materialised["data"], document["data"]); + assert_eq!( + materialised["data"]["access_context"]["identifiers"][0]["value"], + "https://ror.org/013meh722" + ); + assert_eq!( + materialised["data"]["access_context"]["identifiers"][1]["scheme"], + "saml_entity_id" + ); + assert_eq!(materialised["events"].as_array().unwrap().len(), 4); + + // The fixture's `_test_description` is an upstream harness annotation, + // not a member of the document format. It is undefined at the session + // root (spec 5.1.3), so it is recorded rather than dropped (spec 5.7.4). + assert_eq!( + materialised["extensions"]["unrecognised_fields"]["_test_description"], + document["_test_description"] + ); +} + +#[sqlx::test(migrations = "./migrations", fixtures("setup"))] +async fn unknown_identifier_schemes_survive_ingest_and_retrieval(pool: PgPool) { + // Emitters MAY use schemes outside the core three and consumers MUST + // tolerate them (spec 5.1.3), so an unknown scheme is stored and served + // like any other. + let org = agent_org_id(); + let data = serde_json::json!({ + "access_context": { + "identifiers": [ + { "scheme": "example_consortium", "value": "seat-4471" } + ] + }, + "com.example.reporting_period": "2026-08" + }); + + let request: SessionCreateRequest = serde_json::from_value(serde_json::json!({ + "initiator_type": "agent", + "agent_id": "scholar-assistant.example.com", + "data": data.clone(), + })) + .unwrap(); + assert!(conformance::session_data_violation(request.data.as_ref()).is_none()); + + let session = sessions::create_session(&pool, org, &request) + .await + .unwrap(); + let stored = sessions::get_session_with_events(&pool, session.id) + .await + .unwrap() + .unwrap(); + + assert_eq!(standard::standard_document(&stored)["data"], data); +} + +#[test] +fn spec_invalid_access_context_fixtures_are_refused() { + for source in [SPEC_IDENTIFIER_MISSING_VALUE, SPEC_IDENTIFIERS_NOT_ARRAY] { + let document = spec_document(source); + let request: BulkSessionRequest = serde_json::from_value(document.clone()).unwrap(); + let violation = conformance::session_data_violation(request.data.as_ref()); + assert!( + violation.is_some(), + "the standard's invalid fixture should be refused: {}", + document["_test_description"] + ); + } +} diff --git a/crates/content-telemetry-core/tests/spec/README.md b/crates/content-telemetry-core/tests/spec/README.md new file mode 100644 index 0000000..bcf5023 --- /dev/null +++ b/crates/content-telemetry-core/tests/spec/README.md @@ -0,0 +1,18 @@ +# Specification fixtures + +Copied byte-for-byte from the standard's own conformance suite +(`SPUR-Coalition/telemetry`, `tests/valid/` and `tests/invalid/`). They are +the specification's examples, not ours: edit them only by re-copying from +upstream, so a fixture that stops passing here means this server disagrees +with the standard rather than with a local rewrite of it. + +| File | Upstream path | What it is | +|---|---|---| +| `session-access-context.json` | `tests/valid/` | A session document carrying the 5.1.3 `data.access_context` container | +| `access-context-identifier-missing-value.json` | `tests/invalid/` | An identifier with a `scheme` and no `value` | +| `access-context-identifiers-not-array.json` | `tests/invalid/` | `identifiers` as a bare string | + +The `_test_description` and `_expected_error` members are the upstream +harness's annotations. They are not part of the document format, which is +why this server records them as unrecognised top-level fields rather than +interpreting them. diff --git a/crates/content-telemetry-core/tests/spec/access-context-identifier-missing-value.json b/crates/content-telemetry-core/tests/spec/access-context-identifier-missing-value.json new file mode 100644 index 0000000..a0e57a6 --- /dev/null +++ b/crates/content-telemetry-core/tests/spec/access-context-identifier-missing-value.json @@ -0,0 +1,18 @@ +{ + "_test_description": "Session access_context identifier missing its value. The schema requires both scheme and value on every identifier (5.1.3).", + "_expected_error": "/data/access_context/identifiers/0 'value' is a required property", + "document_type": "session", + "schema_version": "1.0", + "session_id": "880e8400-e29b-41d4-a716-446655440090", + "started_at": "2026-08-13T14:02:10Z", + "data": { + "access_context": { + "identifiers": [ + { + "scheme": "ror" + } + ] + } + }, + "events": [] +} diff --git a/crates/content-telemetry-core/tests/spec/access-context-identifiers-not-array.json b/crates/content-telemetry-core/tests/spec/access-context-identifiers-not-array.json new file mode 100644 index 0000000..96fc123 --- /dev/null +++ b/crates/content-telemetry-core/tests/spec/access-context-identifiers-not-array.json @@ -0,0 +1,14 @@ +{ + "_test_description": "Session access_context.identifiers as a bare string. The schema requires an array of {scheme, value} objects (5.1.3).", + "_expected_error": "/data/access_context/identifiers 'https://ror.org/013meh722' is not of type 'array'", + "document_type": "session", + "schema_version": "1.0", + "session_id": "880e8400-e29b-41d4-a716-446655440091", + "started_at": "2026-08-13T14:02:10Z", + "data": { + "access_context": { + "identifiers": "https://ror.org/013meh722" + } + }, + "events": [] +} diff --git a/crates/content-telemetry-core/tests/spec/session-access-context.json b/crates/content-telemetry-core/tests/spec/session-access-context.json new file mode 100644 index 0000000..b2c1422 --- /dev/null +++ b/crates/content-telemetry-core/tests/spec/session-access-context.json @@ -0,0 +1,64 @@ +{ + "_test_description": "Session-level data container (5.1.3): access_context identifies the institution whose entitlement the agent used, required by the governing terms, with turn data at intent level per 5.5. The retrieval event carries content_depth: full (6.1).", + "document_type": "session", + "schema_version": "1.0", + "session_id": "880e8400-e29b-41d4-a716-446655440080", + "agent_id": "scholar-assistant.example.com", + "content_scope": "consortium-agreement-4471", + "started_at": "2026-08-13T14:02:10Z", + "ended_at": "2026-08-13T14:03:44Z", + "data": { + "access_context": { + "identifiers": [ + { "scheme": "ror", "value": "https://ror.org/013meh722" }, + { "scheme": "saml_entity_id", "value": "https://idp.example.ac.uk/shibboleth" } + ] + } + }, + "events": [ + { + "type": "turn_started", + "timestamp": "2026-08-13T14:02:11Z", + "turn_id": "1", + "turn": { + "privacy_level": "intent", + "query_intent": "question", + "topics": ["materials science"] + } + }, + { + "type": "content_retrieved", + "timestamp": "2026-08-13T14:02:14Z", + "source_role": "agent", + "content_telemetry_id": "990e8400-e29b-41d4-a716-446655440081", + "content_url": "https://journals.example.com/article/10.1000/xyz123", + "content_id": "doi:10.1000/xyz123", + "license_ref": "grant-4471-2026", + "data": { + "media_type": "text", + "content_depth": "full" + } + }, + { + "type": "content_grounded", + "timestamp": "2026-08-13T14:02:16Z", + "source_role": "agent", + "turn_id": "1", + "content_id": "doi:10.1000/xyz123", + "data": { + "scope": "turn", + "chars_ingested": 18400 + } + }, + { + "type": "turn_completed", + "timestamp": "2026-08-13T14:03:40Z", + "turn_id": "1", + "turn": { + "privacy_level": "intent", + "query_intent": "question", + "topics": ["materials science"] + } + } + ] +} diff --git a/crates/content-telemetry-server/src/routes.rs b/crates/content-telemetry-server/src/routes.rs index 317dc5a..dba03b9 100644 --- a/crates/content-telemetry-server/src/routes.rs +++ b/crates/content-telemetry-server/src/routes.rs @@ -70,6 +70,8 @@ async fn start_session( Json(mut req): Json, ) -> Result { check_initiator_type(&req.initiator_type)?; + check_session_data(req.data.as_ref())?; + log_unrecognised_fields(&req.unrecognised_fields); // Informational only: the spec forbids rejecting a document over its // conformance level, so a value we do not recognise is logged and stored, @@ -144,6 +146,8 @@ async fn bulk_session( }; check_initiator_type(&req.initiator_type)?; + check_session_data(req.data.as_ref())?; + log_unrecognised_fields(&req.unrecognised_fields); if let Some(outcome) = req.outcome.as_ref() { check_outcome_type(&outcome.outcome_type)?; } @@ -152,6 +156,16 @@ async fn bulk_session( return Err(too_large(req.events.len())); } + // The presented session id is the emitter's, not ours. It is stored as + // the external id under a server-minted primary key, so two emitters + // cannot collide on a chosen id and neither can address the other's row. + // The mapping lives on the document type: the session-level `data` + // container and any top-level members the server does not define travel + // to storage with everything else (spec 5.1.3). + let mut create = req.session_create(); + create.conformance_level = + conformance::normalise_conformance_level(req.conformance_level.as_deref()).value; + let mut events_in = req.events; for (index, event) in events_in.iter_mut().enumerate() { // Normalise before checking: a "0.1" document's migration defaults @@ -160,39 +174,6 @@ async fn bulk_session( validate::check_event(event, line, state.max_event_age_days, index)?; } - let level = conformance::normalise_conformance_level(req.conformance_level.as_deref()); - - // The presented session id is the emitter's, not ours. It is stored as - // the external id under a server-minted primary key, so two emitters - // cannot collide on a chosen id and neither can address the other's row. - let create = SessionCreateRequest { - initiator_type: req.initiator_type, - initiator: req.initiator, - parent_session_id: req.parent_session_id, - content_scope: req.content_scope, - manifest_ref: req.manifest_ref, - conformance_level: level.value, - agent_id: req.agent_id, - external_session_id: Some( - req.external_session_id - .unwrap_or_else(|| req.session_id.to_string()), - ), - user_context: req.user_context, - prior_session_ids: req.prior_session_ids.iter().map(Uuid::to_string).collect(), - platform_id: req.platform_id, - client_type: req.client_type, - client_info: req.client_info, - started_at: req.started_at, - // Withheld when an outcome is present: end_session only updates a - // session whose ended_at is still NULL, so setting it here would - // silently drop the outcome. - ended_at: if req.outcome.is_some() { - None - } else { - req.ended_at - }, - }; - let session = sessions::create_session(&state.pool, org, &create).await?; let created = if events_in.is_empty() { @@ -579,6 +560,43 @@ fn check_initiator_type(value: &str) -> Result<(), ApiError> { } } +/// The session-level `data` container is an extension point, so almost +/// nothing in it is checked — but `access_context` is defined in core (spec +/// 5.1.3) and the schema gives it a shape, so a malformed one is refused +/// rather than stored as a claim nobody can read. +fn check_session_data(data: Option<&Value>) -> Result<(), ApiError> { + match conformance::session_data_violation(data) { + Some(violation) => Err(ApiError::bad_request(violation)), + None => Ok(()), + } +} + +/// Record top-level members the server does not define. +/// +/// The session root is not an extension point (spec 5.1.3) and a consumer +/// MUST tolerate unknown fields without error (spec 5.7.4), so the document +/// is accepted either way. What must not happen is accepting it silently: +/// the members are logged here, stored on the session, and returned under +/// the document's `extensions`. A field the specification adds that this +/// server does not implement yet shows up in all three places instead of +/// disappearing into a 201. +fn log_unrecognised_fields(fields: &serde_json::Map) { + if fields.is_empty() { + return; + } + + let names = fields + .keys() + .map(String::as_str) + .collect::>() + .join(", "); + tracing::warn!( + fields = names, + "session document carries top-level fields this server does not define; stored and \ + returned under extensions.unrecognised_fields (spec 5.1.3, 5.7.4)" + ); +} + fn check_outcome_type(value: &str) -> Result<(), ApiError> { if matches!(value, "conversion" | "abandonment" | "browse") { Ok(()) diff --git a/scripts/smoke.sh b/scripts/smoke.sh index 6e84b05..fb9dbf6 100755 --- a/scripts/smoke.sh +++ b/scripts/smoke.sh @@ -157,6 +157,27 @@ RESP=$(curl -s -X POST "$BASE/sessions/bulk" -H 'content-type: application/json' [ "$(echo "$RESP" | jget 'd.get("events_created","ERR")')" = "1" ] && ok "bulk document ingested" || bad "bulk failed: $RESP" [ "$(echo "$RESP" | jget 'str(d.get("outcome_recorded"))')" = "True" ] && ok "bulk outcome recorded" || bad "bulk outcome not recorded: $RESP" +echo "== session data container (5.1.3)" +BULK=$(curl -s -X POST "$BASE/sessions/bulk" -H 'content-type: application/json' -H "x-organization-id: $AGENT" \ + -d "{\"document_type\":\"session\",\"schema_version\":\"1.0\",\"session_id\":\"$(uuid)\",\"agent_id\":\"demo-agent\", + \"data\":{\"access_context\":{\"identifiers\":[{\"scheme\":\"ror\",\"value\":\"https://ror.org/013meh722\"},{\"scheme\":\"example_local\",\"value\":\"seat-4471\"}]}}, + \"institution_summary\":{\"seats\":3}, + \"events\":[]}") +SDATA=$(echo "$BULK" | jget 'd.get("session_id","")') +[ -n "$SDATA" ] && ok "session document with access_context ingested" || bad "access_context ingest failed: $BULK" + +DOC=$(curl -s "$BASE/sessions/$SDATA/document" -H "x-organization-id: $AGENT") +[ "$(echo "$DOC" | jget 'd["data"]["access_context"]["identifiers"][0]["value"]')" = "https://ror.org/013meh722" ] \ + && ok "access_context served back on the session document" || bad "access_context missing: $DOC" +[ "$(echo "$DOC" | jget 'd["data"]["access_context"]["identifiers"][1]["scheme"]')" = "example_local" ] \ + && ok "unknown identifier scheme preserved" || bad "unknown scheme lost: $DOC" +[ "$(echo "$DOC" | jget 'd["extensions"]["unrecognised_fields"]["institution_summary"]["seats"]')" = "3" ] \ + && ok "undefined top-level field recorded, not discarded" || bad "undefined field lost: $DOC" + +RESP=$(curl -s -X POST "$BASE/sessions/bulk" -H 'content-type: application/json' -H "x-organization-id: $AGENT" \ + -d "{\"schema_version\":\"1.0\",\"session_id\":\"$(uuid)\",\"data\":{\"access_context\":{\"identifiers\":[{\"scheme\":\"ror\"}]}},\"events\":[]}") +echo "$RESP" | grep -q 'requires value' && ok "identifier without a value rejected" || bad "expected identifier error, got: $RESP" + echo if [ "$FAILURES" -eq 0 ]; then echo "ALL CHECKS PASSED"; else echo "$FAILURES CHECK(S) FAILED"; fi exit "$FAILURES"