From c9541032d570aab0d6daaf04869a1a099395f967 Mon Sep 17 00:00:00 2001 From: Ferdinand99 Date: Tue, 27 Oct 2026 17:52:03 +0100 Subject: [PATCH 1/2] fix: privacy and transport hardening from the October review Opting out now takes effect at once: the flush loop checks the switch before every batch, and turning it off cancels the upload in flight. Payload values are cleaned of control characters before they are scrubbed (they used to split an address past the patterns and be removed afterwards), cut to 2,000 characters first, and the patterns' leading runs are bounded so a long value scrubs fast. Shortened IPv6 addresses and card numbers grouped with dots or non-breaking spaces are scrubbed too, and payload keys that look like they hold a secret (password, token, auth, key, card, cookie, cvv, ssn) are dropped with their values. Transport: http:// is refused except for localhost and debug builds, redirects are not followed, answers over 64 KB are ignored and a request has 60 seconds in all. Only Keen Kestrel's own 202 answer counts as delivered; other 2xx answers retry with backoff, as do 405, 407 and 410 instead of dropping the batch. 401, 403 and 404 pause sending for an hour instead of until relaunch. The command stream is bounded to 1,000 commands, and a full queue file is appended to and compacted now and then instead of rewritten for every signal. --- README.md | 32 +++- Sources/KestrelKit/Client.swift | 66 ++++++-- Sources/KestrelKit/DiskQueue.swift | 42 ++++-- Sources/KestrelKit/Engine.swift | 28 +++- Sources/KestrelKit/Kestrel.swift | 29 ++++ Sources/KestrelKit/Limits.swift | 15 +- Sources/KestrelKit/Scrubber.swift | 63 ++++++-- Sources/KestrelKit/Uploader.swift | 166 +++++++++++++++++++-- Tests/KestrelKitTests/DeliveryTests.swift | 174 ++++++++++++++++++++-- Tests/KestrelKitTests/FacadeTests.swift | 29 ++++ Tests/KestrelKitTests/Helpers.swift | 17 ++- Tests/KestrelKitTests/LimitsTests.swift | 118 ++++++++++++++- Tests/KestrelKitTests/QueueTests.swift | 29 ++++ docs/protocol.md | 42 ++++-- 14 files changed, 750 insertions(+), 100 deletions(-) diff --git a/README.md b/README.md index 28da388..de5501d 100644 --- a/README.md +++ b/README.md @@ -55,6 +55,10 @@ The key is the project's public key: the 32 hex characters at the start of the p (`https://@kestrel.example.com/1`), on the project's settings page in Kestrel. It's the same key the Sentry SDK uses, so one project gets both errors and analytics. +KestrelKit only talks to `https://` addresses. `http://` works for `localhost`, and in debug builds +for any host (a Kestrel on your own network); in a release build any other `http://` address is +refused and nothing is stored or sent. + ### SwiftUI ```swift @@ -117,12 +121,21 @@ Each signal has: **Never collected:** the IDFA or IDFV, the device name, the user's name or email, location, contacts, IP addresses (Kestrel doesn't store them), or anything to fingerprint a device. -Payloads are filtered on the device, with the same rules Kestrel applies again on the server: +Payloads are filtered on the device, with the same rules Kestrel applies again on the server (the +shortened IPv6 forms and the dot and non-breaking-space card groupings below are only caught on the +device for now). Control characters are removed first, and a value is cut to 2,000 characters before +it is filtered, so a very long value is cheap: -- values that look like an email address, a phone number, a card number, an IP address, a - `Bearer`/`Basic` credential or a `token=…`-style secret are replaced by `[Filtered]`; +- values that look like an email address, a phone number, a card number (13 to 19 digits grouped by + spaces, dashes, dots or non-breaking spaces, and passing the Luhn check), an IP address (IPv4, and + IPv6 written out or shortened with `::`, such as `2001:db8::1` and `::1`), a `Bearer`/`Basic` + credential or a `token=…`-style secret are replaced by `[Filtered]`; - keys that name an IP address or a user agent (`ip`, `client_ip`, `x-forwarded-for`, `user_agent` - and the like) are dropped with their values. + and the like) are dropped with their values; +- keys that look like they hold a secret are dropped with their values: any key containing `pass` + (`password`, `passwd`), `secret`, `token`, `auth`, `key`, `card`, `cookie` or `cvv`, and `ssn` as a + word of its own, ignoring case. This is blunt on purpose: `keyboard_layout` and `author` go too, so + give such keys other names (`layout`, `writer`). Payloads are limited to 20 keys of up to 40 characters, with values of up to 200 characters. Keep personal data out of them; the filtering is a safety net, not a licence. @@ -157,8 +170,15 @@ choice across launches. Nothing is sent before `Kestrel.configure` is called eit survive restarts and offline periods. Each batch holds up to 100 signals or 256 KB, and is gzipped when that makes it smaller. - When the server is busy or unreachable, KestrelKit backs off: from 30 seconds up to an hour, or as - long as the server's `Retry-After` says. A wrong or disabled key stops sending until the next - launch. + long as the server's `Retry-After` says. A wrong or disabled key (or a server without app + analytics) pauses sending for an hour. Only Keen Kestrel's own answer (`202` with `{"accepted": n}`) + counts as delivered; any other answer, a captive portal's `200` for example, keeps the batch and + retries with backoff. +- Redirects are never followed, answers bigger than 64 KB are ignored, and a request gets 60 seconds + at most. +- Turning `Kestrel.isEnabled` off cancels an upload in progress and sends nothing more. +- Up to 1,000 calls can wait for the queue at once while it is busy; when more arrive, the oldest are + dropped. - On iOS, tvOS and visionOS, KestrelKit asks for a little background time to finish the upload when the app moves to the background. On the Mac, "background" means another app is active. - TestFlight builds are recognised by their sandbox App Store receipt. This doesn't work for Mac diff --git a/Sources/KestrelKit/Client.swift b/Sources/KestrelKit/Client.swift index a8ebd2b..4bf31ae 100644 --- a/Sources/KestrelKit/Client.swift +++ b/Sources/KestrelKit/Client.swift @@ -12,6 +12,18 @@ enum Command: Sendable { case foreground(time: Date) case optOut case barrier(CheckedContinuation) + + /// Lets whoever waits for this command go, for a command that will never be handled. + func abandon() { + switch self { + case let .flush(done), let .barrier(done): + done.resume() + case let .background(_, done): + done?.resume() + case .signal, .tick, .foreground, .optOut: + break + } + } } /// The outside world, swappable in tests. @@ -39,12 +51,13 @@ struct Environment: Sendable { let configuration = URLSessionConfiguration.ephemeral configuration.timeoutIntervalForRequest = 30 + configuration.timeoutIntervalForResource = 60 configuration.httpCookieStorage = nil configuration.urlCache = nil return Environment( store: store, directory: directory, - session: URLSession(configuration: configuration), + session: TransportDelegate.makeSession(configuration), device: device ) } @@ -53,23 +66,29 @@ struct Environment: Sendable { /// One configured KestrelKit: takes calls from any thread without blocking and hands them, /// in order, to the ``Engine`` actor through an `AsyncStream`. final class Client: Sendable { + /// The most commands waiting for the engine. When the engine is stuck (a slow network + /// call, say) and an app sends signals faster than that, the oldest commands go first. + static let maxPendingCommands = 1_000 + let configuration: Kestrel.Configuration let environment: Environment private let continuation: AsyncStream.Continuation + private let uploader: Uploader private let timer: Task? init(baseURL: URL, key: String, configuration: Kestrel.Configuration, environment: Environment) { self.configuration = configuration self.environment = environment - let (stream, continuation) = AsyncStream.makeStream(of: Command.self) + let (stream, continuation) = AsyncStream.makeStream( + of: Command.self, + bufferingPolicy: .bufferingNewest(Self.maxPendingCommands) + ) self.continuation = continuation - let engine = Engine( - uploader: Uploader(endpoint: Uploader.endpoint(for: baseURL), key: key, session: environment.session), - configuration: configuration, - environment: environment - ) + let uploader = Uploader(endpoint: Uploader.endpoint(for: baseURL), key: key, session: environment.session) + self.uploader = uploader + let engine = Engine(uploader: uploader, configuration: configuration, environment: environment) Task { for await command in stream { await engine.handle(command) @@ -82,7 +101,7 @@ final class Client: Sendable { while !Task.isCancelled { try? await Task.sleep(nanoseconds: UInt64(interval * 1_000_000_000)) if Task.isCancelled { break } - if case .terminated = continuation.yield(.tick) { break } + if case .terminated = Self.enqueue(.tick, on: continuation) { break } } } } else { @@ -91,9 +110,9 @@ final class Client: Sendable { if !environment.store.isEnabled { // Opted out in an earlier run: make sure nothing is left behind. - continuation.yield(.optOut) + Self.enqueue(.optOut, on: continuation) } else if configuration.automaticSignals { - continuation.yield(.signal(type: "App.launched", payload: [:], time: environment.now())) + Self.enqueue(.signal(type: "App.launched", payload: [:], time: environment.now()), on: continuation) } } @@ -107,7 +126,7 @@ final class Client: Sendable { func signal(_ type: String, payload: [String: String] = [:]) { guard isEnabled else { return } - continuation.yield(.signal(type: type, payload: payload, time: environment.now())) + Self.enqueue(.signal(type: type, payload: payload, time: environment.now()), on: continuation) } func flush() async { @@ -122,12 +141,14 @@ final class Client: Sendable { /// The app is back: a new session starts if it was away for 5 minutes or more. func willEnterForeground() { - continuation.yield(.foreground(time: environment.now())) + Self.enqueue(.foreground(time: environment.now()), on: continuation) } - /// Deletes the queue and the install id. + /// Deletes the queue and the install id, and stops the upload in flight at once. The engine + /// is busy while it waits for the server, so the opt-out can't wait for its turn. func optOut() { - continuation.yield(.optOut) + uploader.control.cancel() + Self.enqueue(.optOut, on: continuation) } /// Returns once everything sent to the engine before it has been handled. @@ -140,9 +161,24 @@ final class Client: Sendable { continuation.finish() } + /// Hands a command to the engine. When the stream is full, the oldest command is dropped: + /// its waiter is let go, and an opt-out is put back, because that one must not get lost. + @discardableResult + static func enqueue( + _ command: Command, + on continuation: AsyncStream.Continuation + ) -> AsyncStream.Continuation.YieldResult { + let result = continuation.yield(command) + guard case let .dropped(oldest) = result else { return result } + oldest.abandon() + guard case .optOut = oldest else { return result } + if case .optOut = command { return result } + return enqueue(oldest, on: continuation) + } + private func send(_ command: (CheckedContinuation) -> Command) async { await withCheckedContinuation { done in - if case .terminated = continuation.yield(command(done)) { + if case .terminated = Self.enqueue(command(done), on: continuation) { done.resume() } } diff --git a/Sources/KestrelKit/DiskQueue.swift b/Sources/KestrelKit/DiskQueue.swift index 55df55c..1e2abf5 100644 --- a/Sources/KestrelKit/DiskQueue.swift +++ b/Sources/KestrelKit/DiskQueue.swift @@ -3,28 +3,35 @@ import Foundation /// Signals waiting to be sent, kept in memory and mirrored to a JSON-lines file /// (`signals.jsonl`, one encoded signal per line) so they survive restarts. /// -/// New signals are appended to the file; sending or dropping signals rewrites it -/// atomically. A line that isn't a JSON object (a write cut short by a crash) is skipped -/// when the file is read. When the queue holds more than `maxCount` signals or `maxBytes` -/// bytes, the oldest are dropped first. Only used from inside the ``Engine`` actor. +/// New signals are appended to the file; sending signals rewrites it atomically. A line that +/// isn't a JSON object (a write cut short by a crash) is skipped when the file is read. When +/// the queue holds more than `maxCount` signals or `maxBytes` bytes, the oldest are dropped +/// first. A full queue drops one signal for every new one, so it forgets them in memory at +/// once but only rewrites the file when `compactAfter` forgotten signals have piled up in +/// it (the next launch drops them again). Only used from inside the ``Engine`` actor. struct DiskQueue { let fileURL: URL let maxCount: Int let maxBytes: Int + let compactAfter: Int private(set) var lines: [Data] = [] private var bytes = 0 + /// Signals dropped from memory that are still in the file. + private var staleLines = 0 + private var staleBytes = 0 - init(directory: URL, maxCount: Int, maxBytes: Int = 5 * 1024 * 1024) { + init(directory: URL, maxCount: Int, maxBytes: Int = 5 * 1024 * 1024, compactAfter: Int? = nil) { self.fileURL = directory.appendingPathComponent("signals.jsonl") self.maxCount = max(1, maxCount) self.maxBytes = max(Limits.maxBodyBytes, maxBytes) + self.compactAfter = max(1, compactAfter ?? max(50, self.maxCount / 10)) Self.prepare(directory) guard let data = try? Data(contentsOf: fileURL) else { return } let all = data.split(separator: UInt8(ascii: "\n"), omittingEmptySubsequences: true).map { Data($0) } lines = all.filter(Self.isSignal) bytes = lines.reduce(0) { $0 + $1.count } - let trimmed = trimToLimits() + let trimmed = trimToLimits().lines if trimmed > 0 || lines.count != all.count || data.last != UInt8(ascii: "\n") { rewrite() } @@ -38,12 +45,15 @@ struct DiskQueue { mutating func append(_ line: Data) -> Int { lines.append(line) bytes += line.count - let dropped = trimToLimits() + let (dropped, droppedBytes) = trimToLimits() + appendToFile(line) if dropped > 0 { Log.debug("The queue is full; dropped the \(dropped) oldest signal(s).") - rewrite() - } else { - appendToFile(line) + staleLines += dropped + staleBytes += droppedBytes + if staleLines >= compactAfter || staleBytes >= maxBytes / 4 { + rewrite() + } } return dropped } @@ -60,20 +70,24 @@ struct DiskQueue { mutating func removeAll() { lines.removeAll() bytes = 0 + staleLines = 0 + staleBytes = 0 try? FileManager.default.removeItem(at: fileURL) } // MARK: - Private - private mutating func trimToLimits() -> Int { + private mutating func trimToLimits() -> (lines: Int, bytes: Int) { var dropped = 0 + var droppedBytes = 0 // The newest signal always stays. while lines.count - dropped > 1, lines.count - dropped > maxCount || bytes > maxBytes { bytes -= lines[dropped].count + droppedBytes += lines[dropped].count dropped += 1 } if dropped > 0 { lines.removeFirst(dropped) } - return dropped + return (dropped, droppedBytes) } private func appendToFile(_ line: Data) { @@ -93,7 +107,9 @@ struct DiskQueue { } } - private func rewrite() { + private mutating func rewrite() { + staleLines = 0 + staleBytes = 0 guard !lines.isEmpty else { try? FileManager.default.removeItem(at: fileURL) return diff --git a/Sources/KestrelKit/Engine.swift b/Sources/KestrelKit/Engine.swift index abdd26a..a56a448 100644 --- a/Sources/KestrelKit/Engine.swift +++ b/Sources/KestrelKit/Engine.swift @@ -6,6 +6,9 @@ import Foundation actor Engine { /// A new session starts after this long in the background. static let sessionTimeout: TimeInterval = 5 * 60 + /// How long sending pauses after 401, 403 or 404, which a server fix or a new key can cure + /// without the app being relaunched. + static let unavailablePause: TimeInterval = 3600 private let uploader: Uploader private let configuration: Kestrel.Configuration @@ -20,8 +23,6 @@ actor Engine { private var backgroundedAt: Date? private var backoff = Backoff() private var retryAt: Date? - /// Set after 401, 403 or 404; cleared by the next launch. - private var stoppedUntilRelaunch = false init(uploader: Uploader, configuration: Kestrel.Configuration, environment: Environment) { self.uploader = uploader @@ -150,13 +151,19 @@ actor Engine { private enum Stop { case unavailable case backoff(retryAfter: TimeInterval?) + /// The app opted out while a batch was on its way. + case disabled } private func flush() async { - guard isEnabled, !stoppedUntilRelaunch else { return } + // Before the check, so that an opt-out that comes after it can't be missed: it + // cancels sending, and nothing resets that again until the next flush. + uploader.control.reset() + guard isEnabled else { return } if let retryAt, environment.now() < retryAt { return } - while true { + // Opting out can happen at any time, so each batch checks again. + while isEnabled { let batch = nextBatch() guard !batch.isEmpty else { return } let (handled, stop) = await deliver(batch) @@ -165,9 +172,12 @@ actor Engine { case nil: backoff.reset() retryAt = nil + case .disabled: + return case .unavailable: - stoppedUntilRelaunch = true - Log.debug("The server refused the key or has app analytics turned off; nothing more is sent until the next launch.") + let pause = Self.unavailablePause * min(max(environment.jitter(), 0), 1) + retryAt = environment.now().addingTimeInterval(pause) + Log.debug("The server refused the key or has app analytics turned off; trying again in \(Int(pause)) s.") return case let .backoff(retryAfter): let delay = backoff.next(retryAfter: retryAfter, jitter: environment.jitter()) @@ -196,8 +206,12 @@ actor Engine { /// Sends `lines`, splitting on 413. Returns how many of the first lines are done with /// (accepted or dropped) and why it stopped early, if it did. private func deliver(_ lines: ArraySlice) async -> (handled: Int, stop: Stop?) { + guard isEnabled else { return (0, .disabled) } let body = Wire.body(lines: lines, sentAt: environment.now()) - switch await uploader.send(body) { + let outcome = await uploader.send(body) + // An opt-out cancels the upload, which then looks like a failure: not one to back off for. + guard isEnabled else { return (0, .disabled) } + switch outcome { case .accepted: return (lines.count, nil) case .rejected: diff --git a/Sources/KestrelKit/Kestrel.swift b/Sources/KestrelKit/Kestrel.swift index eac85f0..4e5234f 100644 --- a/Sources/KestrelKit/Kestrel.swift +++ b/Sources/KestrelKit/Kestrel.swift @@ -26,6 +26,10 @@ public enum Kestrel { /// /// Nothing is sent before this call. Later calls are ignored. /// + /// KestrelKit only sends over HTTPS. An `http://` address is accepted for `localhost` and + /// in debug builds (for a Kestrel on your own network); in a release build any other + /// `http://` address is refused, and nothing is stored or sent. + /// /// - Parameters: /// - url: Your Keen Kestrel's address, such as `https://kestrel.example.com`. /// - key: The project's public key: the 32 hex characters from its DSN @@ -37,6 +41,10 @@ public enum Kestrel { Log.debug("Kestrel.configure was called again; the first configuration stays.") return nil } + guard isAllowedEndpoint(url) else { + Log.debug("KestrelKit sends over https:// only (http:// is allowed for localhost and in debug builds); \(url.absoluteString) is refused and nothing is sent.") + return nil + } let key = key.trimmingCharacters(in: .whitespacesAndNewlines) if key.count != 32 || !key.allSatisfy(\.isHexDigit) { Log.debug("The key should be the 32 hex characters from the project's DSN.") @@ -100,6 +108,27 @@ public enum Kestrel { } extension Kestrel { + #if DEBUG + private static let isDebugBuild = true + #else + private static let isDebugBuild = false + #endif + + /// Whether signals may be sent to `url`: `https://`, or `http://` for `localhost` and (when + /// `allowsInsecure`, which debug builds default to) any other host. A key and the signals + /// in clear text would be readable on the way. + static func isAllowedEndpoint(_ url: URL, allowsInsecure: Bool = isDebugBuild) -> Bool { + guard let host = url.host, !host.isEmpty else { return false } + switch url.scheme?.lowercased() { + case "https": + return true + case "http": + return allowsInsecure || ["localhost", "127.0.0.1", "::1", "[::1]"].contains(host.lowercased()) + default: + return false + } + } + /// Forgets the configured client, for tests. static func reset(store: any SettingsStore, makeEnvironment: (@Sendable (any SettingsStore) -> Environment)?) { let old = state.withLock { state in diff --git a/Sources/KestrelKit/Limits.swift b/Sources/KestrelKit/Limits.swift index e5e6b06..225ae32 100644 --- a/Sources/KestrelKit/Limits.swift +++ b/Sources/KestrelKit/Limits.swift @@ -10,6 +10,9 @@ enum Limits { static let maxPayloadKeys = 20 static let maxKeyLength = 40 static let maxValueLength = 200 + /// A payload value is cut to this many characters before it is scrubbed, so a huge value + /// costs little. Everything past `maxValueLength` is dropped anyway. + static let maxRawValueLength = 2_000 /// For the other strings (bundle id, versions, model, locale); P4 in the protocol. static let maxFieldLength = 200 @@ -32,7 +35,8 @@ enum Limits { /// The payload to send: the signal's own keys first (sorted), then the default payload's /// keys it doesn't have (sorted), at most 20, each key and value cleaned, scrubbed and cut. - /// Keys that name an IP address or a user agent are dropped, as the server would drop them. + /// Keys that name an IP address or a user agent are dropped, as the server would drop them, + /// and so are keys that look like they hold a secret (`password`, `token`, `card`…). static func payload(_ own: [String: String], defaults: [String: String] = [:]) -> [String: String] { let ordered = own.keys.sorted().map { ($0, own[$0]!) } + defaults.keys.filter { own[$0] == nil }.sorted().map { ($0, defaults[$0]!) } @@ -48,7 +52,14 @@ enum Limits { Log.debug("The payload key \"\(key)\" names an IP address or a user agent, which Kestrel doesn't keep; dropped.") continue } - result[key] = truncate(clean(Scrubber.scrub(rawValue)), to: maxValueLength) + if Scrubber.isSensitiveKey(key) { + Log.debug("The payload key \"\(key)\" looks like it holds a secret; dropped.") + continue + } + // Control characters go first: scrubbing "john@exa\tmple.com" as it is would miss + // the address, and removing the tab afterwards would put it back together. + let value = clean(truncate(rawValue, to: maxRawValueLength)) + result[key] = truncate(Scrubber.scrub(value), to: maxValueLength) } return result } diff --git a/Sources/KestrelKit/Scrubber.swift b/Sources/KestrelKit/Scrubber.swift index 1bc461e..dd639f4 100644 --- a/Sources/KestrelKit/Scrubber.swift +++ b/Sources/KestrelKit/Scrubber.swift @@ -5,26 +5,37 @@ import Foundation /// scrubbing (`src/analytics/clean.rs` with `src/event/scrub.rs`), which runs again on the /// server, so a value missed here is still caught there. In this order: /// -/// - Card numbers: 13–19 digits, optionally grouped by single spaces or dashes, that pass -/// the Luhn check. +/// - Card numbers: 13–19 digits, optionally grouped by single spaces, dashes, dots or +/// non-breaking spaces, that pass the Luhn check. /// - `Bearer …` and `Basic …` credentials. /// - `name=value` pairs whose name contains `pass`, `secret`, `token`, `auth`, `key`, `card` /// or `cookie`: `token=abc` becomes `token=[Filtered]`. /// - Email addresses. -/// - IP addresses: IPv4, and IPv6 written out with at least four groups -/// (`2001:db8:0:0:1:0:0:1`). Shortened IPv6 forms with `::` are kept, as on the server. +/// - IP addresses: IPv4, IPv6 written out with at least four groups +/// (`2001:db8:0:0:1:0:0:1`), and IPv6 shortened with `::` (`2001:db8::1`, `::1`). /// - Phone numbers: 9–15 digits that start with `+` or `00`, or are written in at least /// three groups. Plain digit runs (order numbers, ids) and dates are kept. +/// +/// Callers remove control characters and cut the value to ``Limits/maxRawValueLength`` +/// before scrubbing (see ``Limits/payload(_:defaults:)``): control characters would otherwise +/// split an address past the patterns, and the runs below are bounded so that a long value +/// can't make a pattern slow. enum Scrubber { static let filtered = "[Filtered]" - private static let card = Pattern(#"\b(?:\d[ -]?){12,18}\d\b"#) + private static let card = Pattern(#"\b(?:\d[ .\-\x{00A0}\x{2007}\x{202F}]?){12,18}\d\b"#) private static let credential = Pattern(#"(?i)\b(bearer|basic)\s+[A-Za-z0-9\-._~+/]+=*"#) private static let sensitivePair = Pattern( - #"(?i)\b([\w.\-]*(?:pass|secret|token|auth|key|card|cookie)[\w.\-]*)=([^&\s'",;]+)"# + #"(?i)([\w.\-]{0,40}(?:pass|secret|token|auth|key|card|cookie)[\w.\-]{0,40})=([^&\s'",;]+)"# + ) + // A local part is at most 64 characters and a label 63; longer runs only shorten the match. + private static let email = Pattern( + #"(?i)[a-z0-9._%+\-]{1,64}@[a-z0-9\-]{1,63}(?:\.[a-z0-9\-]{1,63}){0,8}\.[a-z]{2,63}"# ) - private static let email = Pattern(#"(?i)[a-z0-9._%+\-]+@[a-z0-9\-]+(?:\.[a-z0-9\-]+)*\.[a-z]{2,}"#) private static let ip = Pattern(#"(?i)\b(?:(?:\d{1,3}\.){3}\d{1,3}|(?:[0-9a-f]{1,4}:){3,7}[0-9a-f]{1,4})\b"#) + /// A run of hex digits and colons on its own (not part of a word, and not the start of an + /// IPv4 tail such as `::ffff:1.2.3.4`); ``isCompressedIPv6(_:)`` decides. + private static let ipv6Candidate = Pattern(#"(?i)(? Bool { - // Only A–Z are lowered, like Rust's `eq_ignore_ascii_case`, so no other character - // (the Kelvin sign, say) can turn into a listed key. - let lowered = String(decoding: key.utf8.map { (65...90).contains($0) ? $0 + 32 : $0 }, as: UTF8.self) + let lowered = lowercasedASCII(key) return ipKeys.contains(lowered) || userAgentKeys.contains(lowered) } + /// Parts of a payload key that mark its value as a secret: the server's list for web + /// properties (`SENSITIVE_KEY`), plus `cvv`, lower-case. `ssn` counts only as a word of its + /// own, not inside one (`classname`). + static let sensitiveKeyParts = ["pass", "secret", "token", "auth", "key", "card", "cookie", "cvv"] + + /// Whether a payload key (already cleaned and cut) looks like it holds a secret or a card + /// detail: it contains one of ``sensitiveKeyParts`` or is a word `ssn` (`ssn`, `user_ssn`), + /// ignoring ASCII case. The value is dropped with the key, because no pattern can tell what + /// a secret looks like. + static func isSensitiveKey(_ key: String) -> Bool { + let lowered = lowercasedASCII(key) + if sensitiveKeyParts.contains(where: { lowered.contains($0) }) { return true } + return lowered.split { !($0.isASCII && ($0.isLetter || $0.isNumber)) }.contains("ssn") + } + + /// Only A–Z are lowered, like Rust's `eq_ignore_ascii_case`, so no other character + /// (the Kelvin sign, say) can turn into a listed key. + private static func lowercasedASCII(_ text: String) -> String { + String(decoding: text.utf8.map { (65...90).contains($0) ? $0 + 32 : $0 }, as: UTF8.self) + } + + /// Whether `candidate` (hex digits and colons) is an IPv6 address shortened with one `::`: + /// one to seven groups of one to four hex digits around it. + static func isCompressedIPv6(_ candidate: String) -> Bool { + let halves = candidate.components(separatedBy: "::") + guard halves.count == 2 else { return false } + let groups = halves.flatMap { $0.isEmpty ? [] : $0.split(separator: ":", omittingEmptySubsequences: false) } + guard (1...7).contains(groups.count) else { return false } + return groups.allSatisfy { (1...4).contains($0.count) && $0.allSatisfy(\.isHexDigit) } + } + /// The Luhn checksum over the digits in `text`. static func luhn(_ text: String) -> Bool { let digits = text.compactMap(\.wholeNumberValue) diff --git a/Sources/KestrelKit/Uploader.swift b/Sources/KestrelKit/Uploader.swift index 4a44bef..0fb9b9b 100644 --- a/Sources/KestrelKit/Uploader.swift +++ b/Sources/KestrelKit/Uploader.swift @@ -5,25 +5,28 @@ import FoundationNetworking /// What the server said about a batch, reduced to what the SDK does next (docs/protocol.md). enum UploadOutcome: Equatable, Sendable { - /// 2xx: done, also when the server dropped some of the signals (`"dropped": m`). + /// `202` with Keen Kestrel's answer (`{"accepted": n}`): done, also when the server dropped + /// some of the signals (`"dropped": m`). case accepted - /// 400 and other 4xx: the batch can never succeed; drop it and don't retry. + /// 400 and other 4xx that trying again won't fix: drop the batch. case rejected /// 401, 403 (wrong or disabled key, analytics off) and 404 (a server without app - /// analytics): stop sending until the next launch, keep the queue. + /// analytics): pause sending for an hour, keep the queue. case unavailable /// 413: split the batch and send the halves. case tooLarge /// 429 and 503: wait `Retry-After` seconds, or back off when it's missing. case slowDown(retryAfter: TimeInterval?) - /// Other 5xx, 408, or no answer: keep the batch and retry with backoff. + /// Everything else: a 2xx that isn't Kestrel's answer (a captive portal, a proxy), a + /// redirect, 405, 407, 408, 410, other 5xx, or no answer. Keep the batch and retry with backoff. case failed - static func classify(status: Int, retryAfter: String?) -> UploadOutcome { + static func classify(status: Int, retryAfter: String?, body: Data = Data()) -> UploadOutcome { switch status { - case 200..<300: return .accepted + case 202: return isKestrelAnswer(body) ? .accepted : .failed case 401, 403, 404: return .unavailable - case 408: return .failed + // Not Kestrel's verdict on the batch: a proxy, a gateway or a server that moved. + case 405, 407, 408, 410: return .failed case 413: return .tooLarge case 429, 503: return .slowDown(retryAfter: retryAfter.flatMap(parseRetryAfter)) case 400..<500: return .rejected @@ -39,6 +42,16 @@ enum UploadOutcome: Equatable, Sendable { return seconds } + /// Whether a `202` body is what Keen Kestrel answers: a JSON object with the number of + /// stored signals, `{"accepted": 8}` or `{"accepted": 8, "dropped": 2}`. + static func isKestrelAnswer(_ body: Data) -> Bool { + struct Answer: Decodable { + var accepted: Int + var dropped: Int? + } + return (try? JSONDecoder().decode(Answer.self, from: body)).map { $0.accepted >= 0 } ?? false + } + /// How many signals a `202` body says the server dropped (`{"accepted": 8, "dropped": 2}`), /// or `nil` when it doesn't say. Only for the debug log: the batch is done either way. static func droppedCount(in body: Data) -> Int? { @@ -49,11 +62,127 @@ enum UploadOutcome: Equatable, Sendable { } } +/// Lets the app abort the upload in flight from any thread (the opt-out), and makes sure none +/// starts afterwards. +final class UploadControl: Sendable { + private let state = Locked<(task: URLSessionTask?, cancelled: Bool)>((task: nil, cancelled: false)) + + /// Remembers the task about to run. `false` when sending was cancelled: don't start it. + func start(_ task: URLSessionTask) -> Bool { + state.withLock { state in + guard !state.cancelled else { return false } + state.task = task + return true + } + } + + func finish() { + state.withLock { $0.task = nil } + } + + /// Cancels the upload in flight and refuses new ones until ``reset()``. + func cancel() { + state.withLock { state in + state.cancelled = true + state.task?.cancel() + } + } + + /// Allows uploads again (sending was switched back on). + func reset() { + state.withLock { $0.cancelled = false } + } +} + +/// The session's delegate: refuses redirects, caps the size of answers and hands the answer +/// of each task to the ``Uploader`` that is waiting for it. +final class TransportDelegate: NSObject, URLSessionDataDelegate, @unchecked Sendable { + /// Kestrel's answers are a few dozen bytes; anything bigger isn't from Kestrel. + static let maxResponseBytes = 64 * 1024 + + struct Answer { + var status: Int + var retryAfter: String? + var body: Data + } + + private struct Pending { + var continuation: CheckedContinuation + var body = Data() + var oversized = false + } + + private let pending = Locked<[Int: Pending]>([:]) + + /// A session with this delegate. The session keeps it alive. + static func makeSession(_ configuration: URLSessionConfiguration) -> URLSession { + URLSession(configuration: configuration, delegate: TransportDelegate(), delegateQueue: nil) + } + + func register(_ task: URLSessionTask, _ continuation: CheckedContinuation) { + pending.withLock { $0[task.taskIdentifier] = Pending(continuation: continuation) } + } + + func unregister(_ task: URLSessionTask) -> CheckedContinuation? { + pending.withLock { $0.removeValue(forKey: task.taskIdentifier)?.continuation } + } + + // The signals and the key are only meant for the configured address, so a redirect is + // never followed: its response counts as the answer, and it isn't a `202`. + func urlSession( + _ session: URLSession, + task: URLSessionTask, + willPerformHTTPRedirection response: HTTPURLResponse, + newRequest request: URLRequest, + completionHandler: @escaping (URLRequest?) -> Void + ) { + completionHandler(nil) + } + + func urlSession( + _ session: URLSession, + dataTask: URLSessionDataTask, + didReceive response: URLResponse, + completionHandler: @escaping (URLSession.ResponseDisposition) -> Void + ) { + completionHandler(response.expectedContentLength > Int64(Self.maxResponseBytes) ? .cancel : .allow) + } + + func urlSession(_ session: URLSession, dataTask: URLSessionDataTask, didReceive data: Data) { + let oversized = pending.withLock { pending -> Bool in + guard var entry = pending[dataTask.taskIdentifier] else { return false } + entry.body.append(data) + if entry.body.count > Self.maxResponseBytes { + entry.body = Data() + entry.oversized = true + } + pending[dataTask.taskIdentifier] = entry + return entry.oversized + } + if oversized { dataTask.cancel() } + } + + func urlSession(_ session: URLSession, task: URLSessionTask, didCompleteWithError error: Error?) { + guard let entry = pending.withLock({ $0.removeValue(forKey: task.taskIdentifier) }) else { return } + guard error == nil, !entry.oversized, let http = task.response as? HTTPURLResponse else { + entry.continuation.resume(returning: nil) + return + } + entry.continuation.resume(returning: Answer( + status: http.statusCode, + retryAfter: Uploader.header("Retry-After", in: http), + body: entry.body + )) + } +} + /// Posts batches to `{baseURL}/ingest/v1/signals`. struct Uploader: Sendable { let endpoint: URL let key: String + /// Made by ``TransportDelegate/makeSession(_:)``. let session: URLSession + let control = UploadControl() static func endpoint(for baseURL: URL) -> URL { baseURL.appendingPathComponent("ingest").appendingPathComponent("v1").appendingPathComponent("signals") @@ -78,26 +207,31 @@ struct Uploader: Sendable { } func send(_ body: Data) async -> UploadOutcome { + guard let delegate = session.delegate as? TransportDelegate else { + Log.debug("The session has no TransportDelegate; nothing is sent.") + return .failed + } let request = request(for: body) - let answer: (status: Int, retryAfter: String?, body: Data)? = await withCheckedContinuation { continuation in - let task = session.dataTask(with: request) { data, response, _ in - guard let http = response as? HTTPURLResponse else { - continuation.resume(returning: nil) - return - } - continuation.resume(returning: (http.statusCode, Self.header("Retry-After", in: http), data ?? Data())) + let answer: TransportDelegate.Answer? = await withCheckedContinuation { continuation in + let task = session.dataTask(with: request) + delegate.register(task, continuation) + guard control.start(task) else { + // Sending was cancelled while this call was on its way. + delegate.unregister(task)?.resume(returning: nil) + return } task.resume() } + control.finish() guard let answer else { return .failed } - let outcome = UploadOutcome.classify(status: answer.status, retryAfter: answer.retryAfter) + let outcome = UploadOutcome.classify(status: answer.status, retryAfter: answer.retryAfter, body: answer.body) if outcome == .accepted, let dropped = UploadOutcome.droppedCount(in: answer.body), dropped > 0 { Log.debug("The server dropped \(dropped) signal(s) of a batch because they broke the protocol's limits.") } return outcome } - private static func header(_ name: String, in response: HTTPURLResponse) -> String? { + static func header(_ name: String, in response: HTTPURLResponse) -> String? { for (key, value) in response.allHeaderFields { if let key = key as? String, key.caseInsensitiveCompare(name) == .orderedSame { return value as? String diff --git a/Tests/KestrelKitTests/DeliveryTests.swift b/Tests/KestrelKitTests/DeliveryTests.swift index a8ad816..488b38a 100644 --- a/Tests/KestrelKitTests/DeliveryTests.swift +++ b/Tests/KestrelKitTests/DeliveryTests.swift @@ -29,12 +29,10 @@ struct DeliveryTests { #expect(harness.queuedLines == 0) } - @Test("A 202 that reports dropped signals (or says nothing readable) still ends the batch", arguments: [ + @Test("A 202 that reports dropped signals still ends the batch", arguments: [ #"{"accepted":1,"dropped":1}"#, #"{"accepted":0,"dropped":2}"#, #"{"accepted":2}"#, - "", - "not json", ]) func acceptedWithDropped(body: String) async { let harness = Harness() @@ -51,6 +49,141 @@ struct DeliveryTests { #expect(harness.server.requests.map(\.types) == [["Kept", "Broken"], ["Next"]]) } + @Test("Only Kestrel's own answer ends a batch; any other 2xx is retried", arguments: [ + Reply.status(200, body: #"{"accepted":1}"#), + .status(200), + .status(204), + .status(202, body: ""), + .status(202, body: "not json"), + .status(202, body: "Welcome to the hotel wifi"), + .status(202, body: #"{"dropped":1}"#), + .status(202, body: #"{"accepted":"1"}"#), + .status(202, body: "[]"), + ]) + func notKestrelsAnswer(reply: Reply) async { + let harness = Harness() + harness.server.script([reply]) + let client = harness.client() + client.signal("A") + await client.flush() + #expect(harness.queuedLines == 1) + + harness.clock.advance(by: 31) + await client.flush() + #expect(harness.server.requests.map(\.types) == [["A"], ["A"]]) + #expect(harness.queuedLines == 0) + } + + @Test("405, 407 and 410 keep the batch and retry with backoff", arguments: [405, 407, 410]) + func notVerdicts(status: Int) async { + let harness = Harness() + harness.server.script([.status(status)]) + let client = harness.client() + client.signal("A") + await client.flush() + #expect(harness.queuedLines == 1) + + harness.clock.advance(by: 29) + await client.flush() + #expect(harness.server.requests.count == 1) + + harness.clock.advance(by: 2) + await client.flush() + #expect(harness.server.requests.map(\.types) == [["A"], ["A"]]) + #expect(harness.queuedLines == 0) + } + + // Only on Apple platforms: swift-corelibs-foundation can't redirect from a mock URLProtocol. + #if canImport(Darwin) + @Test("A redirect is not followed and the batch is kept") + func redirect() async { + let target = MockServer() + let harness = Harness() + harness.server.respond { _ in .redirect(to: "https://\(target.host)/ingest/v1/signals") } + let client = harness.client() + client.signal("A") + let finished = Locked(false) + Task { + await client.flush() + finished.withLock { $0 = true } + } + #expect(await eventually { finished.withLock { $0 } }) + #expect(target.requests.isEmpty) + #expect(harness.queuedLines == 1) + } + #endif + + @Test("An answer bigger than 64 KB is not Kestrel's; the batch is kept") + func hugeAnswer() async { + let harness = Harness() + harness.server.script([.status(202, body: #"{"accepted":1}"# + String(repeating: " ", count: 100_000))]) + let client = harness.client() + client.signal("A") + await client.flush() + #expect(harness.queuedLines == 1) + + harness.clock.advance(by: 31) + await client.flush() + #expect(harness.server.requests.count == 2) + #expect(harness.queuedLines == 0) + } + + @Test("Opting out cancels the upload in flight and nothing more is sent") + func optOutDuringUpload() async { + let harness = Harness() + harness.server.respond { _ in .hang } + let client = harness.client() + for n in 0..<150 { client.signal("S\(n)") } + let finished = Locked(false) + Task { + await client.flush() + finished.withLock { $0 = true } + } + #expect(await eventually { harness.server.requests.count == 1 }) + + harness.store.set(false, forKey: SettingsKey.isEnabled) + client.optOut() + // The hanging request is cancelled at once (its timeout is 30 s) and the flush ends + // without trying the second batch. + #expect(await eventually(timeout: 5) { finished.withLock { $0 } }) + await client.drain() + #expect(harness.server.requests.count == 1) + #expect(harness.queuedLines == 0) + + // Back on: sending works again. + harness.store.set(true, forKey: SettingsKey.isEnabled) + harness.server.respond { _ in .accepted } + client.signal("After") + await client.flush() + #expect(harness.server.requests.last?.types == ["After"]) + } + + @Test("A full command stream drops the oldest commands, lets their waiters go and keeps an opt-out") + func fullCommandStream() async { + let (stream, continuation) = AsyncStream.makeStream(of: Command.self, bufferingPolicy: .bufferingNewest(3)) + let time = Date() + // Would wait forever if the barrier were dropped without being let go. + await withCheckedContinuation { done in + Client.enqueue(.barrier(done), on: continuation) + Client.enqueue(.optOut, on: continuation) + for n in 0..<3 { + Client.enqueue(.signal(type: "S\(n)", payload: [:], time: time), on: continuation) + } + } + continuation.finish() + var kept: [String] = [] + for await command in stream { + switch command { + case let .signal(type, _, _): kept.append(type) + case .optOut: kept.append("optOut") + default: kept.append("other") + } + } + #expect(kept.count == 3) + #expect(kept.contains("optOut")) + #expect(kept.contains("S2")) + } + @Test("The dropped count is read from the 202 body when it is there") func droppedCount() { #expect(UploadOutcome.droppedCount(in: Data(#"{"accepted":8,"dropped":2}"#.utf8)) == 2) @@ -77,7 +210,7 @@ struct DeliveryTests { let harness = Harness() let client = harness.client() var payload: [String: String] = [:] - for key in 0..<20 { payload["key\(key)"] = String(repeating: "abcdefghij", count: 20) } + for key in 0..<20 { payload["item\(key)"] = String(repeating: "abcdefghij", count: 20) } for _ in 0..<80 { client.signal("Big.signal", payload: payload) } await client.flush() @@ -101,7 +234,7 @@ struct DeliveryTests { #expect(harness.queuedLines == 0) } - @Test("401, 403 and 404 stop sending until the next launch", arguments: [401, 403, 404]) + @Test("401, 403 and 404 pause sending for an hour", arguments: [401, 403, 404]) func unavailable(status: Int) async { let harness = Harness() harness.server.script([.status(status)]) @@ -109,18 +242,32 @@ struct DeliveryTests { client.signal("A") await client.flush() client.signal("B") - harness.clock.advance(by: 7200) + harness.clock.advance(by: 3599) await client.flush() #expect(harness.server.requests.count == 1) await client.drain() #expect(harness.queuedLines == 2) + + // An hour later it tries again, without a relaunch, and sends everything that was kept. + harness.clock.advance(by: 2) + await client.flush() + #expect(harness.server.requests.count == 2) + #expect(harness.server.requests.last?.types == ["A", "B"]) + #expect(harness.queuedLines == 0) + } + + @Test("A relaunch tries again at once after 401, 403 and 404", arguments: [401, 403, 404]) + func unavailableThenRelaunch(status: Int) async { + let harness = Harness() + harness.server.script([.status(status)]) + let client = harness.client() + client.signal("A") + await client.flush() client.shutdown() - // The next launch tries again and sends everything that was kept. let relaunched = harness.client() await relaunched.flush() - #expect(harness.server.requests.count == 2) - #expect(harness.server.requests.last?.types == ["A", "B"]) + #expect(harness.server.requests.map(\.types) == [["A"], ["A"]]) } @Test("413 splits the batch until the halves fit") @@ -246,7 +393,14 @@ struct DeliveryTests { @Test("Response codes map to the protocol's actions") func classification() { - #expect(UploadOutcome.classify(status: 202, retryAfter: nil) == .accepted) + let answer = Data(#"{"accepted":3}"#.utf8) + #expect(UploadOutcome.classify(status: 202, retryAfter: nil, body: answer) == .accepted) + #expect(UploadOutcome.classify(status: 202, retryAfter: nil) == .failed) + #expect(UploadOutcome.classify(status: 200, retryAfter: nil, body: answer) == .failed) + #expect(UploadOutcome.classify(status: 301, retryAfter: nil) == .failed) + #expect(UploadOutcome.classify(status: 405, retryAfter: nil) == .failed) + #expect(UploadOutcome.classify(status: 407, retryAfter: nil) == .failed) + #expect(UploadOutcome.classify(status: 410, retryAfter: nil) == .failed) #expect(UploadOutcome.classify(status: 400, retryAfter: nil) == .rejected) #expect(UploadOutcome.classify(status: 422, retryAfter: nil) == .rejected) #expect(UploadOutcome.classify(status: 401, retryAfter: nil) == .unavailable) diff --git a/Tests/KestrelKitTests/FacadeTests.swift b/Tests/KestrelKitTests/FacadeTests.swift index 6bac81f..117ae42 100644 --- a/Tests/KestrelKitTests/FacadeTests.swift +++ b/Tests/KestrelKitTests/FacadeTests.swift @@ -39,6 +39,35 @@ struct FacadeTests { #expect(signals[2].payload == ["pages": "12", "pdf": "true", "ratio": "0.5", "name": "report"]) } + @Test("Plain http is refused except for localhost, and in debug builds") + func endpoints() { + func allowed(_ url: String, insecure: Bool) -> Bool { + Kestrel.isAllowedEndpoint(URL(string: url)!, allowsInsecure: insecure) + } + for insecure in [false, true] { + #expect(allowed("https://kestrel.example.com", insecure: insecure)) + #expect(allowed("HTTPS://kestrel.example.com:8443/base", insecure: insecure)) + #expect(allowed("http://localhost:3001", insecure: insecure)) + #expect(allowed("http://127.0.0.1:3001", insecure: insecure)) + #expect(allowed("http://[::1]:3001", insecure: insecure)) + #expect(!allowed("ftp://kestrel.example.com", insecure: insecure)) + #expect(!allowed("kestrel.example.com", insecure: insecure)) + } + #expect(!allowed("http://kestrel.example.com", insecure: false)) + #expect(!allowed("http://10.0.3.26:3001", insecure: false)) + #expect(!allowed("http://localhost.evil.example", insecure: false)) + #expect(allowed("http://10.0.3.26:3001", insecure: true)) + } + + @Test("configure refuses an address it won't send to") + func refusedEndpoint() async { + Kestrel.configure(url: URL(string: "ftp://kestrel.example.com")!, key: TestKey.value) + #expect(Kestrel.client == nil) + Kestrel.signal("A") + await Kestrel.flush() + #expect(!FileManager.default.fileExists(atPath: harness.directory.path)) + } + @Test("A second configure is ignored") func configureOnce() async { let other = MockServer() diff --git a/Tests/KestrelKitTests/Helpers.swift b/Tests/KestrelKitTests/Helpers.swift index 5b25671..696a114 100644 --- a/Tests/KestrelKitTests/Helpers.swift +++ b/Tests/KestrelKitTests/Helpers.swift @@ -64,6 +64,10 @@ enum Reply: Sendable { /// `body` is sent as given; without one, a 202 says `{"accepted":1}` and others say nothing. case status(Int, headers: [String: String] = [:], body: String? = nil) case networkError + /// Never answers; the request stays in flight until it is cancelled. + case hang + /// A 302 to `location`, which KestrelKit must not follow. + case redirect(to: String) static let accepted = Reply.status(202) } @@ -138,6 +142,17 @@ final class MockURLProtocol: URLProtocol { client?.urlProtocolDidFinishLoading(self) case .networkError: client?.urlProtocol(self, didFailWithError: URLError(.notConnectedToInternet)) + case .hang: + break + case let .redirect(location): + let response = HTTPURLResponse(url: url, statusCode: 302, httpVersion: "HTTP/1.1", headerFields: ["Location": location])! + #if canImport(Darwin) + client?.urlProtocol(self, wasRedirectedTo: URLRequest(url: URL(string: location)!), redirectResponse: response) + #else + // swift-corelibs-foundation traps on redirects from a custom URLProtocol. + _ = response + client?.urlProtocol(self, didFailWithError: URLError(.badServerResponse)) + #endif } } @@ -158,7 +173,7 @@ final class MockURLProtocol: URLProtocol { func mockSession() -> URLSession { let configuration = URLSessionConfiguration.ephemeral configuration.protocolClasses = [MockURLProtocol.self] - return URLSession(configuration: configuration) + return TransportDelegate.makeSession(configuration) } // MARK: - Clients diff --git a/Tests/KestrelKitTests/LimitsTests.swift b/Tests/KestrelKitTests/LimitsTests.swift index 3d87af0..1aed543 100644 --- a/Tests/KestrelKitTests/LimitsTests.swift +++ b/Tests/KestrelKitTests/LimitsTests.swift @@ -89,6 +89,109 @@ struct LimitsTests { #expect(payload == ["email": "[Filtered]", "plan": "pro"]) } + @Test("Control characters can't split an address past the scrubbing", arguments: [ + ("john@exa\tmple.com", "[Filtered]"), + ("john@example.com\n", "[Filtered]"), + ("jo\u{0}hn@exam\u{7}ple.com", "[Filtered]"), + ("4111 1111 11\n11 1111", "[Filtered]"), + ("41\t11 1111 1111 1111", "[Filtered]"), + ("ip 203.0.\r113.7", "ip [Filtered]"), + ("Bearer\u{0} abc.def", "Bearer [Filtered]"), + ("tok\u{1}en=abc", "token=[Filtered]"), + ]) + func controlCharacters(input: String, expected: String) { + #expect(Limits.payload(["note": input]) == ["note": expected]) + } + + @Test("Card numbers grouped with dots or non-breaking spaces are scrubbed", arguments: [ + "4111.1111.1111.1111", + "4111\u{00A0}1111\u{00A0}1111\u{00A0}1111", + "4111\u{202F}1111\u{202F}1111\u{202F}1111", + "4111-1111.1111 1111", + ]) + func cardSeparators(input: String) { + #expect(Scrubber.scrub(input) == "[Filtered]") + #expect(Scrubber.scrub("paid with \(input) today") == "paid with [Filtered] today") + } + + @Test("Shortened IPv6 addresses are replaced", arguments: [ + ("2001:db8::1", "[Filtered]"), + ("::1", "[Filtered]"), + ("fe80::1", "[Filtered]"), + ("FE80::202:B3FF:FE1E:8329", "[Filtered]"), + ("2001:db8:0:0:1::", "[Filtered]"), + ("from 2001:db8::8a2e:370:7334, then", "from [Filtered], then"), + ("/ip/::1/x", "/ip/[Filtered]/x"), + ("[2001:db8::1]:8080", "[[Filtered]]:8080"), + ]) + func shortenedIPv6(input: String, expected: String) { + #expect(Scrubber.scrub(input) == expected) + } + + @Test("Things that only look a bit like a shortened IPv6 address are kept", arguments: [ + "::", + ":::1", + "1::2::3", + "std::fmt::Display", + "Namespace::Item", + "12:30:45", + "a:b", + "db8::12345", + ]) + func notShortenedIPv6(input: String) { + #expect(Scrubber.scrub(input) == input) + } + + @Test("Keys that look like they hold a secret are dropped with their values", arguments: [ + "password", "Password", "passwd", "user_pass", "secret", "client_secret", "token", "apiToken", + "auth", "Authorization", "key", "api_key", "apiKey", "card", "cardNumber", "cvv", "CVV2", + "ssn", "SSN", "user_ssn", "ssn-last4", "cookie", "session_cookie", + ]) + func sensitiveKeys(key: String) { + #expect(Scrubber.isSensitiveKey(key), "\(key)") + #expect(Limits.payload([key: "hunter2", "plan": "pro"]) == ["plan": "pro"]) + } + + @Test("The default payload's secret keys are dropped too") + func sensitiveDefaultKeys() { + #expect(Limits.payload(["plan": "pro"], defaults: ["api_token": "abc", "channel": "beta"]) == ["plan": "pro", "channel": "beta"]) + } + + @Test("Ordinary keys are kept", arguments: ["plan", "className", "classname", "screen", "count", "name", "email", "item00", "\u{212A}ey"]) + func ordinaryKeys(key: String) { + #expect(!Scrubber.isSensitiveKey(key), "\(key)") + #expect(Limits.payload([key: "x"]) == [key: "x"]) + } + + @Test("A huge value is cut before it is scrubbed and scrubs fast", arguments: [ + String(repeating: "a", count: 20_000), + String(repeating: "a.", count: 10_000), + String(repeating: "a@", count: 10_000), + String(repeating: "1 ", count: 10_000), + String(repeating: "token", count: 4_000), + String(repeating: "pass-", count: 4_000), + String(repeating: "1:", count: 10_000), + "x@" + String(repeating: "a.", count: 10_000), + String(repeating: "a", count: 20_000) + "@example.com", + ]) + func hugeValues(value: String) { + let clock = ContinuousClock() + let elapsed = clock.measure { + let payload = Limits.payload(["note": value]) + #expect((payload["note"]?.unicodeScalars.count ?? 0) <= Limits.maxValueLength) + } + #expect(elapsed < .seconds(2), "took \(elapsed)") + } + + @Test("A value is cut to 2,000 characters before scrubbing, then to 200") + func rawCap() { + let payload = Limits.payload(["note": String(repeating: "x", count: 2_100) + "jane@example.com"]) + #expect(payload["note"] == String(repeating: "x", count: 200)) + // An address in the first 200 characters is still found when the value is long. + let early = Limits.payload(["note": "jane@example.com " + String(repeating: "x", count: 5_000)]) + #expect(early["note"] == "[Filtered] " + String(repeating: "x", count: 189)) + } + @Test("Luhn") func luhn() { #expect(Scrubber.luhn("4111111111111111")) @@ -111,13 +214,11 @@ struct LimitsTests { #expect(Scrubber.scrub(input) == expected) } - @Test("Shortened IPv6 and things that only look a bit like addresses are kept", arguments: [ + @Test("Things that only look a bit like addresses are kept", arguments: [ "1.4.0", "/v/1.2.3", "12:30", "12:30:45", - "::1", - "fe80::1", ]) func notIPAddresses(input: String) { #expect(Scrubber.scrub(input) == input) @@ -170,6 +271,8 @@ struct LimitsTests { let payload = Limits.payload([ "email": "jane.doe@example.com", "phone": "+47 912 34 567", + "payment": "4111 1111 1111 1111", + "header": "Authorization: Bearer abc.def-123", "card": "4111 1111 1111 1111", "auth": "Authorization: Bearer abc.def-123", "note": "order 123456789", @@ -183,8 +286,9 @@ struct LimitsTests { ]) #expect(payload["email"] == "[Filtered]") #expect(payload["phone"] == "[Filtered]") - #expect(payload["card"] == "[Filtered]") - #expect(payload["auth"] == "Authorization: Bearer [Filtered]") + #expect(payload["payment"] == "[Filtered]") + #expect(payload["header"] == "Authorization: Bearer [Filtered]") + #expect(payload["card"] == nil && payload["auth"] == nil) #expect(payload["note"] == "order 123456789") #expect(payload["pages"] == "12" && payload["pdf"] == "true") #expect(payload["ip"] == nil && payload["user_agent"] == nil) @@ -197,10 +301,10 @@ struct LimitsTests { @Test("Dropped keys don't use up the 20, and are cleaned before the check") func droppedKeysInPayloads() { var own: [String: String] = [" IP\u{0007} ": "203.0.113.7", "User-Agent": "Mozilla/5.0"] - for i in 0..<20 { own["key\(i < 10 ? "0" : "")\(i)"] = "\(i)" } + for i in 0..<20 { own["item\(i < 10 ? "0" : "")\(i)"] = "\(i)" } let payload = Limits.payload(own, defaults: ["x-forwarded-for": "10.0.0.1"]) #expect(payload.count == 20) - #expect(payload.keys.allSatisfy { $0.hasPrefix("key") }) + #expect(payload.keys.allSatisfy { $0.hasPrefix("item") }) } @Test("Every type KestrelKit sends is one the server accepts as it is", arguments: [ diff --git a/Tests/KestrelKitTests/QueueTests.swift b/Tests/KestrelKitTests/QueueTests.swift index ff6326b..179178e 100644 --- a/Tests/KestrelKitTests/QueueTests.swift +++ b/Tests/KestrelKitTests/QueueTests.swift @@ -44,6 +44,35 @@ struct QueueTests { #expect(DiskQueue(directory: directory, maxCount: 3).lines == [line(2), line(3), line(4)]) } + @Test("A full queue appends to the file and rewrites it only now and then") + func fullQueueRewritesRarely() throws { + func fileLines(_ queue: DiskQueue) throws -> Int { + try Data(contentsOf: queue.fileURL).split(separator: UInt8(ascii: "\n")).count + } + var queue = DiskQueue(directory: directory, maxCount: 3, compactAfter: 4) + for n in 0..<5 { queue.append(line(n)) } + #expect(queue.lines == [line(2), line(3), line(4)]) + // Two signals were dropped from memory, but the file still has them: no rewrite yet. + #expect(try fileLines(queue) == 5) + + queue.append(line(5)) + #expect(try fileLines(queue) == 6) + // The fourth dropped signal compacts the file. + queue.append(line(6)) + #expect(queue.lines == [line(4), line(5), line(6)]) + #expect(try fileLines(queue) == 3) + #expect(DiskQueue(directory: directory, maxCount: 3).lines == [line(4), line(5), line(6)]) + } + + @Test("Many signals into a full queue keep the file small") + func fullQueueStaysBounded() throws { + var queue = DiskQueue(directory: directory, maxCount: 100) + for n in 0..<1_000 { queue.append(line(n)) } + let fileLines = try Data(contentsOf: queue.fileURL).split(separator: UInt8(ascii: "\n")).count + #expect(fileLines <= 100 + queue.compactAfter) + #expect(DiskQueue(directory: directory, maxCount: 100).lines == (900..<1_000).map(line)) + } + @Test("A smaller cap on the next launch trims the file") func capOnLoad() { var queue = DiskQueue(directory: directory, maxCount: 10) diff --git a/docs/protocol.md b/docs/protocol.md index 2f112bc..2ab1e42 100644 --- a/docs/protocol.md +++ b/docs/protocol.md @@ -25,6 +25,10 @@ POST {baseURL}/ingest/v1/signals No cookies are sent or accepted (P12). Apps send no `Origin` header. **Server:** if a request has one, it must be one of the key's allowed origins, or the answer is `403`. +SDKs send only over `https://`. KestrelKit accepts `http://` for `localhost` and, in debug builds, +for any host; it never follows a redirect, reads at most 64 KB of an answer, and gives a request +60 seconds at most. + KestrelKit compresses bodies of 1 KB or more with gzip, and sends the compressed form only when it is smaller (P3). It uses no other encoding. @@ -79,8 +83,8 @@ sends: is sent as `Paywall_shown`), then cuts it to 100 characters. An empty type isn't sent. This matters, because the server drops a signal whose type has other characters (see below). - It converts numbers and booleans in payloads to strings (`12`, `true`), puts the signal's own keys - before the default payload's, and drops keys that name an IP address or a user agent (see - [Scrubbing](#scrubbing-p13)). + before the default payload's, and drops keys that name an IP address or a user agent, or look + like they hold a secret (see [Scrubbing](#scrubbing-p13)). - It removes control characters from the other strings, trims them and cuts them to 200. ### Lenient per signal, strict per batch (P5) @@ -101,7 +105,7 @@ has more than 100 signals or is too large. Otherwise it fixes or drops each sign | has numbers or booleans as payload values | stores them as strings (`12`, `true`). | | has objects, arrays or `null` as payload values | drops those keys. | | has more than 20 payload keys | keeps the first 20 it reads. | -| has payload keys over 40 characters | cuts them to 40. Empty keys and keys that name an IP address or a user agent are dropped. | +| has payload keys over 40 characters | cuts them to 40. Empty keys and keys that name an IP address or a user agent are dropped. A key that looks like it holds a secret is dropped too (see [Scrubbing](#scrubbing-p13)). | The `202` answer counts the signals that were stored, and says how many were dropped when some were (see [Responses](#responses)). @@ -120,21 +124,25 @@ project says otherwise) are dropped. Both sides replace payload values that look like personal data with `[Filtered]`, with the same rules (Keen Kestrel's `src/analytics/clean.rs` and `src/event/scrub.rs`, KestrelKit's -`Scrubber.swift`), in this order: +`Scrubber.swift`). Before the values are scrubbed, both remove control characters (so +`john@example.com` can't slip past the patterns) and, in KestrelKit, cut the value to 2,000 +characters (it is cut to 200 afterwards). Then, in this order: 1. Card numbers: 13–19 digits, optionally grouped by single spaces or dashes, that pass the Luhn - check. + check. KestrelKit also accepts dots and non-breaking spaces (U+00A0, U+2007, U+202F) as separators; + the server doesn't yet. 2. `Bearer …` and `Basic …` credentials. The scheme stays: `Bearer [Filtered]`. 3. `name=value` pairs whose name contains `pass`, `secret`, `token`, `auth`, `key`, `card` or `cookie`: `token=abc` becomes `token=[Filtered]`. 4. Email addresses. 5. IP addresses: IPv4, and IPv6 written out with at least four groups (`2001:db8:0:0:1:0:0:1`). - Shortened IPv6 with `::` isn't recognised. + KestrelKit also recognises IPv6 shortened with one `::` (`2001:db8::1`, `::1`, `fe80::1`); the + server doesn't yet. 6. Phone numbers: 9–15 digits that start with `+` or `00`, or are written in at least three groups. Plain digit runs (order numbers, ids), dates and versions are kept. -Payload keys that name an IP address or a user agent are dropped with their values. The keys, -compared ignoring ASCII case, are: +Payload keys that name an IP address or a user agent, or look like they hold a secret, are dropped +with their values. The IP address and user agent keys, compared ignoring ASCII case, are: - `ip`, `ip_address`, `client_ip`, `remote_addr`; - the forwarding headers `x-forwarded-for`, `x-real-ip`, `forwarded`, `cf-connecting-ip`, @@ -143,6 +151,12 @@ compared ignoring ASCII case, are: `http_true_client_ip`, `http_x_client_ip`; - `user-agent`, `user_agent`, `http_user_agent`. +A key looks like it holds a secret when, ignoring ASCII case, it contains `pass`, `secret`, `token`, +`auth`, `key`, `card`, `cookie` or `cvv` (so `password`, `apiKey`, `cardNumber` and `Authorization`, +but also `keyboard_layout` and `author`), or has `ssn` as a word of its own (`ssn`, `user_ssn`; not +`classname`). Both sides drop such keys (the server's own list for web event properties is the same +without `cvv` and `ssn`). + KestrelKit scrubs on the device, before a signal is queued. The server scrubs again, so older or other SDKs are covered too. Scrubbing is a safety net: apps should keep personal data out of payloads. @@ -152,20 +166,22 @@ payloads. | Status | Meaning | What KestrelKit does | |---|---|---| | `202 {"accepted": n}` | Done: `n` signals were stored. When some were dropped (P5), the body also has `"dropped": m`. | Forgets the batch. Logs `dropped` in debug builds; never resends. | -| other `2xx` | Done. | Forgets the batch. | +| other `2xx`, or a `202` without `"accepted"` | Not Kestrel's answer: a captive portal, a proxy. | Keeps the batch and retries with backoff. | +| `3xx` | A redirect. SDKs don't follow redirects. | Keeps the batch and retries with backoff. | | `400` | The batch is malformed: not JSON, no `signals` array, or an unknown or broken `Content-Encoding`. | Drops it and doesn't retry. | -| `401` | No key, or an unknown one. | Stops sending until the next launch and keeps the queue. Logs in debug builds only. | +| `401` | No key, or an unknown one. | Pauses sending for an hour and keeps the queue. Logs in debug builds only. | | `403` | The key is disabled, app analytics is off for the project, or the `Origin` header isn't allowed. | As `401`. | | `404` | The server has no app-signal ingest (an older Kestrel). | As `401`, so the signals arrive once the server is upgraded (P7). | -| `408` | Request timeout. | Retries like a `5xx` (P7). | +| `405`, `407`, `408`, `410` | Not verdicts on the batch: a proxy, a gateway or a server that moved. | Retries like a `5xx` (P7). | | `413` | More than 100 signals, or more than 256 KB, compressed or not. | Splits the batch and retries the halves; drops a single signal that is still too large. | | `429` | A rate limit: the server's per-IP limit or the key's `rate_limit_per_minute`. Both count requests, not signals. | Waits `Retry-After` seconds. | | `503` | The server's write queue is full, or it briefly can't take signals. | Waits `Retry-After` seconds. | | other `4xx` | | As `400` (P7). | | other `5xx`, network error | Something went wrong. | Keeps the batch and retries with backoff. | -SDKs treat any `2xx` as done, whatever `n` and `m` are, so they never resend a batch the server has -seen (P5, P7). Error responses have a `{"detail": "…"}` body meant for people; SDKs don't need to +SDKs treat a `202` with an `accepted` count as done, whatever `n` and `m` are, so they never resend a +batch the server has seen (P5, P7). Any other answer to a successful request can't be Kestrel's, and +the batch is kept. Error responses have a `{"detail": "…"}` body meant for people; SDKs don't need to read it. All answers carry `Cache-Control: no-store`. **Retry-After (P8).** `429` and `503` carry `Retry-After` in seconds. KestrelKit reads it as a number From 3f8a1fb718a97cab96ef23852c9201fb5d409091 Mon Sep 17 00:00:00 2001 From: Ferdinand99 Date: Tue, 27 Oct 2026 18:11:56 +0100 Subject: [PATCH 2/2] test: let the mock answer a refused redirect with the 302 itself --- Tests/KestrelKitTests/Helpers.swift | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/Tests/KestrelKitTests/Helpers.swift b/Tests/KestrelKitTests/Helpers.swift index 696a114..f9da62a 100644 --- a/Tests/KestrelKitTests/Helpers.swift +++ b/Tests/KestrelKitTests/Helpers.swift @@ -148,6 +148,10 @@ final class MockURLProtocol: URLProtocol { let response = HTTPURLResponse(url: url, statusCode: 302, httpVersion: "HTTP/1.1", headerFields: ["Location": location])! #if canImport(Darwin) client?.urlProtocol(self, wasRedirectedTo: URLRequest(url: URL(string: location)!), redirectResponse: response) + // When the delegate refuses to follow, the 302 itself is the answer. (A followed + // redirect would stop this load and start a new one on the target.) + client?.urlProtocol(self, didReceive: response, cacheStoragePolicy: .notAllowed) + client?.urlProtocolDidFinishLoading(self) #else // swift-corelibs-foundation traps on redirects from a custom URLProtocol. _ = response