diff --git a/README.md b/README.md index 28da388..de5501d 100644 --- a/README.md +++ b/README.md @@ -55,6 +55,10 @@ The key is the project's public key: the 32 hex characters at the start of the p (`https://@kestrel.example.com/1`), on the project's settings page in Kestrel. It's the same key the Sentry SDK uses, so one project gets both errors and analytics. +KestrelKit only talks to `https://` addresses. `http://` works for `localhost`, and in debug builds +for any host (a Kestrel on your own network); in a release build any other `http://` address is +refused and nothing is stored or sent. + ### SwiftUI ```swift @@ -117,12 +121,21 @@ Each signal has: **Never collected:** the IDFA or IDFV, the device name, the user's name or email, location, contacts, IP addresses (Kestrel doesn't store them), or anything to fingerprint a device. -Payloads are filtered on the device, with the same rules Kestrel applies again on the server: +Payloads are filtered on the device, with the same rules Kestrel applies again on the server (the +shortened IPv6 forms and the dot and non-breaking-space card groupings below are only caught on the +device for now). Control characters are removed first, and a value is cut to 2,000 characters before +it is filtered, so a very long value is cheap: -- values that look like an email address, a phone number, a card number, an IP address, a - `Bearer`/`Basic` credential or a `token=…`-style secret are replaced by `[Filtered]`; +- values that look like an email address, a phone number, a card number (13 to 19 digits grouped by + spaces, dashes, dots or non-breaking spaces, and passing the Luhn check), an IP address (IPv4, and + IPv6 written out or shortened with `::`, such as `2001:db8::1` and `::1`), a `Bearer`/`Basic` + credential or a `token=…`-style secret are replaced by `[Filtered]`; - keys that name an IP address or a user agent (`ip`, `client_ip`, `x-forwarded-for`, `user_agent` - and the like) are dropped with their values. + and the like) are dropped with their values; +- keys that look like they hold a secret are dropped with their values: any key containing `pass` + (`password`, `passwd`), `secret`, `token`, `auth`, `key`, `card`, `cookie` or `cvv`, and `ssn` as a + word of its own, ignoring case. This is blunt on purpose: `keyboard_layout` and `author` go too, so + give such keys other names (`layout`, `writer`). Payloads are limited to 20 keys of up to 40 characters, with values of up to 200 characters. Keep personal data out of them; the filtering is a safety net, not a licence. @@ -157,8 +170,15 @@ choice across launches. Nothing is sent before `Kestrel.configure` is called eit survive restarts and offline periods. Each batch holds up to 100 signals or 256 KB, and is gzipped when that makes it smaller. - When the server is busy or unreachable, KestrelKit backs off: from 30 seconds up to an hour, or as - long as the server's `Retry-After` says. A wrong or disabled key stops sending until the next - launch. + long as the server's `Retry-After` says. A wrong or disabled key (or a server without app + analytics) pauses sending for an hour. Only Keen Kestrel's own answer (`202` with `{"accepted": n}`) + counts as delivered; any other answer, a captive portal's `200` for example, keeps the batch and + retries with backoff. +- Redirects are never followed, answers bigger than 64 KB are ignored, and a request gets 60 seconds + at most. +- Turning `Kestrel.isEnabled` off cancels an upload in progress and sends nothing more. +- Up to 1,000 calls can wait for the queue at once while it is busy; when more arrive, the oldest are + dropped. - On iOS, tvOS and visionOS, KestrelKit asks for a little background time to finish the upload when the app moves to the background. On the Mac, "background" means another app is active. - TestFlight builds are recognised by their sandbox App Store receipt. This doesn't work for Mac diff --git a/Sources/KestrelKit/Client.swift b/Sources/KestrelKit/Client.swift index a8ebd2b..4bf31ae 100644 --- a/Sources/KestrelKit/Client.swift +++ b/Sources/KestrelKit/Client.swift @@ -12,6 +12,18 @@ enum Command: Sendable { case foreground(time: Date) case optOut case barrier(CheckedContinuation) + + /// Lets whoever waits for this command go, for a command that will never be handled. + func abandon() { + switch self { + case let .flush(done), let .barrier(done): + done.resume() + case let .background(_, done): + done?.resume() + case .signal, .tick, .foreground, .optOut: + break + } + } } /// The outside world, swappable in tests. @@ -39,12 +51,13 @@ struct Environment: Sendable { let configuration = URLSessionConfiguration.ephemeral configuration.timeoutIntervalForRequest = 30 + configuration.timeoutIntervalForResource = 60 configuration.httpCookieStorage = nil configuration.urlCache = nil return Environment( store: store, directory: directory, - session: URLSession(configuration: configuration), + session: TransportDelegate.makeSession(configuration), device: device ) } @@ -53,23 +66,29 @@ struct Environment: Sendable { /// One configured KestrelKit: takes calls from any thread without blocking and hands them, /// in order, to the ``Engine`` actor through an `AsyncStream`. final class Client: Sendable { + /// The most commands waiting for the engine. When the engine is stuck (a slow network + /// call, say) and an app sends signals faster than that, the oldest commands go first. + static let maxPendingCommands = 1_000 + let configuration: Kestrel.Configuration let environment: Environment private let continuation: AsyncStream.Continuation + private let uploader: Uploader private let timer: Task? init(baseURL: URL, key: String, configuration: Kestrel.Configuration, environment: Environment) { self.configuration = configuration self.environment = environment - let (stream, continuation) = AsyncStream.makeStream(of: Command.self) + let (stream, continuation) = AsyncStream.makeStream( + of: Command.self, + bufferingPolicy: .bufferingNewest(Self.maxPendingCommands) + ) self.continuation = continuation - let engine = Engine( - uploader: Uploader(endpoint: Uploader.endpoint(for: baseURL), key: key, session: environment.session), - configuration: configuration, - environment: environment - ) + let uploader = Uploader(endpoint: Uploader.endpoint(for: baseURL), key: key, session: environment.session) + self.uploader = uploader + let engine = Engine(uploader: uploader, configuration: configuration, environment: environment) Task { for await command in stream { await engine.handle(command) @@ -82,7 +101,7 @@ final class Client: Sendable { while !Task.isCancelled { try? await Task.sleep(nanoseconds: UInt64(interval * 1_000_000_000)) if Task.isCancelled { break } - if case .terminated = continuation.yield(.tick) { break } + if case .terminated = Self.enqueue(.tick, on: continuation) { break } } } } else { @@ -91,9 +110,9 @@ final class Client: Sendable { if !environment.store.isEnabled { // Opted out in an earlier run: make sure nothing is left behind. - continuation.yield(.optOut) + Self.enqueue(.optOut, on: continuation) } else if configuration.automaticSignals { - continuation.yield(.signal(type: "App.launched", payload: [:], time: environment.now())) + Self.enqueue(.signal(type: "App.launched", payload: [:], time: environment.now()), on: continuation) } } @@ -107,7 +126,7 @@ final class Client: Sendable { func signal(_ type: String, payload: [String: String] = [:]) { guard isEnabled else { return } - continuation.yield(.signal(type: type, payload: payload, time: environment.now())) + Self.enqueue(.signal(type: type, payload: payload, time: environment.now()), on: continuation) } func flush() async { @@ -122,12 +141,14 @@ final class Client: Sendable { /// The app is back: a new session starts if it was away for 5 minutes or more. func willEnterForeground() { - continuation.yield(.foreground(time: environment.now())) + Self.enqueue(.foreground(time: environment.now()), on: continuation) } - /// Deletes the queue and the install id. + /// Deletes the queue and the install id, and stops the upload in flight at once. The engine + /// is busy while it waits for the server, so the opt-out can't wait for its turn. func optOut() { - continuation.yield(.optOut) + uploader.control.cancel() + Self.enqueue(.optOut, on: continuation) } /// Returns once everything sent to the engine before it has been handled. @@ -140,9 +161,24 @@ final class Client: Sendable { continuation.finish() } + /// Hands a command to the engine. When the stream is full, the oldest command is dropped: + /// its waiter is let go, and an opt-out is put back, because that one must not get lost. + @discardableResult + static func enqueue( + _ command: Command, + on continuation: AsyncStream.Continuation + ) -> AsyncStream.Continuation.YieldResult { + let result = continuation.yield(command) + guard case let .dropped(oldest) = result else { return result } + oldest.abandon() + guard case .optOut = oldest else { return result } + if case .optOut = command { return result } + return enqueue(oldest, on: continuation) + } + private func send(_ command: (CheckedContinuation) -> Command) async { await withCheckedContinuation { done in - if case .terminated = continuation.yield(command(done)) { + if case .terminated = Self.enqueue(command(done), on: continuation) { done.resume() } } diff --git a/Sources/KestrelKit/DiskQueue.swift b/Sources/KestrelKit/DiskQueue.swift index 55df55c..1e2abf5 100644 --- a/Sources/KestrelKit/DiskQueue.swift +++ b/Sources/KestrelKit/DiskQueue.swift @@ -3,28 +3,35 @@ import Foundation /// Signals waiting to be sent, kept in memory and mirrored to a JSON-lines file /// (`signals.jsonl`, one encoded signal per line) so they survive restarts. /// -/// New signals are appended to the file; sending or dropping signals rewrites it -/// atomically. A line that isn't a JSON object (a write cut short by a crash) is skipped -/// when the file is read. When the queue holds more than `maxCount` signals or `maxBytes` -/// bytes, the oldest are dropped first. Only used from inside the ``Engine`` actor. +/// New signals are appended to the file; sending signals rewrites it atomically. A line that +/// isn't a JSON object (a write cut short by a crash) is skipped when the file is read. When +/// the queue holds more than `maxCount` signals or `maxBytes` bytes, the oldest are dropped +/// first. A full queue drops one signal for every new one, so it forgets them in memory at +/// once but only rewrites the file when `compactAfter` forgotten signals have piled up in +/// it (the next launch drops them again). Only used from inside the ``Engine`` actor. struct DiskQueue { let fileURL: URL let maxCount: Int let maxBytes: Int + let compactAfter: Int private(set) var lines: [Data] = [] private var bytes = 0 + /// Signals dropped from memory that are still in the file. + private var staleLines = 0 + private var staleBytes = 0 - init(directory: URL, maxCount: Int, maxBytes: Int = 5 * 1024 * 1024) { + init(directory: URL, maxCount: Int, maxBytes: Int = 5 * 1024 * 1024, compactAfter: Int? = nil) { self.fileURL = directory.appendingPathComponent("signals.jsonl") self.maxCount = max(1, maxCount) self.maxBytes = max(Limits.maxBodyBytes, maxBytes) + self.compactAfter = max(1, compactAfter ?? max(50, self.maxCount / 10)) Self.prepare(directory) guard let data = try? Data(contentsOf: fileURL) else { return } let all = data.split(separator: UInt8(ascii: "\n"), omittingEmptySubsequences: true).map { Data($0) } lines = all.filter(Self.isSignal) bytes = lines.reduce(0) { $0 + $1.count } - let trimmed = trimToLimits() + let trimmed = trimToLimits().lines if trimmed > 0 || lines.count != all.count || data.last != UInt8(ascii: "\n") { rewrite() } @@ -38,12 +45,15 @@ struct DiskQueue { mutating func append(_ line: Data) -> Int { lines.append(line) bytes += line.count - let dropped = trimToLimits() + let (dropped, droppedBytes) = trimToLimits() + appendToFile(line) if dropped > 0 { Log.debug("The queue is full; dropped the \(dropped) oldest signal(s).") - rewrite() - } else { - appendToFile(line) + staleLines += dropped + staleBytes += droppedBytes + if staleLines >= compactAfter || staleBytes >= maxBytes / 4 { + rewrite() + } } return dropped } @@ -60,20 +70,24 @@ struct DiskQueue { mutating func removeAll() { lines.removeAll() bytes = 0 + staleLines = 0 + staleBytes = 0 try? FileManager.default.removeItem(at: fileURL) } // MARK: - Private - private mutating func trimToLimits() -> Int { + private mutating func trimToLimits() -> (lines: Int, bytes: Int) { var dropped = 0 + var droppedBytes = 0 // The newest signal always stays. while lines.count - dropped > 1, lines.count - dropped > maxCount || bytes > maxBytes { bytes -= lines[dropped].count + droppedBytes += lines[dropped].count dropped += 1 } if dropped > 0 { lines.removeFirst(dropped) } - return dropped + return (dropped, droppedBytes) } private func appendToFile(_ line: Data) { @@ -93,7 +107,9 @@ struct DiskQueue { } } - private func rewrite() { + private mutating func rewrite() { + staleLines = 0 + staleBytes = 0 guard !lines.isEmpty else { try? FileManager.default.removeItem(at: fileURL) return diff --git a/Sources/KestrelKit/Engine.swift b/Sources/KestrelKit/Engine.swift index abdd26a..a56a448 100644 --- a/Sources/KestrelKit/Engine.swift +++ b/Sources/KestrelKit/Engine.swift @@ -6,6 +6,9 @@ import Foundation actor Engine { /// A new session starts after this long in the background. static let sessionTimeout: TimeInterval = 5 * 60 + /// How long sending pauses after 401, 403 or 404, which a server fix or a new key can cure + /// without the app being relaunched. + static let unavailablePause: TimeInterval = 3600 private let uploader: Uploader private let configuration: Kestrel.Configuration @@ -20,8 +23,6 @@ actor Engine { private var backgroundedAt: Date? private var backoff = Backoff() private var retryAt: Date? - /// Set after 401, 403 or 404; cleared by the next launch. - private var stoppedUntilRelaunch = false init(uploader: Uploader, configuration: Kestrel.Configuration, environment: Environment) { self.uploader = uploader @@ -150,13 +151,19 @@ actor Engine { private enum Stop { case unavailable case backoff(retryAfter: TimeInterval?) + /// The app opted out while a batch was on its way. + case disabled } private func flush() async { - guard isEnabled, !stoppedUntilRelaunch else { return } + // Before the check, so that an opt-out that comes after it can't be missed: it + // cancels sending, and nothing resets that again until the next flush. + uploader.control.reset() + guard isEnabled else { return } if let retryAt, environment.now() < retryAt { return } - while true { + // Opting out can happen at any time, so each batch checks again. + while isEnabled { let batch = nextBatch() guard !batch.isEmpty else { return } let (handled, stop) = await deliver(batch) @@ -165,9 +172,12 @@ actor Engine { case nil: backoff.reset() retryAt = nil + case .disabled: + return case .unavailable: - stoppedUntilRelaunch = true - Log.debug("The server refused the key or has app analytics turned off; nothing more is sent until the next launch.") + let pause = Self.unavailablePause * min(max(environment.jitter(), 0), 1) + retryAt = environment.now().addingTimeInterval(pause) + Log.debug("The server refused the key or has app analytics turned off; trying again in \(Int(pause)) s.") return case let .backoff(retryAfter): let delay = backoff.next(retryAfter: retryAfter, jitter: environment.jitter()) @@ -196,8 +206,12 @@ actor Engine { /// Sends `lines`, splitting on 413. Returns how many of the first lines are done with /// (accepted or dropped) and why it stopped early, if it did. private func deliver(_ lines: ArraySlice) async -> (handled: Int, stop: Stop?) { + guard isEnabled else { return (0, .disabled) } let body = Wire.body(lines: lines, sentAt: environment.now()) - switch await uploader.send(body) { + let outcome = await uploader.send(body) + // An opt-out cancels the upload, which then looks like a failure: not one to back off for. + guard isEnabled else { return (0, .disabled) } + switch outcome { case .accepted: return (lines.count, nil) case .rejected: diff --git a/Sources/KestrelKit/Kestrel.swift b/Sources/KestrelKit/Kestrel.swift index eac85f0..4e5234f 100644 --- a/Sources/KestrelKit/Kestrel.swift +++ b/Sources/KestrelKit/Kestrel.swift @@ -26,6 +26,10 @@ public enum Kestrel { /// /// Nothing is sent before this call. Later calls are ignored. /// + /// KestrelKit only sends over HTTPS. An `http://` address is accepted for `localhost` and + /// in debug builds (for a Kestrel on your own network); in a release build any other + /// `http://` address is refused, and nothing is stored or sent. + /// /// - Parameters: /// - url: Your Keen Kestrel's address, such as `https://kestrel.example.com`. /// - key: The project's public key: the 32 hex characters from its DSN @@ -37,6 +41,10 @@ public enum Kestrel { Log.debug("Kestrel.configure was called again; the first configuration stays.") return nil } + guard isAllowedEndpoint(url) else { + Log.debug("KestrelKit sends over https:// only (http:// is allowed for localhost and in debug builds); \(url.absoluteString) is refused and nothing is sent.") + return nil + } let key = key.trimmingCharacters(in: .whitespacesAndNewlines) if key.count != 32 || !key.allSatisfy(\.isHexDigit) { Log.debug("The key should be the 32 hex characters from the project's DSN.") @@ -100,6 +108,27 @@ public enum Kestrel { } extension Kestrel { + #if DEBUG + private static let isDebugBuild = true + #else + private static let isDebugBuild = false + #endif + + /// Whether signals may be sent to `url`: `https://`, or `http://` for `localhost` and (when + /// `allowsInsecure`, which debug builds default to) any other host. A key and the signals + /// in clear text would be readable on the way. + static func isAllowedEndpoint(_ url: URL, allowsInsecure: Bool = isDebugBuild) -> Bool { + guard let host = url.host, !host.isEmpty else { return false } + switch url.scheme?.lowercased() { + case "https": + return true + case "http": + return allowsInsecure || ["localhost", "127.0.0.1", "::1", "[::1]"].contains(host.lowercased()) + default: + return false + } + } + /// Forgets the configured client, for tests. static func reset(store: any SettingsStore, makeEnvironment: (@Sendable (any SettingsStore) -> Environment)?) { let old = state.withLock { state in diff --git a/Sources/KestrelKit/Limits.swift b/Sources/KestrelKit/Limits.swift index e5e6b06..225ae32 100644 --- a/Sources/KestrelKit/Limits.swift +++ b/Sources/KestrelKit/Limits.swift @@ -10,6 +10,9 @@ enum Limits { static let maxPayloadKeys = 20 static let maxKeyLength = 40 static let maxValueLength = 200 + /// A payload value is cut to this many characters before it is scrubbed, so a huge value + /// costs little. Everything past `maxValueLength` is dropped anyway. + static let maxRawValueLength = 2_000 /// For the other strings (bundle id, versions, model, locale); P4 in the protocol. static let maxFieldLength = 200 @@ -32,7 +35,8 @@ enum Limits { /// The payload to send: the signal's own keys first (sorted), then the default payload's /// keys it doesn't have (sorted), at most 20, each key and value cleaned, scrubbed and cut. - /// Keys that name an IP address or a user agent are dropped, as the server would drop them. + /// Keys that name an IP address or a user agent are dropped, as the server would drop them, + /// and so are keys that look like they hold a secret (`password`, `token`, `card`…). static func payload(_ own: [String: String], defaults: [String: String] = [:]) -> [String: String] { let ordered = own.keys.sorted().map { ($0, own[$0]!) } + defaults.keys.filter { own[$0] == nil }.sorted().map { ($0, defaults[$0]!) } @@ -48,7 +52,14 @@ enum Limits { Log.debug("The payload key \"\(key)\" names an IP address or a user agent, which Kestrel doesn't keep; dropped.") continue } - result[key] = truncate(clean(Scrubber.scrub(rawValue)), to: maxValueLength) + if Scrubber.isSensitiveKey(key) { + Log.debug("The payload key \"\(key)\" looks like it holds a secret; dropped.") + continue + } + // Control characters go first: scrubbing "john@exa\tmple.com" as it is would miss + // the address, and removing the tab afterwards would put it back together. + let value = clean(truncate(rawValue, to: maxRawValueLength)) + result[key] = truncate(Scrubber.scrub(value), to: maxValueLength) } return result } diff --git a/Sources/KestrelKit/Scrubber.swift b/Sources/KestrelKit/Scrubber.swift index 1bc461e..dd639f4 100644 --- a/Sources/KestrelKit/Scrubber.swift +++ b/Sources/KestrelKit/Scrubber.swift @@ -5,26 +5,37 @@ import Foundation /// scrubbing (`src/analytics/clean.rs` with `src/event/scrub.rs`), which runs again on the /// server, so a value missed here is still caught there. In this order: /// -/// - Card numbers: 13–19 digits, optionally grouped by single spaces or dashes, that pass -/// the Luhn check. +/// - Card numbers: 13–19 digits, optionally grouped by single spaces, dashes, dots or +/// non-breaking spaces, that pass the Luhn check. /// - `Bearer …` and `Basic …` credentials. /// - `name=value` pairs whose name contains `pass`, `secret`, `token`, `auth`, `key`, `card` /// or `cookie`: `token=abc` becomes `token=[Filtered]`. /// - Email addresses. -/// - IP addresses: IPv4, and IPv6 written out with at least four groups -/// (`2001:db8:0:0:1:0:0:1`). Shortened IPv6 forms with `::` are kept, as on the server. +/// - IP addresses: IPv4, IPv6 written out with at least four groups +/// (`2001:db8:0:0:1:0:0:1`), and IPv6 shortened with `::` (`2001:db8::1`, `::1`). /// - Phone numbers: 9–15 digits that start with `+` or `00`, or are written in at least /// three groups. Plain digit runs (order numbers, ids) and dates are kept. +/// +/// Callers remove control characters and cut the value to ``Limits/maxRawValueLength`` +/// before scrubbing (see ``Limits/payload(_:defaults:)``): control characters would otherwise +/// split an address past the patterns, and the runs below are bounded so that a long value +/// can't make a pattern slow. enum Scrubber { static let filtered = "[Filtered]" - private static let card = Pattern(#"\b(?:\d[ -]?){12,18}\d\b"#) + private static let card = Pattern(#"\b(?:\d[ .\-\x{00A0}\x{2007}\x{202F}]?){12,18}\d\b"#) private static let credential = Pattern(#"(?i)\b(bearer|basic)\s+[A-Za-z0-9\-._~+/]+=*"#) private static let sensitivePair = Pattern( - #"(?i)\b([\w.\-]*(?:pass|secret|token|auth|key|card|cookie)[\w.\-]*)=([^&\s'",;]+)"# + #"(?i)([\w.\-]{0,40}(?:pass|secret|token|auth|key|card|cookie)[\w.\-]{0,40})=([^&\s'",;]+)"# + ) + // A local part is at most 64 characters and a label 63; longer runs only shorten the match. + private static let email = Pattern( + #"(?i)[a-z0-9._%+\-]{1,64}@[a-z0-9\-]{1,63}(?:\.[a-z0-9\-]{1,63}){0,8}\.[a-z]{2,63}"# ) - private static let email = Pattern(#"(?i)[a-z0-9._%+\-]+@[a-z0-9\-]+(?:\.[a-z0-9\-]+)*\.[a-z]{2,}"#) private static let ip = Pattern(#"(?i)\b(?:(?:\d{1,3}\.){3}\d{1,3}|(?:[0-9a-f]{1,4}:){3,7}[0-9a-f]{1,4})\b"#) + /// A run of hex digits and colons on its own (not part of a word, and not the start of an + /// IPv4 tail such as `::ffff:1.2.3.4`); ``isCompressedIPv6(_:)`` decides. + private static let ipv6Candidate = Pattern(#"(?i)(? Bool { - // Only A–Z are lowered, like Rust's `eq_ignore_ascii_case`, so no other character - // (the Kelvin sign, say) can turn into a listed key. - let lowered = String(decoding: key.utf8.map { (65...90).contains($0) ? $0 + 32 : $0 }, as: UTF8.self) + let lowered = lowercasedASCII(key) return ipKeys.contains(lowered) || userAgentKeys.contains(lowered) } + /// Parts of a payload key that mark its value as a secret: the server's list for web + /// properties (`SENSITIVE_KEY`), plus `cvv`, lower-case. `ssn` counts only as a word of its + /// own, not inside one (`classname`). + static let sensitiveKeyParts = ["pass", "secret", "token", "auth", "key", "card", "cookie", "cvv"] + + /// Whether a payload key (already cleaned and cut) looks like it holds a secret or a card + /// detail: it contains one of ``sensitiveKeyParts`` or is a word `ssn` (`ssn`, `user_ssn`), + /// ignoring ASCII case. The value is dropped with the key, because no pattern can tell what + /// a secret looks like. + static func isSensitiveKey(_ key: String) -> Bool { + let lowered = lowercasedASCII(key) + if sensitiveKeyParts.contains(where: { lowered.contains($0) }) { return true } + return lowered.split { !($0.isASCII && ($0.isLetter || $0.isNumber)) }.contains("ssn") + } + + /// Only A–Z are lowered, like Rust's `eq_ignore_ascii_case`, so no other character + /// (the Kelvin sign, say) can turn into a listed key. + private static func lowercasedASCII(_ text: String) -> String { + String(decoding: text.utf8.map { (65...90).contains($0) ? $0 + 32 : $0 }, as: UTF8.self) + } + + /// Whether `candidate` (hex digits and colons) is an IPv6 address shortened with one `::`: + /// one to seven groups of one to four hex digits around it. + static func isCompressedIPv6(_ candidate: String) -> Bool { + let halves = candidate.components(separatedBy: "::") + guard halves.count == 2 else { return false } + let groups = halves.flatMap { $0.isEmpty ? [] : $0.split(separator: ":", omittingEmptySubsequences: false) } + guard (1...7).contains(groups.count) else { return false } + return groups.allSatisfy { (1...4).contains($0.count) && $0.allSatisfy(\.isHexDigit) } + } + /// The Luhn checksum over the digits in `text`. static func luhn(_ text: String) -> Bool { let digits = text.compactMap(\.wholeNumberValue) diff --git a/Sources/KestrelKit/Uploader.swift b/Sources/KestrelKit/Uploader.swift index 4a44bef..0fb9b9b 100644 --- a/Sources/KestrelKit/Uploader.swift +++ b/Sources/KestrelKit/Uploader.swift @@ -5,25 +5,28 @@ import FoundationNetworking /// What the server said about a batch, reduced to what the SDK does next (docs/protocol.md). enum UploadOutcome: Equatable, Sendable { - /// 2xx: done, also when the server dropped some of the signals (`"dropped": m`). + /// `202` with Keen Kestrel's answer (`{"accepted": n}`): done, also when the server dropped + /// some of the signals (`"dropped": m`). case accepted - /// 400 and other 4xx: the batch can never succeed; drop it and don't retry. + /// 400 and other 4xx that trying again won't fix: drop the batch. case rejected /// 401, 403 (wrong or disabled key, analytics off) and 404 (a server without app - /// analytics): stop sending until the next launch, keep the queue. + /// analytics): pause sending for an hour, keep the queue. case unavailable /// 413: split the batch and send the halves. case tooLarge /// 429 and 503: wait `Retry-After` seconds, or back off when it's missing. case slowDown(retryAfter: TimeInterval?) - /// Other 5xx, 408, or no answer: keep the batch and retry with backoff. + /// Everything else: a 2xx that isn't Kestrel's answer (a captive portal, a proxy), a + /// redirect, 405, 407, 408, 410, other 5xx, or no answer. Keep the batch and retry with backoff. case failed - static func classify(status: Int, retryAfter: String?) -> UploadOutcome { + static func classify(status: Int, retryAfter: String?, body: Data = Data()) -> UploadOutcome { switch status { - case 200..<300: return .accepted + case 202: return isKestrelAnswer(body) ? .accepted : .failed case 401, 403, 404: return .unavailable - case 408: return .failed + // Not Kestrel's verdict on the batch: a proxy, a gateway or a server that moved. + case 405, 407, 408, 410: return .failed case 413: return .tooLarge case 429, 503: return .slowDown(retryAfter: retryAfter.flatMap(parseRetryAfter)) case 400..<500: return .rejected @@ -39,6 +42,16 @@ enum UploadOutcome: Equatable, Sendable { return seconds } + /// Whether a `202` body is what Keen Kestrel answers: a JSON object with the number of + /// stored signals, `{"accepted": 8}` or `{"accepted": 8, "dropped": 2}`. + static func isKestrelAnswer(_ body: Data) -> Bool { + struct Answer: Decodable { + var accepted: Int + var dropped: Int? + } + return (try? JSONDecoder().decode(Answer.self, from: body)).map { $0.accepted >= 0 } ?? false + } + /// How many signals a `202` body says the server dropped (`{"accepted": 8, "dropped": 2}`), /// or `nil` when it doesn't say. Only for the debug log: the batch is done either way. static func droppedCount(in body: Data) -> Int? { @@ -49,11 +62,127 @@ enum UploadOutcome: Equatable, Sendable { } } +/// Lets the app abort the upload in flight from any thread (the opt-out), and makes sure none +/// starts afterwards. +final class UploadControl: Sendable { + private let state = Locked<(task: URLSessionTask?, cancelled: Bool)>((task: nil, cancelled: false)) + + /// Remembers the task about to run. `false` when sending was cancelled: don't start it. + func start(_ task: URLSessionTask) -> Bool { + state.withLock { state in + guard !state.cancelled else { return false } + state.task = task + return true + } + } + + func finish() { + state.withLock { $0.task = nil } + } + + /// Cancels the upload in flight and refuses new ones until ``reset()``. + func cancel() { + state.withLock { state in + state.cancelled = true + state.task?.cancel() + } + } + + /// Allows uploads again (sending was switched back on). + func reset() { + state.withLock { $0.cancelled = false } + } +} + +/// The session's delegate: refuses redirects, caps the size of answers and hands the answer +/// of each task to the ``Uploader`` that is waiting for it. +final class TransportDelegate: NSObject, URLSessionDataDelegate, @unchecked Sendable { + /// Kestrel's answers are a few dozen bytes; anything bigger isn't from Kestrel. + static let maxResponseBytes = 64 * 1024 + + struct Answer { + var status: Int + var retryAfter: String? + var body: Data + } + + private struct Pending { + var continuation: CheckedContinuation + var body = Data() + var oversized = false + } + + private let pending = Locked<[Int: Pending]>([:]) + + /// A session with this delegate. The session keeps it alive. + static func makeSession(_ configuration: URLSessionConfiguration) -> URLSession { + URLSession(configuration: configuration, delegate: TransportDelegate(), delegateQueue: nil) + } + + func register(_ task: URLSessionTask, _ continuation: CheckedContinuation) { + pending.withLock { $0[task.taskIdentifier] = Pending(continuation: continuation) } + } + + func unregister(_ task: URLSessionTask) -> CheckedContinuation? { + pending.withLock { $0.removeValue(forKey: task.taskIdentifier)?.continuation } + } + + // The signals and the key are only meant for the configured address, so a redirect is + // never followed: its response counts as the answer, and it isn't a `202`. + func urlSession( + _ session: URLSession, + task: URLSessionTask, + willPerformHTTPRedirection response: HTTPURLResponse, + newRequest request: URLRequest, + completionHandler: @escaping (URLRequest?) -> Void + ) { + completionHandler(nil) + } + + func urlSession( + _ session: URLSession, + dataTask: URLSessionDataTask, + didReceive response: URLResponse, + completionHandler: @escaping (URLSession.ResponseDisposition) -> Void + ) { + completionHandler(response.expectedContentLength > Int64(Self.maxResponseBytes) ? .cancel : .allow) + } + + func urlSession(_ session: URLSession, dataTask: URLSessionDataTask, didReceive data: Data) { + let oversized = pending.withLock { pending -> Bool in + guard var entry = pending[dataTask.taskIdentifier] else { return false } + entry.body.append(data) + if entry.body.count > Self.maxResponseBytes { + entry.body = Data() + entry.oversized = true + } + pending[dataTask.taskIdentifier] = entry + return entry.oversized + } + if oversized { dataTask.cancel() } + } + + func urlSession(_ session: URLSession, task: URLSessionTask, didCompleteWithError error: Error?) { + guard let entry = pending.withLock({ $0.removeValue(forKey: task.taskIdentifier) }) else { return } + guard error == nil, !entry.oversized, let http = task.response as? HTTPURLResponse else { + entry.continuation.resume(returning: nil) + return + } + entry.continuation.resume(returning: Answer( + status: http.statusCode, + retryAfter: Uploader.header("Retry-After", in: http), + body: entry.body + )) + } +} + /// Posts batches to `{baseURL}/ingest/v1/signals`. struct Uploader: Sendable { let endpoint: URL let key: String + /// Made by ``TransportDelegate/makeSession(_:)``. let session: URLSession + let control = UploadControl() static func endpoint(for baseURL: URL) -> URL { baseURL.appendingPathComponent("ingest").appendingPathComponent("v1").appendingPathComponent("signals") @@ -78,26 +207,31 @@ struct Uploader: Sendable { } func send(_ body: Data) async -> UploadOutcome { + guard let delegate = session.delegate as? TransportDelegate else { + Log.debug("The session has no TransportDelegate; nothing is sent.") + return .failed + } let request = request(for: body) - let answer: (status: Int, retryAfter: String?, body: Data)? = await withCheckedContinuation { continuation in - let task = session.dataTask(with: request) { data, response, _ in - guard let http = response as? HTTPURLResponse else { - continuation.resume(returning: nil) - return - } - continuation.resume(returning: (http.statusCode, Self.header("Retry-After", in: http), data ?? Data())) + let answer: TransportDelegate.Answer? = await withCheckedContinuation { continuation in + let task = session.dataTask(with: request) + delegate.register(task, continuation) + guard control.start(task) else { + // Sending was cancelled while this call was on its way. + delegate.unregister(task)?.resume(returning: nil) + return } task.resume() } + control.finish() guard let answer else { return .failed } - let outcome = UploadOutcome.classify(status: answer.status, retryAfter: answer.retryAfter) + let outcome = UploadOutcome.classify(status: answer.status, retryAfter: answer.retryAfter, body: answer.body) if outcome == .accepted, let dropped = UploadOutcome.droppedCount(in: answer.body), dropped > 0 { Log.debug("The server dropped \(dropped) signal(s) of a batch because they broke the protocol's limits.") } return outcome } - private static func header(_ name: String, in response: HTTPURLResponse) -> String? { + static func header(_ name: String, in response: HTTPURLResponse) -> String? { for (key, value) in response.allHeaderFields { if let key = key as? String, key.caseInsensitiveCompare(name) == .orderedSame { return value as? String diff --git a/Tests/KestrelKitTests/DeliveryTests.swift b/Tests/KestrelKitTests/DeliveryTests.swift index a8ad816..488b38a 100644 --- a/Tests/KestrelKitTests/DeliveryTests.swift +++ b/Tests/KestrelKitTests/DeliveryTests.swift @@ -29,12 +29,10 @@ struct DeliveryTests { #expect(harness.queuedLines == 0) } - @Test("A 202 that reports dropped signals (or says nothing readable) still ends the batch", arguments: [ + @Test("A 202 that reports dropped signals still ends the batch", arguments: [ #"{"accepted":1,"dropped":1}"#, #"{"accepted":0,"dropped":2}"#, #"{"accepted":2}"#, - "", - "not json", ]) func acceptedWithDropped(body: String) async { let harness = Harness() @@ -51,6 +49,141 @@ struct DeliveryTests { #expect(harness.server.requests.map(\.types) == [["Kept", "Broken"], ["Next"]]) } + @Test("Only Kestrel's own answer ends a batch; any other 2xx is retried", arguments: [ + Reply.status(200, body: #"{"accepted":1}"#), + .status(200), + .status(204), + .status(202, body: ""), + .status(202, body: "not json"), + .status(202, body: "Welcome to the hotel wifi"), + .status(202, body: #"{"dropped":1}"#), + .status(202, body: #"{"accepted":"1"}"#), + .status(202, body: "[]"), + ]) + func notKestrelsAnswer(reply: Reply) async { + let harness = Harness() + harness.server.script([reply]) + let client = harness.client() + client.signal("A") + await client.flush() + #expect(harness.queuedLines == 1) + + harness.clock.advance(by: 31) + await client.flush() + #expect(harness.server.requests.map(\.types) == [["A"], ["A"]]) + #expect(harness.queuedLines == 0) + } + + @Test("405, 407 and 410 keep the batch and retry with backoff", arguments: [405, 407, 410]) + func notVerdicts(status: Int) async { + let harness = Harness() + harness.server.script([.status(status)]) + let client = harness.client() + client.signal("A") + await client.flush() + #expect(harness.queuedLines == 1) + + harness.clock.advance(by: 29) + await client.flush() + #expect(harness.server.requests.count == 1) + + harness.clock.advance(by: 2) + await client.flush() + #expect(harness.server.requests.map(\.types) == [["A"], ["A"]]) + #expect(harness.queuedLines == 0) + } + + // Only on Apple platforms: swift-corelibs-foundation can't redirect from a mock URLProtocol. + #if canImport(Darwin) + @Test("A redirect is not followed and the batch is kept") + func redirect() async { + let target = MockServer() + let harness = Harness() + harness.server.respond { _ in .redirect(to: "https://\(target.host)/ingest/v1/signals") } + let client = harness.client() + client.signal("A") + let finished = Locked(false) + Task { + await client.flush() + finished.withLock { $0 = true } + } + #expect(await eventually { finished.withLock { $0 } }) + #expect(target.requests.isEmpty) + #expect(harness.queuedLines == 1) + } + #endif + + @Test("An answer bigger than 64 KB is not Kestrel's; the batch is kept") + func hugeAnswer() async { + let harness = Harness() + harness.server.script([.status(202, body: #"{"accepted":1}"# + String(repeating: " ", count: 100_000))]) + let client = harness.client() + client.signal("A") + await client.flush() + #expect(harness.queuedLines == 1) + + harness.clock.advance(by: 31) + await client.flush() + #expect(harness.server.requests.count == 2) + #expect(harness.queuedLines == 0) + } + + @Test("Opting out cancels the upload in flight and nothing more is sent") + func optOutDuringUpload() async { + let harness = Harness() + harness.server.respond { _ in .hang } + let client = harness.client() + for n in 0..<150 { client.signal("S\(n)") } + let finished = Locked(false) + Task { + await client.flush() + finished.withLock { $0 = true } + } + #expect(await eventually { harness.server.requests.count == 1 }) + + harness.store.set(false, forKey: SettingsKey.isEnabled) + client.optOut() + // The hanging request is cancelled at once (its timeout is 30 s) and the flush ends + // without trying the second batch. + #expect(await eventually(timeout: 5) { finished.withLock { $0 } }) + await client.drain() + #expect(harness.server.requests.count == 1) + #expect(harness.queuedLines == 0) + + // Back on: sending works again. + harness.store.set(true, forKey: SettingsKey.isEnabled) + harness.server.respond { _ in .accepted } + client.signal("After") + await client.flush() + #expect(harness.server.requests.last?.types == ["After"]) + } + + @Test("A full command stream drops the oldest commands, lets their waiters go and keeps an opt-out") + func fullCommandStream() async { + let (stream, continuation) = AsyncStream.makeStream(of: Command.self, bufferingPolicy: .bufferingNewest(3)) + let time = Date() + // Would wait forever if the barrier were dropped without being let go. + await withCheckedContinuation { done in + Client.enqueue(.barrier(done), on: continuation) + Client.enqueue(.optOut, on: continuation) + for n in 0..<3 { + Client.enqueue(.signal(type: "S\(n)", payload: [:], time: time), on: continuation) + } + } + continuation.finish() + var kept: [String] = [] + for await command in stream { + switch command { + case let .signal(type, _, _): kept.append(type) + case .optOut: kept.append("optOut") + default: kept.append("other") + } + } + #expect(kept.count == 3) + #expect(kept.contains("optOut")) + #expect(kept.contains("S2")) + } + @Test("The dropped count is read from the 202 body when it is there") func droppedCount() { #expect(UploadOutcome.droppedCount(in: Data(#"{"accepted":8,"dropped":2}"#.utf8)) == 2) @@ -77,7 +210,7 @@ struct DeliveryTests { let harness = Harness() let client = harness.client() var payload: [String: String] = [:] - for key in 0..<20 { payload["key\(key)"] = String(repeating: "abcdefghij", count: 20) } + for key in 0..<20 { payload["item\(key)"] = String(repeating: "abcdefghij", count: 20) } for _ in 0..<80 { client.signal("Big.signal", payload: payload) } await client.flush() @@ -101,7 +234,7 @@ struct DeliveryTests { #expect(harness.queuedLines == 0) } - @Test("401, 403 and 404 stop sending until the next launch", arguments: [401, 403, 404]) + @Test("401, 403 and 404 pause sending for an hour", arguments: [401, 403, 404]) func unavailable(status: Int) async { let harness = Harness() harness.server.script([.status(status)]) @@ -109,18 +242,32 @@ struct DeliveryTests { client.signal("A") await client.flush() client.signal("B") - harness.clock.advance(by: 7200) + harness.clock.advance(by: 3599) await client.flush() #expect(harness.server.requests.count == 1) await client.drain() #expect(harness.queuedLines == 2) + + // An hour later it tries again, without a relaunch, and sends everything that was kept. + harness.clock.advance(by: 2) + await client.flush() + #expect(harness.server.requests.count == 2) + #expect(harness.server.requests.last?.types == ["A", "B"]) + #expect(harness.queuedLines == 0) + } + + @Test("A relaunch tries again at once after 401, 403 and 404", arguments: [401, 403, 404]) + func unavailableThenRelaunch(status: Int) async { + let harness = Harness() + harness.server.script([.status(status)]) + let client = harness.client() + client.signal("A") + await client.flush() client.shutdown() - // The next launch tries again and sends everything that was kept. let relaunched = harness.client() await relaunched.flush() - #expect(harness.server.requests.count == 2) - #expect(harness.server.requests.last?.types == ["A", "B"]) + #expect(harness.server.requests.map(\.types) == [["A"], ["A"]]) } @Test("413 splits the batch until the halves fit") @@ -246,7 +393,14 @@ struct DeliveryTests { @Test("Response codes map to the protocol's actions") func classification() { - #expect(UploadOutcome.classify(status: 202, retryAfter: nil) == .accepted) + let answer = Data(#"{"accepted":3}"#.utf8) + #expect(UploadOutcome.classify(status: 202, retryAfter: nil, body: answer) == .accepted) + #expect(UploadOutcome.classify(status: 202, retryAfter: nil) == .failed) + #expect(UploadOutcome.classify(status: 200, retryAfter: nil, body: answer) == .failed) + #expect(UploadOutcome.classify(status: 301, retryAfter: nil) == .failed) + #expect(UploadOutcome.classify(status: 405, retryAfter: nil) == .failed) + #expect(UploadOutcome.classify(status: 407, retryAfter: nil) == .failed) + #expect(UploadOutcome.classify(status: 410, retryAfter: nil) == .failed) #expect(UploadOutcome.classify(status: 400, retryAfter: nil) == .rejected) #expect(UploadOutcome.classify(status: 422, retryAfter: nil) == .rejected) #expect(UploadOutcome.classify(status: 401, retryAfter: nil) == .unavailable) diff --git a/Tests/KestrelKitTests/FacadeTests.swift b/Tests/KestrelKitTests/FacadeTests.swift index 6bac81f..117ae42 100644 --- a/Tests/KestrelKitTests/FacadeTests.swift +++ b/Tests/KestrelKitTests/FacadeTests.swift @@ -39,6 +39,35 @@ struct FacadeTests { #expect(signals[2].payload == ["pages": "12", "pdf": "true", "ratio": "0.5", "name": "report"]) } + @Test("Plain http is refused except for localhost, and in debug builds") + func endpoints() { + func allowed(_ url: String, insecure: Bool) -> Bool { + Kestrel.isAllowedEndpoint(URL(string: url)!, allowsInsecure: insecure) + } + for insecure in [false, true] { + #expect(allowed("https://kestrel.example.com", insecure: insecure)) + #expect(allowed("HTTPS://kestrel.example.com:8443/base", insecure: insecure)) + #expect(allowed("http://localhost:3001", insecure: insecure)) + #expect(allowed("http://127.0.0.1:3001", insecure: insecure)) + #expect(allowed("http://[::1]:3001", insecure: insecure)) + #expect(!allowed("ftp://kestrel.example.com", insecure: insecure)) + #expect(!allowed("kestrel.example.com", insecure: insecure)) + } + #expect(!allowed("http://kestrel.example.com", insecure: false)) + #expect(!allowed("http://10.0.3.26:3001", insecure: false)) + #expect(!allowed("http://localhost.evil.example", insecure: false)) + #expect(allowed("http://10.0.3.26:3001", insecure: true)) + } + + @Test("configure refuses an address it won't send to") + func refusedEndpoint() async { + Kestrel.configure(url: URL(string: "ftp://kestrel.example.com")!, key: TestKey.value) + #expect(Kestrel.client == nil) + Kestrel.signal("A") + await Kestrel.flush() + #expect(!FileManager.default.fileExists(atPath: harness.directory.path)) + } + @Test("A second configure is ignored") func configureOnce() async { let other = MockServer() diff --git a/Tests/KestrelKitTests/Helpers.swift b/Tests/KestrelKitTests/Helpers.swift index 5b25671..f9da62a 100644 --- a/Tests/KestrelKitTests/Helpers.swift +++ b/Tests/KestrelKitTests/Helpers.swift @@ -64,6 +64,10 @@ enum Reply: Sendable { /// `body` is sent as given; without one, a 202 says `{"accepted":1}` and others say nothing. case status(Int, headers: [String: String] = [:], body: String? = nil) case networkError + /// Never answers; the request stays in flight until it is cancelled. + case hang + /// A 302 to `location`, which KestrelKit must not follow. + case redirect(to: String) static let accepted = Reply.status(202) } @@ -138,6 +142,21 @@ final class MockURLProtocol: URLProtocol { client?.urlProtocolDidFinishLoading(self) case .networkError: client?.urlProtocol(self, didFailWithError: URLError(.notConnectedToInternet)) + case .hang: + break + case let .redirect(location): + let response = HTTPURLResponse(url: url, statusCode: 302, httpVersion: "HTTP/1.1", headerFields: ["Location": location])! + #if canImport(Darwin) + client?.urlProtocol(self, wasRedirectedTo: URLRequest(url: URL(string: location)!), redirectResponse: response) + // When the delegate refuses to follow, the 302 itself is the answer. (A followed + // redirect would stop this load and start a new one on the target.) + client?.urlProtocol(self, didReceive: response, cacheStoragePolicy: .notAllowed) + client?.urlProtocolDidFinishLoading(self) + #else + // swift-corelibs-foundation traps on redirects from a custom URLProtocol. + _ = response + client?.urlProtocol(self, didFailWithError: URLError(.badServerResponse)) + #endif } } @@ -158,7 +177,7 @@ final class MockURLProtocol: URLProtocol { func mockSession() -> URLSession { let configuration = URLSessionConfiguration.ephemeral configuration.protocolClasses = [MockURLProtocol.self] - return URLSession(configuration: configuration) + return TransportDelegate.makeSession(configuration) } // MARK: - Clients diff --git a/Tests/KestrelKitTests/LimitsTests.swift b/Tests/KestrelKitTests/LimitsTests.swift index 3d87af0..1aed543 100644 --- a/Tests/KestrelKitTests/LimitsTests.swift +++ b/Tests/KestrelKitTests/LimitsTests.swift @@ -89,6 +89,109 @@ struct LimitsTests { #expect(payload == ["email": "[Filtered]", "plan": "pro"]) } + @Test("Control characters can't split an address past the scrubbing", arguments: [ + ("john@exa\tmple.com", "[Filtered]"), + ("john@example.com\n", "[Filtered]"), + ("jo\u{0}hn@exam\u{7}ple.com", "[Filtered]"), + ("4111 1111 11\n11 1111", "[Filtered]"), + ("41\t11 1111 1111 1111", "[Filtered]"), + ("ip 203.0.\r113.7", "ip [Filtered]"), + ("Bearer\u{0} abc.def", "Bearer [Filtered]"), + ("tok\u{1}en=abc", "token=[Filtered]"), + ]) + func controlCharacters(input: String, expected: String) { + #expect(Limits.payload(["note": input]) == ["note": expected]) + } + + @Test("Card numbers grouped with dots or non-breaking spaces are scrubbed", arguments: [ + "4111.1111.1111.1111", + "4111\u{00A0}1111\u{00A0}1111\u{00A0}1111", + "4111\u{202F}1111\u{202F}1111\u{202F}1111", + "4111-1111.1111 1111", + ]) + func cardSeparators(input: String) { + #expect(Scrubber.scrub(input) == "[Filtered]") + #expect(Scrubber.scrub("paid with \(input) today") == "paid with [Filtered] today") + } + + @Test("Shortened IPv6 addresses are replaced", arguments: [ + ("2001:db8::1", "[Filtered]"), + ("::1", "[Filtered]"), + ("fe80::1", "[Filtered]"), + ("FE80::202:B3FF:FE1E:8329", "[Filtered]"), + ("2001:db8:0:0:1::", "[Filtered]"), + ("from 2001:db8::8a2e:370:7334, then", "from [Filtered], then"), + ("/ip/::1/x", "/ip/[Filtered]/x"), + ("[2001:db8::1]:8080", "[[Filtered]]:8080"), + ]) + func shortenedIPv6(input: String, expected: String) { + #expect(Scrubber.scrub(input) == expected) + } + + @Test("Things that only look a bit like a shortened IPv6 address are kept", arguments: [ + "::", + ":::1", + "1::2::3", + "std::fmt::Display", + "Namespace::Item", + "12:30:45", + "a:b", + "db8::12345", + ]) + func notShortenedIPv6(input: String) { + #expect(Scrubber.scrub(input) == input) + } + + @Test("Keys that look like they hold a secret are dropped with their values", arguments: [ + "password", "Password", "passwd", "user_pass", "secret", "client_secret", "token", "apiToken", + "auth", "Authorization", "key", "api_key", "apiKey", "card", "cardNumber", "cvv", "CVV2", + "ssn", "SSN", "user_ssn", "ssn-last4", "cookie", "session_cookie", + ]) + func sensitiveKeys(key: String) { + #expect(Scrubber.isSensitiveKey(key), "\(key)") + #expect(Limits.payload([key: "hunter2", "plan": "pro"]) == ["plan": "pro"]) + } + + @Test("The default payload's secret keys are dropped too") + func sensitiveDefaultKeys() { + #expect(Limits.payload(["plan": "pro"], defaults: ["api_token": "abc", "channel": "beta"]) == ["plan": "pro", "channel": "beta"]) + } + + @Test("Ordinary keys are kept", arguments: ["plan", "className", "classname", "screen", "count", "name", "email", "item00", "\u{212A}ey"]) + func ordinaryKeys(key: String) { + #expect(!Scrubber.isSensitiveKey(key), "\(key)") + #expect(Limits.payload([key: "x"]) == [key: "x"]) + } + + @Test("A huge value is cut before it is scrubbed and scrubs fast", arguments: [ + String(repeating: "a", count: 20_000), + String(repeating: "a.", count: 10_000), + String(repeating: "a@", count: 10_000), + String(repeating: "1 ", count: 10_000), + String(repeating: "token", count: 4_000), + String(repeating: "pass-", count: 4_000), + String(repeating: "1:", count: 10_000), + "x@" + String(repeating: "a.", count: 10_000), + String(repeating: "a", count: 20_000) + "@example.com", + ]) + func hugeValues(value: String) { + let clock = ContinuousClock() + let elapsed = clock.measure { + let payload = Limits.payload(["note": value]) + #expect((payload["note"]?.unicodeScalars.count ?? 0) <= Limits.maxValueLength) + } + #expect(elapsed < .seconds(2), "took \(elapsed)") + } + + @Test("A value is cut to 2,000 characters before scrubbing, then to 200") + func rawCap() { + let payload = Limits.payload(["note": String(repeating: "x", count: 2_100) + "jane@example.com"]) + #expect(payload["note"] == String(repeating: "x", count: 200)) + // An address in the first 200 characters is still found when the value is long. + let early = Limits.payload(["note": "jane@example.com " + String(repeating: "x", count: 5_000)]) + #expect(early["note"] == "[Filtered] " + String(repeating: "x", count: 189)) + } + @Test("Luhn") func luhn() { #expect(Scrubber.luhn("4111111111111111")) @@ -111,13 +214,11 @@ struct LimitsTests { #expect(Scrubber.scrub(input) == expected) } - @Test("Shortened IPv6 and things that only look a bit like addresses are kept", arguments: [ + @Test("Things that only look a bit like addresses are kept", arguments: [ "1.4.0", "/v/1.2.3", "12:30", "12:30:45", - "::1", - "fe80::1", ]) func notIPAddresses(input: String) { #expect(Scrubber.scrub(input) == input) @@ -170,6 +271,8 @@ struct LimitsTests { let payload = Limits.payload([ "email": "jane.doe@example.com", "phone": "+47 912 34 567", + "payment": "4111 1111 1111 1111", + "header": "Authorization: Bearer abc.def-123", "card": "4111 1111 1111 1111", "auth": "Authorization: Bearer abc.def-123", "note": "order 123456789", @@ -183,8 +286,9 @@ struct LimitsTests { ]) #expect(payload["email"] == "[Filtered]") #expect(payload["phone"] == "[Filtered]") - #expect(payload["card"] == "[Filtered]") - #expect(payload["auth"] == "Authorization: Bearer [Filtered]") + #expect(payload["payment"] == "[Filtered]") + #expect(payload["header"] == "Authorization: Bearer [Filtered]") + #expect(payload["card"] == nil && payload["auth"] == nil) #expect(payload["note"] == "order 123456789") #expect(payload["pages"] == "12" && payload["pdf"] == "true") #expect(payload["ip"] == nil && payload["user_agent"] == nil) @@ -197,10 +301,10 @@ struct LimitsTests { @Test("Dropped keys don't use up the 20, and are cleaned before the check") func droppedKeysInPayloads() { var own: [String: String] = [" IP\u{0007} ": "203.0.113.7", "User-Agent": "Mozilla/5.0"] - for i in 0..<20 { own["key\(i < 10 ? "0" : "")\(i)"] = "\(i)" } + for i in 0..<20 { own["item\(i < 10 ? "0" : "")\(i)"] = "\(i)" } let payload = Limits.payload(own, defaults: ["x-forwarded-for": "10.0.0.1"]) #expect(payload.count == 20) - #expect(payload.keys.allSatisfy { $0.hasPrefix("key") }) + #expect(payload.keys.allSatisfy { $0.hasPrefix("item") }) } @Test("Every type KestrelKit sends is one the server accepts as it is", arguments: [ diff --git a/Tests/KestrelKitTests/QueueTests.swift b/Tests/KestrelKitTests/QueueTests.swift index ff6326b..179178e 100644 --- a/Tests/KestrelKitTests/QueueTests.swift +++ b/Tests/KestrelKitTests/QueueTests.swift @@ -44,6 +44,35 @@ struct QueueTests { #expect(DiskQueue(directory: directory, maxCount: 3).lines == [line(2), line(3), line(4)]) } + @Test("A full queue appends to the file and rewrites it only now and then") + func fullQueueRewritesRarely() throws { + func fileLines(_ queue: DiskQueue) throws -> Int { + try Data(contentsOf: queue.fileURL).split(separator: UInt8(ascii: "\n")).count + } + var queue = DiskQueue(directory: directory, maxCount: 3, compactAfter: 4) + for n in 0..<5 { queue.append(line(n)) } + #expect(queue.lines == [line(2), line(3), line(4)]) + // Two signals were dropped from memory, but the file still has them: no rewrite yet. + #expect(try fileLines(queue) == 5) + + queue.append(line(5)) + #expect(try fileLines(queue) == 6) + // The fourth dropped signal compacts the file. + queue.append(line(6)) + #expect(queue.lines == [line(4), line(5), line(6)]) + #expect(try fileLines(queue) == 3) + #expect(DiskQueue(directory: directory, maxCount: 3).lines == [line(4), line(5), line(6)]) + } + + @Test("Many signals into a full queue keep the file small") + func fullQueueStaysBounded() throws { + var queue = DiskQueue(directory: directory, maxCount: 100) + for n in 0..<1_000 { queue.append(line(n)) } + let fileLines = try Data(contentsOf: queue.fileURL).split(separator: UInt8(ascii: "\n")).count + #expect(fileLines <= 100 + queue.compactAfter) + #expect(DiskQueue(directory: directory, maxCount: 100).lines == (900..<1_000).map(line)) + } + @Test("A smaller cap on the next launch trims the file") func capOnLoad() { var queue = DiskQueue(directory: directory, maxCount: 10) diff --git a/docs/protocol.md b/docs/protocol.md index 2f112bc..2ab1e42 100644 --- a/docs/protocol.md +++ b/docs/protocol.md @@ -25,6 +25,10 @@ POST {baseURL}/ingest/v1/signals No cookies are sent or accepted (P12). Apps send no `Origin` header. **Server:** if a request has one, it must be one of the key's allowed origins, or the answer is `403`. +SDKs send only over `https://`. KestrelKit accepts `http://` for `localhost` and, in debug builds, +for any host; it never follows a redirect, reads at most 64 KB of an answer, and gives a request +60 seconds at most. + KestrelKit compresses bodies of 1 KB or more with gzip, and sends the compressed form only when it is smaller (P3). It uses no other encoding. @@ -79,8 +83,8 @@ sends: is sent as `Paywall_shown`), then cuts it to 100 characters. An empty type isn't sent. This matters, because the server drops a signal whose type has other characters (see below). - It converts numbers and booleans in payloads to strings (`12`, `true`), puts the signal's own keys - before the default payload's, and drops keys that name an IP address or a user agent (see - [Scrubbing](#scrubbing-p13)). + before the default payload's, and drops keys that name an IP address or a user agent, or look + like they hold a secret (see [Scrubbing](#scrubbing-p13)). - It removes control characters from the other strings, trims them and cuts them to 200. ### Lenient per signal, strict per batch (P5) @@ -101,7 +105,7 @@ has more than 100 signals or is too large. Otherwise it fixes or drops each sign | has numbers or booleans as payload values | stores them as strings (`12`, `true`). | | has objects, arrays or `null` as payload values | drops those keys. | | has more than 20 payload keys | keeps the first 20 it reads. | -| has payload keys over 40 characters | cuts them to 40. Empty keys and keys that name an IP address or a user agent are dropped. | +| has payload keys over 40 characters | cuts them to 40. Empty keys and keys that name an IP address or a user agent are dropped. A key that looks like it holds a secret is dropped too (see [Scrubbing](#scrubbing-p13)). | The `202` answer counts the signals that were stored, and says how many were dropped when some were (see [Responses](#responses)). @@ -120,21 +124,25 @@ project says otherwise) are dropped. Both sides replace payload values that look like personal data with `[Filtered]`, with the same rules (Keen Kestrel's `src/analytics/clean.rs` and `src/event/scrub.rs`, KestrelKit's -`Scrubber.swift`), in this order: +`Scrubber.swift`). Before the values are scrubbed, both remove control characters (so +`john@example.com` can't slip past the patterns) and, in KestrelKit, cut the value to 2,000 +characters (it is cut to 200 afterwards). Then, in this order: 1. Card numbers: 13–19 digits, optionally grouped by single spaces or dashes, that pass the Luhn - check. + check. KestrelKit also accepts dots and non-breaking spaces (U+00A0, U+2007, U+202F) as separators; + the server doesn't yet. 2. `Bearer …` and `Basic …` credentials. The scheme stays: `Bearer [Filtered]`. 3. `name=value` pairs whose name contains `pass`, `secret`, `token`, `auth`, `key`, `card` or `cookie`: `token=abc` becomes `token=[Filtered]`. 4. Email addresses. 5. IP addresses: IPv4, and IPv6 written out with at least four groups (`2001:db8:0:0:1:0:0:1`). - Shortened IPv6 with `::` isn't recognised. + KestrelKit also recognises IPv6 shortened with one `::` (`2001:db8::1`, `::1`, `fe80::1`); the + server doesn't yet. 6. Phone numbers: 9–15 digits that start with `+` or `00`, or are written in at least three groups. Plain digit runs (order numbers, ids), dates and versions are kept. -Payload keys that name an IP address or a user agent are dropped with their values. The keys, -compared ignoring ASCII case, are: +Payload keys that name an IP address or a user agent, or look like they hold a secret, are dropped +with their values. The IP address and user agent keys, compared ignoring ASCII case, are: - `ip`, `ip_address`, `client_ip`, `remote_addr`; - the forwarding headers `x-forwarded-for`, `x-real-ip`, `forwarded`, `cf-connecting-ip`, @@ -143,6 +151,12 @@ compared ignoring ASCII case, are: `http_true_client_ip`, `http_x_client_ip`; - `user-agent`, `user_agent`, `http_user_agent`. +A key looks like it holds a secret when, ignoring ASCII case, it contains `pass`, `secret`, `token`, +`auth`, `key`, `card`, `cookie` or `cvv` (so `password`, `apiKey`, `cardNumber` and `Authorization`, +but also `keyboard_layout` and `author`), or has `ssn` as a word of its own (`ssn`, `user_ssn`; not +`classname`). Both sides drop such keys (the server's own list for web event properties is the same +without `cvv` and `ssn`). + KestrelKit scrubs on the device, before a signal is queued. The server scrubs again, so older or other SDKs are covered too. Scrubbing is a safety net: apps should keep personal data out of payloads. @@ -152,20 +166,22 @@ payloads. | Status | Meaning | What KestrelKit does | |---|---|---| | `202 {"accepted": n}` | Done: `n` signals were stored. When some were dropped (P5), the body also has `"dropped": m`. | Forgets the batch. Logs `dropped` in debug builds; never resends. | -| other `2xx` | Done. | Forgets the batch. | +| other `2xx`, or a `202` without `"accepted"` | Not Kestrel's answer: a captive portal, a proxy. | Keeps the batch and retries with backoff. | +| `3xx` | A redirect. SDKs don't follow redirects. | Keeps the batch and retries with backoff. | | `400` | The batch is malformed: not JSON, no `signals` array, or an unknown or broken `Content-Encoding`. | Drops it and doesn't retry. | -| `401` | No key, or an unknown one. | Stops sending until the next launch and keeps the queue. Logs in debug builds only. | +| `401` | No key, or an unknown one. | Pauses sending for an hour and keeps the queue. Logs in debug builds only. | | `403` | The key is disabled, app analytics is off for the project, or the `Origin` header isn't allowed. | As `401`. | | `404` | The server has no app-signal ingest (an older Kestrel). | As `401`, so the signals arrive once the server is upgraded (P7). | -| `408` | Request timeout. | Retries like a `5xx` (P7). | +| `405`, `407`, `408`, `410` | Not verdicts on the batch: a proxy, a gateway or a server that moved. | Retries like a `5xx` (P7). | | `413` | More than 100 signals, or more than 256 KB, compressed or not. | Splits the batch and retries the halves; drops a single signal that is still too large. | | `429` | A rate limit: the server's per-IP limit or the key's `rate_limit_per_minute`. Both count requests, not signals. | Waits `Retry-After` seconds. | | `503` | The server's write queue is full, or it briefly can't take signals. | Waits `Retry-After` seconds. | | other `4xx` | | As `400` (P7). | | other `5xx`, network error | Something went wrong. | Keeps the batch and retries with backoff. | -SDKs treat any `2xx` as done, whatever `n` and `m` are, so they never resend a batch the server has -seen (P5, P7). Error responses have a `{"detail": "…"}` body meant for people; SDKs don't need to +SDKs treat a `202` with an `accepted` count as done, whatever `n` and `m` are, so they never resend a +batch the server has seen (P5, P7). Any other answer to a successful request can't be Kestrel's, and +the batch is kept. Error responses have a `{"detail": "…"}` body meant for people; SDKs don't need to read it. All answers carry `Cache-Control: no-store`. **Retry-After (P8).** `429` and `503` carry `Retry-After` in seconds. KestrelKit reads it as a number