From e24b326e7e2e3a458a63d705aa6c247be7a51316 Mon Sep 17 00:00:00 2001 From: Michael Yeack Date: Fri, 29 May 2026 19:39:42 -0700 Subject: [PATCH] Move iMessage sync into BlackbookServer (un-sandboxed) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit iMessage Sync in the sandboxed main macOS app could never read ~/Library/Messages/chat.db: NSHomeDirectory() resolved to the sandbox container, and the App Sandbox blocks the Messages DB regardless of Full Disk Access. De-sandboxing the main app isn't an option — App Store Connect rejects non-sandboxed macOS builds (ITMS-90296), which would break the TestFlight pipeline. BlackbookServer is already un-sandboxed, always-running, owns the master SwiftData store, and syncs to all devices, so the iMessage reader belongs there. Interactions it creates reach iOS + macOS via /sync/changes (the pull query serves by updatedAt). It also works when the Mac app is closed. Main app: - Restore sandboxed entitlements / project.yml (revert exploratory de-sandbox); macOS build re-embeds com.apple.security.app-sandbox. - Remove the dead main-app IMessageSyncService, its BlackbookApp wiring, the SettingsView section, and the unused AppConstants.IMessageSync enum. BlackbookServer: - New IMessageSyncService: polls chat.db, matches contacts by phone AND email (handles routed to Apple-ID emails now match), 30-day backfill, (contactId, second, summary) dedup guard, and an unmatched-handles diagnostic. Path uses NSUserName() (sandbox-independent). - Wire into ServerStatusModel (starts if enabled) and add the controls to the menu-bar popover: Log iMessages toggle, Sync Last 30 Days, Open Full Disk Access, Unmatched handles. Requires granting Full Disk Access to "Blackbook Server" (not "Blackbook"). BlackbookServer is not in TestFlight; rebuild/reinstall locally per docs/test-scenarios/TEST_SCENARIOS.md. Co-Authored-By: Claude Opus 4.8 --- Blackbook.xcodeproj/project.pbxproj | 8 +- Blackbook/App/BlackbookApp.swift | 9 - Blackbook/Services/IMessageSyncService.swift | 270 ------------ Blackbook/Utilities/Constants.swift | 7 - Blackbook/Views/Settings/SettingsView.swift | 70 --- BlackbookServer/App/IMessageSyncService.swift | 405 ++++++++++++++++++ BlackbookServer/App/ServerMenuView.swift | 82 ++++ BlackbookServer/App/ServerStatusModel.swift | 8 + docs/test-scenarios/TEST_SCENARIOS.md | 54 +++ 9 files changed, 553 insertions(+), 360 deletions(-) delete mode 100644 Blackbook/Services/IMessageSyncService.swift create mode 100644 BlackbookServer/App/IMessageSyncService.swift diff --git a/Blackbook.xcodeproj/project.pbxproj b/Blackbook.xcodeproj/project.pbxproj index 76d5d98..6946edb 100644 --- a/Blackbook.xcodeproj/project.pbxproj +++ b/Blackbook.xcodeproj/project.pbxproj @@ -138,7 +138,6 @@ E93AF93D1470F43091DF3613 /* AddNoteView.swift in Sources */ = {isa = PBXBuildFile; fileRef = B0BBA6EAAA528870946A9A68 /* AddNoteView.swift */; }; E99E54A243049CBE59F488DD /* MoreView.swift in Sources */ = {isa = PBXBuildFile; fileRef = 9070FA35720DE1331E514BFF /* MoreView.swift */; }; E9C12D0FA76386A83E75A5A7 /* BackupMetadata.swift in Sources */ = {isa = PBXBuildFile; fileRef = 8C6A1F78A9CDFC1FC5ADC17B /* BackupMetadata.swift */; }; - EA77718EDE2F2C9BAFD02C27 /* IMessageSyncService.swift in Sources */ = {isa = PBXBuildFile; fileRef = 51B78E4C7B87A207BBC1A65A /* IMessageSyncService.swift */; }; EC143B5E71184A068145E022 /* CalendarPickerView.swift in Sources */ = {isa = PBXBuildFile; fileRef = BBD49AF1F502FF3C26EC5CF1 /* CalendarPickerView.swift */; }; EFC657A3B7FB38203943BAF7 /* RelationshipScoreEngine.swift in Sources */ = {isa = PBXBuildFile; fileRef = 56EBAC598F8585225B1F09E4 /* RelationshipScoreEngine.swift */; }; F0207B4239B9859DC6331039 /* LocationListView.swift in Sources */ = {isa = PBXBuildFile; fileRef = 1837970957580D296D714FDF /* LocationListView.swift */; }; @@ -147,6 +146,7 @@ F90B2184862AC79EBA241556 /* Contact.swift in Sources */ = {isa = PBXBuildFile; fileRef = 8222BEE7020B6711480F644E /* Contact.swift */; }; F92545D544AD003815547A34 /* DeviceIdentity.swift in Sources */ = {isa = PBXBuildFile; fileRef = 535A0087C38A75EF251D28DF /* DeviceIdentity.swift */; }; F9C8E6C6BBBDE1468720591E /* ContactListViewModelTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = ECBCBC491C2BFAA0AABA68B3 /* ContactListViewModelTests.swift */; }; + FA0527F27C39C281C7287EBE /* IMessageSyncService.swift in Sources */ = {isa = PBXBuildFile; fileRef = 530F5990544546A519FE2390 /* IMessageSyncService.swift */; }; FC428EE351F1C7F125A3E819 /* AIInsightsView.swift in Sources */ = {isa = PBXBuildFile; fileRef = 4C58C1200FB5DE674799040E /* AIInsightsView.swift */; }; FC4D35EB1F0F93963E0499E2 /* DateHelpersTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = B242CFE0D00227BB630A5801 /* DateHelpersTests.swift */; }; FD356F0646511FD58F59CCEF /* RelationshipScoreEngineTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = AB7B07D8F95B7197892AB8D6 /* RelationshipScoreEngineTests.swift */; }; @@ -207,7 +207,7 @@ 4C58C1200FB5DE674799040E /* AIInsightsView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = AIInsightsView.swift; sourceTree = ""; }; 4DCA116B785721C8942FD325 /* FeatureGatingTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = FeatureGatingTests.swift; sourceTree = ""; }; 4E45E53C6F0C7F7A6808F6EE /* LocalSyncServer.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = LocalSyncServer.swift; sourceTree = ""; }; - 51B78E4C7B87A207BBC1A65A /* IMessageSyncService.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = IMessageSyncService.swift; sourceTree = ""; }; + 530F5990544546A519FE2390 /* IMessageSyncService.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = IMessageSyncService.swift; sourceTree = ""; }; 535A0087C38A75EF251D28DF /* DeviceIdentity.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = DeviceIdentity.swift; sourceTree = ""; }; 56EBAC598F8585225B1F09E4 /* RelationshipScoreEngine.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = RelationshipScoreEngine.swift; sourceTree = ""; }; 59D5A4C8F3FE7AFE76757690 /* Group.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = Group.swift; sourceTree = ""; }; @@ -343,6 +343,7 @@ 8C6A1F78A9CDFC1FC5ADC17B /* BackupMetadata.swift */, 208C79A4DEA3EB7672FDF510 /* BackupServer.swift */, F49097698C68BE415965A964 /* BlackbookServerApp.swift */, + 530F5990544546A519FE2390 /* IMessageSyncService.swift */, 107F0516F43606F715A04CAC /* ServerMenuView.swift */, 4AB554426E75CDDDB20498E2 /* ServerModelContainer.swift */, 9B9D910910FBACB1A4CF56B3 /* ServerStatusModel.swift */, @@ -428,7 +429,6 @@ 67485481A0ED6072989C3A0D /* ContactMergeService.swift */, DC159C3515CC1B41D178A551 /* ContactSyncService.swift */, 2382E25FE6BDF339DA44560D /* GoogleCalendarService.swift */, - 51B78E4C7B87A207BBC1A65A /* IMessageSyncService.swift */, D876B79CE3D85BAABBEBD128 /* LocalServerSyncService.swift */, 4E45E53C6F0C7F7A6808F6EE /* LocalSyncServer.swift */, 682862BCC7F0F257DF35CEB8 /* NetworkGraphEngine.swift */, @@ -821,7 +821,6 @@ 460F7B5C6218C225C8C7B812 /* GroupIconSuggestionView.swift in Sources */, CAD3A03B2BCDC59E1693ADC5 /* GroupListView.swift in Sources */, 9EF211A7D945367755381495 /* GroupManagerView.swift in Sources */, - EA77718EDE2F2C9BAFD02C27 /* IMessageSyncService.swift in Sources */, 5D10E3A07EE83D1C213B9CBD /* IconAndColorPickers.swift in Sources */, DF0FF2772A83C8B373CB36BD /* Interaction.swift in Sources */, 047D658F5CE553E21B993DF3 /* InteractionLogView.swift in Sources */, @@ -916,6 +915,7 @@ 25F4C43A32CF59B6B592D404 /* DateHelpers.swift in Sources */, 3F759DE748EA4246FBBBA456 /* DeviceIdentity.swift in Sources */, 1D1FA4DA1805F09735832F43 /* Group.swift in Sources */, + FA0527F27C39C281C7287EBE /* IMessageSyncService.swift in Sources */, 13B68B5D4ABD5BDC2C1AC8B7 /* Interaction.swift in Sources */, 113034766F531D2FF71921B2 /* LocalSyncProtocol.swift in Sources */, 27956EEE4BB572147FBD8494 /* Location.swift in Sources */, diff --git a/Blackbook/App/BlackbookApp.swift b/Blackbook/App/BlackbookApp.swift index e14a612..d38476f 100644 --- a/Blackbook/App/BlackbookApp.swift +++ b/Blackbook/App/BlackbookApp.swift @@ -16,9 +16,6 @@ struct BlackbookApp: App { let modelContainer: ModelContainer @State private var authService = AuthenticationService() @State private var googleCalendarService = GoogleCalendarService() - #if os(macOS) - @State private var iMessageService = IMessageSyncService() - #endif init() { UserActionLogger.shared.setUserEmail(UserDefaults.standard.string(forKey: "auth.userEmail")) @@ -68,12 +65,6 @@ struct BlackbookApp: App { AuthGateView() .environment(authService) .environment(googleCalendarService) - #if os(macOS) - .environment(iMessageService) - .onAppear { - iMessageService.startIfEnabled(with: modelContainer.mainContext) - } - #endif } .modelContainer(modelContainer) } diff --git a/Blackbook/Services/IMessageSyncService.swift b/Blackbook/Services/IMessageSyncService.swift deleted file mode 100644 index f1dc477..0000000 --- a/Blackbook/Services/IMessageSyncService.swift +++ /dev/null @@ -1,270 +0,0 @@ -#if os(macOS) -import Foundation -import Observation -import SwiftData -import SQLite3 -import os - -private let logger = Logger(subsystem: "com.blackbookdevelopment.app", category: "IMessageSync") - -/// Continuously polls the local iMessage database (chat.db) and creates -/// Interaction records for messages sent to or received from Blackbook contacts. -@Observable -final class IMessageSyncService { - private var db: OpaquePointer? - private var pollTimer: Timer? - private weak var modelContext: ModelContext? - - var isRunning = false - var lastSyncDate: Date? - var syncError: String? - var messagesProcessed: Int = 0 - - private var lastProcessedROWID: Int64 { - get { Int64(UserDefaults.standard.integer(forKey: AppConstants.IMessageSync.lastProcessedROWIDKey)) } - set { UserDefaults.standard.set(Int(newValue), forKey: AppConstants.IMessageSync.lastProcessedROWIDKey) } - } - - var isEnabled: Bool { - get { UserDefaults.standard.bool(forKey: AppConstants.IMessageSync.enabledKey) } - set { - UserDefaults.standard.set(newValue, forKey: AppConstants.IMessageSync.enabledKey) - if newValue { start() } else { stop() } - } - } - - deinit { - stop() - } - - // MARK: - Lifecycle - - /// Starts polling if the feature is enabled. Call once from app startup with the main ModelContext. - func startIfEnabled(with context: ModelContext) { - self.modelContext = context - guard isEnabled else { - logger.info("iMessage sync disabled — skipping") - return - } - start() - } - - func start() { - guard !isRunning else { return } - - guard openDatabase() else { return } - - // If first launch, seed the ROWID to now so we don't import the entire history - if lastProcessedROWID == 0 { - seedLastROWID() - } - - isRunning = true - syncError = nil - logger.info("iMessage sync started — polling every \(AppConstants.IMessageSync.pollIntervalSeconds)s") - - // Run immediately, then on timer - pollNewMessages() - - pollTimer = Timer.scheduledTimer( - withTimeInterval: AppConstants.IMessageSync.pollIntervalSeconds, - repeats: true - ) { [weak self] _ in - self?.pollNewMessages() - } - } - - func stop() { - pollTimer?.invalidate() - pollTimer = nil - closeDatabase() - isRunning = false - logger.info("iMessage sync stopped") - } - - // MARK: - SQLite Access - - private func openDatabase() -> Bool { - let path = AppConstants.IMessageSync.chatDBPath - - guard FileManager.default.fileExists(atPath: path) else { - syncError = "chat.db not found. Ensure Messages is set up on this Mac." - logger.error("chat.db not found at \(path)") - return false - } - - let flags = SQLITE_OPEN_READONLY | SQLITE_OPEN_NOMUTEX - let result = sqlite3_open_v2(path, &db, flags, nil) - guard result == SQLITE_OK else { - let msg = String(cString: sqlite3_errmsg(db)) - syncError = "Cannot open chat.db: \(msg). Grant Full Disk Access to this app in System Settings." - logger.error("sqlite3_open_v2 failed: \(msg)") - db = nil - return false - } - - logger.info("Opened chat.db read-only") - return true - } - - private func closeDatabase() { - if let db { - sqlite3_close(db) - self.db = nil - } - } - - /// Sets the ROWID cursor to the current max so we only process future messages. - private func seedLastROWID() { - guard let db else { return } - var stmt: OpaquePointer? - let sql = "SELECT MAX(ROWID) FROM message" - if sqlite3_prepare_v2(db, sql, -1, &stmt, nil) == SQLITE_OK, - sqlite3_step(stmt) == SQLITE_ROW { - let maxID = sqlite3_column_int64(stmt, 0) - lastProcessedROWID = maxID - logger.info("Seeded lastProcessedROWID to \(maxID)") - } - sqlite3_finalize(stmt) - } - - // MARK: - Polling - - private func pollNewMessages() { - guard let db, let modelContext else { return } - - let sql = """ - SELECT m.ROWID, m.text, m.is_from_me, m.date, h.id - FROM message m - JOIN handle h ON m.handle_id = h.ROWID - WHERE m.ROWID > ?1 AND m.text IS NOT NULL AND m.text != '' - ORDER BY m.ROWID ASC - LIMIT 500 - """ - - var stmt: OpaquePointer? - guard sqlite3_prepare_v2(db, sql, -1, &stmt, nil) == SQLITE_OK else { - let msg = String(cString: sqlite3_errmsg(db)) - syncError = "Query failed: \(msg)" - logger.error("prepare failed: \(msg)") - return - } - defer { sqlite3_finalize(stmt) } - - sqlite3_bind_int64(stmt, 1, lastProcessedROWID) - - // Fetch all contacts once per poll cycle for matching - let contacts: [Contact] - do { - contacts = try modelContext.fetch(FetchDescriptor()) - } catch { - syncError = "Failed to fetch contacts: \(error.localizedDescription)" - logger.error("Contact fetch failed: \(error)") - return - } - - // Build a lookup: normalized phone digits (last 10) → Contact - let phoneLookup = buildPhoneLookup(contacts: contacts) - - var highestROWID = lastProcessedROWID - var newCount = 0 - - while sqlite3_step(stmt) == SQLITE_ROW { - let rowid = sqlite3_column_int64(stmt, 0) - let text = sqlite3_column_text(stmt, 1).map { String(cString: $0) } ?? "" - let isFromMe = sqlite3_column_int(stmt, 2) == 1 - let dateValue = sqlite3_column_int64(stmt, 3) - let handleId = sqlite3_column_text(stmt, 4).map { String(cString: $0) } ?? "" - - highestROWID = max(highestROWID, rowid) - - // Normalize handle and look up contact - let normalizedHandle = normalizePhoneNumber(handleId) - guard let contact = phoneLookup[normalizedHandle] else { - continue // Not a Blackbook contact - } - - // Convert iMessage date (nanoseconds since 2001-01-01) to Date - let messageDate = dateFromIMesssageTimestamp(dateValue) - - let interaction = Interaction( - contact: contact, - type: .text, - date: messageDate, - summary: text - ) - interaction.messageDirection = isFromMe ? .sent : .received - - modelContext.insert(interaction) - - // Update contact's last interaction date - if contact.lastInteractionDate == nil || messageDate > contact.lastInteractionDate! { - contact.lastInteractionDate = messageDate - contact.updatedAt = Date() - } - - newCount += 1 - } - - if newCount > 0 { - do { - try modelContext.save() - messagesProcessed += newCount - logger.info("Synced \(newCount) new iMessage interaction(s)") - } catch { - syncError = "Failed to save: \(error.localizedDescription)" - logger.error("Save failed: \(error)") - return - } - } - - lastProcessedROWID = highestROWID - lastSyncDate = Date() - syncError = nil - } - - // MARK: - Phone Normalization - - /// Builds a dictionary mapping normalized phone numbers (last 10 digits) to contacts. - private func buildPhoneLookup(contacts: [Contact]) -> [String: Contact] { - var lookup: [String: Contact] = [:] - for contact in contacts { - for phone in contact.phones { - let normalized = normalizePhoneNumber(phone) - if !normalized.isEmpty { - lookup[normalized] = contact - } - } - } - return lookup - } - - /// Strips a phone string to its last 10 digits for comparison. - /// Handles formats like +1 (555) 867-5309, 15558675309, etc. - private func normalizePhoneNumber(_ raw: String) -> String { - let digits = raw.filter(\.isWholeNumber) - if digits.count >= 10 { - return String(digits.suffix(10)) - } - return digits - } - - // MARK: - Date Conversion - - /// Converts an iMessage timestamp to a Swift Date. - /// Modern macOS stores dates as nanoseconds since 2001-01-01. - /// Older databases used seconds. We detect by magnitude. - private func dateFromIMesssageTimestamp(_ timestamp: Int64) -> Date { - let reference = Date(timeIntervalSinceReferenceDate: 0) // 2001-01-01 - - if timestamp > 1_000_000_000 { - // Nanoseconds since 2001-01-01 - let seconds = TimeInterval(timestamp) / 1_000_000_000 - return reference.addingTimeInterval(seconds) - } else { - // Seconds since 2001-01-01 (older format) - return reference.addingTimeInterval(TimeInterval(timestamp)) - } - } -} -#endif diff --git a/Blackbook/Utilities/Constants.swift b/Blackbook/Utilities/Constants.swift index 5d7ebda..2354e63 100644 --- a/Blackbook/Utilities/Constants.swift +++ b/Blackbook/Utilities/Constants.swift @@ -109,13 +109,6 @@ enum AppConstants { static let serverURL = "https://sync.libersecretorum.com" } - enum IMessageSync { - static let chatDBPath = "\(NSHomeDirectory())/Library/Messages/chat.db" - static let pollIntervalSeconds: TimeInterval = 30 - static let lastProcessedROWIDKey = "iMessageSync.lastProcessedROWID" - static let enabledKey = "iMessageSync.enabled" - } - enum Defaults { static let reminderLeadTimeDays = 1 static let autoReminderThreshold: Double = 25.0 diff --git a/Blackbook/Views/Settings/SettingsView.swift b/Blackbook/Views/Settings/SettingsView.swift index c27951c..f00ff30 100644 --- a/Blackbook/Views/Settings/SettingsView.swift +++ b/Blackbook/Views/Settings/SettingsView.swift @@ -57,16 +57,10 @@ struct SettingsView: View { @State private var showGoogleSignOutConfirm = false @State private var showImportOptions = false @State private var showSelectiveImport = false - #if os(macOS) - @Environment(IMessageSyncService.self) private var iMessageService - #endif var body: some View { Form { contactsSyncSection - #if os(macOS) - iMessageSyncSection - #endif hiddenContactsSection securitySection dataSection @@ -186,70 +180,6 @@ struct SettingsView: View { return "Not synced" } - // MARK: iMessage Sync (macOS) - - #if os(macOS) - private var iMessageSyncSection: some View { - Section { - SettingsRow( - icon: "message.fill", - iconColor: .green, - title: "iMessage Sync", - subtitle: iMessageSyncSubtitle - ) { - Toggle("", isOn: Binding( - get: { iMessageService.isEnabled }, - set: { newValue in - iMessageService.isEnabled = newValue - if newValue { - iMessageService.startIfEnabled(with: modelContext) - } - } - )) - .labelsHidden() - } - - if iMessageService.isRunning { - SettingsRow( - icon: "checkmark.circle.fill", - iconColor: .green, - title: "Messages Logged", - subtitle: nil - ) { - Text("\(iMessageService.messagesProcessed)") - .foregroundStyle(.secondary) - } - } - - if let err = iMessageService.syncError { - HStack(spacing: 8) { - Image(systemName: "exclamationmark.triangle.fill") - .foregroundStyle(.red) - .font(.caption) - Text(err) - .font(.caption) - .foregroundStyle(.red) - } - .padding(.leading, 40) - } - } header: { - Text("iMessage") - } footer: { - Text("Automatically logs sent and received iMessages as interactions for matching contacts. Requires Full Disk Access.") - } - } - - private var iMessageSyncSubtitle: String { - if iMessageService.isRunning, let date = iMessageService.lastSyncDate { - return "Last checked: \(date.relativeDescription)" - } - if iMessageService.isRunning { - return "Running" - } - return "Disabled" - } - #endif - // MARK: Hidden Contacts private var hiddenContactsSection: some View { diff --git a/BlackbookServer/App/IMessageSyncService.swift b/BlackbookServer/App/IMessageSyncService.swift new file mode 100644 index 0000000..2854dc8 --- /dev/null +++ b/BlackbookServer/App/IMessageSyncService.swift @@ -0,0 +1,405 @@ +import Foundation +import Observation +import SwiftData +import SQLite3 +import os + +private let logger = Logger(subsystem: "com.blackbookdevelopment.server", category: "IMessageSync") + +/// Polls the local iMessage database (chat.db) and creates Interaction records in the master +/// store for messages sent to or received from Blackbook contacts. +/// +/// This lives in **BlackbookServer** (un-sandboxed, always-running) rather than the main app: +/// the main macOS app ships sandboxed via TestFlight and cannot read `~/Library/Messages/chat.db`. +/// The server is un-sandboxed and only needs Full Disk Access granted once. Interactions it creates +/// in the master store flow to iOS and macOS clients automatically via `/sync/changes` (the pull +/// query serves records by `updatedAt`, so freshly-created interactions are picked up). +@Observable +final class IMessageSyncService { + + enum Const { + // chat.db lives in the real user home. NSUserName() is correct regardless of sandbox state. + static let chatDBPath = "/Users/\(NSUserName())/Library/Messages/chat.db" + static let pollIntervalSeconds: TimeInterval = 30 + static let batchLimit: Int64 = 500 + static let lastProcessedROWIDKey = "iMessageSync.lastProcessedROWID" + static let enabledKey = "iMessageSync.enabled" + } + + @ObservationIgnored private var db: OpaquePointer? + @ObservationIgnored private var pollTimer: Timer? + @ObservationIgnored private let modelContainer: ModelContainer? + @ObservationIgnored private let context: ModelContext? + @ObservationIgnored private let defaults = UserDefaults(suiteName: "com.blackbookdevelopment.server") ?? .standard + + var isRunning = false + var lastSyncDate: Date? + var syncError: String? + var messagesProcessed: Int = 0 + var isBackfilling = false + /// Handles seen in the most recent poll that did not match any contact (capped at 20). + /// Lets the user diagnose "why isn't Nick logging?" — his handle shows here if unmatched. + var unmatchedHandlesLastPoll: [String] = [] + + private var lastProcessedROWID: Int64 { + get { Int64(defaults.integer(forKey: Const.lastProcessedROWIDKey)) } + set { defaults.set(Int(newValue), forKey: Const.lastProcessedROWIDKey) } + } + + var isEnabled: Bool { + get { defaults.bool(forKey: Const.enabledKey) } + set { + defaults.set(newValue, forKey: Const.enabledKey) + if newValue { start() } else { stop() } + } + } + + init(modelContainer: ModelContainer?) { + self.modelContainer = modelContainer + self.context = modelContainer.map { ModelContext($0) } + } + + deinit { + stop() + } + + // MARK: - Lifecycle + + /// Starts polling if the feature is enabled. Call once from server startup. + func startIfEnabled() { + guard isEnabled else { + logger.info("iMessage sync disabled — skipping") + return + } + start() + } + + func start() { + guard !isRunning else { return } + guard context != nil else { + syncError = "Master store unavailable — cannot log iMessages." + return + } + guard openDatabase() else { return } + + // On first run, seed the cursor to the current max so we don't import the whole history. + if lastProcessedROWID == 0 { + seedLastROWID() + } + + isRunning = true + syncError = nil + logger.info("iMessage sync started — polling every \(Const.pollIntervalSeconds)s") + + pollNewMessages() + pollTimer = Timer.scheduledTimer( + withTimeInterval: Const.pollIntervalSeconds, + repeats: true + ) { [weak self] _ in + self?.pollNewMessages() + } + } + + func stop() { + pollTimer?.invalidate() + pollTimer = nil + closeDatabase() + isRunning = false + logger.info("iMessage sync stopped") + } + + // MARK: - SQLite Access + + private func openDatabase() -> Bool { + let path = Const.chatDBPath + + guard FileManager.default.fileExists(atPath: path) else { + syncError = "chat.db not found at \(path). Ensure Messages is set up on this Mac." + logger.error("chat.db not found at \(path)") + return false + } + + let flags = SQLITE_OPEN_READONLY | SQLITE_OPEN_NOMUTEX + let result = sqlite3_open_v2(path, &db, flags, nil) + guard result == SQLITE_OK else { + let msg = String(cString: sqlite3_errmsg(db)) + syncError = "Cannot open chat.db: \(msg). Grant Full Disk Access to Blackbook Server in System Settings." + logger.error("sqlite3_open_v2 failed: \(msg)") + db = nil + return false + } + + logger.info("Opened chat.db read-only") + return true + } + + private func closeDatabase() { + if let db { + sqlite3_close(db) + self.db = nil + } + } + + /// Sets the ROWID cursor to the current max so we only process future messages. + private func seedLastROWID() { + guard let db else { return } + var stmt: OpaquePointer? + if sqlite3_prepare_v2(db, "SELECT MAX(ROWID) FROM message", -1, &stmt, nil) == SQLITE_OK, + sqlite3_step(stmt) == SQLITE_ROW { + let maxID = sqlite3_column_int64(stmt, 0) + lastProcessedROWID = maxID + logger.info("Seeded lastProcessedROWID to \(maxID)") + } + sqlite3_finalize(stmt) + } + + // MARK: - Backfill + + /// Resets the cursor to import every message from the last N days, then runs poll cycles + /// until the cursor catches up. + @MainActor + func backfill(daysBack: Int) async { + guard isRunning, let db else { + syncError = "Enable iMessage Sync before running a backfill." + return + } + guard !isBackfilling else { return } + isBackfilling = true + defer { isBackfilling = false } + + let cutoff = Date().addingTimeInterval(-Double(daysBack) * 86_400) + let cutoffTimestamp = imessageTimestamp(from: cutoff) + + var stmt: OpaquePointer? + guard sqlite3_prepare_v2(db, "SELECT MIN(ROWID) FROM message WHERE date >= ?1", -1, &stmt, nil) == SQLITE_OK else { + syncError = "Backfill query failed: \(String(cString: sqlite3_errmsg(db)))" + return + } + sqlite3_bind_int64(stmt, 1, cutoffTimestamp) + var startROWID: Int64 = 0 + if sqlite3_step(stmt) == SQLITE_ROW { + startROWID = sqlite3_column_int64(stmt, 0) + } + sqlite3_finalize(stmt) + + guard startROWID > 0 else { + logger.info("Backfill: no messages within the last \(daysBack) days") + lastSyncDate = Date() + return + } + + lastProcessedROWID = startROWID - 1 + logger.info("Backfill: resetting cursor to \(startROWID - 1), importing \(daysBack) days") + + var iterations = 0 + while iterations < 1000 { + let processed = pollNewMessages() + iterations += 1 + if processed < Const.batchLimit { break } + await Task.yield() + } + logger.info("Backfill complete: \(self.messagesProcessed) messages logged across \(iterations) batches") + } + + // MARK: - Polling + + /// Polls for new messages above the cursor, inserts matching Interactions, advances the cursor. + /// Returns the number of rows fetched (matched + unmatched) so the backfill loop knows whether + /// another batch remains. + @discardableResult + private func pollNewMessages() -> Int64 { + guard let db, let context else { return 0 } + + let sql = """ + SELECT m.ROWID, m.text, m.is_from_me, m.date, h.id + FROM message m + JOIN handle h ON m.handle_id = h.ROWID + WHERE m.ROWID > ?1 AND m.text IS NOT NULL AND m.text != '' + ORDER BY m.ROWID ASC + LIMIT \(Const.batchLimit) + """ + + var stmt: OpaquePointer? + guard sqlite3_prepare_v2(db, sql, -1, &stmt, nil) == SQLITE_OK else { + let msg = String(cString: sqlite3_errmsg(db)) + syncError = "Query failed: \(msg)" + logger.error("prepare failed: \(msg)") + return 0 + } + defer { sqlite3_finalize(stmt) } + + sqlite3_bind_int64(stmt, 1, lastProcessedROWID) + + let contacts: [Contact] + do { + contacts = try context.fetch(FetchDescriptor()) + } catch { + syncError = "Failed to fetch contacts: \(error.localizedDescription)" + logger.error("Contact fetch failed: \(error)") + return 0 + } + + let lookup = buildHandleLookup(contacts: contacts) + + struct PendingRow { + let text: String + let isFromMe: Bool + let date: Date + let contact: Contact + } + + var pending: [PendingRow] = [] + var highestROWID = lastProcessedROWID + var rowCount: Int64 = 0 + var unmatched: [String] = [] + + while sqlite3_step(stmt) == SQLITE_ROW { + rowCount += 1 + let rowid = sqlite3_column_int64(stmt, 0) + let text = sqlite3_column_text(stmt, 1).map { String(cString: $0) } ?? "" + let isFromMe = sqlite3_column_int(stmt, 2) == 1 + let dateValue = sqlite3_column_int64(stmt, 3) + let handleId = sqlite3_column_text(stmt, 4).map { String(cString: $0) } ?? "" + + highestROWID = max(highestROWID, rowid) + + guard let contact = lookup(handleId) else { + if unmatched.count < 20, !unmatched.contains(handleId) { + unmatched.append(handleId) + } + continue + } + + pending.append(PendingRow( + text: text, + isFromMe: isFromMe, + date: dateFromIMesssageTimestamp(dateValue), + contact: contact + )) + } + + unmatchedHandlesLastPoll = unmatched + + guard !pending.isEmpty else { + lastProcessedROWID = highestROWID + lastSyncDate = Date() + syncError = nil + return rowCount + } + + // Dedup: skip any (contact, second-truncated date, summary) we already stored. Protects the + // backfill path and any future ROWID reset (e.g. a chat.db rebuild). + let earliest = pending.map(\.date).min() ?? Date() + let latest = pending.map(\.date).max() ?? Date() + var existingKeys = Set() + do { + var descriptor = FetchDescriptor( + predicate: #Predicate { $0.date >= earliest && $0.date <= latest } + ) + descriptor.fetchLimit = 5000 + for ix in try context.fetch(descriptor) { + guard let cid = ix.contact?.id else { continue } + existingKeys.insert(dedupKey(contactId: cid, date: ix.date, summary: ix.summary)) + } + } catch { + logger.warning("Dedup fetch failed (\(error.localizedDescription)) — proceeding without dedup") + } + + var newCount = 0 + for row in pending { + let key = dedupKey(contactId: row.contact.id, date: row.date, summary: row.text) + if existingKeys.contains(key) { continue } + existingKeys.insert(key) + + let interaction = Interaction( + contact: row.contact, + type: .text, + date: row.date, + summary: row.text + ) + interaction.messageDirection = row.isFromMe ? .sent : .received + context.insert(interaction) + + if row.contact.lastInteractionDate == nil || row.date > row.contact.lastInteractionDate! { + row.contact.lastInteractionDate = row.date + row.contact.updatedAt = Date() + } + newCount += 1 + } + + if newCount > 0 { + do { + try context.save() + messagesProcessed += newCount + logger.info("Logged \(newCount) new iMessage interaction(s)") + } catch { + syncError = "Failed to save: \(error.localizedDescription)" + logger.error("Save failed: \(error)") + return rowCount + } + } + + lastProcessedROWID = highestROWID + lastSyncDate = Date() + syncError = nil + return rowCount + } + + private func dedupKey(contactId: UUID, date: Date, summary: String?) -> String { + let seconds = Int64(date.timeIntervalSince1970) + return "\(contactId.uuidString)|\(seconds)|\(summary ?? "")" + } + + // MARK: - Handle Matching + + /// Returns a closure mapping a raw chat.db `handle.id` to a Contact, indexing both phone + /// numbers (last-10-digits) and emails (lowercased) so Apple-ID-email threads also match. + private func buildHandleLookup(contacts: [Contact]) -> (String) -> Contact? { + var phoneMap: [String: Contact] = [:] + var emailMap: [String: Contact] = [:] + for contact in contacts { + for phone in contact.phones { + let normalized = normalizePhoneNumber(phone) + if !normalized.isEmpty { phoneMap[normalized] = contact } + } + for email in contact.emails { + let normalized = normalizeEmail(email) + if !normalized.isEmpty { emailMap[normalized] = contact } + } + } + return { handleId in + if handleId.contains("@") { + return emailMap[self.normalizeEmail(handleId)] + } else { + return phoneMap[self.normalizePhoneNumber(handleId)] + } + } + } + + /// Strips a phone string to its last 10 digits for comparison. + private func normalizePhoneNumber(_ raw: String) -> String { + let digits = raw.filter(\.isWholeNumber) + return digits.count >= 10 ? String(digits.suffix(10)) : digits + } + + private func normalizeEmail(_ raw: String) -> String { + raw.lowercased().trimmingCharacters(in: .whitespacesAndNewlines) + } + + // MARK: - Date Conversion + + /// Modern chat.db stores message dates as nanoseconds since 2001-01-01; older ones used seconds. + private func dateFromIMesssageTimestamp(_ timestamp: Int64) -> Date { + let reference = Date(timeIntervalSinceReferenceDate: 0) + if timestamp > 1_000_000_000 { + return reference.addingTimeInterval(TimeInterval(timestamp) / 1_000_000_000) + } else { + return reference.addingTimeInterval(TimeInterval(timestamp)) + } + } + + /// Inverse of `dateFromIMesssageTimestamp` — nanoseconds-since-2001 form used by modern chat.db. + private func imessageTimestamp(from date: Date) -> Int64 { + Int64(date.timeIntervalSinceReferenceDate * 1_000_000_000) + } +} diff --git a/BlackbookServer/App/ServerMenuView.swift b/BlackbookServer/App/ServerMenuView.swift index d7e51ec..97a7311 100644 --- a/BlackbookServer/App/ServerMenuView.swift +++ b/BlackbookServer/App/ServerMenuView.swift @@ -82,6 +82,10 @@ struct ServerMenuView: View { Divider() + iMessageSection + + Divider() + Button("Quit Blackbook Server") { NSApplication.shared.terminate(nil) } @@ -96,6 +100,84 @@ struct ServerMenuView: View { } } + @ViewBuilder + private var iMessageSection: some View { + let im = model.imessage + VStack(alignment: .leading, spacing: 8) { + Toggle(isOn: Binding( + get: { im.isEnabled }, + set: { im.isEnabled = $0 } + )) { + Label("Log iMessages", systemImage: "message") + .font(.caption) + } + + if im.isRunning { + HStack { + Text("\(im.messagesProcessed) logged") + .font(.caption) + .foregroundStyle(.secondary) + if let date = im.lastSyncDate { + Spacer() + Text("checked \(date.formatted(.relative(presentation: .numeric)))") + .font(.caption2) + .foregroundStyle(.secondary) + .lineLimit(1) + } + } + + Button { + Task { await im.backfill(daysBack: 30) } + } label: { + HStack(spacing: 6) { + if im.isBackfilling { + ProgressView().controlSize(.small) + Text("Importing…") + } else { + Image(systemName: "clock.arrow.circlepath") + Text("Sync Last 30 Days") + } + } + .font(.caption) + } + .disabled(im.isBackfilling) + } + + if let err = im.syncError { + Text(err) + .font(.caption2) + .foregroundStyle(.red) + .fixedSize(horizontal: false, vertical: true) + Button("Open Full Disk Access") { + if let url = URL(string: "x-apple.systempreferences:com.apple.preference.security?Privacy_AllFiles") { + NSWorkspace.shared.open(url) + } + } + .font(.caption2) + } + + if !im.unmatchedHandlesLastPoll.isEmpty { + DisclosureGroup { + VStack(alignment: .leading, spacing: 2) { + ForEach(im.unmatchedHandlesLastPoll, id: \.self) { handle in + Text(handle) + .font(.caption2.monospaced()) + .foregroundStyle(.secondary) + .textSelection(.enabled) + } + Text("Add as a phone or email on the matching contact to log their messages.") + .font(.caption2) + .foregroundStyle(.secondary) + } + } label: { + Text("Unmatched handles (\(im.unmatchedHandlesLastPoll.count))") + .font(.caption2) + .foregroundStyle(.secondary) + } + } + } + } + private func saveEmail() { let trimmed = editingEmail.trimmingCharacters(in: .whitespacesAndNewlines) guard !trimmed.isEmpty else { return } diff --git a/BlackbookServer/App/ServerStatusModel.swift b/BlackbookServer/App/ServerStatusModel.swift index 3befcd8..23fc10a 100644 --- a/BlackbookServer/App/ServerStatusModel.swift +++ b/BlackbookServer/App/ServerStatusModel.swift @@ -14,6 +14,11 @@ final class ServerStatusModel { /// Master SwiftData container. Nil if container init failed (sync routes will return 503 /// in that case; backups and logs continue to work since they're filesystem-only). private let modelContainer: ModelContainer? + + /// Reads the local iMessage database and logs interactions into the master store. + /// Runs independently of the backup/sync server (it only needs the master container + + /// Full Disk Access). Interactions it creates reach all devices on their next pull. + let imessage: IMessageSyncService var launchAtLogin: Bool { get { SMAppService.mainApp.status == .enabled } set { @@ -34,9 +39,12 @@ final class ServerStatusModel { init(modelContainer: ModelContainer? = nil) { self.modelContainer = modelContainer + self.imessage = IMessageSyncService(modelContainer: modelContainer) if !email.isEmpty { startServer() } + // iMessage logging is independent of the email/sync server — start it if the user enabled it. + imessage.startIfEnabled() } // Source of truth is the main Blackbook app's logged-in user. diff --git a/docs/test-scenarios/TEST_SCENARIOS.md b/docs/test-scenarios/TEST_SCENARIOS.md index e720884..9c152e2 100644 --- a/docs/test-scenarios/TEST_SCENARIOS.md +++ b/docs/test-scenarios/TEST_SCENARIOS.md @@ -307,3 +307,57 @@ rm -rf /Applications/BlackbookServer.app cp -R /tmp/blackbook-server-build/Build/Products/Release/BlackbookServer.app /Applications/ open /Applications/BlackbookServer.app ``` + +--- + +## 2026-05-29 — iMessage Sync moved into BlackbookServer (un-sandboxed) + email matching, backfill, dedup, diagnostics + +**Context:** Toggling "iMessage Sync" in the macOS app did nothing even with Full Disk Access. Root cause: the main macOS app is sandboxed (required for its TestFlight distribution), so it (a) resolved `chat.db` to a non-existent sandbox-container path and (b) couldn't read `~/Library/Messages/` even with FDA. De-sandboxing the main app would break TestFlight (App Store Connect rejects non-sandboxed macOS builds, ITMS-90296). Resolution: the iMessage reader was moved into **BlackbookServer**, which is already un-sandboxed, always-running (login item), holds the master SwiftData store, and syncs to all devices. Interactions it creates flow to iOS + macOS via `/sync/changes` (server pull serves by `updatedAt`). + +The main app's iMessage Sync UI/service was **removed**. All iMessage controls now live in the BlackbookServer menu-bar popover. + +### Setup (one-time) +1. Rebuild + reinstall BlackbookServer from this branch (see deploy note below). The menu-bar "server.rack" icon should appear. +2. Grant **Full Disk Access to "Blackbook Server"** (not "Blackbook") in System Settings → Privacy & Security → Full Disk Access. This is the key difference — FDA must be on the server app. +3. Click the menu-bar icon → toggle **"Log iMessages"** on. + +### Scenario 1 — Toggle works in the server +1. Open the BlackbookServer menu. Flip "Log iMessages" on. +2. It should show "N logged" and "checked