CBL-8743 : Add binary test targets for the Objective-C tests

[pasin]

Runs the Objective-C tests against a shipped CouchbaseLite binary instead of a build from the working tree, so a test run certifies a released artifact. This is a part of the tasks to improve CI system for CBL-iOS.

• Add CBL_EE_ObjC_Binary_Tests and CBL_ObjC_Binary_Tests, which compile and link against frameworks staged in BinaryTests/Frameworks. Each runs on macOS and iOS. The community target leaves out the tests for Enterprise edition.

• Add Scripts/prepare_binary_test.sh to download and copy built binaries from latestbuild.

• Move download_vector_search_extension.sh into CE repo from the EE repo as it is used by prepare_binary_tests.sh to dowload the vector search extension. Update the script to detect and skip downloading if the extension's already downloaded.

• Set CBL_BINARY_TEST in the targets' xcconfig. Tests that need internal API are guarded with #ifndef CBL_BINARY_TEST.

• Limit the targets to the public API. They have no internal header search paths, and USE_HEADERMAP = NO so that a forgetting internal import fails to compile.

• Replace internal API in the shared tests with public equivalents: a CBLJSONUtil owned by the tests, SecItem keychain cleanup, and a N1QL query on meta().deleted for document expiration.

• Delete internal API that no test uses now: Foundation+CBL, activeServiceCount, isClosedLocked and the fromCollections:config: initializer.

• Add an iOS host application, shared by both editions. iOS tests need one because keychain calls fail without an app, and its AppDelegate sets the default that enables the keychain tests. It links no frameworks, so the only CouchbaseLite loaded is the one embedded in the test bundle target.

• Add a scheme per edition and platform. The macOS ones run without a host app. The community targets build in Debug and Release and the enterprise ones in Debug_EE and Release_EE, which keeps their products apart.

• Companion PR : https://github.com/couchbaselabs/couchbase-lite-ios-ee/pull/350

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds an Objective-C “binary (black-box) test” setup that runs the test suite against staged CouchbaseLite xcframeworks (instead of building from source), and refactors tests to separate public-API coverage from internal white-box checks.

Changes:

• Introduces new ObjC binary test xcconfigs/schemes plus a lightweight iOS host app for keychain-requiring tests.
• Adds scripts to download/stage CouchbaseLite (+ vector search extension) binaries for repeatable binary test runs.
• Refactors tests/utilities to avoid internal headers in binary tests (guards, new JSON/date helpers, keychain cleanup changes).

Reviewed changes

Copilot reviewed 61 out of 62 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
xcconfigs/CBL_ObjC_Tests_iOS_App.xcconfig	Enables perf-test compilation for the optimized-build host app.
xcconfigs/CBL_ObjC_Binary_Tests_iOS_App.xcconfig	New xcconfig for the iOS host app used by binary tests.
xcconfigs/CBL_ObjC_Binary_Tests.xcconfig	New xcconfig defining binary-test build constraints (no headermap, public API only).
Scripts/prepare_binary_test.sh	New script to download and stage CouchbaseLite binaries for testing.
Scripts/download_vector_search_extension.sh	New script to download the vector search extension for tests.
Objective-C/Tests/iOS/AppDelegate.m	Runs perf tests only when explicitly enabled and not in Debug.
Objective-C/Tests/VectorSearchTest+Lazy.m	Switches date parsing to test-owned JSON utilities.
Objective-C/Tests/Util/CBLWordEmbeddingModel.h	Routes imports through shared test common header for binary/source compatibility.
Objective-C/Tests/Util/CBLTestCustomLogSink.h	Routes imports through shared test common header for binary/source compatibility.
Objective-C/Tests/Util/CBLMockConnectionErrorLogic.m	Routes imports through shared test common header for binary/source compatibility.
Objective-C/Tests/Util/CBLMockConnection.m	Removes internal imports to support binary tests.
Objective-C/Tests/Util/CBLMockConnection.h	Routes imports through shared test common header for binary/source compatibility.
Objective-C/Tests/Util/CBLJSONUtil.m	New Foundation-only JSON/date helper for tests (works in binary tests).
Objective-C/Tests/Util/CBLJSONUtil.h	Declares Foundation-only JSON/date helper for tests.
Objective-C/Tests/Util/CBLBlockConflictResolver.h	Routes imports through shared test common header for binary/source compatibility.
Objective-C/Tests/UnnestArrayIndexTest.m	Guards internal-only index-info checks; adds public-API verification for binary tests.
Objective-C/Tests/URLEndpointListenerTest.m	Reworks keychain cleanup to avoid internal APIs and improve robustness.
Objective-C/Tests/URLEndpointListenerTest.h	Adds anonymous identity common-name constant and new cleanup method signatures.
Objective-C/Tests/URLEndpointListenerTest+Main.m	Guards internal-only tests and updates cleanup calls.
Objective-C/Tests/TrustCheckTest.m	Hard-errors when included in binary tests due to internal API usage.
Objective-C/Tests/TLSIdentityTest.m	Guards internal-only tests from running in binary tests.
Objective-C/Tests/ReplicatorTest.m	Moves Sync Gateway/internal logic out to internal-only compilation unit.
Objective-C/Tests/ReplicatorTest.h	Guards enterprise internal header import from binary tests.
Objective-C/Tests/ReplicatorTest+SG.m	New internal-only SG helper/test file (excluded from binary tests).
Objective-C/Tests/ReplicatorTest+PendingDocIds.m	Removes internal header dependency.
Objective-C/Tests/ReplicatorTest+MessageEndPoint.m	Removes internal header dependencies.
Objective-C/Tests/ReplicatorTest+Main.m	Moves internal-only tests behind binary-test guards; updates config init usage.
Objective-C/Tests/ReplicatorTest+CustomConflict.m	Moves internal-only test cases behind binary-test guards; reduces internal imports.
Objective-C/Tests/ReplicatorTest+Collection.m	Avoids removed internal convenience API; configures per-collection explicitly.
Objective-C/Tests/ReplicatorTest+Backgrounding.m	New internal-only backgrounding tests split into separate file.
Objective-C/Tests/QueryTest+Main.m	Uses Foundation-only JSON util; moves internal-only tests behind guards.
Objective-C/Tests/PredictiveQueryTest.m	Uses Foundation-only JSON util for date JSON formatting.
Objective-C/Tests/PredictiveQueryTest+CoreML.m	Splits internal-only CoreML conversion tests from black-box tests.
Objective-C/Tests/NotificationTest.m	Removes internal import to support binary tests.
Objective-C/Tests/MultipeerReplicatorTest.m	Hard-errors when included in binary tests due to internal API usage.
Objective-C/Tests/MiscTest.m	Hard-errors when included in binary tests due to internal API usage.
Objective-C/Tests/MiscCppTest.mm	Hard-errors when included in binary tests due to internal API usage.
Objective-C/Tests/MigrationTest.m	Uses exception-breakpoint helper instead of manipulating LiteCore exception counter directly.
Objective-C/Tests/LogTest.m	Hard-errors when included in binary tests due to internal API usage.
Objective-C/Tests/FragmentTest.m	Uses Foundation-only JSON util for date JSON formatting.
Objective-C/Tests/DocumentTest.m	Uses Foundation-only JSON util for date parsing/formatting and JSON structural compares.
Objective-C/Tests/DocumentExpirationTest.m	Adjusts expiration/purge tests for stronger assertions and new purge checks.
Objective-C/Tests/DictionaryTest.m	Uses Foundation-only JSON util for JSON structural compares.
Objective-C/Tests/DatabaseTest.m	Guards internal-only assertions; removes reliance on internal active-service counters.
Objective-C/Tests/DatabaseEncryptionTest.m	Removes internal import to support binary tests.
Objective-C/Tests/ConcurrentTest.m	Removes dependency on CollectionUtils my_map for binary/source portability.
Objective-C/Tests/CBLTestCase.m	Adds macros to make LiteCore exception counter usage a no-op for binary tests.
Objective-C/Tests/CBLTestCase.h	Imports shared test common header; hides exception counter declarations for binary tests.
Objective-C/Tests/AuthenticatorTest.m	Moves internal-only authenticator verification behind binary-test guards.
Objective-C/Tests/ArrayTest.m	Uses Foundation-only JSON util for date parsing/formatting and JSON structural compares.
Objective-C/Internal/Replicator/CBLCollectionConfiguration+Internal.h	Removes internal convenience API declaration.
Objective-C/Internal/Foundation+CBL.mm	Replaced with header-like content related to test/common imports.
Objective-C/Internal/Foundation+CBL.h	Replaced with xcconfig includes (appears unrelated to original purpose).
Objective-C/Internal/CBLDatabase+Internal.h	Removes internal-only testing helpers (active-service count, isClosedLocked).
Objective-C/CBLDatabase.mm	Removes internal-only testing helpers (active-service count, isClosedLocked).
Objective-C/CBLCollectionConfiguration.m	Removes internal convenience API implementation.
CouchbaseLite.xcodeproj/xcshareddata/xcschemes/CBL_ObjC_Binary_Tests_iOS_App.xcscheme	New scheme to build/run the iOS host app + binary tests.
CouchbaseLite.xcodeproj/xcshareddata/xcschemes/CBL_ObjC_Binary_Tests.xcscheme	New scheme to run ObjC binary tests.
CouchbaseLite.xcodeproj/xcshareddata/xcschemes/CBL_EE_ObjC_Binary_Tests_iOS_App.xcscheme	New EE scheme for iOS host app + binary tests.
CouchbaseLite.xcodeproj/xcshareddata/xcschemes/CBL_EE_ObjC_Binary_Tests.xcscheme	New EE scheme to run ObjC binary tests.
CouchbaseLite.xcodeproj/project.pbxproj	Adds new targets, build phases, resources, xcconfigs, and staged xcframework references.
.gitignore	Ignores staged binary frameworks and vector-search download marker file.

Suppressed comments (8)

Objective-C/Tests/Util/CBLJSONUtil.m:1

• NSDateFormatter instances are not thread-safe. Since this is a shared static formatter, concurrent test execution (or any utility usage from multiple threads) can lead to crashes or incorrect parsing/formatting. Prefer a thread-safe approach such as NSISO8601DateFormatter configured for fractional seconds, or create a formatter per call / per-thread (e.g., thread-local storage) or guard formatter access with a lock/serial queue.
  Objective-C/Tests/Util/CBLJSONUtil.m:1
• NSCAssert is typically compiled out in Release builds, which can make invalid JSON silently return nil and cause misleading downstream failures. Since this is test infrastructure, prefer always-on validation (e.g., throwing an exception, or returning nil plus an out-NSError**, or using the project’s test assertion macros) so failures are deterministic across build configurations.
  Scripts/prepare_binary_test.sh:1
• The for xcf in $(find ...) construct will split paths on whitespace/newlines and can break if any directory names contain spaces. Prefer a null-delimited find loop (e.g., find ... -print0 with while IFS= read -r -d '') to handle paths safely. Also consider adding set -o pipefail (and optionally -u) at the top for more reliable failure reporting in pipelines/subshells used within the script.
  Scripts/prepare_binary_test.sh:1
• The for xcf in $(find ...) construct will split paths on whitespace/newlines and can break if any directory names contain spaces. Prefer a null-delimited find loop (e.g., find ... -print0 with while IFS= read -r -d '') to handle paths safely. Also consider adding set -o pipefail (and optionally -u) at the top for more reliable failure reporting in pipelines/subshells used within the script.
  Scripts/download_vector_search_extension.sh:1
• The download/extract steps are not robust to redirects and whitespace/shell-expansion issues (unquoted variables). For reliability in CI and across environments, consider using curl -fL and quoting ${URL} / ${ZIP_FILENAME}. Also consider downloading into a temp directory and cleaning up on failure to avoid partial/corrupted artifacts in Tests/Extensions.
  Scripts/download_vector_search_extension.sh:1
• The download/extract steps are not robust to redirects and whitespace/shell-expansion issues (unquoted variables). For reliability in CI and across environments, consider using curl -fL and quoting ${URL} / ${ZIP_FILENAME}. Also consider downloading into a temp directory and cleaning up on failure to avoid partial/corrupted artifacts in Tests/Extensions.
  Objective-C/Tests/URLEndpointListenerTest.m:1
• The cleanup deletes identities and certificates by label, but it doesn’t attempt to clean up any associated/private key items. Over time (especially with interrupted runs), this can leave orphaned keys in the keychain and cause flaky behavior or bloat. Consider also deleting kSecClassKey items associated with the same label (or using the same strategy as the prior CBLTLSIdentity helper if it correctly removes key material), while still tolerating errSecItemNotFound / permission-related statuses.
  Objective-C/Tests/ReplicatorTest+SG.m:1
• Typo in comment: 'offse' should be 'offset'.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.