SOLIDPOD: readpod returning undecrypted files

[jesscmoore]

Describe the Bug

[A clear and concise description of the bug.]

I'm now testing PersonalLibrary app with 20 book records in the Pod.
With small numbers of books, readpod() is returning decrypted content. However when there are 20 files to decrypt, which are sequentially decrypted, it is regularly failing to decrypt 1 or more files.

Claude's diagnosis of the problem:

Repeated testing shows that quite often getOwnBooksList() is returning at least 1 undecrypted file. Why would that happen?

Now I have a clear picture. The root cause is in solidpod's read_pod.dart line 137–139:


return encKey != null
    ? decryptData(encDataStr, encKey, IV.fromBase64(ivStr))
    : fileContent;   // ← silently returns the raw encrypted TTL
encKey is null when retrieveEncKey → KeyManager.getIndividualKey → IndividualKeyManager.getIndividualKey cannot find the file's URL in _indKeyMap.

The _indKeyMap is loaded from ind-keys.ttl via readIndKeyFile. There is a known CSS server version inconsistency (noted in solidpod's own comment on line 141 of key_helper.dart): in newer CSS versions, SPARQL-inserted triples use a relative path as subject instead of a full URL. When readIndKeyFile reads those entries back, it converts the stored relative pathPred value to a full URL via getFileUrl. If that URL doesn't exactly match the URL produced by generateResourceUrlFromPath inside readPod (different trailing slash, encoding, etc.), _indKeyMap.containsKey(resourceUrl) returns false → key is null → raw encrypted content is returned silently instead of throwing.

The fix at this level — detect the encrypted content and treat it as a parse failure so those books appear in unparseableBooks rather than silently disappearing:

Read [rest_api.dart](vscode-webview://0co9cdg0mngivmc6p9ut2v0l38n7cq4si584evnbch394fbjkqes/index.html?id=08445335-a656-4934-9360-bd7af6d69a2d&parentId=14&origin=842d49cc-ddb2-4d56-9657-3b24d1410a19&swVersion=4&extensionId=Anthropic.claude-code&platform=electron&vscode-resource-base-authority=vscode-resource.vscode-cdn.net&parentOrigin=vscode-file%3A%2F%2Fvscode-app&session=0513cc2f-2e90-4ce9-9fd4-19d20d53b744#) (lines 94-118)
Edit [rest_api.dart](vscode-webview://0co9cdg0mngivmc6p9ut2v0l38n7cq4si584evnbch394fbjkqes/index.html?id=08445335-a656-4934-9360-bd7af6d69a2d&parentId=14&origin=842d49cc-ddb2-4d56-9657-3b24d1410a19&swVersion=4&extensionId=Anthropic.claude-code&platform=electron&vscode-resource-base-authority=vscode-resource.vscode-cdn.net&parentOrigin=vscode-file%3A%2F%2Fvscode-app&session=0513cc2f-2e90-4ce9-9fd4-19d20d53b744#)
Added 19 lines
Summary:

Root cause: solidpod's readPod has a silent fallback — when the decryption key can't be found in _indKeyMap, it returns the raw encrypted TTL instead of throwing. The key lookup fails due to a URL mismatch: newer CSS server versions store SPARQL-inserted triples with a relative path as subject rather than a full URL, and the conversion back to a full URL via getFileUrl doesn't always produce an identical string to what generateResourceUrlFromPath generates during the read.

Fix: Added an 'encData' guard before parsing. This is the solidpod predicate name for encrypted file content — its presence in the returned string unambiguously identifies undecrypted content. Those files are routed to unparseableBooks (visible to the user) instead of being silently dropped.

To Reproduce

To reproduce, add 20 books to personallibrary. Probably would also occur in other pod apps that sequentially run readpod on a list of more than 15 files. Click Library. To reload Library, click a different menu button and return to Library.

flutter run --debug -d macos
....
flutter: 20 files in Pod
flutter: Undecrypted content detected in: book-9781460759837.ttl
flutter: Undecrypted content detected in: book-9781460756805.ttl
flutter: Found 2 unparseable or empty files

Screenshots

Context

Where has the issue been observed:

• Android
• Chrome
• iOS
• Linux
• macOS
• Web
• Windows

App Version: personallibrary version: 0.0.11+1
Flutter Version: 3.41.6

Closing Criteria

Checklist for closing the issue:

• No errors from make prep
• All tests pass make qtest
• All fles decrypted by readpod()

[jesscmoore]

The particular files that are not decrypted varies.

[jesscmoore]

Ironically, today I am unable to replicate the instance of one or more files in a folder not decrypting.

However, I've reviewed the code in solidpod relating to individual key retrieval for readPod(), as it relates to running readPod() on a list of files, each individually encrypted with an individual key.

readPod() fetches the encryption key with:

final encKey = await retrieveEncKey(fileUrl, inheritKeyUrl: inheritKeyUrl)

which calls KeyManager.getIndividualKey()
which gets the master key with. getMasterKey() as an argument of the individual key manager

IndividualKeyManager.getIndividualKey(
      resourceUrl,
      await getMasterKey(),
    )

This runs loadIndividualKeys() if the state variable _indKeyMap == null || _indKeyMap!.isEmpty
to fetch the map of keys, where the resourceUrl is the key to the json map.

loadIndividualKeys() only runs is _indKeyMap not null and forceReload == false (default), and calls readIndKeyFile() in key_helper.dart

readIndKeyFile() returns the map of IndividualKeyRecords for every file in the folder, where the key to the map is the fileUrl. This process to make _indKeyMap needs to fully complete, not just be not null and not empty, for a key to be found to decrypt any file in the folder. I suspect that we are missing an await somewhere in the checks of the above process, which could have caused the failure to decrypt. There does not seem to be any issue with the formation of the fileUrl key.

In theory, providing the full _indKeyMap is generated, subsequent readPod() processes could be done synchronously, which would make decrypting lists of files much quicker.

[jesscmoore]

Rather than allowing return on the non-decrypted file, suggest we require an encryption key to be found to allow decryption, to make it easier to identify this problem - have submitted PR for this. Noting, testing in personalLibrary app continues to now load all files correctly. I suspect this is because, I have switched from synchronous to sequential calls of readPod() for all the book record files in the pod.

    if (!_indKeyMap!.containsKey(resourceUrl)) {
      return null;
    }

[jesscmoore]

Showing files decrypting successfully in personalLibrary 0.0.11+1 pointed at solidpod branch jess/626_fix_paths: