Skip to content

fix(lightroom-plugin): icon, anti-duplicate republish, hash-based selection matching - #27

Open
manuzzi wants to merge 6 commits into
markusthiel:mainfrom
manuzzi-photo:feat/lightroom-plugin-icon-dedupe-hash-matching
Open

manuzzi wants to merge 6 commits into
markusthiel:mainfrom
manuzzi-photo:feat/lightroom-plugin-icon-dedupe-hash-matching

Conversation

@manuzzi

@manuzzi manuzzi commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Closes #26

Summary

  • Ship the publish service's icon (icon.png/icon@2x.png, generated from the existing Lumio brand mark). It was referenced in LumioPublishService.lua but never added to the plug-in folder, so it showed up blank/broken in Lightroom's Publishing Services panel.
  • Stop duplicate files piling up on re-publish: uploadOnePhoto now deletes the old remote file (via rendition.publishedPhotoId, set by LR from the previous publish) before uploading the new one, instead of only ever adding new files.
  • Make Selection-Import matching survive renames and same-basename ambiguity: the Publish-Service embeds the MD5 of the original master file into a custom XMP segment of the uploaded JPEG (new JpegXmp.lua). The worker extracts it via the existing exiftool call and stores it under the already-present (and previously unused) exif.lumio.originalMd5 JSON field — no schema migration. ImportSelectionTask.lua uses it to auto-resolve ambiguous filename matches and, with a new opt-in dialog option, to recover files renamed after publishing.
  • "Show in Lumio" (right-click a published collection) now opens the gallery's Studio management view (/studio/<galleryId>) instead of the public customer-facing gallery link (/g/<slug>) — a photographer clicking this wants to manage the gallery, not see the customer's view. This also drops the slug-from-cache lookup that only existed to support the old public link.
  • Bump the plug-in to v0.3, translate all plug-in comments to English (previously a mix of German/Finnish/English), update the plug-in's README/known-limitations docs, add a CHANGELOG entry.

Test plan

  • All 9 plug-in .lua files pass luac -p syntax check
  • Round-tripped the XMP embedding against a real JPEG: exiftool -G1 correctly reads back XMP-lumio:OriginalMD5 with no custom config needed
  • apps/worker/exif_meta.py's new extract_original_md5/extract_metadata verified against that same stamped JPEG; existing 13-test test_exif_meta.py suite still passes unmodified
  • Verified the exif JSONB merge (COALESCE(exif,'{}') || COALESCE(new,'{}')) against a throwaway Postgres instance: sets the value, leaves other exif keys untouched when no hash is given, and correctly overwrites only the lumio key on a second write
  • tsc --noEmit passes on apps/api with the updated plugin.ts
  • Manual end-to-end verification in a real Lightroom Classic instance (icon rendering, LrMD5.digest return format, re-publish flow, Selection-Import with a renamed file, the new Studio deep link) — I don't have a full LrC install in this environment, would appreciate a second pair of eyes here

🤖 Generated with Claude Code

…lection matching

Three fixes/improvements to the Lightroom Classic plug-in, found and
fixed while dogfooding it against a real LrC instance:

- Ship the publish service's icon (icon.png/icon@2x.png). It was
  referenced in code but never added to the plug-in folder, so it
  showed up blank/broken in Lightroom's Publishing Services panel.

- Stop duplicate files piling up on re-publish. uploadOnePhoto always
  created a new Lumio file and never removed the previous one, so
  editing a photo and republishing (or a manual "Republish") left
  several versions of the same photo in the online gallery. The
  plug-in now deletes the old remote file (via
  rendition.publishedPhotoId) before uploading the new one.

- Make Selection-Import matching survive renames and same-basename
  ambiguity. The Publish-Service now embeds the MD5 of the original
  master file into a custom XMP segment of the uploaded JPEG
  (JpegXmp.lua). The worker extracts it via the existing exiftool call
  and stores it under the already-present (and previously unused)
  exif.lumio.originalMd5 JSON field -- no schema migration needed.
  Selection-Import uses it to auto-resolve ambiguous filename matches
  and, with a new opt-in option, to recover files renamed after
  publishing.

Also: bump the plug-in to v0.3, translate all plug-in comments to
English (previously a mix of German/Finnish/English), and update the
plug-in's own README/known-limitations docs to match.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings August 31, 2026 16:11

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

There are correctness/documentation issues that should be addressed before merge (JSONB merge semantics for exif.lumio.*, misleading changelog wording, and JPEG XMP segment insertion position compatibility risk).

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR updates the Lightroom Classic plug-in and supporting server/worker plumbing to (1) ship the missing publish-service icon, (2) prevent duplicate remote files on re-publish by deleting the previously published remote file first, and (3) make Selection-Import matching robust to renames/ambiguous basenames by embedding + round-tripping an original-master MD5 via a custom XMP tag.

Changes:

  • Lightroom plug-in: add XMP-stamping (JpegXmp.lua) + hash-based disambiguation/rename recovery; delete old remote file on re-publish; bump plug-in version.
  • Worker/API: extract/surface originalMd5 (stored in File.exif.lumio.originalMd5) through plugin endpoints; update processing tasks to persist it.
  • Docs/changelog/readmes: document the new behavior and plug-in structure; translate plug-in comments to English in touched files.
File summaries
File Description
CHANGELOG.md Adds Unreleased notes for the plug-in icon, re-publish behavior, and hash-based matching.
apps/worker/tasks/process_raw.py Switches EXIF extraction to extract_metadata() and persists original_md5 when available.
apps/worker/tasks/process_file.py Switches EXIF extraction to extract_metadata() and persists original_md5 when available.
apps/worker/exif_meta.py Adds original-master MD5 extraction from custom XMP and a combined extract_metadata() helper.
apps/worker/db.py Extends mark_file_ready() to persist original_md5 into the existing exif JSONB field.
apps/lightroom-plugin/README.md Updates known limitations + folder structure to reflect hash matching and new files.
apps/lightroom-plugin/README.it.md Updates Italian README with the new hash-matching behavior and file list.
apps/lightroom-plugin/lumio.lrdevplugin/PluginManager.lua Translates/clarifies Plug-in Manager UI comments.
apps/lightroom-plugin/lumio.lrdevplugin/LumioPublishService.lua Embeds original MD5 into JPEG XMP and deletes previous remote file on re-publish.
apps/lightroom-plugin/lumio.lrdevplugin/LumioApi.lua Translates comments/messages and keeps retry logic; supports delete call used by re-publish.
apps/lightroom-plugin/lumio.lrdevplugin/Logger.lua Translates header comments.
apps/lightroom-plugin/lumio.lrdevplugin/Json.lua Translates header comments for embedded JSON helper.
apps/lightroom-plugin/lumio.lrdevplugin/JpegXmp.lua New module to inject an APP1 XMP segment containing OriginalMD5 into rendered JPEGs.
apps/lightroom-plugin/lumio.lrdevplugin/Info.lua Bumps plug-in version to 0.3 and updates documentation comments.
apps/lightroom-plugin/lumio.lrdevplugin/ImportSelectionTask.lua Adds hash-based disambiguation and opt-in rename recovery while applying selection to the catalog.
apps/lightroom-plugin/lumio.lrdevplugin/ImportSelectionDialog.lua Adds opt-in checkbox to enable hash-based recovery of renamed files.
apps/api/src/routes/plugin.ts Extracts and returns originalMd5 from exif.lumio.originalMd5 on plugin endpoints.
apps/api/prisma/schema.prisma Documents the exif.lumio.originalMd5 convention in the Prisma schema comments.
Review details

Suppressed comments (1)

CHANGELOG.md:43

  • This changelog bullet duplicates the same English sentence twice (plain + italic after ·). It looks like an accidental double entry and makes the section noisy.
- Lightroom plug-in: the Publish-Service now embeds the MD5 hash of the original master file into a custom XMP field of the uploaded JPEG, and the Selection-Import side uses it to automatically resolve ambiguous filename matches (e.g. the same shot published as both `.NEF` and `.DNG`) and, with a new opt-in option, to recover files that were renamed in Lightroom after publishing. Only available for files published from this plug-in version onward. · *The Lightroom plug-in's Publish-Service now embeds the original master file's MD5 hash into a custom XMP field of the uploaded JPEG, and Selection-Import uses it to automatically resolve ambiguous filename matches (e.g. the same shot published as both `.NEF` and `.DNG`) and, with a new opt-in option, to recover files renamed in Lightroom after publishing. Only available for files published from this plug-in version onward.*
  • Files reviewed: 18/20 changed files
  • Comments generated: 6
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread CHANGELOG.md Outdated
Comment on lines +32 to +34
A pull + regular redeploy is enough for the server/worker (the new
`originalMd5` column is picked up by the existing `prisma migrate deploy`
step). **Anyone using the Lightroom plug-in has to reinstall it:** the
Comment thread apps/worker/db.py Outdated
Comment on lines +86 to +89
SET status = 'ready', width = %s, height = %s,
sha256 = COALESCE(%s, sha256),
exif = COALESCE(exif, '{}'::jsonb) || COALESCE(%s::jsonb, '{}'::jsonb),
"updatedAt" = NOW()
Comment thread apps/worker/exif_meta.py
Comment on lines +179 to +183
def extract_original_md5(src_path: str) -> str | None:
"""MD5 of the ORIGINAL master file, as embedded by the Lightroom
plug-in's Publish-Service into a custom XMP field of the uploaded
JPEG. Only present for files published via that plug-in from this
feature onward; None for everything else -- never throws.
Comment on lines +118 to +121
Original-Masters im hochgeladenen JPEG hinterlegt. Damit werden
mehrdeutige Treffer automatisch aufgelöst, und mit der Option
„Rinominati per hash" lassen sich auch umbenannte Files wiederfinden.
Für älter veröffentlichte Files (oder Uploads über Browser/Upload-Link)
end
local segment = string.char(0xFF, 0xE1) .. u16be(segmentLen) .. payload

local newData = data:sub(1, 2) .. segment .. data:sub(3)
Comment on lines +276 to +296
local targets = {} -- md5 -> file
for _, file in ipairs(missingCandidates) do
if file.originalMd5 then targets[file.originalMd5] = file end
end
if next(targets) then
progress:setCaption("Recomputing hashes to find renamed files…")
for i, photo in ipairs(pool) do
if progress:isCanceled() then canceled = true break end
if i % 20 == 0 then progress:setPortionComplete(i, #pool) end
local okPath, path = LrTasks.pcall(function() return photo:getRawMetadata("path") end)
if okPath and path then
local okHash, hash = LrTasks.pcall(fileMd5, path)
if okHash and hash and targets[hash] then
local file = targets[hash]
resolved[file] = { photo }
targets[hash] = nil -- consumed, keep scanning for the rest
resolvedByHashCount = resolvedByHashCount + 1
end
end
end
end
… public gallery

goToPublishedCollection built the public customer-facing link
(host + /g/<slug>). A photographer clicking "Show in Lumio" from
Lightroom wants to manage the gallery, not see the customer's view, so
it now opens the Studio management page (host + /studio/<galleryId>)
instead. The Studio route is keyed by gallery id, not slug, so this
also drops the now-unneeded slug-from-cache lookup and the gallerySlug
tracking in processRenderedPhotos that only existed to support it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@markusthiel

Copy link
Copy Markdown
Owner

Big one, thanks. The XMP route for the hash is a good call — solving the rename problem without a schema migration is better than the column I'd have reached for.

Two of the Copilot findings I'd want fixed before merge:

APP1 before APP0. Inserting the XMP segment directly after SOI when a JFIF APP0 follows can trip strict readers. This one touches real customer files that go on to print labs and third-party software, so the conservative order seems worth it: if the first segment is APP0, insert after it, otherwise keep the current behaviour. Unless you've already checked how the labs' intake handles it and it's a non-issue — you'd know better than me there.

targets collision in the rename-recovery pass. A md5 -> file map means only one file survives per hash. Virtual copies of the same master aren't unusual in a Lightroom catalog, so this would silently skip work in a fairly normal setup. md5 -> [files] fixes it.

The third one I read differently, and I'd like your view. Copilot says the db.py docstring's claim is wrong, but COALESCE(exif,'{}') || patch is a shallow top-level merge — other exif keys do survive, which is what the docstring actually claims. The concern only bites once lumio holds more than one field, and today it holds exactly one. My instinct is a line in the docstring saying a second lumio.* field would need jsonb_set, rather than rewriting the query now. Do you see a near-term second field coming? If you do, better to do it properly in this PR than to leave a trap.

Three smaller things:

  • The CHANGELOG wording about a new column being picked up by prisma migrate deploy — Copilot is right that it's misleading, since there's no migration here. Worth correcting so self-hosters don't go looking for one.
  • apps/lightroom-plugin/README.md still has the Italian option label ("Rinominati per hash") in the German text.
  • mark_file_ready now writes exif = COALESCE(exif,'{}') || '{}' on every call without a hash, so files that previously kept exif NULL end up with {}. Harmless as far as I can tell — auto_tag.py reads it defensively — but it's a quiet change on every processed file, so I'd rather have it in the PR text than discover it in six months.

A worker test for extract_original_md5 would be welcome too, along the lines Copilot suggested. The XMP tag name is the kind of thing that breaks silently.

On the "Show in Lumio" change: I follow the reasoning and mostly agree a photographer clicking it wants the management view. One thing gives me pause — the studio link needs an active studio session in the browser, where the public link worked regardless, and there are moments where you genuinely want to see what the client sees. Not asking you to revert it; what did you weigh there? If you considered keeping the customer link reachable and decided against, I'm happy to take your read on it.

The end-to-end check in a real Lightroom Classic is the part I don't want to skip. @canja006, you were in the plug-in loading path in #13 — any chance you can run the icon, re-publish and Selection-Import flows against your install?

@canja006

Copy link
Copy Markdown
Contributor

Happy to — I'll run the end-to-end flows against my install (Lightroom Classic on macOS + self-hosted instance): plugin icon, publish → re-publish (anti-duplicate), and Selection-Import both normally and against a renamed file to exercise the XMP hash path. I'll wait for the two fixes above to land first and expect to have results within a day of the push.

- JpegXmp.lua: insert the XMP hash segment after a leading APP0/JFIF
  segment when present, instead of unconditionally right after SOI.
  Verified byte-for-byte against a real JPEG with and without a
  leading APP0.
- ImportSelectionTask.lua: the rename-recovery hash map was
  hash -> file (single), silently dropping every file after the first
  one sharing a hash -- normal when virtual copies of one master were
  all published. Now hash -> list of files, all resolved to the same
  matched photo.
- LumioPublishService.lua: add goToPublishedPhoto (per-photo "Show
  public gallery"), opening the public customer-facing gallery link.
  Complements the collection-level "Show in Lumio" now pointing at the
  Studio view, for when you do want to see the client's side -- and
  doesn't need an active Studio session the way the Studio link does.
  Restores the gallerySlug tracking this needs.
- db.py: mark_file_ready no longer touches the exif column at all when
  there's no original_md5 to write (previously flipped NULL to {} on
  every file). When there is one, it's written via jsonb_set over the
  existing 'lumio' object instead of a flat merge, so a future second
  field under exif.lumio survives being written alongside originalMd5.
  Verified both the previous silent-{} behaviour and the new jsonb_set
  merge against a throwaway Postgres instance.
- README.md (DE): fixed an Italian checkbox label ("Rinominati per
  hash") accidentally left in the German option description.
- Added worker tests for extract_original_md5/extract_metadata,
  including one asserting a single exiftool call for both values.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@manuzzi

manuzzi commented Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the thorough read-through. Pushed 822ed66 addressing everything:

APP1/APP0 ordering — fixed: JpegXmp.lua now inserts after a leading APP0/JFIF segment when present, falling back to right-after-SOI otherwise. Verified byte-for-byte (segment offsets dumped manually) against a real JPEG both with and without a leading APP0, plus exiftool confirming the tag still reads back correctly in both cases.

targets collision — fixed with exactly the md5 -> [files] you suggested: every Lumio file sharing a hash (virtual copies of one master will all hash identically) now resolves to the same matched candidate photo, instead of only the last one registered.

db.py docstring / second lumio.* field — there is a near-term second field planned, so rather than leave a note for later I implemented the proper fix now: exif is updated via jsonb_set over the existing lumio object (COALESCE(exif->'lumio','{}') || jsonb_build_object('originalMd5', ...), written back at the one-level path {lumio}) instead of a flat top-level merge. A second lumio.* field will survive an originalMd5 write and vice versa. Verified against a throwaway Postgres instance, including the specific case of an existing sibling field under lumio surviving an originalMd5 update.

CHANGELOG wording — this one was already fixed in b335d0b, before this review ran (Copilot's finding predates that push). Re-checked, current wording is accurate: no migration, reuses the existing exif field.

README Italian label — fixed, replaced with the actual German option text.

exif NULL -> {} on every file — took the stricter option here instead of just documenting it: mark_file_ready now skips the exif clause entirely when there's no original_md5 to write, so every file that doesn't go through the Lightroom plug-in keeps exif exactly as it was (including staying NULL). Zero behavior change for the vast majority of files.

Worker test — added test_original_md5_read_back, _absent_returns_none, _malformed_value_rejected, _is_lowercased, _missing_file_returns_none, and one asserting extract_metadata() makes a single exiftool call (not two) by counting subprocess.run invocations. All 19 tests in test_exif_meta.py pass.

"Show in Lumio" -> Studio — to answer directly: this was an explicit request from the project owner, not something I inferred on my own. Your underlying concern (needs an active Studio session; sometimes you want the client's view) is fair though, so rather than just explain it away I added goToPublishedPhoto — a new per-photo "Show public gallery" entry (right-click a published photo) that opens the public /g/<slug> link, no Studio session required. So: right-click the collection for management, right-click a photo for the customer's view.

@canja006 the two blocking fixes are in, whenever you get a chance to run the LrC flows.

@manuzzi

manuzzi commented Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

Further explanation for the Show in Lumio:
I personally prefer to be able to get fast to the gallery management, I find this very useful in other provider's plugin like Pixieset as I find it very useful to manage passwords and links after the uploads.
My typical workflow is:

  1. Upload the photos to Lumio
  2. Open gallery on Lumio in Studio
  3. Set links and passwords
  4. Enable the gallery
  5. Send the link via mail/whatsapp

This is just my workflow, for sure I'm bringing some influence from Pixieset

@canja006

canja006 commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Ran the end-to-end flows against my install — Lightroom Classic 15.5.1 on macOS, self-hosted Lumio on v0.74.1 with this PR at 822ed66, worker rebuilt so the exiftool extraction is live. Catalog is 2774 photos, mostly NEF/DNG.

Five of the six work. The sixth could not have worked on any machine.

Icon — renders in the Publishing Services panel.

Anti-duplicate re-publish — republished a 70-file gallery. Per photo the order is uploads/init → S3 PUT → uploads/complete → DELETE <old id>, so the new file is safely up before the old one goes. Afterwards the gallery still holds 70 files, and grouping originalFilename per gallery across the whole instance returns no duplicates.

The XMP hash round trip — all 70 files carry exif.lumio.originalMd5, every value matches ^[0-9a-f]{32}$, and all 70 are distinct. LrMD5.digest returns lowercase hex, so that worry was unfounded.

Selection import by filename — 70/70, both with and without the hash option.

Selection import after a rename — renamed a published photo in the catalog, then: hash off → Imported for 68 of 70 with both unmatched names listed; hash on → Imported for 70 of 70, 2 renamed file(s) were recovered by content hash. The feature does what it claims.

"Show in Lumio"/studio/<galleryId>. Correct.

"Show public gallery" never reaches the collection settings

goToPublishedPhoto always fails with "Gallery slug or host is missing". I logged what the SDK actually passes:

info keys: photo=table, publishService=table, publishedCollectionInfo=table,
           publishedPhoto=table, remoteId=string
publishedCollectionInfo keys: isDefaultCollection=boolean, name=string, parents=table

There is no publishedCollection object, and publishedCollectionInfo carries no collectionSettings — so collSettings is {} and resolveGallerySlug returns nil on its first line.

Worth stressing that this is not a slug-availability problem: the gallery cache held the right entry (1b57f711… → 8qw6h3vkcxug) and the collection settings did hold gallerySlug. Neither could be reached.

LrPublishedPhoto has no getPublishedCollection() either — attempt to call method 'getPublishedCollection' (a nil value). The only route I found is through the photo, and it has to run in an async task, because getContainedPublishedCollections() yields and yielding inside a plain pcall throws "Yielding is not allowed within a C or metamethod call".

function exportServiceProvider.goToPublishedPhoto(publishSettings, info)
    local LrHttp = import "LrHttp"
    info = info or {}

    -- Reading the catalog yields, and yielding is not allowed inside a plain
    -- pcall (a C call), so this runs in an async task with LrTasks.pcall.
    LrTasks.startAsyncTask(function()
        -- publishedCollectionInfo carries only {isDefaultCollection, name,
        -- parents}, and LrPublishedPhoto has no getPublishedCollection(),
        -- so the settings have to come from the photo.
        local wantedName = (type(info.publishedCollectionInfo) == "table")
                           and info.publishedCollectionInfo.name or nil

        local collSettings = {}
        if info.photo then
            local ok, colls = LrTasks.pcall(function()
                return info.photo:getContainedPublishedCollections()
            end)
            if ok and type(colls) == "table" then
                local fallback
                for _, coll in ipairs(colls) do
                    local okn, nm = LrTasks.pcall(function() return coll:getName() end)
                    local oks, summary = LrTasks.pcall(function()
                        return coll:getCollectionInfoSummary()
                    end)
                    local st = oks and summary and summary.collectionSettings or nil
                    if type(st) == "table" and st.galleryId and st.galleryId ~= "" then
                        if okn and wantedName and nm == wantedName then
                            collSettings = st
                            break
                        end
                        fallback = fallback or st
                    end
                end
                if next(collSettings) == nil and fallback then collSettings = fallback end
            end
        end

        local host = (LrPrefs.prefsForPlugin().host or ""):gsub("/+$", "")
        local slug = resolveGallerySlug(collSettings)
        if not slug or slug == "" or host == "" then
            LrDialogs.message("Lumio",
                "Gallery slug or host is missing. Publish the collection once, " ..
                "or set the server address in Plug-in Manager.", "warning")
            return
        end
        LrHttp.openUrlInBrowser(host .. "/g/" .. slug)
    end)
end

With that, the log reads contained published collections: 1 / galleryId=1b57f711… gallerySlug=8qw6h3vkcxug, and the browser opens /g/8qw6h3vkcxug.

The hash pass is expensive by default

Same import, same result, two scopes:

Scope Time
matchScope = "collection" (70 photos) 3.8 s
matchScope = "library" (2774 photos) — the default 4 min 12 s

Lightroom became visibly sluggish during the second one and memory climbed throughout. Two causes:

  1. The scan never exits early. targets[hash] = nil consumes a match, but the loop keeps walking the rest of the pool. Both files I was looking for were found early; it then hashed the remaining ~2700 photos for nothing. if not next(targets) then break end after consuming would have ended it in seconds.
  2. fileMd5 does f:read("*all") — every master is read into memory whole. For RAW that is 25–80 MB per allocation, repeated across the catalog.

Photos already resolved by filename in pass 1 are hashed again too.

Since "library" is the default and the checkbox only says "(slower)", the first person to tick it on a real catalog will get the four-minute version.

Two small things

  • Logger.lua's header says the log lands in ~/Documents/LrClassicLogs/. On LrC 15.5 / macOS it is ~/Library/Logs/Adobe/Lightroom/LrClassicLogs/Lumio.log.
  • gallerySlug is only written into the collection settings in the create-a-new-gallery branch (LumioPublishService.lua:517-527). A collection bound to an existing gallery never gets one, leaving goToPublishedPhoto dependent on galleryCacheJson — which refreshes only when the "Edit Lumio Gallery" dialog is opened, so it goes stale the moment a gallery is created in Studio. Mine was stale exactly that way. Not the cause of the failure above, but it would be the next one.

… speed up rename recovery

Addresses the real end-to-end test report from @canja006 on PR markusthiel#27
(LrC 15.5.1, 2774-photo catalog).

- goToPublishedPhoto ("Show public gallery") always failed: it assumed
  info.publishedCollectionInfo had a .collectionSettings field like
  getCollectionInfoSummary() elsewhere in this file. On a real install
  it only ever carries {isDefaultCollection, name, parents}, and
  LrPublishedPhoto has no getPublishedCollection() either. Adopts
  @canja006's tested fix: fetch the collection through
  photo:getContainedPublishedCollections() instead, wrapped in
  LrTasks.startAsyncTask since that call yields.

- gallerySlug was only ever captured when a NEW gallery was created
  from the plug-in dialog; a collection bound to an EXISTING gallery
  (picked from the dropdown) never got one, leaving the public-gallery
  link dependent on a cache that goes stale as soon as the gallery is
  edited in Studio. processRenderedPhotos now self-heals it (same
  proven catalog:withWriteAccessDo pattern already used for gallery
  creation) on the next publish to any such collection.

- Rename-recovery hash pass took 4m12s on the full 2774-photo catalog
  (the default scope) vs 3.8s on a 70-photo collection, with Lightroom
  visibly sluggish and memory climbing. Three fixes: stop scanning once
  every target hash is found (it never exited early before), skip
  photos pass 1 already matched by filename (they were hashed again
  for nothing), and -- the structural fix -- embed the original
  master's byte size alongside its MD5 (free: the file is already
  fully in RAM to hash it) so a rename-recovery candidate can be ruled
  out with a plain LrFileUtils.fileAttributes stat instead of being
  fully read into memory. Exact byte-size collisions between different
  photos are rare; identity is still always confirmed by the real hash.
  exif_meta.py/db.py/plugin.ts extended to carry originalSize alongside
  originalMd5 through the same pipeline, same reasoning as the
  original hash (no schema change, jsonb_set-based merge).

- Logger.lua/README: corrected the macOS log path for current
  Lightroom Classic (confirmed on 15.5: ~/Library/Logs/Adobe/Lightroom/
  LrClassicLogs/, not ~/Documents/LrClassicLogs/).

Verified: XMP round-trip with the new OriginalSize field against a
real JPEG + exiftool (with and without a size, matching the existing
OriginalMD5 test pattern); worker db.py's extended jsonb_set query
against a throwaway Postgres instance including the actual
mark_file_ready() function end-to-end (exif=NULL populated with both
fields when given, an existing exif with sibling keys left completely
untouched when no hash is given); 4 new + all 19 existing worker tests
pass; tsc --noEmit clean; luac -p clean on all 9 plug-in files.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@manuzzi

manuzzi commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

@canja006 thank you for this — an exceptionally thorough report, and exactly the kind of real-device signal I can't produce here. Pushed 052b0f9 addressing all of it:

goToPublishedPhoto — adopted your fix essentially as-is (credited in the code and CHANGELOG): fetch the collection through photo:getContainedPublishedCollections() inside LrTasks.startAsyncTask, matching on publishedCollectionInfo.name when a photo belongs to more than one collection. My original assumption about publishedCollectionInfo having .collectionSettings was simply wrong, as your logging showed conclusively.

gallerySlug stalenessprocessRenderedPhotos now self-heals it: when a collection is bound to an existing gallery and gallerySlug is still missing, it resolves it (same cache fallback) and persists it via the same withWriteAccessDo pattern already proven for newly-created galleries. Self-heals on the very next publish.

The hash-pass performance — went with a structural fix rather than just the two obvious patches. Now:

  1. Stops scanning the pool the moment every target hash is found (previously it kept going regardless).
  2. Skips photos pass 1 already matched by filename.
  3. The one that actually matters at scale: LumioPublishService.lua now embeds the original master's byte size alongside its MD5 (free — the file's already fully in RAM to hash it), and ImportSelectionTask.lua checks a candidate's size via a plain LrFileUtils.fileAttributes stat before reading+hashing it. Only size-matching candidates get the expensive full read. Your 2774-photo / 25-80MB-RAW numbers are exactly the case this targets — a renamed file's candidates get found by a stat scan, not a content-read scan, and identity is still always confirmed by the real hash, never by size alone.

This does mean originalMd5/originalSize from an OLD-plug-in-version publish won't have a size to filter on — those candidates just don't benefit from the pre-filter (no correctness impact, only no speedup for that specific file).

Logger.lua/README log path — corrected to ~/Library/Logs/Adobe/Lightroom/LrClassicLogs/ per your confirmation on 15.5.

Verified from this end: XMP round-trip with the new OriginalSize field against a real JPEG + exiftool; the extended jsonb_set query against a throwaway Postgres instance running the actual mark_file_ready() (confirmed an existing exif with unrelated sibling keys survives untouched when no hash is given, and gets both fields when one is); 4 new worker tests plus all 19 previous ones pass; tsc/luac -p clean.

Whenever you get a chance: would appreciate a re-run of goToPublishedPhoto, confirming gallerySlug self-heals on a collection bound to an existing gallery, and timing matchScope = "library" again on the same 2774-photo catalog to see where it lands now.

Scoped to the Python files touched by this PR (db.py, process_file.py,
process_raw.py, test_exif_meta.py) -- exif_meta.py was already in
English. Comments only, no functional changes.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@canja006

canja006 commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Re-ran everything against 052b0f9 — Lumio rebuilt from that commit (api + worker), plug-in reinstalled from the branch, Lightroom restarted so the new Lua actually loaded. Same machine, same 2774-photo catalog as before.

All three hold up.

goToPublishedPhoto — opens the public gallery. Fixed.

gallerySlug self-heal — tested on the case it was written for: a collection bound to a gallery that was created in Studio (2026-08-15) and never published to, so the create-a-new-gallery branch had never run for it and no slug had ever been written. Published three photos; the catalog's collection settings now carry gallerySlug = "atgkt32f86yf". Self-heals as described, on the first publish.

Rename-recovery timing — same catalog, same matchScope = "library", same checkbox:

822ed66 052b0f9
hash off (baseline) 0.5 s 0.6 s
hash on, whole catalog 4 min 12 s 1.44 s

Both correct: 69 of 70 with the option off and the renamed file listed, 70 of 70 and 1 renamed file(s) were recovered by content hash with it on. Lightroom stayed responsive throughout and memory did not climb — the size pre-filter is doing exactly what you designed it to.

originalSize coverage is 70/70 in that gallery after a single republish. The old-publish caveat you flagged is visible in another gallery here — 60 files, 1 with originalMd5, none with a size — and behaves as you said: no speedup for those, no correctness impact.

One thing worth knowing for anyone else testing this on a real install: the plug-in file was in place on disk at 17:55, but Lightroom was still executing the previously loaded copy at 20:00 — I could only tell because my own build had logging in it that yours doesn't. A full Lightroom restart at 20:06 picked up the new file. Two of my earlier results were against stale in-memory Lua before I caught it.

Nothing outstanding from my side.

@manuzzi

manuzzi commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @canja006 for all the tests.
Regarding the very last comment about Lr plugin version:
This is a known (at least for me) issue on LrC for all the plugin, I've found out that even if you reload the new plugin through LrC plugin interface the behaviour is not guaranteed, therefore a fully app restart is needed.
I think we should mention this on the plugin readme, what do you think @markusthiel ?

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

There are a couple of correctness issues in the new hash-recovery path and tests that should be fixed before merging.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details
  • Files reviewed: 19/21 changed files
  • Comments generated: 3
  • Review effort level: Lite

Comment on lines +321 to +345
local sizesToCheck = {}
for _, file in ipairs(missingCandidates) do
if file.originalMd5 then
targets[file.originalMd5] = targets[file.originalMd5] or {}
table.insert(targets[file.originalMd5], file)
if file.originalSize then
sizesToCheck[file.originalSize] = true
end
end
end
if next(targets) then
progress:setCaption("Recomputing hashes to find renamed files…")
for i, photo in ipairs(pool) do
if progress:isCanceled() then canceled = true break end
if not usedPhotos[photo] then
if i % 20 == 0 then progress:setPortionComplete(i, #pool) end
local okPath, path = LrTasks.pcall(function() return photo:getRawMetadata("path") end)
if okPath and path then
local attrs = LrFileUtils.fileAttributes(path)
local size = attrs and attrs.fileSize
-- Cheap pre-filter (see PERFORMANCE above):
-- only hash candidates whose size could
-- possibly match a hash we're still after.
if size and sizesToCheck[size] then
local okHash, hash = LrTasks.pcall(fileMd5, path)
Comment thread CHANGELOG.md Outdated
Comment on lines +43 to +59
- Lightroom plug-in: the Publish-Service now embeds the MD5 hash of the original master file into a custom XMP field of the uploaded JPEG, and the Selection-Import side uses it to automatically resolve ambiguous filename matches (e.g. the same shot published as both `.NEF` and `.DNG`) and, with a new opt-in option, to recover files that were renamed in Lightroom after publishing. Only available for files published from this plug-in version onward. · *The Lightroom plug-in's Publish-Service now embeds the original master file's MD5 hash into a custom XMP field of the uploaded JPEG, and Selection-Import uses it to automatically resolve ambiguous filename matches (e.g. the same shot published as both `.NEF` and `.DNG`) and, with a new opt-in option, to recover files renamed in Lightroom after publishing. Only available for files published from this plug-in version onward.*

### Changed

- Lightroom plug-in: "Show in Lumio" (right-click a published collection) now opens the gallery's Studio management view instead of the public customer-facing gallery link. A new per-photo "Show public gallery" entry (right-click a published photo) covers the case where you do want the customer's view. · *"Show in Lumio" (right-click a published collection) now opens the gallery's Studio management view instead of the public customer-facing gallery link. A new per-photo "Show public gallery" entry (right-click a published photo) covers the case where you do want the customer's view.*

### Fixed

- Lightroom plug-in: the publish service's icon was referenced but never shipped, so it showed up blank/broken in Lightroom's Publishing Services panel. · *The publish service's icon was referenced but never shipped, so it showed up blank/broken in Lightroom's Publishing Services panel.*
- Lightroom plug-in: re-publishing a photo (after an edit, or via "Republish") uploaded a new file without ever removing the previous one, so the online gallery accumulated several versions of the same photo. The plug-in now deletes the old remote file before uploading the new one. · *Re-publishing a photo (after an edit, or via "Republish") uploaded a new file without ever removing the previous one, so the online gallery accumulated several versions of the same photo. The plug-in now deletes the old remote file before uploading the new one.*
- Lightroom plug-in: the embedded original-file hash could land before a leading JFIF (APP0) segment in the uploaded JPEG, which some strict readers (print lab intake, some third-party tools) don't tolerate. It's now inserted after APP0 when one is present. · *The embedded original-file hash could land before a leading JFIF (APP0) segment in the uploaded JPEG, which some strict readers (print lab intake, some third-party tools) don't tolerate. It's now inserted after APP0 when one is present.*
- Lightroom plug-in: the renamed-file recovery pass could silently skip files when several Lumio uploads shared the same original-file hash (e.g. virtual copies of one master) — only one of them was ever attempted. All of them are now resolved. · *The renamed-file recovery pass could silently skip files when several Lumio uploads shared the same original-file hash (e.g. virtual copies of one master) — only one of them was ever attempted. All of them are now resolved.*
- Worker: processing a file with no original-file hash to report (i.e. almost every file, since only the Lightroom plug-in ever produces one) was quietly turning `exif` from `NULL` into `{}`. The `exif` column is now left untouched when there's nothing to write into it. · *Processing a file with no original-file hash to report (i.e. almost every file, since only the Lightroom plug-in ever produces one) was quietly turning `exif` from `NULL` into `{}`. The `exif` column is now left untouched when there's nothing to write into it.*
- Lightroom plug-in: the per-photo "Show public gallery" entry always failed ("Gallery slug or host is missing"), because the collection info Lightroom actually hands to that hook has no `collectionSettings` field to read from. Fixed by fetching the collection's settings through the photo instead (credit: [@canja006](https://github.com/canja006), tested against a real Lightroom Classic install). · *The per-photo "Show public gallery" entry always failed ("Gallery slug or host is missing"), because the collection info Lightroom actually hands to that hook has no `collectionSettings` field to read from. Fixed by fetching the collection's settings through the photo instead (credit: [@canja006](https://github.com/canja006), tested against a real Lightroom Classic install).*
- Lightroom plug-in: a published collection bound to an *existing* Lumio gallery (picked from the dropdown, rather than created from the plug-in) never had its gallery slug saved, leaving the public-gallery link dependent on a cache that goes stale as soon as the gallery is edited in Studio. It now self-heals on the next publish. · *A published collection bound to an existing Lumio gallery (picked from the dropdown, rather than created from the plug-in) never had its gallery slug saved, leaving the public-gallery link dependent on a cache that goes stale as soon as the gallery is edited in Studio. It now self-heals on the next publish.*
- Lightroom plug-in: recovering renamed files by content hash took over 4 minutes on a 2774-photo catalog (the default search scope), with Lightroom visibly sluggish throughout — measured on real hardware. The pass now stops as soon as every renamed file has been found, skips photos already matched by filename, and, most importantly, embeds the original file's byte size alongside its hash so most catalog photos can be ruled out with a plain file-size check instead of being fully read into memory just to be hashed. · *Recovering renamed files by content hash took over 4 minutes on a 2774-photo catalog (the default search scope), with Lightroom visibly sluggish throughout — measured on real hardware. The pass now stops as soon as every renamed file has been found, skips photos already matched by filename, and, most importantly, embeds the original file's byte size alongside its hash so most catalog photos can be ruled out with a plain file-size check instead of being fully read into memory just to be hashed.*
- Lightroom plug-in: the log-file location documented for macOS was wrong for current Lightroom Classic versions (confirmed on 15.5: `~/Library/Logs/Adobe/Lightroom/LrClassicLogs/Lumio.log`, not `~/Documents/LrClassicLogs/`). · *The log-file location documented for macOS was wrong for current Lightroom Classic versions (confirmed on 15.5: `~/Library/Logs/Adobe/Lightroom/LrClassicLogs/Lumio.log`, not `~/Documents/LrClassicLogs/`).*
Comment thread apps/worker/tests/test_exif_meta.py Outdated
Comment on lines +92 to +94
data = open(path, "rb").read()
with open(path, "wb") as f:
f.write(data[:2] + segment + data[2:])
@markusthiel

Copy link
Copy Markdown
Owner

Thanks both — @canja006, that report was worth more than any amount of review from my side, and the goToPublishedPhoto fix especially.

Copilot re-reviewed on the 8th, five days after the last push, so three things are still open. One of them matters.

The size pre-filter breaks recovery for targets without originalSize. Not just a missing speedup — sizesToCheck is only filled from files that carry a size, and the scan gate is if size and sizesToCheck[size]. A target with originalMd5 and no size contributes nothing to the set, so its renamed file never gets hashed and targets[hash] is never reached. It fails silently, where 822ed66 was slow but correct.

Small blast radius, since originalMd5 only exists as of this PR — in practice it's files published by an intermediate build of this branch. But the fix is a couple of lines: set a flag when any target lacks a size and skip the gate in that case, keeping the fast path for the normal case where every target has one.

Worth flagging how this slipped through: @canja006 had the exact case in hand (60 files, one hash, no sizes) and reported it as behaving as described — but that was reading the data, not running a recovery in that gallery. Easy trap, and my own review missed it too.

CHANGELOG is meant to be German + English. The entries here are English, then ·, then the same English again in italics. The shape is right, the first half should be German. It matters beyond tidiness: scripts/release-notes.mjs extracts the English half for the release page, so as it stands the notes would come out duplicated.

Third, trivial: open(...).read() without a context manager in _stamp_original_md5.

On the restart question — yes, please put it in the plug-in README. It cost two of @canja006's results here, and the next person testing a plug-in change will hit the same thing. Worth being blunt about it: a full Lightroom restart, "Reload" in the Plug-in Manager isn't enough.

Once those are in and @canja006 has had a look at the recovery case specifically, this is good to merge from my side.

@canja006

Copy link
Copy Markdown
Contributor

Fair point — that was read off the data, not run. I have the exact case here (one file with originalMd5 and no originalSize, gallery of 63), so once the fix lands I'll rename that master, run the import with hash matching on and library scope, and report whether it comes back recovered. Full Lightroom restart between builds, as before.

Three findings from Copilot's 2026-09-08 review on PR markusthiel#27, left open
by the previous fix commit:

- ImportSelectionTask.lua: the rename-recovery size pre-filter added
  in the last commit skipped hashing any candidate whose size didn't
  match sizesToCheck. But sizesToCheck is only populated from targets
  that carry an originalSize -- a target with originalMd5 but no
  originalSize (older embeddings, from before this PR's last commit
  added size) was silently unrecoverable, since no candidate size
  would ever match it. Now tracked via sizelessRemaining: the size
  filter only excludes a candidate once every size-less target has
  already been resolved.

- CHANGELOG.md: every [Unreleased] bullet duplicated its English
  sentence a second time in italics after " · ". Every other version
  section in this file (0.74.1, 0.74.0, 0.73.5, ...) uses a single
  plain-English sentence -- the bilingual " · *...*" format is only
  ever used in the file's header/preamble, never in a version's own
  entries. Stripped the duplicate from all 9 affected bullets.

- test_exif_meta.py: _stamp_original_md5 read the JPEG via
  open(path, "rb").read() without closing the handle. Wrapped in a
  context manager.

Verified: luac -p on ImportSelectionTask.lua; full worker test suite
(23/23) passes unmodified.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@manuzzi

manuzzi commented Sep 10, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the remaining findings from Copilot's 2026-09-08 review:

  • ImportSelectionTask.lua: fixed the rename-recovery pre-filter -- candidates whose originalMd5 has no matching originalSize (older embeddings, before this PR's last commit added size) are no longer silently unrecoverable. The size pre-filter now only kicks in once every size-less target has been resolved.
  • CHANGELOG.md: removed the duplicated English sentence (plain + italic) from every [Unreleased] bullet -- matches the single-sentence format used by every other version section in this file.
  • test_exif_meta.py: closed the file handle in _stamp_original_md5 (open(...).read() -> context manager).

Verified: luac -p on the changed Lua file, full worker test suite (23/23) still green.

@canja006

Copy link
Copy Markdown
Contributor

Ran it, before and after, on the exact case.

Setup: the gallery of 63 files, one of them (DSC_3634-2.jpg) carrying exif.lumio.originalMd5 with no originalSize — embedded by an intermediate build of this branch. Renamed that master in Lightroom so the filename pass can't find it, then imported with "recover renamed files by content hash" on, library scope, 2774-photo catalog. Full Lightroom restart between the two runs.

Build Result Time
7d04d7c (pre-fix gate) Imported for 62 of 63 filesNot found: • DSC_3634-2.jpg 1.4 s
e41e987 Imported for 63 of 63 files + 1 renamed file(s) were recovered by content hash 38.8 s

So it reproduced exactly as described — silently unrecoverable, and fast precisely because it never hashed anything — and sizelessRemaining fixes it.

The 38.8 s is worth recording next to the 1.44 s from the all-sizes case: while a size-less target is outstanding the pre-filter is off, so the scan is back to reading candidates in full, and it ends as soon as that target is found. Correct-then-fast rather than fast-then-wrong, which is the right way round, and it only costs anything for files published by an intermediate build.

Two things about my test case rather than your code, for the record:

  • My catalog holds a second master that is byte-identical to the renamed one (same MD5), so two photos could satisfy that hash and this run can't have distinguished them. It doesn't change the pass/fail — the target was recovered either way — but it's the situation your md5 -> [files] comment describes, seen from the other side: one hash, several candidates. Recovery resolves to whichever the pool reaches first.
  • This gallery carries no picks or ratings, so what the run verifies is the matching, not the writing of selection state. The recovery line comes from the matcher, which is the part that was broken.

Nothing else outstanding from me on this one.

@manuzzi

manuzzi commented Sep 15, 2026

Copy link
Copy Markdown
Contributor Author

@canja006 thank you for running the exact case, before and after — that table is exactly the confirmation this needed. The byte-identical-second-master note is a fair callout on your test setup rather than the code: pass/fail didn't depend on which of the two identical files got matched, and that's the "one hash, several candidates" situation the md5 -> [files] comment already accounts for.

Nothing outstanding from my side either. @markusthiel this one's fully closed out now — two rounds of Copilot findings addressed, three rounds of real-device testing from canja006, all clean. Ready whenever you get to it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Lightroom plug-in: broken publish icon, duplicate files on re-publish, fragile filename-only matching

4 participants