Add DriveThruCards Integration#367
Draft
bwsinger wants to merge 76 commits into
Draft
Conversation
- Automate login, product setup, PDF upload, and checkout navigation - Fix card dimensions to 2.73" x 3.71" for Premium Euro Poker with bleed - Use Ghostscript -dNOSAFER flag for PDF/X-1a conversion - Add placeholder cover image for product form - Replace sleep timers with dynamic element waits
Author
bwsinger
marked this pull request as draft
January 30, 2026 07:56
Collaborator
|
thanks - i haven't looked at the code yet but this sounds exciting! i'll look into the PDF formatting issue a bit and will report back if i find anything. not very keen to bundle/install a third-party command tool alongside autofill, i'd much prefer for the autofill binary to be totally standalone. |
JPEG has a 64KB limit for XMP data in a single APP1 segment. Some source images have XMP data exceeding this limit, causing "XMP data is too long" errors when saving. Since XMP is purely metadata and doesn't affect the visual content, we strip it before saving.
- Replace standard selenium Chrome driver with undetected-chromedriver - Add aggressive polling (500ms) for Cloudflare challenge detection - Add auto-click attempt for Turnstile checkbox in iframe - Fix login modal selector to target correct link - Remove set_network_conditions calls (potential detection vector) - Add setuptools dependency for Python 3.13 compatibility
- Remove unused ChromeOptions import - Add _detect_chrome_version() to query installed Chrome version - Support macOS, Windows, and Linux version detection - Pass detected version to undetected-chromedriver to fix version mismatch
- Add JavaScript patches to hide automation traces (webdriver property, chrome object, plugins, languages, ChromeDriver detection variables) - Add --disable-blink-features=AutomationControlled flag - Remove ineffective focus workaround that caused unnecessary delays
…ring polling Selenium's implicit wait (5s) was causing long delays between solving the Cloudflare CAPTCHA and clicking the login button. Fixed by temporarily disabling implicit wait in polling methods: - _try_click_turnstile_checkbox() - _is_cloudflare_challenge_active() - _is_site_loaded() - is_dtc_user_authenticated() - click_element_polling()
These methods were part of a failed attempt to bypass bot detection by triggering visibility/focus events. They are no longer called: - _simulate_human_behavior() - _trigger_visibility_change_via_minimize() - _perform_tab_switch_workaround() - _perform_login_focus_workaround() Also added commit/cleanup workflow guidance to CLAUDE.md.
The 'Click here after uploading your files' button starts with an onclick handler that shows an error and returns false. After server-side upload validation, the page JS replaces this with the actual form submit handler. Previously the click fired before this replacement, causing the page to not navigate.
fill_dtc_product_form, submit_dtc_description_page, open_dtc_upload_page and select_card_type_and_upload_pdf previously reduced missing required elements (title, price, cover, filters, submit, file input, upload confirmation, continue/complete/buy-now) to warnings and returned early, so 'order setup complete!' printed even when no PDF was uploaded. These failures now raise and abort the order with step context. Adds negative fake-DOM tests asserting each step aborts instead of reporting success.
The previous Ghostscript invocation used bare -dPDFX (PDF/X-3 semantics) with no definition file, so the output was a plain CMYK PDF 1.3 that was reported as PDF/X-1a. Now generates a PDF/X-1a definition file (per Ghostscript's lib/PDFX_def.ps) declaring GTS_PDFXVersion/Conformance, Trapped and a US Web Coated (SWOP) output intent with the ICC profile embedded when available, invokes gs with -dPDFX=1, and preflights the output for the PDF/X-1a markers before accepting it. Verified against Ghostscript 10.07.1 (real-gs test included; gs adds the required ArtBox automatically).
Adobe's Color Profile License permits embedding the profile in image files but forbids redistributing it bundled with application software, so remove USWebCoatedSWOP.icc and the EULA PDF from the repo. At runtime the tool now finds an installed copy (Adobe/ColorSync/system colour directories) or, with the user's consent, downloads Adobe's own end-user profile bundle (checksum-verified, byte-identical to the previously bundled profile) and caches it in ~/.mpc-autofill. Falls back to Ghostscript's default CMYK conversion with a warning. Also fixes the Nuitka onefile path bug: the old bundled-profile lookup resolved from sys.argv[0] where compiled builds extract data files beside __file__, so packaged builds could never find the profile.
The DriveThruCards branch previously combined confirm_ghostscript_ install_for_dtc, a gs_missing flag and ensure_ghostscript_available's ask_for_install_confirmation parameter, producing a confusing double prompt. Now a single detect -> ask -> install -> re-check loop asks for explicit permission before any package manager runs; declining prints per-platform install instructions instead.
- save_processed_image_to_bytes: never called - MM_PER_INCH: never read - unused io import in pdf_maker - DriveThruCardsSelectors upload/quantity/cart/continue fields and DriveThruCardsSite.base_url: never read - merge _resolve_ghostscript_path into get_ghostscript_path (the public function was a pass-through wrapper)
- Extract a no_implicit_wait contextmanager, replacing the implicitly_wait(0)/implicitly_wait(5) dance repeated across six methods; delete the unused wait_for_selector helper. - Share export-directory derivation between PdfExporter and the --skip-pdf-if-exists reuse path via pdf_maker.get_export_directory so they can't drift; drop get_site_picker_choices (enum order already lists DriveThruCards last). - --skip-pdf-if-exists staleness now compares card images against the PDF/X output specifically, so a stale _pdfx.pdf is not silently reused when some other PDF is newer. - A DriveThruCards order whose PDF/X conversion produced no output now aborts the run instead of logging and exiting successfully, and the tip-jar handoff only shows when a browser session actually ran.
- processing.py: PEP 585 builtin generics instead of typing.Tuple/Dict - driver.py: hoist the function-local os imports to module level - web_server.py: direct attribute access instead of getattr - order.py: move is_image_valid/remove_if_exists from nested closures to module level
- CI: run the desktop tool test suite unconditionally; tests requiring the Google Drive API key now skip themselves (via a client_secrets skipif marker) instead of the whole workflow being skipped on fork pull requests where the secret is unavailable. - Replace placeholder_cover.png (was JPEG bytes with a .png extension and old Photoshop metadata, unknown provenance) with a project-owned Pillow-generated PNG; record its origin in assets/README.md. - Move the wiki addendum and publisher-workflow HTML out of the tree (they belong in the PR description / upstream wiki) and drop the test that asserted their contents. - Revert the .python-version patch-level pin and trim .gitignore to build outputs (drop contributor-specific AI-instruction/plan-file ignores).
No production config sets output_directory/output_extension, so the get_post_processed_path redirect was an identity no-op and the local-file post-processing elif was unreachable (its guard implied the preceding branch's condition). DTC passes download_config=None anyway; its per-card resize happens in PdfExporter.add_image. Corruption validation and the 2-attempt Drive retry are kept.
add_image re-ran the resize/JPEG-encode for every slot and wrote a uniquely named temp file each time, so a card repeated across slots (especially the shared cardback) was processed N times and, because fpdf caches embedded images by path, its JPEG data was embedded N times in the output PDF. Cache the processed temp file per source path for the exporter run and clean the temp files up when execute() finishes.
The placeholder cover is DriveThruCards' own file, provided by DTC for this purpose (the README wrongly described it as generated with Pillow).
Author
|
Really the only things left to do at this point are testing on Windows 11 and Ubuntu. Once that's hammered out I'll also complete any remaining documentation in the wiki. I've placed multiple real orders on DriveThruCards using this fork. Once I get it to it's polished state and confirm it works on Windows, my final test will be asking some non-programmer friends to try placing an order on their own. |
- Interactive onboarding prompts (InquirerPy) with DTC question skipping - Lazy imports and Nuitka cached onefile extraction for faster startup - Bundled certifi TLS fix with --check-tls CI smoke tests - Image download failure gating (ImageDownloadError) - PDF progress bar fix
- Log an explanation when explicit --no-auto-save/--image-post-processing flags are overridden on the DriveThruCards path (click ParameterSource). - Stop printing ImageDownloadError twice on exit. - Share CRASH_LOG_FILENAME constant between logging and exc modules. - Pin pre-commit mypy hook to python3.13 to match CI and fix local env build.
fork PR CI runs have no GOOGLE_DRIVE_API_KEY secret, and 9 tests that download fixture images from Google Drive were failing there; marking them with requires_google_drive_credentials makes the workflow's self-skip promise true (verified locally: hiding client_secrets.json yields 101 passed / 21 skipped / 0 failures).
- PdfExporter now stops its enlighten manager when execute() finishes or aborts, freezing completed counters into scrollback and releasing terminal rows so later bars cannot collide with them - Its transient State row is cleared (leave=False) - AutofillDriver no longer creates the per-image upload/download and project counters on the DriveThruCards path (they only ever showed 0/0) - just the State/Action row - DTC order completion now sets State: Finished instead of leaving a stale state - Also includes a black reformat of a test signature that the previous commit missed
… for all sites - ConsoleFormatter strips exception tracebacks from console output at non-DEBUG levels while the crash log still records them in full - execute_order now completes all image downloads before any browser automation and raises ImageDownloadError listing the failed cards, so no site gets a partially-uploaded project - failed-download collection moved to CardOrder.get_failed_downloads and shared with PdfExporter - ImageDownloadError message generalised to cover both PDF and upload flows
Nuitka's cached onefile extraction rewrites changed files in place when a new build reuses the cache directory; on macOS this invalidates the kernel's cached code signature of the previously-executed binary and the process dies with SIGKILL and no output. Extracting each build version to its own directory ({VERSION} in --onefile-tempdir-spec, fed by --file-version=1.0.<run number> in CI) sidesteps the in-place rewrite entirely, and the tool prunes stale version directories at startup so they do not accumulate. Builds without --file-version now fail loudly instead of silently reintroducing the bug.
per-image failure logs (HTTP error lines, failed-download lines with links, local-file problems) now carry a FILE_ONLY extra that a console-handler filter drops at non-DEBUG levels - the crash log still records everything and the ImageDownloadError summary remains the single user-facing report; --download-images-only now raises the same summary so silencing per-image lines leaves no path without feedback; PdfExporter closes its transient status row explicitly so leave=False actually clears it on stop.
… editor progress bars are now fully transient - live while their phase runs, cleared (leave=False + close(clear=True)) when it ends - so final output reads as naturally-scrolled log text with no frozen bar rows pinned to the bottom of the window; the failed-download message now points at https://mpcfill.com/editor to distinguish the web app from the desktop tool.
this test previously survived fork-PR CI without credentials because failed downloads were merely skipped at upload time; the new stop-before-upload gate correctly aborts it, so it needs the requires_google_drive_credentials marker like the other secret-backed tests (verified: full suite without client_secrets.json now passes 101/skips 27).
The signed desktop-tool-ci workflow cannot be dispatched on the fork (not on the default branch), so the build workflow now also produces the Windows executable; the smoke test step declares bash so its syntax works on the Windows runner.
the step uses test/subshell syntax that pwsh cannot parse, which failed the first Windows build job.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR adds DriveThruCards (DTC) as a target site for the desktop tool. DTC doesn't work like the MPC-family sites: there's no web editor to fill card by card. Instead, you upload one print-ready PDF/X-1a file through their publisher tools. So this PR is really two things: a pipeline that builds that PDF (Pillow + fpdf + Ghostscript), and Selenium automation for DTC's publisher/product/upload/checkout flow. I also made a few fixes to the shared download/export code along the way that apply to every target site.
New dependencies
Important
Ghostscript (external CLI, only needed for DriveThruCards). DTC wants PDF/X-1a, which is normally something you get out of Adobe Acrobat or similar paid software. I didn't want a paid, inconsistent dependency, and there's no Python library that can produce compliant PDF/X-1a, so I used Ghostscript. I also didn't want to bundle OS-specific Ghostscript binaries with autofill, so when you pick DriveThruCards the tool checks for Ghostscript and asks permission to install it via your package manager (Homebrew / winget / apt / dnf / yum). If you decline, it prints manual install instructions and re-checks when you're ready. The other target sites never touch any of this.
Important
Adobe's US Web Coated (SWOP) ICC profile (for accurate CMYK conversion). Adobe's license lets you use the profile but not redistribute it bundled with software, so it's not in the repo. The tool first looks for a copy already installed in the standard OS colour-profile folders. If it can't find one, it offers to download Adobe's own end-user bundle, checks the profile against a pinned SHA-256 hash, and caches it in
~/.mpc-autofill/. If you decline or the download fails, it falls back to Ghostscript's default CMYK handling and warns that print colours may differ. There's a--dtc-icc-profileflag if you'd rather point at your own profile.Smaller new dependencies:
undetected-chromedriver(new Python dependency): DTC sits behind Cloudflare bot detection that blocks standard Selenium. It's only used when targeting DriveThruCards; every other site keeps its existing driver, and a test pins the standard driver factories to their upstream signatures.setuptoolscomes along with it because undetected-chromedriver needsdistutils, which Python 3.13 removed.assets/placeholder_cover.png: DTC's product form requires a cover image, which doesn't matter for a personal print order, so I bundled a generic placeholder image.dtc-post-launch.html: a DTC variant of the existing post-launch instruction page, served by the same local web server.The DriveThruCards workflow
Selecting
--site DriveThruCardsruns a pipeline that looks quite different from the existing per-image upload flow:PDFX_def.ps) declaring PDF/X-1a:2001 conformance and a US Web Coated (SWOP) output intent, and embeds the ICC profile as the destination profile when one is available. The conversion is written atomically and then verified: the output has to actually contain the PDF/X-1a conformance marker and an output intent, because a zero exit code from Ghostscript alone doesn't prove compliance. If conversion or verification fails, the DTC flow stops with an error rather than uploading a non-compliant file.A few deliberate design decisions:
--sitepicker on purpose, so the existing MPC-family default experience is unchanged.New opt-in CLI flags, all defaulting to prior behaviour:
--skip-pdf-if-exists: reuse existing exported PDFs; if images incards/are newer than the export, the tool asks before reusing stale output. For DTC this specifically requires a fresh PDF/X-1a file, not just any PDF. (Also works with plain--exportpdf.)--browser-profile-path/--browser-profile-name: reuse an existing Chromium user profile (cookies, password manager). Currently only the DTC driver honours these.--skip-dtc-instructions: skip thedtc-post-launch.htmlinstructions page and go straight to login.--dtc-icc-profile: path to your own local .icc color profile used for the RGB to CMYK conversion.Changes to the general MPC workflow
These apply to all target sites, not just DTC:
PdfExporternow feeds fpdf raw image bytes instead of file paths, so fpdf's cache keys on content rather than path and refreshed images actually get picked up when re-exporting.Automated tests
Roughly 60 new tests, all offline by default:
test_drivethrucards_driver.pysuite (23 tests) exercises the DTC Selenium flow against mocked drivers: step sequencing, login/auth detection, Cloudflare wait behaviour, publisher onboarding, upload-URL extraction, and the fail-loudly paths for missing form elements.test_desktop_tool.py(37 tests) cover Ghostscript detection and consent-gated install, ICC profile lookup/download/checksum rejection, atomic PDF/X conversion and its verification, PDF reuse/staleness logic, and the new CLI flags. One integration test runs real Ghostscript end-to-end and checks the output is actually PDF/X-1a; it skips itself when Ghostscript isn't installed.Checklist
pre-commitand installed the hooks withpre-commit installbefore creating any commits.On macOS, the full DriveThruCards workflow runs end-to-end without manual intervention other than account sign-in: