feat: authenticated file access (drive=), silent-HTML guard, GDFiles - #8
Merged
Merged
Conversation
Three related gaps that together made private-Drive ingest impossible. Authenticated single-file fetch (#5) get_bytes gains a keyword-only drive=. With it, the download goes through the authenticated API (_download_via_api -> GetContentIOBuffer, streamed and joined; byte-exact, no temp file). Without it, behaviour is unchanged. get_bytes also now accepts a bare file id, not just a URL. Silent-HTML failure (#6) Drive serves its sign-in interstitial with HTTP 200, so the public path was returning the login page as if it were the file -- plausible-looking bytes that only fail later, in whatever parses them. _looks_like_html now sniffs the content type and body prefix, and NotPubliclyShared is raised with an actionable message. allow_html=True opts out for genuine HTML downloads. GDFiles (#7) A file-level Mapping keyed by file id or Drive URL (normalised through _extract_file_id), values bytes over the authenticated API. Iteration requires a folder_url scope -- listing a whole Drive is unbounded and paginated, so __iter__/__len__ raise NotImplementedError naming the fix rather than silently paginating. Lookup works either way. __contains__ answers from a metadata probe, never a download; a genuine 404 becomes KeyError while 5xx/auth failures propagate as themselves. Metadata without downloading get_metadata(url, drive=...) plus GDFiles.metadata: title/fileSize/mimeType/ modifiedDate, so a caller can decide whether an 18MB fetch is worth it. fileSize is coerced to int; absent for Google-native files. Also - GDReader.__getitem__ went through GetContentString(...).encode('latin-1'), which decodes as utf-8 first and so corrupted (or raised on) every real binary. It now shares _download_via_api. - Folder traversal extracted to _iter_folder_files, shared by GDFiles and GDReader instead of duplicated. - .gitignore covers service-account keys, client_secrets.json, tokens. - README: private-file quickstart, GDFiles, and the service-account setup steps (including sharing the folder with the account's client_email). 48 new tests, all offline: a fake GoogleDrive and a fake requests.Session cover the HTML guard, the authenticated paths, metadata, and GDFiles. Closes #5 Closes #6 Closes #7 Claude-Session: https://claude.ai/code/session_01Vb8moaBd5Yg8b4Abo77nHf
GDReader builds keys with os.path.join, so the nested-file key uses a backslash on Windows. The Windows CI job is continue-on-error, but a red check is still noise. Claude-Session: https://claude.ai/code/session_01Vb8moaBd5Yg8b4Abo77nHf
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #5, closes #6, closes #7.
Motivated by
Trufflepig-Travel/snout, which ingests client.xlsxexports from a privateDrive folder shared with the user's account. Today that is impossible:
get_bytesisunauthenticated and returns Google's sign-in page as if it were the file.
1. Authenticated single-file fetch (#5)
With
drive=, the download goes through the API —CreateFile({'id': ...})→GetContentIOBuffer(), streamed and joined. Byte-exact, no temp file, nostrround trip.local_pathanduse_cachebehave identically on both paths.drive=Nonekeeps today'spublic behaviour exactly.
get_bytesnow also accepts a bare file id.2. The silent-HTML failure (#6)
Drive serves its sign-in interstitial with HTTP 200, so the existing status check never
fired and the login page was returned as file content — plausible-looking bytes that only blow
up much later, in whatever tries to parse them. (Observed: a private 18.5 MB xlsx came back as
902 KB of HTML, no exception.)
_looks_like_htmlnow sniffs the declared content type and the body prefix (toleratingleading whitespace and a BOM, case-insensitive), and the public path raises
NotPubliclyShared— a dedicated, catchable exception whose message names thedrive=fix andthe
client_emailsharing step.allow_html=Trueopts out when the file genuinely is HTML.3.
GDFiles(#7)File-level Mapping keyed by file id or Drive URL (both normalised through
_extract_file_id, so they are one entry); values are bytes over the authenticated API.__iter__/__len__require a scope. Unscoped, the mapping addresses the whole Drive —unbounded and paginated, never what a caller wants, so offering it would be a trap. They raise
NotImplementedErrorwith a message namingfolder_url=; with afolder_urlthey yield thatfolder's file ids using the same traversal
GDReaderuses. Lookup works either way, which isthe primary use case and needs no listing.
__contains__answers from a metadata probe, never a download. A genuine 404 becomesKeyError(Mapping contract), while 5xx / auth / network failures propagate as themselves rather than
being disguised as "absent".
4. Metadata without downloading
title/fileSize/mimeType/modifiedDate— enough to decide whether an 18 MB fetch isworth making.
fileSizeis coerced toint(Drive sends a string) and is absent forGoogle-native files, which have no stored byte size.
Drive-by fixes
GDReader.__getitem__corrupted binaries. It usedGetContentString(mimetype='application/octet-stream').encode('latin-1');GetContentStringdecodes as utf-8, so for any real binary this raised
UnicodeDecodeErroror silentlymangled the bytes — latin-1 cannot undo a utf-8 decode. Now shares
_download_via_api._iter_folder_files, shared byGDFilesandGDReaderinstead of duplicated. Semantics preserved exactly (regression-tested).
.gitignorenow covers service-account keys,client_secrets.json, tokens,settings.yaml.Nothing sensitive was tracked before or is tracked now.
GDFiles,NotPubliclySharedtroubleshooting, and thefull service-account setup — including the step people forget, sharing the file/folder with
the service account's
client_email.Tests
48 new tests in
pydrivedol/tests/test_authenticated_access.py, all offline — nocredentials, no network. A
FakeDriveimplements the slice of the PyDrive2 API this packagecalls; a
FakeSessionstands in forrequests.Session. Covered: HTML detection (includingfalse-negative and false-positive cases, and a realistic sign-in-page fixture), the
authenticated download with
local_path/use_cache,drive=Nonenever touching the API,metadata field selection and int coercion, all of
GDFiles, and a regression test thatGDReader.__getitem__returns exact bytes.Full suite: 53 passed, 14 skipped (the skips are the pre-existing live-Drive tests).
ruff checkandruff format --checkclean.Not verified
No Google credentials available in this environment, so nothing was exercised against a real
Drive. Specifically unverified end to end: that
GetContentIOBuffer()behaves as assumedagainst the live API, and the exact Drive-v2 field names in
DEFAULT_METADATA_FIELDS.Follow-up (not in this PR)
[tool.pytest.ini_options] testpaths = ["pydrivedol"]excludes the repo-roottests/directory, so
tests/test_convert.py(4 tests) never runs in CI. They pass when invokeddirectly. Left alone here to keep this PR to one concern.
https://claude.ai/code/session_01Vb8moaBd5Yg8b4Abo77nHf