Make data.sqlite upload explicit and optional - #28
Merged
Merged
Conversation
The local database has always travelled to morph.io as a side effect of "upload everything in the directory", but nothing said so and there was no way to leave it behind. Wire up the dormant database_path helper so the upload announces the database when it's included, and add a --skip-data option to upload just the code when the database is big and the wait isn't worth it. Also simplify the timeout lookup to Hash#fetch while trimming execute to fit the existing RuboCop metrics; behaviour is unchanged, including an explicit nil disabling the timeout. Resolves #2 Assisted-by: OpenCode:anthropic.claude-fable-5 Signed-off-by: Ben Fairless <ben@oaf.org.au>
benrfairless
marked this pull request as ready for review
August 24, 2026 04:00
benrfairless
enabled auto-merge
August 24, 2026 04:00
ianheggie-oaf
approved these changes
Aug 24, 2026
ianheggie-oaf
left a comment
Member
There was a problem hiding this comment.
Approved - Makes sense to me. You may with to also add a --skip-spec (spec, coverage) or --skip-files=file,dir,file,*.bak as spec caused it to be too large in my previous attempts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Makes the upload of the local
data.sqlitedatabase explicit and optional. The database has always been included in the upload as a side effect of "upload everything in the directory", but nothing said so and there was no way to leave it out. Now theUploading...line announces the database when it is included (Uploading 21.00 B (including data.sqlite)...), and a new--skip-dataoption uploads just the code. This wires up thedatabase_pathhelper that was added in 2014 as groundwork for this feature but never used.The timeout lookup is also simplified to
Hash#fetchwith identical behaviour (an explicitnilstill disables the timeout), to keepexecutewithin the repository's RuboCop metrics rather than loosening.rubocop_todo.yml.Motivation and Context
Scrapers use
data.sqliteto continue from the data they already have, so sending the local database with the run matters for incremental scraping - but it should be visible and skippable, since a large local database makes every upload slow.Resolves #2
How Has This Been Tested?
Three new specs cover the announcement, the
--skip-dataexclusion (asserting the database bytes are absent from the multipart request body via WebMock) and the CLI option plumbing. All four CI jobs pass locally:bundle exec rspec(32 examples, 0 failures, coverage above the 90% SimpleCov floor),bundle exec rubocop(no offenses),bundle exec bundler-audit check --updateandbundle exec rake build.Screenshots (if appropriate):
Types of Changes
Checklist:
AI assistance disclosure
This change was written with AI assistance: OpenCode:anthropic.claude-fable-5. The same disclosure is on the commit as an
Assisted-bytrailer. All generated code and documentation has been reviewed by a human before submission. No code was adapted from external sources.