Skip to content

Make data.sqlite upload explicit and optional - #28

Merged
benrfairless merged 1 commit into
mainfrom
feature/2-upload-sqlite-database
Aug 24, 2026
Merged

benrfairless merged 1 commit into
mainfrom
feature/2-upload-sqlite-database

Conversation

@benrfairless

Copy link
Copy Markdown
Member

Description

Makes the upload of the local data.sqlite database explicit and optional. The database has always been included in the upload as a side effect of "upload everything in the directory", but nothing said so and there was no way to leave it out. Now the Uploading... line announces the database when it is included (Uploading 21.00 B (including data.sqlite)...), and a new --skip-data option uploads just the code. This wires up the database_path helper that was added in 2014 as groundwork for this feature but never used.

The timeout lookup is also simplified to Hash#fetch with identical behaviour (an explicit nil still disables the timeout), to keep execute within the repository's RuboCop metrics rather than loosening .rubocop_todo.yml.

Motivation and Context

Scrapers use data.sqlite to continue from the data they already have, so sending the local database with the run matters for incremental scraping - but it should be visible and skippable, since a large local database makes every upload slow.

Resolves #2

How Has This Been Tested?

  • Checked affected area manually on my own / staging system
  • Ran automated tests on my own system
  • Confirmed it passed the GitHub actions tests

Three new specs cover the announcement, the --skip-data exclusion (asserting the database bytes are absent from the multipart request body via WebMock) and the CLI option plumbing. All four CI jobs pass locally: bundle exec rspec (32 examples, 0 failures, coverage above the 90% SimpleCov floor), bundle exec rubocop (no offenses), bundle exec bundler-audit check --update and bundle exec rake build.

Screenshots (if appropriate):

Types of Changes

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.

AI assistance disclosure

This change was written with AI assistance: OpenCode:anthropic.claude-fable-5. The same disclosure is on the commit as an Assisted-by trailer. All generated code and documentation has been reviewed by a human before submission. No code was adapted from external sources.

The local database has always travelled to morph.io as a side effect of
"upload everything in the directory", but nothing said so and there was
no way to leave it behind. Wire up the dormant database_path helper so
the upload announces the database when it's included, and add a
--skip-data option to upload just the code when the database is big and
the wait isn't worth it.

Also simplify the timeout lookup to Hash#fetch while trimming execute
to fit the existing RuboCop metrics; behaviour is unchanged, including
an explicit nil disabling the timeout.

Resolves #2

Assisted-by: OpenCode:anthropic.claude-fable-5
Signed-off-by: Ben Fairless <ben@oaf.org.au>
@benrfairless benrfairless self-assigned this Aug 24, 2026
@benrfairless
benrfairless marked this pull request as ready for review August 24, 2026 04:00
@benrfairless
benrfairless requested a review from a team as a code owner August 24, 2026 04:00
@benrfairless
benrfairless enabled auto-merge August 24, 2026 04:00

@ianheggie-oaf ianheggie-oaf left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved - Makes sense to me. You may with to also add a --skip-spec (spec, coverage) or --skip-files=file,dir,file,*.bak as spec caused it to be too large in my previous attempts

@benrfairless
benrfairless merged commit c8323c9 into main Aug 24, 2026
9 checks passed
@benrfairless
benrfairless deleted the feature/2-upload-sqlite-database branch August 24, 2026 05:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Upload sqlite database

2 participants