Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 11 additions & 3 deletions .github/workflows/ruby_test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,20 +12,28 @@ jobs:
name: Rspec
runs-on: ubuntu-latest

strategy:
fail-fast: false
matrix:
# Due to https://github.com/actions/runner/issues/849, we have to use quotes for '3.0'
ruby: [ "3.0", "3.1", "3.2", "3.3", "3.4", "4.0" ]
steps:
- name: Checkout Repository
uses: sanger/.github/.github/actions/setup/checkout@master

- name: Setup Ruby
uses: sanger/.github/.github/actions/setup/ruby@master
- name: Setup Ruby ${{ matrix.ruby }}
uses: ruby/setup-ruby@v1
with:
ruby-version: ${{ matrix.ruby }}
bundler-cache: true # runs 'bundle install' and caches installed gems automatically

- name: Run rspec
run: bundle exec rspec

- name: Upload coverage reports to Codecov
uses: sanger/.github/.github/actions/tests/codecov@master
with:
name: ${{ github.run_id }}_${{ github.job }}_${{ github.event_name }}
name: ${{ github.run_id }}_${{ github.job }}_${{ github.event_name }}_${{ matrix.ruby }}
token: ${{ secrets.CODECOV_TOKEN }}
flags: ${{ github.event_name }}
disable-search: true
Expand Down
5 changes: 3 additions & 2 deletions .rubocop.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
---

inherit_from: ".rubocop_todo.yml"

# The behaviour of RuboCop can be controlled via the .rubocop.yml
# configuration file. It makes it possible to enable/disable
# certain cops (checks) and to alter their behaviour if they accept
Expand All @@ -13,12 +11,15 @@ inherit_from: ".rubocop_todo.yml"
#
# See https://docs.rubocop.org/rubocop/latest/configuration.html

inherit_from: ".rubocop_todo.yml"

plugins:
- rubocop-rake
- rubocop-rspec

AllCops:
NewCops: enable
TargetRubyVersion: 3.0

Gemspec/DevelopmentDependencies:
EnforcedStyle: gemspec
Expand Down
9 changes: 1 addition & 8 deletions .rubocop_todo.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# This configuration was generated by
# `rubocop --auto-gen-config`
# on 2026-04-29 17:06:59 UTC using RuboCop version 1.86.1.
# on 2026-04-29 18:48:04 UTC using RuboCop version 1.86.1.
# The point is for the user to remove these configuration records
# one by one as the offenses are removed from the code base.
# Note that changes in the inspected code, or installation of new
Expand All @@ -12,13 +12,6 @@ Gemspec/RequireMFA:
Exclude:
- 'record_loader.gemspec'

# Offense count: 1
# This cop supports safe autocorrection (--autocorrect).
# Configuration parameters: MinBodyLength, AllowConsecutiveConditionals.
Style/GuardClause:
Exclude:
- 'record_loader.gemspec'

# Offense count: 3
# This cop supports unsafe autocorrection (--autocorrect-all).
Style/ReduceToHash:
Expand Down
19 changes: 16 additions & 3 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,29 @@
# Changelog

Keeps track of notable changes. Please remember to add new behaviours to the
Unreleased section to make new releases easy.
Unreleased section to make new releases easy.
Follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [1.0.0]

- [Changed] Version numbering to follow [Ruby Gem versioning guidelines](https://guides.rubygems.org/patterns/)
- [Changed] Explicitly require and use Psych gem for YAML parsing
- [Added] Add support for Ruby 3.1 - 4.0
- [Documentation] Update documentation to reflect changes in versioning and supported Ruby versions

## [0.3.0]

- [Breaking] Remove support for Ruby 2.5
- [Added] Add support for Ruby 3.0

## [0.2.0]

- [Added] `RecordLoader.export_attributes` for easy generation of yaml from
existing data
existing data
- [Added] Improved feedback if exceptions raised during record creation
- [Added] Improved templated yml files to use attributes from table
- [Changed] Update name of yaml files generated as part of tests.
No changes are required to existing loaders.
No changes are required to existing loaders.
- [Fixed] Default yaml files correctly templated

## [0.1.1]
Expand Down
16 changes: 13 additions & 3 deletions Gemfile.lock
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
PATH
remote: .
specs:
record_loader (0.3.0)
record_loader (1.0.0)
psych (~> 3.0)

GEM
remote: https://rubygems.org/
Expand All @@ -13,7 +14,11 @@ GEM
json (2.19.4)
language_server-protocol (3.17.0.5)
lint_roller (1.1.0)
logger (1.7.0)
mdtoc (0.3.1)
sorbet-runtime
method_source (1.0.0)
ostruct (0.6.3)
parallel (1.28.0)
parser (3.3.11.1)
ast (~> 2.4.1)
Expand All @@ -22,6 +27,7 @@ GEM
pry (0.14.2)
coderay (~> 1.1)
method_source (~> 1.0)
psych (3.3.4)
racc (1.8.1)
rainbow (3.1.1)
rake (13.0.6)
Expand Down Expand Up @@ -67,6 +73,7 @@ GEM
simplecov-html (0.13.2)
simplecov-lcov (0.9.0)
simplecov_json_formatter (0.1.4)
sorbet-runtime (0.6.13169)
unicode-display_width (3.2.0)
unicode-emoji (~> 4.1)
unicode-emoji (4.2.0)
Expand All @@ -76,7 +83,10 @@ PLATFORMS
ruby

DEPENDENCIES
bundler (~> 2.3)
bundler (~> 2.5)
logger (~> 1.7)
mdtoc (~> 0.3.1)
ostruct (~> 0.6.3)
pry (~> 0.14)
rake (~> 13.0)
record_loader!
Expand All @@ -89,4 +99,4 @@ DEPENDENCIES
yard (~> 0.9)

BUNDLED WITH
2.3.21
2.5.9
144 changes: 101 additions & 43 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,28 @@ your production and development environments.

While written with ActiveRecord/Rails in mind, it is possible to use RecordLoader in different environments.

<!-- mdtoc -->
* [RecordLoader](README.md#recordloader)
* [Key features](README.md#key-features)
* [Version compatibility](README.md#version-compatibility)
* [Installation](README.md#installation)
* [How to run](README.md#how-to-run)
* [How to generate new records loaders for your project (Rails)](README.md#how-to-generate-new-records-loaders-for-your-project-rails)
* [An example loader](README.md#an-example-loader)
* [Dev and Wip files](README.md#dev-and-wip-files)
* [RecordLoader Dependencies](README.md#recordloader-dependencies)
* [Triggering on deployment](README.md#triggering-on-deployment)
* [Within Sanger PSD](README.md#within-sanger-psd)
* [Non Rails Environments](README.md#non-rails-environments)
* [Development](README.md#development)
* [Setup](README.md#setup)
* [Testing](README.md#testing)
* [Releasing](README.md#releasing)
* [Contributing](README.md#contributing)
* [License](README.md#license)
* [Updating the table of contents](README.md#updating-the-table-of-contents)
<!-- mdtoc-end -->

## Key features

- Produce testable, reproducible data migrations across multiple environments
Expand All @@ -18,6 +40,24 @@ While written with ActiveRecord/Rails in mind, it is possible to use RecordLoade
- Keep work-in-progress isolated with .wip.yml files
- Rails generators to quickly create new record loaders

## Version compatibility

If using with Rails:

| Rails Version | RecordLoader Version |
| ------------- | -------------------- |
| 6.x and below | 0.3.0\*, 1.0.0 |

_\* Pin psych < 4_

If using with Ruby directly:

| Ruby Version | RecordLoader Version |
| ------------ | -------------------- |
| 4.x | 1.x |
| 3.x | 0.3.0, 1.x |
| 2.5 - 2.7 | 0.2.0 |

## Installation

Add this line to your application's Gemfile:
Expand Down Expand Up @@ -82,62 +122,62 @@ Suppose you want to create a loader to maintain a selection of product types. Yo

This will create several files:

#### `lib/tasks/record_loader.rake`
- `lib/tasks/record_loader.rake`

Adds the record_loader:all rake task which can be used to trigger all record loaders.
Adds the record_loader:all rake task which can be used to trigger all record loaders.

#### `lib/record_loader/application_record_loader.rb`
- `lib/record_loader/application_record_loader.rb`

Application specific base class for customization.
Application specific base class for customization.

#### `config/record_loader/product_types/default_records.yml`
- `config/record_loader/product_types/default_records.yml`

Example yaml file to begin populating with your record information. Record Loaders will load all yaml files from within
this directory, so it is possible to separate your records into multiple different files for better organization.
In addition yaml files ending in `.dev.yml` and `.wip.yml` exhibit special behaviour.
See [dev and wip files](#dev-and-wip).
Example yaml file to begin populating with your record information. Record Loaders will load all yaml files from within
this directory, so it is possible to separate your records into multiple different files for better organization.
In addition yaml files ending in `.dev.yml` and `.wip.yml` exhibit special behaviour.
See [dev and wip files](#dev-and-wip).

#### `lib/record_loader/product_type_loader.rb`
- `lib/record_loader/product_type_loader.rb`

The actual loader. It will look something like this:
The actual loader. It will look something like this:

```ruby
# frozen_string_literal: true
# This file was automatically generated via `rails g record_loader`

# RecordLoader handles automatic population and updating of database records
# across different environments
# @see https://rubydoc.info/github/sanger/record_loader/
module RecordLoader
# Creates the specified plate types if they are not present
class ProductTypeLoader < ApplicationRecordLoader
config_folder 'product_types'

def create_or_update!(name, options)
ProductType.create_with(options).find_or_create_by!(name: name)
```ruby
# frozen_string_literal: true
# This file was automatically generated via `rails g record_loader`

# RecordLoader handles automatic population and updating of database records
# across different environments
# @see https://rubydoc.info/github/sanger/record_loader/
module RecordLoader
# Creates the specified plate types if they are not present
class ProductTypeLoader < ApplicationRecordLoader
config_folder 'product_types'

def create_or_update!(name, options)
ProductType.create_with(options).find_or_create_by!(name: name)
end
end
end
end
```
```

The `config_folder` specifies which directory under `config/record_loader` will be used to source the yaml files.
The method `create_or_update!` will create the actual records, and should be idempotent (ie. calling it multiple times will
have the same effect as calling it once). `create_or_update!` will be called once for each entry in the yaml files,
with the first argument being the key, and the second argument being the value, usually a hash of options.
The `config_folder` specifies which directory under `config/record_loader` will be used to source the yaml files.
The method `create_or_update!` will create the actual records, and should be idempotent (ie. calling it multiple times will
have the same effect as calling it once). `create_or_update!` will be called once for each entry in the yaml files,
with the first argument being the key, and the second argument being the value, usually a hash of options.

#### `lib/record_loader/tasks/record_loader/product_type.rake`
- `lib/record_loader/tasks/record_loader/product_type.rake`

This contains the `record_loader:product_type` which will trigger the record loader, and also ensures that
`record_loader:product_type` will get invoked on calling `record_loader:all`.
This contains the `record_loader:product_type` which will trigger the record loader, and also ensures that
`record_loader:product_type` will get invoked on calling `record_loader:all`.

#### `spec/data/record_loader/product_types/product_types_basic.yml`
- `spec/data/record_loader/product_types/product_types_basic.yml`

A basic configuration for testing the loader. Tests use a separate directory to avoid coupling your specs to the data.
A basic configuration for testing the loader. Tests use a separate directory to avoid coupling your specs to the data.

#### `spec/lib/record_loader/product_type_loader_spec.rb`
- `spec/lib/record_loader/product_type_loader_spec.rb`

A basic rspec spec file for testing your loader. By default this just confirms that your loader creates the
expected number of records, and that it is idempotent.
A basic rspec spec file for testing your loader. By default this just confirms that your loader creates the
expected number of records, and that it is idempotent.

## Dev and Wip files

Expand Down Expand Up @@ -206,11 +246,20 @@ See {RecordLoader::Adapter} for information about custom adapters.

## Development

After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can
also run `bin/console` for an interactive prompt that will allow you to experiment.
### Setup

After checking out the repo, run `bin/setup` to install dependencies.

### Testing

Then, run `bundle rspec` to run the unit tests.
You can also run `bin/console` for an interactive prompt that will allow you to experiment.
To install this gem onto your local machine, run `bundle exec rake install`.

To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the
version number in `version.rb`, ensure the CHANGELOG.md is updated and that everything is committed.
### Releasing

To release a new version, update the version number in `version.rb`, ensure the CHANGELOG.md is updated
and that everything is committed.
Then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push
the `.gem` file to [rubygems.org](https://rubygems.org).

Expand All @@ -221,3 +270,12 @@ Bug reports and pull requests are welcome on GitHub at <https://github.com/sange
## License

The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

## Updating the table of contents

After updating this README, please update the table of contents.
To do this automatically with the [mdtoc](https://github.com/andornaut/mdtoc) gem, run:

```shell
bundle exec mdtoc -o README.md README.md
```
2 changes: 1 addition & 1 deletion lib/record_loader/base.rb
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,7 @@ def default_path
#
def load_config
@config = @files.each_with_object({}) do |file, store|
latest_file = YAML.load_file(file)
latest_file = Psych.load_file(file)
duplicate_keys = store.keys & latest_file.keys
adapter.logger.warn "Duplicate keys in #{@path}: #{duplicate_keys}" unless duplicate_keys.empty?
store.merge!(latest_file)
Expand Down
2 changes: 1 addition & 1 deletion lib/record_loader/version.rb
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,5 @@

module RecordLoader
# @return [String] The current sem-ver version number
VERSION = '0.3.0'
VERSION = '1.0.0'
end
Loading
Loading