config: add optional s3 storage_class and sftp host-key validation params#122
Merged
Conversation
…rams s3 storage_class passes through to rclone's s3 storage_class config key (backend-specific values). sftp known_hosts_file and host_key_algorithms map to the rclone sftp options of the same name; pointing rclone at a known_hosts file is what enables server host-key validation, which sftp destinations otherwise skip. All three are optionalString passthroughs confined to their backend type by the existing unknown-field check.
End-to-end WriteRcloneConfig tests confirming the new optional params land in the written rclone.conf.
Note that sftp destinations skip host-key validation unless known_hosts_file is set, and recommend setting it.
There was a problem hiding this comment.
Pull request overview
This PR extends the destination config schema to allow passing through additional rclone backend options on a per-destination basis, specifically adding optional parameters for S3 storage class selection and SFTP host-key verification-related settings. It also updates documentation and adds tests to ensure these parameters parse and render into the generated rclone.conf.
Changes:
- Add optional
storage_classpassthrough fors3destinations. - Add optional
known_hosts_fileandhost_key_algorithmspassthroughs forsftpdestinations. - Add/extend config + sync tests and README documentation for the new parameters.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 10 comments.
| File | Description |
|---|---|
config/destinations.go |
Extends per-type schema optional string params for s3 and sftp. |
config/config_test.go |
Adds parsing/unknown-field tests for the new destination params. |
sync/rclone_test.go |
Adds end-to-end WriteRcloneConfig tests ensuring params reach rclone.conf. |
README.md |
Documents the new optional destination parameters with configuration examples. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| Some optional params are specific to one backend type and rejected on the others (as an unknown field): | ||
|
|
||
| - **`sftp` host-key validation** — `known_hosts_file` points rclone at a known_hosts file so it validates the server's host key before transferring; `host_key_algorithms` is rclone's space-separated list pinning the accepted host-key algorithms. Both map to the rclone sftp options of the same name. **Without `known_hosts_file`, rclone does not validate the server's host key** and will connect to whatever host answers — set it (recommended) so a redirected or impersonated server is rejected. |
| user = "martin" | ||
| password = { env = "NAS_PASSWORD" } | ||
| root = "/volume1/squirrel" | ||
| known_hosts_file = "~/.ssh/known_hosts" # validate the server host key (recommended) |
Comment on lines
+46
to
+51
| // known_hosts_file points rclone at a known_hosts file so it | ||
| // validates the server's host key before transferring; absent, rclone | ||
| // accepts whatever host key the server presents. host_key_algorithms | ||
| // pins the accepted host-key algorithms (rclone's space-separated | ||
| // list). Both map straight to the rclone sftp options of the same | ||
| // name. The unknown-field check confines them to this type. |
Comment on lines
+450
to
+454
| // TestLoadDestinationSFTPHostKeyValidation parses the optional sftp | ||
| // known_hosts_file and host_key_algorithms params and confirms both render | ||
| // verbatim into the sftp section. Pointing rclone at a known_hosts file is | ||
| // what turns on server host-key validation; absent, rclone accepts any host | ||
| // key the server presents. |
| user = "u" | ||
| root = "/r" | ||
| password = "p" | ||
| known_hosts_file = "~/.ssh/known_hosts" |
Comment on lines
+471
to
+473
| if d.Params["known_hosts_file"] != "~/.ssh/known_hosts" { | ||
| t.Fatalf("known_hosts_file not resolved: %v", d.Params) | ||
| } |
Comment on lines
+478
to
+481
| for _, want := range []string{ | ||
| "known_hosts_file = ~/.ssh/known_hosts", | ||
| "host_key_algorithms = ssh-ed25519 ssh-rsa", | ||
| } { |
Comment on lines
+162
to
+165
| // TestWriteRcloneConfigRendersSFTPHostKeyValidation confirms the optional | ||
| // sftp host-key params reach the written rclone.conf: known_hosts_file is | ||
| // what enables server host-key validation, and host_key_algorithms pins the | ||
| // accepted algorithms. Absent these, rclone does no host-key validation. |
| user = "martin" | ||
| root = "/data" | ||
| password = "p" | ||
| known_hosts_file = "~/.ssh/known_hosts" |
Comment on lines
+183
to
+186
| for _, want := range []string{ | ||
| "known_hosts_file = ~/.ssh/known_hosts", | ||
| "host_key_algorithms = ssh-ed25519 ssh-rsa", | ||
| } { |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds two optional, backward-compatible per-destination params that surfaced while drafting a deployment config. Both are generic rclone capabilities expressed as such — no provider-specific behavior is baked in.
What
s3storage_class— maps to rclone's s3storage_classconfig key. Accepts whatever value the backend supports (commonlySTANDARDand archive tiers such asGLACIER/DEEP_ARCHIVE); absent, the backend default applies. Rendered verbatim into the s3rclone.confsection when set.sftphost-key validation —known_hosts_fileandhost_key_algorithms, mapping to the rclone sftp options of the same name. Rendered into the sftp section when set.Security rationale (host-key validation)
squirrel's sftp destinations currently connect with no host-key validation — rclone accepts whatever host key the server presents, so a redirected or impersonated server is not detected. Pointing rclone at a
known_hosts_fileis what turns on server host-key validation;host_key_algorithmsadditionally pins the accepted algorithms. The param is optional and backward-compatible (absent → today's behavior), and the README recommends settingknown_hosts_file.How it's wired
All three are plain
optionalStringpassthroughs in the existingdestSchema(config/destinations.go). The schema is flat per backend type, and the existing strict unknown-field check confines each param to the right type for free:storage_classonly ons3,known_hosts_file/host_key_algorithmsonly onsftp— anything else is rejected as an unknown field at load time.RcloneSectionalready renders everyoptionalStringkey verbatim into the section, so no rendering code changed.Tests
storage_classparses + renders into the s3 section; rejected on a non-s3 (sftp) destination as an unknown field.known_hosts_file+host_key_algorithmsparse + render into the sftp section;known_hosts_filerejected on a non-sftp (s3) destination.WriteRcloneConfigtests confirming both reach the writtenrclone.conf.go vet ./...,go test ./...,golangci-lint runall clean.Judgment calls
storage_class/known_hosts_fileare simply listed under the one type'soptionalString, matching howregion/endpoint/key_filealready work. No new code path.host_key_algorithmsalongsideknown_hosts_filesince it is the natural companion sftp host-key option, but the security-relevant one isknown_hosts_file(it is what enables validation).