|
| 1 | +# http-pull |
| 2 | + |
| 3 | +A lightweight Go service that periodically downloads files via HTTP(S), stores them locally, and runs configurable hooks afterward. |
| 4 | + |
| 5 | +## Features |
| 6 | + |
| 7 | +- **Scheduled HTTP pulls** — download files at configurable intervals per target |
| 8 | +- **Atomic writes** — files are written to a temp file first, then renamed into place |
| 9 | +- **Built-in hooks** — run shell commands or move files after each download |
| 10 | +- **Extensible hook interface** — third-party hooks via a public Go interface in `pkg/hook` |
| 11 | +- **Live reload** — send `SIGHUP` to re-read configuration without restarting |
| 12 | +- **Graceful shutdown** — in-flight downloads and hooks complete before exit |
| 13 | +- **Structured logging** — JSON logs via `slog`, configurable level and output target |
| 14 | + |
| 15 | +## Installation |
| 16 | + |
| 17 | +Requires Go 1.24+. |
| 18 | + |
| 19 | +```sh |
| 20 | +go build -o http-pull ./cmd/http-pull/ |
| 21 | +``` |
| 22 | + |
| 23 | +## Usage |
| 24 | + |
| 25 | +```sh |
| 26 | +http-pull --config /path/to/config.yaml |
| 27 | +``` |
| 28 | + |
| 29 | +| Flag | Default | Description | |
| 30 | +|------|---------|-------------| |
| 31 | +| `--config` | `/etc/http-pull/config.yaml` | Path to the configuration file | |
| 32 | + |
| 33 | +### Signals |
| 34 | + |
| 35 | +| Signal | Behaviour | |
| 36 | +|--------|-----------| |
| 37 | +| `SIGHUP` | Reload configuration and apply changes live | |
| 38 | +| `SIGINT` / `SIGTERM` | Graceful shutdown (30s timeout for in-flight work) | |
| 39 | + |
| 40 | +## Configuration |
| 41 | + |
| 42 | +```yaml |
| 43 | +log: |
| 44 | + level: INFO # DEBUG, INFO, WARN, or ERROR |
| 45 | + target: stdout # stdout or file |
| 46 | + file: /var/log/http-pull.log # only used when target is "file" |
| 47 | + |
| 48 | +targets: |
| 49 | + - name: example |
| 50 | + url: https://example.com/data.txt |
| 51 | + interval: 30s |
| 52 | + destination: /tmp/data.txt |
| 53 | + http_request: # optional |
| 54 | + method: GET # default: GET |
| 55 | + headers: # optional |
| 56 | + - name: Authorization |
| 57 | + value: Bearer my-token |
| 58 | + basic_auth: # optional |
| 59 | + username: user |
| 60 | + password: pass |
| 61 | + follow_redirects: true # default: true |
| 62 | + hooks: # optional |
| 63 | + - type: shell |
| 64 | + command: echo "Downloaded successfully" |
| 65 | + on: # default: [success] |
| 66 | + - success |
| 67 | + - failure |
| 68 | + - type: move |
| 69 | + destination: /opt/data/data.txt |
| 70 | +``` |
| 71 | +
|
| 72 | +### Defaults |
| 73 | +
|
| 74 | +| Setting | Default | |
| 75 | +|---------|---------| |
| 76 | +| `log.level` | `INFO` | |
| 77 | +| `log.target` | `stdout` | |
| 78 | +| `http_request.method` | `GET` | |
| 79 | +| `http_request.follow_redirects` | `true` | |
| 80 | +| `hooks[].on` | `["success"]` | |
| 81 | +| `User-Agent` header | `http-pull/1.0` (overridable via `headers`) | |
| 82 | + |
| 83 | +## Hooks |
| 84 | + |
| 85 | +### Built-in hooks |
| 86 | + |
| 87 | +**`shell`** — Executes a command via `/bin/sh -c` after the pull completes. |
| 88 | + |
| 89 | +```yaml |
| 90 | +- type: shell |
| 91 | + command: systemctl reload nginx |
| 92 | + on: [success] |
| 93 | +``` |
| 94 | + |
| 95 | +**`move`** — Moves the downloaded file to another path. Uses atomic rename when possible, falls back to copy+delete for cross-device moves. |
| 96 | + |
| 97 | +```yaml |
| 98 | +- type: move |
| 99 | + destination: /etc/app/config.json |
| 100 | + on: [success] |
| 101 | +``` |
| 102 | + |
| 103 | +### Hook triggers |
| 104 | + |
| 105 | +| Value | Description | |
| 106 | +|-------|-------------| |
| 107 | +| `success` | Run when the download succeeds (HTTP status < 400) | |
| 108 | +| `failure` | Run when the download fails (HTTP status >= 400 or network error) | |
| 109 | + |
| 110 | +### Writing custom hooks |
| 111 | + |
| 112 | +The hook interface is in `pkg/hook` so it can be imported by external projects: |
| 113 | + |
| 114 | +```go |
| 115 | +import "http-pull/pkg/hook" |
| 116 | +
|
| 117 | +type MyHook struct{} |
| 118 | +
|
| 119 | +func (h *MyHook) Execute(ctx context.Context, result hook.Result) error { |
| 120 | + if result.Success { |
| 121 | + // handle successful download at result.FilePath |
| 122 | + } |
| 123 | + return nil |
| 124 | +} |
| 125 | +``` |
| 126 | + |
| 127 | +## Project structure |
| 128 | + |
| 129 | +``` |
| 130 | +cmd/http-pull/ Entry point and signal handling |
| 131 | +internal/ |
| 132 | + config/ Configuration loading and validation (Viper) |
| 133 | + logging/ Structured logging setup (slog) |
| 134 | + hook/ Built-in hook implementations and registry |
| 135 | + puller/ HTTP download with atomic file writes |
| 136 | + runner/ Worker orchestration, reload, and graceful shutdown |
| 137 | +pkg/hook/ Public hook interface for third-party extensions |
| 138 | +``` |
| 139 | +
|
| 140 | +## License |
| 141 | +
|
| 142 | +See [LICENSE](LICENSE) for details. |
0 commit comments