You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
A lightweight Go service that periodically downloads files via HTTP(S), stores them locally, and runs configurable hooks afterward.
2
+
Simple service that pulls files via http(s) in configurable intervals, stores them on your system and executes hooks after each successful and/or failed download. Configurable via YAML.
4
3
5
4
## Features
6
-
7
-
-**Scheduled HTTP pulls** — download files at configurable intervals per target
8
-
-**Atomic writes** — files are written to a temp file first, then renamed into place
9
-
-**Built-in hooks** — run shell commands or move files after each download
10
-
-**Extensible hook interface** — third-party hooks via a public Go interface in `pkg/hook`
11
-
-**Live reload** — send `SIGHUP` to re-read configuration without restarting
12
-
-**Graceful shutdown** — in-flight downloads and hooks complete before exit
13
-
-**Structured logging** — JSON logs via `slog`, configurable level and output target
5
+
-**Easily Configurable** - Configure which files to pull in which interval and what to do with them in a single configuration file
6
+
-**Sub-Minute Intervals** - Pull files as often (or as infrequent) as you need to
7
+
-**Atomic Writes** - Files are downloaded into a temporary file, first, then atomically moved into place (`inotify`-compatible)
8
+
-**Hooks** - Configure hooks to be executed after a pull was completed successfully or after it failed
9
+
-**Extensible** - Easily develop own hooks by satisfying a simple Go interface with just one method
14
10
15
11
## Installation
12
+
You can download a [pre-built binary](https://github.com/rescaled/http-pull/releases) for macOS, Linux or Windows from GitHub.
16
13
17
-
Requires Go 1.24+.
14
+
Ubuntu or Debian users can also install `http-pull` as package.
15
+
16
+
## Build
17
+
Building this project requires Go 1.24+ to be installed on your system.
18
18
19
19
```sh
20
+
git clone git@github.com:rescaled/http-pull.git
21
+
cd http-pull
20
22
go build -o http-pull ./cmd/http-pull/
21
23
```
22
24
23
25
## Usage
26
+
Running the service is as easy as pointing to a configuration file. By default, `http-pull` assumes that the configuration file is stored at `/etc/http-pull/config.yaml`. It's recommended to run the software in the background or as service with e.g. systemd. You can find a [systemd service template](https://github.com/rescaled/http-pull/blob/main/packaging/systemd/http-pull.service) in the `packaging/` directory that is being used to build the .deb packages for Ubuntu and Debian and adapt it to your needs, if necessary.
24
27
25
28
```sh
26
29
http-pull --config /path/to/config.yaml
27
30
```
28
31
29
-
| Flag | Default | Description |
30
-
|------|---------|-------------|
31
-
|`--config`|`/etc/http-pull/config.yaml`| Path to the configuration file |
32
-
33
-
### Signals
34
-
35
-
| Signal | Behaviour |
36
-
|--------|-----------|
37
-
|`SIGHUP`| Reload configuration and apply changes live |
file: /var/log/http-pull.log # only used when target is "file"
36
+
level: INFO # DEBUG, INFO, WARN, or ERROR
37
+
target: stdout # stdout or file
38
+
file: /var/log/http-pull.log # only used when target is "file"
47
39
48
40
targets:
49
-
- name: example
50
-
url: https://example.com/data.txt
51
-
interval: 30s
52
-
destination: /tmp/data.txt
53
-
http_request: # optional
54
-
method: GET # default: GET
55
-
headers: # optional
41
+
- name: example# unique target name
42
+
url: https://example.com/data.txt# URL to download from
43
+
interval: 30s# interval as Go-compatible expression
44
+
destination: /tmp/data.txt# final destination for the downloaded file
45
+
http_request: # optional
46
+
method: GET # default: GET
47
+
headers: # optional
56
48
- name: Authorization
57
49
value: Bearer my-token
58
-
basic_auth: # optional
59
-
username: user
60
-
password: pass
61
-
follow_redirects: true # default: true
62
-
hooks: # optional
63
-
- type: shell
50
+
basic_auth: # optional
51
+
username: username
52
+
password: password
53
+
follow_redirects: true # default: true
54
+
hooks: # optional
55
+
- type: shell# `shell` or `move`
64
56
command: echo "Downloaded successfully"
65
-
on: # default: [success]
57
+
on: # default: [success]
66
58
- success
67
59
- failure
68
-
- type: move
60
+
- type: move
69
61
destination: /opt/data/data.txt
70
62
```
71
63
72
-
### Defaults
73
-
74
-
| Setting | Default |
75
-
|---------|---------|
76
-
| `log.level` | `INFO` |
77
-
| `log.target` | `stdout` |
78
-
| `http_request.method` | `GET` |
79
-
| `http_request.follow_redirects` | `true` |
80
-
| `hooks[].on` | `["success"]` |
81
-
| `User-Agent` header | `http-pull/1.0` (overridable via `headers`) |
82
-
83
64
## Hooks
84
65
85
-
### Built-in hooks
86
-
87
66
**`shell`** — Executes a command via `/bin/sh -c` after the pull completes.
88
67
89
68
```yaml
@@ -100,16 +79,8 @@ targets:
100
79
on: [success]
101
80
```
102
81
103
-
### Hook triggers
104
-
105
-
| Value | Description |
106
-
|-------|-------------|
107
-
| `success` | Run when the download succeeds (HTTP status < 400) |
108
-
| `failure` | Run when the download fails (HTTP status >= 400 or network error) |
109
-
110
-
### Writing custom hooks
111
-
112
-
The hook interface is in `pkg/hook` so it can be imported by external projects:
82
+
## Custom Hooks
83
+
The hook interface is in `pkg/hook` so it can be imported by external projects. Writing your own hook is as easy as satisfying the interface's single method.
0 commit comments