Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion PixivServer/app.py
Original file line number Diff line number Diff line change
Expand Up @@ -68,8 +68,12 @@ async def request_metrics_middleware(request: Request, call_next):
response = await call_next(request)
duration = time.perf_counter() - start

# fastapi >= 0.137 no longer flattens included routers, so scope["route"] is the inner
# APIRoute and its .path has lost the include_router() prefix. The full path lives on the
# effective route context. Drop this once https://github.com/fastapi/fastapi/issues/16176 lands.
ctx = (request.scope.get("fastapi") or {}).get("effective_route_context")
route = request.scope.get("route")
endpoint = route.path if route and hasattr(route, "path") else None
endpoint = getattr(ctx, "path", None) or getattr(route, "path", None)
if endpoint is None:
return response
status_class = f"{response.status_code // 100}xx"
Expand Down
4 changes: 2 additions & 2 deletions PixivUtilClient/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,15 @@ build-backend = "uv_build"

[project]
name = "pixivutil-server-client"
version = "0.1.2"
version = "0.1.3"
description = "Async aiohttp client SDK for PixivUtil Server"
readme = "README.md"
requires-python = ">=3.12"
authors = [
{ name = "psilabs-dev" }
]
dependencies = [
"aiohttp>=3.13.3",
"aiohttp>=3.14.3",
"pydantic>=2.12.4",
"pixivutil-server-common>=0.1.0",
]
Expand Down
39 changes: 31 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,16 +22,15 @@ Build and run with Docker compose. A pixiv cookie is required at `PIXIVUTIL_COOK
docker compose up --build --remove-orphans
```

This project uses `uv` inside the Docker image to install and run the app.
Make a request to download an artwork:
This project uses `uv` inside the Docker image to install and run the app. Make a request to download an artwork:

```sh
curl -X POST http://localhost:8000/api/queue/download/artwork/{artwork-id-here}
```

`/api/download/*` remains available as a deprecated compatibility alias.

If `PIXIVUTIL_SERVER_API_KEY` is not set (or is empty), API key authentication is disabled.
If `PIXIVUTIL_SERVER_API_KEY` is not set (or is empty), API key authentication is disabled. If it is set, the request above needs an `Authorization: Bearer <your-api-key>` header.

An nginx [configuration file](/nginx/default.conf) is attached for your reverse proxy reference.

Expand All @@ -50,8 +49,7 @@ For example, the server supports the following endpoints:

#### [Dead letter queue (DLQ)](/docs/api/dlq.md)

API endpoints to inspect, replay, and purge failed worker messages in the dead
letter queue.
API endpoints to inspect, replay, and purge failed worker messages in the dead letter queue.

#### [Download queueing](/docs/api/download.md)

Expand All @@ -62,6 +60,9 @@ The following jobs are supported via PixivUtil server API:
- Download artworks by member ID
- Download artwork by image ID
- Download artworks by tag
- Delete an artwork from the database and filesystem

All queue endpoints accept an optional `priority` query parameter (`1`-`3`, where `3` is highest). Defaults are chosen per endpoint so that cheap single-artwork requests outrank long bulk crawls.

#### [Health](/docs/api/health.md)

Expand All @@ -73,6 +74,10 @@ API endpoints to queue metadata downloads for worker from server. Metadata inclu

This is helpful when you have an artwork downloaded, but it's old/outdated and you want to re-fetch only the metadata.

#### [Metrics](/docs/api/metrics.md)

Prometheus metrics endpoint, covering database counts, disk and system usage, queue/DLQ depth, and HTTP request statistics.

#### [Server](/docs/api/server.md)

Server-related API endpoints, such as get cookie, update cookie, delete database, and delete downloads.
Expand All @@ -83,7 +88,17 @@ PixivUtil server applies a downstream server flavor of PixivUtil2 called "Server

For further configuration, apply them at `.pixivUtil2/conf/conf.ini` (refer to [Pixivutil2](https://github.com/Nandaka/PixivUtil2) configuration options). You should shutdown PixivUtil server and remove the backup `.ini` file before applying the changes and restarting.

Supported environment variable overrides. See `PixivServer/configuration/pixivutil.py`.
Supported environment variables, defined across `PixivServer/config/`:

| Variable | Default | Purpose |
| --- | --- | --- |
| `PIXIVUTIL_COOKIE` | (none) | Pixiv session cookie. Required. |
| `PIXIVUTIL_SERVER_API_KEY` | (unset) | Enables API key authentication when set. |
| `PIXIVUTIL_SERVER_ENV` | `production` | `production` or `development`. See [Environments](#environments). |
| `RABBITMQ_BROKER_URL` | `amqp://guest:guest@rabbitmq:5672` | Celery broker. |
| `RABBITMQ_MANAGEMENT_URL` | `http://guest:guest@rabbitmq:15672` | Used to collect queue depth metrics. |

`PUID`, `PGID`, and `CELERYBEAT_SCHEDULE` are consumed by the container entrypoint and Celery beat rather than by the application config objects; see `docker-compose.yml`.

### User Configuration

Expand All @@ -108,9 +123,16 @@ Header format:
Authorization: Bearer <your-api-key>
```

## Architecture and Development
### Environments

`PIXIVUTIL_SERVER_ENV` selects server environment.

- `production` (default)
- `development`

When running PixivUtil server in development, set `PIXIVUTIL_SERVER_ENV=development`. This will enable debug logging level.
If set to `development`, enables debug logging and mounts `/api/dev/*` router with a set of testing endpoints.

## Architecture and Development

PixivUtil server is a Python project based on PixivUtil2 as its API client engine. PixivUtil2 is a separate git repository added to this as a submodule.

Expand Down Expand Up @@ -142,6 +164,7 @@ PixivUtil Server applies server-managed SQLite runtime settings. These settings
uv sync --extra pixivutil2 # sync dev + PixivUtil2 dependencies
uv run pytest tests # run tests
uv run ruff check . # run ruff lint check
uv run pyright # run type checks
```

The project also uses `uv` as the build runtime with `uv_build`, which significantly speeds up build times. On a raspberry pi, building the Dockerfile with `uv_build` takes ~5m, 3m less than with default `pip`.
Expand Down
18 changes: 15 additions & 3 deletions docs/api/download.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,19 @@ Authentication:
- Requires `Authorization: Bearer <api-key>` when `PIXIVUTIL_SERVER_API_KEY` is set.
- If `PIXIVUTIL_SERVER_API_KEY` is unset/empty, authentication is disabled.

All endpoints here are queue operations: they enqueue a job for the single worker and return immediately with a `task_id`. They do not wait for the download.

## Queue priority

| Endpoint | Default priority |
| --- | --- |
| `POST /api/queue/download/artwork/{artwork_id}` | `3` |
| `POST /api/queue/download/member/{member_id}` | `2` |
| `POST /api/queue/download/tag/{tag}` | `1` |
| `DELETE /api/queue/download/artwork/{artwork_id}` | `2` |

## Endpoints

`POST /api/queue/download/artwork/{artwork_id}`

Queue download of artwork by ID.
Expand All @@ -14,7 +27,6 @@ Queue download of a member's artworks by member ID.

`POST /api/queue/download/tag/{tag}`

Queue download of all artworks with a given tag (tags should be URL encoded).
Queue download of all artworks with a given tag. The tag is URL-decoded by the server, so it should be URL encoded by the caller and may contain special characters.

> Compatibility note: `/api/download/*` endpoints are still available but
> deprecated. Use `/api/queue/download/*` as the canonical path.
> Compatibility note: `/api/download/*` endpoints are still available but deprecated. Use `/api/queue/download/*` as the canonical path.
10 changes: 8 additions & 2 deletions docs/api/health.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,18 @@
# Health API

Authentication:
- `/api/health/*` endpoints are always public and do not require an API key.
- `GET /api/health/` is always public and never requires an API key. Use for container healthcheck.
- `GET /api/health/pixiv` requires `Authorization: Bearer <api-key>` when `PIXIVUTIL_SERVER_API_KEY` is set.

`GET /api/health/`

Check if the server is running.
Check if the server is running. Returns `success`.

`GET /api/health/pixiv`

Check if the Pixiv cookie is effective.

Responses:
- `200`: `Pixiv login works!`
- `403`: `Pixiv login failed.` (cookie is missing, expired, or rejected)
- `401`: API key missing or invalid (only when authentication is enabled)
19 changes: 15 additions & 4 deletions docs/api/metadata.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,19 @@ Authentication:
- Requires `Authorization: Bearer <api-key>` when `PIXIVUTIL_SERVER_API_KEY` is set.
- If `PIXIVUTIL_SERVER_API_KEY` is unset/empty, authentication is disabled.

All endpoints here are queue operations: they enqueue a job for the single worker and return immediately with a `task_id`. They do not wait for the metadata fetch.

## Queue priority

| Endpoint | Default priority |
| --- | --- |
| `POST /api/queue/metadata/artwork/{artwork_id}` | `2` |
| `POST /api/queue/metadata/member/{member_id}` | `2` |
| `POST /api/queue/metadata/series/{series_id}` | `1` |
| `POST /api/queue/metadata/tag/{tag}` | `1` |

## Endpoints

`POST /api/queue/metadata/artwork/{artwork_id}`

Queue download of artwork metadata by ID.
Expand All @@ -18,8 +31,6 @@ Queue download of series metadata by ID.

`POST /api/queue/metadata/tag/{tag}`

Queue download of tag metadata by tag name. Optional query: `filter_mode` in
`none`, `pixpedia`, `translation`, `pixpedia_or_translation`.
Queue download of tag metadata by tag name. The tag is URL-decoded by the server, so it should be URL encoded by the caller. Optional query: `filter_mode` in `none`, `pixpedia`, `translation`, `pixpedia_or_translation`.

> Breaking change: `/api/metadata/*` endpoints were removed. Use
`/api/queue/metadata/*` instead.
> Breaking change: `/api/metadata/*` endpoints were removed. Use `/api/queue/metadata/*` instead.
41 changes: 41 additions & 0 deletions docs/api/metrics.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# Metrics API

Authentication:
- Requires `Authorization: Bearer <api-key>` when `PIXIVUTIL_SERVER_API_KEY` is set.
- If `PIXIVUTIL_SERVER_API_KEY` is unset/empty, authentication is disabled.

`GET /metrics`

Prometheus exposition endpoint, in the standard text format. Note this endpoint is served at the root, not under `/api`, and a Prometheus scrape job must send the bearer token when authentication is enabled.

## Exported metrics

`pixivutil_server_info`: build info, carries a `version` label.

Database counts, refreshed every 60s:
- `pixivutil_db_members_total`
- `pixivutil_db_artworks_total`
- `pixivutil_db_pages_total`
- `pixivutil_db_tags_total`
- `pixivutil_db_series_total`

Disk usage, refreshed every 300s:
- `pixivutil_disk_downloads_bytes`
- `pixivutil_disk_database_bytes`

Host system, refreshed every 15s:
- `pixivutil_cpu_usage_percent`
- `pixivutil_memory_used_bytes`
- `pixivutil_memory_total_bytes`
- `pixivutil_sys_disk_used_bytes`
- `pixivutil_sys_disk_total_bytes`

Worker queue, refreshed every 15s from `RABBITMQ_MANAGEMENT_URL`:
- `pixivutil_queue_depth`: messages pending in the main task queue
- `pixivutil_dlq_depth`: messages in the [dead letter queue](/docs/api/dlq.md)

HTTP traffic, recorded per request by middleware, labelled by `method` and `endpoint` (the route template, not the resolved path). `pixivutil_http_requests_total` is additionally labelled by `status_class`:
- `pixivutil_http_requests_total`
- `pixivutil_http_request_duration_seconds`
- `pixivutil_http_request_size_bytes`
- `pixivutil_http_response_size_bytes`
10 changes: 8 additions & 2 deletions docs/api/server.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,13 @@ Authentication:

Get the cookie in use.

`PUT /api/server/cookie/{cookie}`
`PUT /api/server/cookie`

Update the cookie.
Update the cookie. The cookie is sent in the JSON request body, so it does not end up in URLs or access logs.

```json
{"cookie": "<your-pixiv-cookie>"}
```

`DELETE /api/server/database`

Expand All @@ -19,3 +23,5 @@ Reset the database.
`DELETE /api/server/downloads`

Delete the downloads folder.

> Compatibility note: `PUT /api/server/cookie/{cookie}` is still available but deprecated. Use `PUT /api/server/cookie` with a request body instead.
15 changes: 0 additions & 15 deletions docs/api/subscription.md

This file was deleted.

6 changes: 3 additions & 3 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,11 @@ license = { file = "LICENSE" }
authors = [
{ name = "psilabs-dev" }
]
version = "2.4.6"
version = "2.4.7"
dependencies = [
"aiofiles>=25.1.0",
"celery>=5.5.3",
"fastapi>=0.122.0",
"fastapi>=0.141.1",
"pixivutil-server-common>=0.1.0",
"prometheus-client>=0.24.1",
"psutil>=7.2.2",
Expand All @@ -34,7 +34,7 @@ pixivutil2 = [
"curl-cffi>=0.11.3",
"demjson3>=3.0.0",
"mechanize>=0.4.5",
"pillow>=12.2.0",
"pillow>=12.3.0",
"pysocks>=1.7.1",
]

Expand Down
Loading
Loading