Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -64,3 +64,19 @@ jobs:

- name: Build MkDocs site (--strict)
run: uv run mkdocs build --strict

test:
name: Test (backend matrix via mongomock)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- uses: astral-sh/setup-uv@v5
with:
python-version: "3.12"

- name: Install package and dev tools
run: uv sync --group dev

- name: Run tests (json + mongo backends)
run: uv run task test
4 changes: 4 additions & 0 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ q2google moves assets from **GoPro cloud** (`gopro-api` / `AsyncGoProClient`) in
| `q2google/gphotos/` | Low-level Library v1 HTTP (`GooglePhotosAPI`), OAuth (`GooglePhotosOAuth`), Pydantic models. |
| `q2google/state/base.py` | `SessionState`, `ItemState`, `SyncStateBackend` protocol — persistence contract. |
| `q2google/state/local.py` | `JsonFileBackend` — directory-tree backend; each session is a subdirectory containing `meta.json`, `items/*.json`, and `batches/*.json`. Reads legacy flat-file sessions transparently. |
| `q2google/state/mongo.py` | `MongoBackend` — MongoDB backend; distributes each session across three collections (`sessions`, `items`, `batches`). Requires `pymongo` (`pip install q2google[mongo]`). |
| `q2google/state/__init__.py` | `build_backend(cfg)` — factory that parses `cfg.state_uri` scheme and returns the matching `SyncStateBackend`; defaults to `JsonFileBackend` when `state_uri` is unset. |

## Sync pipeline (`sync_date_range`)

Expand Down Expand Up @@ -61,6 +63,8 @@ class CustomBackend:

`SessionState.to_dict()` / `from_dict()` produce a plain dict suitable for any document store. No changes to `GoProToPhotosSync` are required.

Pass the instance directly or route through `build_backend()` by registering the new scheme there. See the [Backends guide](backends.md) for a full walkthrough.

## Documentation conventions

Public modules, classes, and functions use a one-line summary, optional narrative, and structured
Expand Down
9 changes: 8 additions & 1 deletion docs/api/state.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
# State

Session persistence — protocol, models, and the built-in JSON file backend.
Session persistence — protocol, models, and built-in storage backends.

See the [Backends guide](../backends.md) for usage instructions, configuration, and
collection schemas for each backend.

## Base protocol and models

Expand All @@ -9,3 +12,7 @@ Session persistence — protocol, models, and the built-in JSON file backend.
## JSON file backend

::: q2google.state.local

## MongoDB backend

::: q2google.state.mongo
178 changes: 178 additions & 0 deletions docs/backends.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,178 @@
# State Backends

q2google persists every sync session through a `SyncStateBackend` — a small two-method
protocol that decouples the sync pipeline from any particular storage engine.

```python
class SyncStateBackend(Protocol):
def load(self, session_id: str) -> SessionState | None: ...
def save(self, state: SessionState) -> None: ...
```

The active backend is selected at startup by the `Q2GOOGLE_STATE_URI` environment variable.
When the variable is absent the filesystem backend is used by default, so existing setups
require no changes.

```mermaid
flowchart TD
env["Q2GOOGLE_STATE_URI"]
factory["build_backend(cfg)"]
json["JsonFileBackend\n(default)"]
mongo["MongoBackend\n(mongodb://)"]
future["Future backends\n(redis://, postgresql://…)"]

env -->|absent| factory
env -->|set| factory
factory -->|"no URI"| json
factory -->|"mongodb://"| mongo
factory -->|"new scheme"| future
```

---

## Filesystem backend (default)

No extra packages required. Each session is stored as a directory of JSON files under
`Q2GOOGLE_STATE_DIR` (default `.q2google_sessions`).

### Layout

```
.q2google_sessions/
{session_id}/
meta.json # session metadata + stage statuses
items/
{file_name}.json # one file per ItemState
batches/
{batch_index}.json # one file per BatchState
```

Each file is written atomically via a temp file and `os.replace`, so a crash mid-save
never corrupts an existing checkpoint.

### Configuration

| Variable | Default | Description |
|----------|---------|-------------|
| `Q2GOOGLE_STATE_DIR` | `.q2google_sessions` | Root directory for session subdirectories. |
| `Q2GOOGLE_STATE_URI` | _(absent)_ | Leave unset to use this backend. |

### When to use

- Local CLI use or single-machine deployments.
- No external service dependency.
- Human-readable checkpoints (plain JSON files, easy to inspect or delete).

---

## MongoDB backend

Requires `pymongo`. Each session is distributed across three collections that mirror
the filesystem layout, enabling per-collection queries and clear visibility into
what is running.

### Installation

=== "pip"

```bash
pip install q2google[mongo]
```

=== "uv"

```bash
uv add q2google[mongo]
```

### Configuration

Set `Q2GOOGLE_STATE_URI` to a MongoDB connection string. The database name is taken
from the URI path component; it defaults to `q2google` when absent.

```dotenv
Q2GOOGLE_STATE_URI=mongodb://localhost:27017/q2google
```

| Variable | Example | Description |
|----------|---------|-------------|
| `Q2GOOGLE_STATE_URI` | `mongodb://host:27017/dbname` | Activates MongoBackend; `mongodb+srv://` is also supported. |

### Collection schema

**`sessions`** — one document per session

| Field | Type | Description |
|-------|------|-------------|
| `session_id` | string | Unique session key (index). |
| `schema_version` | int | Document format version. |
| `created_at` | string | ISO 8601 creation timestamp. |
| `updated_at` | string | ISO 8601 last-updated timestamp. |
| `start_date_iso` | string | Capture window start. |
| `end_date_iso` | string | Capture window end. |
| `batch_size` | int | Transfer batch size chosen at session creation. |
| `stages` | object | Map of stage key → status (`pending`/`running`/`completed`/`failed`). |

**`items`** — one document per `(session_id, file_name)`

| Field | Type | Description |
|-------|------|-------------|
| `session_id` | string | Parent session key (compound index with `file_name`). |
| `file_name` | string | GoPro logical filename. |
| `media_id` | string \| null | Remote media identifier once known. |
| `download_url` | string \| null | Resolved CDN URL after discovery. |
| `discovery_status` | string | `pending` / `running` / `completed` / `failed`. |
| `transfer_status` | string | `pending` / `running` / `completed` / `failed`. |
| `create_status` | string | `pending` / `running` / `completed` / `failed` / `skipped`. |
| `upload_token` | string \| null | Resumable upload token before library registration. |
| `errors` | object | Map of stage key → last `ErrorRecord`. |

**`batches`** — one document per `(session_id, batch_index)`

| Field | Type | Description |
|-------|------|-------------|
| `session_id` | string | Parent session key (compound index with `batch_index`). |
| `batch_index` | int | Zero-based index among create batches. |
| `file_names` | array | Filenames included in this batch. |
| `status` | string | `pending` / `running` / `completed` / `failed`. |
| `responses_json` | array \| null | Per-response JSON strings after success. |
| `error` | object \| null | Populated when the batch fails as a whole. |

### When to use

- Multi-machine or container deployments where a shared filesystem is unavailable.
- Observability — query individual collections to see item or batch progress without
reading the full session document.
- When you need indexing, TTL expiry, or aggregation on session data.

---

## Custom backend

Implement `SyncStateBackend` to integrate any storage layer:

```python
from q2google import SessionState, SyncStateBackend
import json


class RedisBackend:
def __init__(self, client):
self._r = client

def load(self, session_id: str) -> SessionState | None:
raw = self._r.get(session_id)
return SessionState.from_dict(json.loads(raw)) if raw else None

def save(self, state: SessionState) -> None:
self._r.set(state.session_id, json.dumps(state.to_dict()))
```

Pass it directly to `GoProToPhotosSync`:

```python
syncer = GoProToPhotosSync(gopro=gopro, photos=photos, state_backend=RedisBackend(redis_client))
```

`SessionState.to_dict()` / `from_dict()` produce a plain dict suitable for any
document or key-value store. No changes to the sync pipeline are required.
3 changes: 2 additions & 1 deletion docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,8 @@ q2google loads the GoPro token into `Q2GoogleSettings.gopro_access_token` and pa

| Variable | Default | Description |
|----------|---------|-------------|
| `Q2GOOGLE_STATE_DIR` | `.q2google_sessions` | Root directory for per-session state; each session is stored as a subdirectory containing `meta.json`, `items/`, and `batches/`. |
| `Q2GOOGLE_STATE_URI` | _(absent)_ | Backend URI whose scheme selects the storage engine. `mongodb://host:port/db` activates MongoBackend; leave unset to use the filesystem backend. Takes precedence over `Q2GOOGLE_STATE_DIR` when set. See [Backends](backends.md) for details. |
| `Q2GOOGLE_STATE_DIR` | `.q2google_sessions` | Root directory for the filesystem backend; each session is stored as a subdirectory containing `meta.json`, `items/`, and `batches/`. Ignored when `Q2GOOGLE_STATE_URI` is set. |
| `Q2GOOGLE_SESSION_ID` | _(auto)_ | Default session identifier when `--session-id` is omitted from the CLI. |

## Transfer tuning
Expand Down
23 changes: 10 additions & 13 deletions docs/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,24 +99,21 @@ responses = await syncer.sync_date_range(
)
```

## Custom state backend
## State backends

Implement `SyncStateBackend` to persist sessions in any storage layer (database, object store, etc.):
By default sessions are saved to the local filesystem under `.q2google_sessions`.
q2google also ships a **MongoDB backend** and supports custom backends via the
`SyncStateBackend` protocol.

```python
from q2google import SessionState, SyncStateBackend


class RedisBackend:
def load(self, session_id: str) -> SessionState | None:
raw = redis_client.get(session_id)
return SessionState.from_dict(json.loads(raw)) if raw else None
Set `Q2GOOGLE_STATE_URI` to switch backends without any code changes:

def save(self, state: SessionState) -> None:
redis_client.set(state.session_id, json.dumps(state.to_dict()))
```dotenv
# MongoDB backend (requires: pip install q2google[mongo])
Q2GOOGLE_STATE_URI=mongodb://localhost:27017/q2google
```

Pass it directly to `GoProToPhotosSync(state_backend=RedisBackend())`. No other changes required.
See the [Backends guide](backends.md) for installation, configuration, collection
schemas, and instructions for writing your own backend.

## Stage completion hook

Expand Down
1 change: 1 addition & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@ nav:
- Getting Started: getting-started.md
- CLI: cli.md
- Configuration: configuration.md
- Backends: backends.md
- Architecture: ARCHITECTURE.md
- API Reference:
- api/index.md
Expand Down
26 changes: 24 additions & 2 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,27 @@ build-backend = "hatchling.build"

[project]
name = "q2google"
version = "0.0.1"
version = "0.0.2"
description = "Sync GoPro cloud media to Google Photos Library."
readme = "README.md"
requires-python = ">=3.10,<3.14"
license = { file = "LICENSE" }
authors = [{ name = "himewel", email = "welberthime@gmail.com" }]
keywords = ["google-photos", "quik", "cloud", "api", "async", "aiohttp", "media", "gopro-api"]
classifiers = [
"Development Status :: 3 - Alpha",
"Environment :: Web Environment",
"Intended Audience :: Developers",
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3 :: Only",
"Topic :: Internet :: WWW/HTTP",
"Topic :: Multimedia :: Graphics :: Capture :: Digital Camera",
]
requires-python = ">=3.10"
dependencies = [
"aiohttp~=3.11.11",
"aiofiles~=25.1.0",
Expand All @@ -28,13 +45,18 @@ docs = [
"mkdocs-material>=9",
"mkdocstrings[python]>=0.27",
]
mongo = [
"pymongo>=4",
]

[dependency-groups]
dev = [
"coverage>=7",
"mike>=2",
"mkdocs-material>=9",
"mkdocstrings[python]>=0.27",
"mongomock>=4",
"pymongo>=4",
"pytest>=8",
"pytest-asyncio>=1",
"pytest-cov>=6",
Expand Down
Loading
Loading