Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ All notable changes to insto. Format follows [Keep a Changelog](https://keepacha

### Fixed

* clarify backend setup names ([#37](https://github.com/subzeroid/insto/issues/37)) ([0ed435a](https://github.com/subzeroid/insto/commit/0ed435ad9ff81daccc706f10845dee58ca7193d1))
* clarify hikerapi setup prompts and aiograpi install hints ([#37](https://github.com/subzeroid/insto/issues/37)) ([0ed435a](https://github.com/subzeroid/insto/commit/0ed435ad9ff81daccc706f10845dee58ca7193d1))


### Documentation
Expand Down
15 changes: 9 additions & 6 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,17 +7,20 @@ Thanks for considering a contribution. This document covers the dev workflow.
```bash
git clone git@github.com:subzeroid/insto.git
cd insto
uv sync --extra dev
uv sync --extra aiograpi
```

Python 3.11+ required (we use `dataclass(slots=True)`, `X | Y` unions, `datetime.fromisoformat` with `Z`).

This matches CI: dev dependencies are installed by default, and the
`aiograpi` extra exercises the optional logged-in backend module.

## Running tests

```bash
uv run pytest # all tests
uv run pytest -k hiker # subset
uv run pytest --cov=insto --cov-report=term-missing
uv run pytest --cov=insto --cov-report=term-missing --cov-fail-under=75
```

The suite is fully offline — no real HikerAPI or Instagram calls. There's also a structured live smoke against the real HikerAPI:
Expand All @@ -28,13 +31,13 @@ HIKERAPI_TOKEN_TEST=<token> uv run python tests/live/smoke.py

Nine REQ checks (resolve / profile / posts / followers / tagged / hashtag / search / quota / 404) plus one OPT check (`/similar`, per-target flaky). Skips with exit 0 if `HIKERAPI_TOKEN_TEST` is unset, so it's safe to wire into release-prep gates. Costs ~10 requests, single-digit cents. **Run before each release tag** — caught a real `iter_hashtag_posts` bug that mocks couldn't.

Coverage targets: keep pure-logic modules at 100% (`models`, `_redact`, `exceptions`, mappers). Everything else: 90%+ on touched code.
Coverage targets: keep pure-logic modules at 100% (`models`, `_redact`, `exceptions`, mappers). CI currently gates the whole package at 75%; touched code should not lower coverage without a clear reason.

## Lint + types

```bash
uv run ruff check insto tests
uv run ruff format --check insto tests
uv run ruff check
uv run ruff format --check
uv run mypy insto
```

Expand Down Expand Up @@ -67,7 +70,7 @@ insto/
├── ui/ # banner, theme, render helpers
├── backends/ # OSINTBackend ABC, HikerBackend, AiograpiBackend, mappers
├── service/ # facade, history, analytics, exporter, watch
└── commands/ # one file per group (target/profile/media/...)
└── commands/ # one file per group (target/profile/media/direct/saved/...)
tests/
├── fakes.py # FakeBackend with per-method error injection
├── fixtures/hiker/ # frozen HikerAPI dict responses per access state
Expand Down
22 changes: 12 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
# insto

Interactive Instagram OSINT CLI on the [HikerAPI](https://hikerapi.com/p/6k1q1388) backend and [aiograpi](https://github.com/subzeroid/aiograpi) lib.
Interactive Instagram OSINT CLI on the [HikerAPI](https://hikerapi.com/p/6k1q1388) backend and optional [aiograpi](https://github.com/subzeroid/aiograpi) logged-in backend.

![demo](docs/demo.gif)

## Design choices

- **HikerAPI is the default backend, not a side mode.** Quota balance is read on REPL startup and surfaced in the bottom toolbar; `with_retry` honours `RateLimited.retry_after`; every mapper raises typed `SchemaDrift(endpoint, missing_field)` when HikerAPI's documented fields move. Logged-in `aiograpi` is one extra (`pipx install 'insto[aiograpi]'`) when you actually need data behind the login wall — but kept off the default path so your account isn't in scope.
- **Async, typed, tested.** Python 3.11, strict mypy, ~93% coverage, ruff-clean. Backends, facade, commands all `async def`.
- **Async, typed, tested.** Python >= 3.11, strict mypy, ruff-clean, and 900+ offline tests. Backends, facade, commands all `async def`.
- **Two surfaces, one grammar.** A prompt-toolkit REPL with slash-popup completion and live `/watch` notifications, *and* a Unix-friendly one-shot mode (`insto @user -c info`). `--json -` and `--csv -` write to stdout; `/batch -` reads targets from stdin.
- **Snapshot / watch / diff.** Persisted in `~/.insto/store.db`. Poll a target on an interval; diff against the last snapshot.
- **Maltego CSV export** out of the box (`--maltego` on any flat-row command, plus full `/dossier`).
Expand Down Expand Up @@ -42,11 +42,12 @@ uv tool install 'insto[aiograpi]'
pipx install 'insto[aiograpi]'
```

If `insto` is already installed through `pipx` and you later switch to
`aiograpi`, add the optional dependency to the existing tool venv:
If `insto` is already installed and you later switch to `aiograpi`, update the
existing tool environment instead of running the bare installer again:

```sh
pipx inject insto aiograpi
pipx inject insto aiograpi # existing pipx install
uv tool install --force 'insto[aiograpi]' # existing uv tool install
```

`insto setup` then offers a `hikerapi | aiograpi` choice and prompts for
Expand Down Expand Up @@ -84,9 +85,10 @@ insto setup
```

Interactive wizard. Writes `~/.insto/config.toml` (mode `0600`) with your
HikerAPI token, output directory, sqlite store path, and optional proxy.
The token is read with `getpass` so it does not echo to the terminal; pass
`-` for the proxy to clear a previously-saved value.
backend choice, HikerAPI token or aiograpi credentials, output directory,
sqlite store path, and optional proxy. The HikerAPI token prompt links to
<https://hikerapi.com/tokens>. Secrets are read with `getpass` so they do not
echo to the terminal; pass `-` for the proxy to clear a previously-saved value.

> 💸 **HikerAPI gives you 100 free requests** to try things out — no card,
> no email-verification dance, just sign up and a token is waiting:
Expand Down Expand Up @@ -124,7 +126,7 @@ supported.
| Maltego CSV export | ✅ | ❌ | ❌ | ❌ |
| Interactive REPL | ✅ prompt-toolkit, slash-popup | ✅ basic shell | ❌ one-shot | ❌ one-shot |
| One-shot / scriptable | ✅ stdin/stdout pipes | ⚠️ shell only | ✅ | ✅ |
| Type-safe / strict mypy | ✅ ~93% coverage | ❌ | ❌ | ❌ |
| Type-safe / strict mypy | ✅ strict mypy + coverage gate | ❌ | ❌ | ❌ |

**When to pick insto** — Instagram-specific OSINT where you want HikerAPI on the default path (no IG session is ever in scope), deep network/geo analytics, snapshots over time, and Maltego-ready export. Modern stack — asyncio, strict mypy, prompt-toolkit REPL with slash-popup completion.

Expand All @@ -151,7 +153,7 @@ $ insto
@nasa
i n s t o ⇋ o s i n t @instagram
instagram tool · open-source intel
hiker · 14.7M requests left · $4,417 · 15 rps cap
hikerapi · 14.7M requests left · $4,417 · 15 rps cap

insto @→ /
```
Expand Down
6 changes: 3 additions & 3 deletions SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ insto is pre-1.0. Only the latest published release receives security fixes.

| Version | Supported |
| ------- | ------------------ |
| 0.1.x | :white_check_mark: |
| Latest published release | :white_check_mark: |

## Reporting a vulnerability

Expand All @@ -33,7 +33,7 @@ In scope:

- Code under `insto/` and `tests/`.
- Default deployment via `pip install insto` / `uv tool install insto`.
- Anything that allows reading secrets (`hikerapi_token`, snapshots) from logs, error output, or `~/.insto/`.
- Anything that allows reading secrets (`hikerapi.token`, aiograpi credentials, snapshots) from logs, error output, or `~/.insto/`.
- Any path-traversal, MIME-confusion, or remote-content-trust issue in the CDN streamer.
- Schema-drift or rate-limit handling that could exfiltrate the operator's IP / token.

Expand All @@ -45,4 +45,4 @@ Out of scope:

## Operator notes

`insto` is offensive-tooling-adjacent: it queries third-party data, downloads remote media, and stores intel on disk. Even fully patched, the local store at `~/.insto/store.db` is not encrypted at rest in 0.1.x. A stolen laptop is still a data-loss event. Treat the output directory like any other source of sensitive material.
`insto` is offensive-tooling-adjacent: it queries third-party data, downloads remote media, and stores intel on disk. Even fully patched, the local store at `~/.insto/store.db` is not encrypted at rest. A stolen laptop is still a data-loss event. Treat the output directory like any other source of sensitive material.
9 changes: 5 additions & 4 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@ Six layers, top to bottom. The rule that holds the design together: each layer t
```text
UI: REPL (prompt_toolkit) │ one-shot CLI (argparse)
Dispatch: parse → validate → run → render
Commands: commands/{target,profile,media,network,content,interactions,batch,watch,operational,dossier}.py
Commands: commands/{target,profile,media,network,content,interactions,discovery,
direct,saved,places,batch,watch,operational,dossier}.py
Service: facade · history · analytics · exporter · watch
Backends: OSINTBackend ABC · HikerBackend · AiograpiBackend
Models: @dataclass(slots=True) DTOs — Profile, Post, Story, User, Comment, Quota, ...
Expand All @@ -15,7 +16,7 @@ Models: @dataclass(slots=True) DTOs — Profile, Post, Story, User, Comment,

- **Async everywhere.** `httpx` (transitive via `hikerapi`), `asyncio` for fan-out, `asyncio.to_thread` for sqlite calls.
- **Backend boundary is a hard wall.** Raw HikerAPI / aiograpi dicts never leave `backends/`. Mappers in `_hiker_map.py` and `_aiograpi_map.py` are the only converters.
- **Lazy backend imports.** `import hikerapi` and `import aiograpi` happen only inside `make_backend(...)`. Import errors stay localized.
- **Lazy backend imports.** `import hikerapi` and `import aiograpi` happen only inside `make_backend(...)`. Missing optional dependencies stay localized and surface with install hints.
- **Retry / backoff lives in one place.** `backends/_retry.py` decorates SDK-method calls inside `HikerBackend`; commands never know retries exist.
- **CDN streaming through a single helper.** `backends/_cdn.py` is the only code that pulls untrusted bytes off the network. Host allowlist, MIME sniff, byte budget, atomic write — every download passes through it.
- **Pagination as `AsyncIterator[T]` + `limit: int | None`.** Every collection method is an async generator. Cursor management lives inside the backend; commands consume one item at a time and stop on `limit`.
Expand Down Expand Up @@ -85,8 +86,8 @@ Session limits: max 3 active watches, 5-minute floor on the interval, all watche

## Test strategy

- 850+ unit + integration tests, no live API calls in CI.
- 900+ unit + integration tests, no live API calls in CI.
- Fixtures: one frozen HikerAPI dict per profile-access state (`public`, `private`, `deleted`, `empty`, `schema_drift`).
- `tests/fakes.py:FakeBackend` implements `OSINTBackend` from fixtures with per-method error injection covering every entry of the error taxonomy.
- 3 e2e flows under `tests/e2e/`: subprocess one-shot, prompt_toolkit pty REPL session, `/watch` tick with `patch_stdout` capture.
- Strict mypy + ruff format + ruff lint as CI gates.
- Strict mypy + ruff format + ruff lint as CI gates; pytest coverage must stay at or above the current CI floor.
14 changes: 11 additions & 3 deletions docs/backends.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ The contract lives in `insto/backends/_base.py:OSINTBackend`. Two implementation
| Sees private accounts you follow | No | Yes |
| Sees DMs / saved feed | No | Read-only Direct threads/messages and saved collections/media exposed; personal feed intentionally not exposed |
| Quota visibility | Yes (`/sys/balance`) | No |
| Install footprint | base | `pip install 'insto[aiograpi]'` |
| Install footprint | base | `uv tool install 'insto[aiograpi]'` / `pipx install 'insto[aiograpi]'` |

## Pick a backend

Expand All @@ -38,7 +38,7 @@ Precedence is **flag > env > toml > default** for every key:

| Key | Flag | Env |
|---|---|---|
| `backend` | _(no flag yet)_ | `INSTO_BACKEND` |
| `backend` | `--backend` | `INSTO_BACKEND` |
| `hikerapi.token` | `--hiker-token` | `HIKERAPI_TOKEN` |
| `hikerapi.proxy` | `--proxy` | `HIKERAPI_PROXY` |
| `aiograpi.username` | _(no flag)_ | `AIOGRAPI_USERNAME` |
Expand Down Expand Up @@ -82,11 +82,19 @@ pipx install 'insto[aiograpi]'
pip install 'insto[aiograpi]' # in a venv only — see installation.md
```

For an existing install, inject or reinstall the tool environment:

```sh
pipx inject insto aiograpi
uv tool install --force 'insto[aiograpi]'
```

Then run `insto setup`, pick `aiograpi`, paste your Instagram username + password, and (optionally) the TOTP seed for 2FA.

What works on aiograpi (>= 0.9.6):

- Every command — `/info`, `/posts`, `/reels`, `/stories`, `/highlights`, `/followers`, `/followings`, `/mutuals`, `/comments`, `/captions`, `/likes`, `/wcommented`, `/hashtags`, `/mentions`, `/locations`, `/tagged`, `/similar`, `/direct`, `/direct-thread`, `/collections`, `/saved`, `/dossier`.
- Most target-scoped commands — `/info`, `/posts`, `/reels`, `/stories`, `/highlights`, `/followers`, `/followings`, `/mutuals`, `/comments`, `/captions`, `/likes`, `/wcommented`, `/hashtags`, `/mentions`, `/locations`, `/tagged`, `/similar`, `/direct`, `/direct-thread`, `/collections`, `/saved`, `/dossier`.
- `/reposts` is hikerapi-only today; Instagram's repost surface is not exposed through aiograpi.
- Reads private profiles you follow.
- Login is **lazy** — the constructor stores credentials, the actual `client.login()` fires on the first network call. The session is then dumped to `~/.insto/aiograpi.session.json` (mode `0600`); subsequent runs reuse it without re-authenticating.

Expand Down
2 changes: 1 addition & 1 deletion docs/cli-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ Inline target: `/info instagram`. Active target used otherwise.
| `/posts [N]` | Last N feed posts (default 12). |
| `/reels [N]` | Last N reels — pulled from feed and filtered (default 10). |
| `/tagged [N]` | Posts where the target is tagged (default 10). |
| `/reposts [N]` | Posts the target reposted via IG's repost surface (HikerAPI only). |
| `/reposts [N]` | Posts the target reposted via IG's repost surface (hikerapi only). |
| `/audio <track_id> [N]` | Clips that use a given audio asset id. |
| `/postinfo <ref>` | Resolve a media URL / shortcode / pk → full `Post` DTO. No active target needed. |

Expand Down
4 changes: 3 additions & 1 deletion docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Interactive Instagram OSINT CLI on the [HikerAPI](https://hikerapi.com/p/6k1q138

```sh
uv tool install insto
insto setup # paste your HikerAPI token
insto setup # choose hikerapi token or aiograpi login
insto # REPL with welcome screen
insto -c info instagram # one-shot lookup, no REPL
```
Expand Down Expand Up @@ -81,4 +81,6 @@ See [Backends](backends.md) for the full breakdown.
- [Installation](installation.md) — `uv tool install insto`, dev sources, shell completion.
- [Basic usage](basic-usage.md) — REPL walkthrough, one-shot patterns, pipelines.
- [CLI reference](cli-reference.md) — every command with flags and examples.
- [Backends](backends.md) — hikerapi vs aiograpi setup, trade-offs, risk.
- [Architecture](architecture.md) — how the layers fit together.
- [Troubleshooting](troubleshooting.md) — install, token, backend, and runtime fixes.
11 changes: 7 additions & 4 deletions docs/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,10 +41,11 @@ pipx install 'insto[aiograpi]'
pip install 'insto[aiograpi]' # in a venv only — see PEP 668 below
```

If `insto` is already installed through `pipx` and you later switch to `aiograpi`, add the optional dependency to the existing tool venv:
If `insto` is already installed and you later switch to `aiograpi`, update the existing tool environment instead of running the bare installer again:

```sh
pipx inject insto aiograpi
pipx inject insto aiograpi # existing pipx install
uv tool install --force 'insto[aiograpi]' # existing uv tool install
```

After install, `insto setup` will offer `backend (hikerapi | aiograpi)` and prompt for the credentials of the chosen backend. See [Backends](backends.md) for the trade-offs and the account-ban risk.
Expand Down Expand Up @@ -82,7 +83,7 @@ insto --print-completion bash | sudo tee /etc/bash_completion.d/insto
```sh
git clone git@github.com:subzeroid/insto.git
cd insto
uv sync --extra dev
uv sync --extra aiograpi
uv run insto --help
```

Expand All @@ -96,8 +97,10 @@ insto setup

Interactive wizard. Writes `~/.insto/config.toml` (mode `0600`) with:

- `backend` — `hikerapi` by default, or `aiograpi` for logged-in access.
- `hikerapi.token` — your [HikerAPI](https://hikerapi.com/tokens) access key.
- `hikerapi.proxy` (optional) — `http://`, `https://`, or `socks5h://` proxy URL.
- `aiograpi.username`, `aiograpi.password`, `aiograpi.totp_seed` — only when you pick `aiograpi`.
- `output_dir` — where downloads and exports land (resolved to absolute).
- `db_path` — where the sqlite store lives (default `~/.insto/store.db`).

Expand All @@ -106,6 +109,6 @@ Token can also live in:
- `--hiker-token <value>` — per-call flag (overrides everything else).
- `HIKERAPI_TOKEN` env — overrides the toml file.

Precedence: **flag > env > toml**. Same shape for `--proxy` / `HIKERAPI_PROXY` / `[hikerapi].proxy`.
Precedence: **flag > env > toml**. Same shape for `--proxy` / `HIKERAPI_PROXY` / `[hikerapi].proxy`. Backend selection follows **flag > env > toml > default** via `--backend`, `INSTO_BACKEND`, and `backend = "..."`.

`~/.insto/` is created mode `0700`; `store.db` and `config.toml` are `0600`. The setup wizard refuses to write a world-readable file.
26 changes: 24 additions & 2 deletions docs/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## "no HIKERAPI_TOKEN configured"

You ran `insto` without setting up a token. Either:
You are using the default `hikerapi` backend without setting up a token. Either:

```sh
insto setup # interactive wizard, writes ~/.insto/config.toml
Expand All @@ -14,6 +14,28 @@ insto -c info instagram --hiker-token=hk_live_...

Precedence is **flag > env > toml**.

If you meant to use aiograpi instead, install the optional dependency and run
setup again:

```sh
pipx inject insto aiograpi # existing pipx install
uv tool install --force 'insto[aiograpi]' # existing uv tool install
insto setup # choose aiograpi
```

## "aiograpi backend requested but the `aiograpi` package is not installed"

`aiograpi` is optional. Choosing it in `insto setup` writes credentials, but it
does not mutate the already-installed tool environment. Add the extra:

```sh
pipx inject insto aiograpi # existing pipx install
pipx install 'insto[aiograpi]' # fresh pipx install
uv tool install --force 'insto[aiograpi]' # uv tool install / reinstall
```

Then run `insto setup`, pick `aiograpi`, and launch `insto` again.

## "balance: pending" in the welcome banner

The startup `/sys/balance` call hasn't returned yet. By default `insto` waits up to 2 s for this; on a slow network the banner falls back to "balance: pending" and the actual figure shows up the next time you run `/quota`.
Expand All @@ -31,7 +53,7 @@ What to do:

1. `/health` shows the schema-drift counter for this session.
2. Open an issue with the command you ran and the failing field name.
3. Workarounds typically land in a `0.1.x` patch.
3. Workarounds typically land in the next patch release.

## "filesystem does not support xattr; tagging skipped"

Expand Down
Loading
Loading