From 9e681cef4030db580780d030e71431b5b22d7ab0 Mon Sep 17 00:00:00 2001 From: subzeroid <143403577+subzeroid@users.noreply.github.com> Date: Thu, 28 May 2026 05:57:05 +0300 Subject: [PATCH] docs: refresh release documentation for v0.7.13 --- CHANGELOG.md | 2 +- CONTRIBUTING.md | 15 ++++++----- README.md | 22 +++++++++------- SECURITY.md | 6 ++--- docs/architecture.md | 9 ++++--- docs/backends.md | 14 +++++++--- docs/cli-reference.md | 2 +- docs/index.md | 4 ++- docs/installation.md | 11 +++++--- docs/troubleshooting.md | 26 +++++++++++++++++-- .../why-instagram-bans-your-osint-account.md | 14 +++++----- uv.lock | 2 +- 12 files changed, 84 insertions(+), 43 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e43aa09..ccb0f3a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,7 +7,7 @@ All notable changes to insto. Format follows [Keep a Changelog](https://keepacha ### Fixed -* clarify backend setup names ([#37](https://github.com/subzeroid/insto/issues/37)) ([0ed435a](https://github.com/subzeroid/insto/commit/0ed435ad9ff81daccc706f10845dee58ca7193d1)) +* clarify hikerapi setup prompts and aiograpi install hints ([#37](https://github.com/subzeroid/insto/issues/37)) ([0ed435a](https://github.com/subzeroid/insto/commit/0ed435ad9ff81daccc706f10845dee58ca7193d1)) ### Documentation diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 6581a50..f649301 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -7,17 +7,20 @@ Thanks for considering a contribution. This document covers the dev workflow. ```bash git clone git@github.com:subzeroid/insto.git cd insto -uv sync --extra dev +uv sync --extra aiograpi ``` Python 3.11+ required (we use `dataclass(slots=True)`, `X | Y` unions, `datetime.fromisoformat` with `Z`). +This matches CI: dev dependencies are installed by default, and the +`aiograpi` extra exercises the optional logged-in backend module. + ## Running tests ```bash uv run pytest # all tests uv run pytest -k hiker # subset -uv run pytest --cov=insto --cov-report=term-missing +uv run pytest --cov=insto --cov-report=term-missing --cov-fail-under=75 ``` The suite is fully offline — no real HikerAPI or Instagram calls. There's also a structured live smoke against the real HikerAPI: @@ -28,13 +31,13 @@ HIKERAPI_TOKEN_TEST= uv run python tests/live/smoke.py Nine REQ checks (resolve / profile / posts / followers / tagged / hashtag / search / quota / 404) plus one OPT check (`/similar`, per-target flaky). Skips with exit 0 if `HIKERAPI_TOKEN_TEST` is unset, so it's safe to wire into release-prep gates. Costs ~10 requests, single-digit cents. **Run before each release tag** — caught a real `iter_hashtag_posts` bug that mocks couldn't. -Coverage targets: keep pure-logic modules at 100% (`models`, `_redact`, `exceptions`, mappers). Everything else: 90%+ on touched code. +Coverage targets: keep pure-logic modules at 100% (`models`, `_redact`, `exceptions`, mappers). CI currently gates the whole package at 75%; touched code should not lower coverage without a clear reason. ## Lint + types ```bash -uv run ruff check insto tests -uv run ruff format --check insto tests +uv run ruff check +uv run ruff format --check uv run mypy insto ``` @@ -67,7 +70,7 @@ insto/ ├── ui/ # banner, theme, render helpers ├── backends/ # OSINTBackend ABC, HikerBackend, AiograpiBackend, mappers ├── service/ # facade, history, analytics, exporter, watch -└── commands/ # one file per group (target/profile/media/...) +└── commands/ # one file per group (target/profile/media/direct/saved/...) tests/ ├── fakes.py # FakeBackend with per-method error injection ├── fixtures/hiker/ # frozen HikerAPI dict responses per access state diff --git a/README.md b/README.md index 8e233d6..d7c51e2 100644 --- a/README.md +++ b/README.md @@ -1,13 +1,13 @@ # insto -Interactive Instagram OSINT CLI on the [HikerAPI](https://hikerapi.com/p/6k1q1388) backend and [aiograpi](https://github.com/subzeroid/aiograpi) lib. +Interactive Instagram OSINT CLI on the [HikerAPI](https://hikerapi.com/p/6k1q1388) backend and optional [aiograpi](https://github.com/subzeroid/aiograpi) logged-in backend. ![demo](docs/demo.gif) ## Design choices - **HikerAPI is the default backend, not a side mode.** Quota balance is read on REPL startup and surfaced in the bottom toolbar; `with_retry` honours `RateLimited.retry_after`; every mapper raises typed `SchemaDrift(endpoint, missing_field)` when HikerAPI's documented fields move. Logged-in `aiograpi` is one extra (`pipx install 'insto[aiograpi]'`) when you actually need data behind the login wall — but kept off the default path so your account isn't in scope. -- **Async, typed, tested.** Python ≥ 3.11, strict mypy, ~93% coverage, ruff-clean. Backends, facade, commands all `async def`. +- **Async, typed, tested.** Python >= 3.11, strict mypy, ruff-clean, and 900+ offline tests. Backends, facade, commands all `async def`. - **Two surfaces, one grammar.** A prompt-toolkit REPL with slash-popup completion and live `/watch` notifications, *and* a Unix-friendly one-shot mode (`insto @user -c info`). `--json -` and `--csv -` write to stdout; `/batch -` reads targets from stdin. - **Snapshot / watch / diff.** Persisted in `~/.insto/store.db`. Poll a target on an interval; diff against the last snapshot. - **Maltego CSV export** out of the box (`--maltego` on any flat-row command, plus full `/dossier`). @@ -42,11 +42,12 @@ uv tool install 'insto[aiograpi]' pipx install 'insto[aiograpi]' ``` -If `insto` is already installed through `pipx` and you later switch to -`aiograpi`, add the optional dependency to the existing tool venv: +If `insto` is already installed and you later switch to `aiograpi`, update the +existing tool environment instead of running the bare installer again: ```sh -pipx inject insto aiograpi +pipx inject insto aiograpi # existing pipx install +uv tool install --force 'insto[aiograpi]' # existing uv tool install ``` `insto setup` then offers a `hikerapi | aiograpi` choice and prompts for @@ -84,9 +85,10 @@ insto setup ``` Interactive wizard. Writes `~/.insto/config.toml` (mode `0600`) with your -HikerAPI token, output directory, sqlite store path, and optional proxy. -The token is read with `getpass` so it does not echo to the terminal; pass -`-` for the proxy to clear a previously-saved value. +backend choice, HikerAPI token or aiograpi credentials, output directory, +sqlite store path, and optional proxy. The HikerAPI token prompt links to +. Secrets are read with `getpass` so they do not +echo to the terminal; pass `-` for the proxy to clear a previously-saved value. > 💸 **HikerAPI gives you 100 free requests** to try things out — no card, > no email-verification dance, just sign up and a token is waiting: @@ -124,7 +126,7 @@ supported. | Maltego CSV export | ✅ | ❌ | ❌ | ❌ | | Interactive REPL | ✅ prompt-toolkit, slash-popup | ✅ basic shell | ❌ one-shot | ❌ one-shot | | One-shot / scriptable | ✅ stdin/stdout pipes | ⚠️ shell only | ✅ | ✅ | -| Type-safe / strict mypy | ✅ ~93% coverage | ❌ | ❌ | ❌ | +| Type-safe / strict mypy | ✅ strict mypy + coverage gate | ❌ | ❌ | ❌ | **When to pick insto** — Instagram-specific OSINT where you want HikerAPI on the default path (no IG session is ever in scope), deep network/geo analytics, snapshots over time, and Maltego-ready export. Modern stack — asyncio, strict mypy, prompt-toolkit REPL with slash-popup completion. @@ -151,7 +153,7 @@ $ insto @nasa i n s t o ⇋ o s i n t @instagram instagram tool · open-source intel - hiker · 14.7M requests left · $4,417 · 15 rps cap + hikerapi · 14.7M requests left · $4,417 · 15 rps cap insto @→ / ``` diff --git a/SECURITY.md b/SECURITY.md index 295e215..9280438 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -6,7 +6,7 @@ insto is pre-1.0. Only the latest published release receives security fixes. | Version | Supported | | ------- | ------------------ | -| 0.1.x | :white_check_mark: | +| Latest published release | :white_check_mark: | ## Reporting a vulnerability @@ -33,7 +33,7 @@ In scope: - Code under `insto/` and `tests/`. - Default deployment via `pip install insto` / `uv tool install insto`. -- Anything that allows reading secrets (`hikerapi_token`, snapshots) from logs, error output, or `~/.insto/`. +- Anything that allows reading secrets (`hikerapi.token`, aiograpi credentials, snapshots) from logs, error output, or `~/.insto/`. - Any path-traversal, MIME-confusion, or remote-content-trust issue in the CDN streamer. - Schema-drift or rate-limit handling that could exfiltrate the operator's IP / token. @@ -45,4 +45,4 @@ Out of scope: ## Operator notes -`insto` is offensive-tooling-adjacent: it queries third-party data, downloads remote media, and stores intel on disk. Even fully patched, the local store at `~/.insto/store.db` is not encrypted at rest in 0.1.x. A stolen laptop is still a data-loss event. Treat the output directory like any other source of sensitive material. +`insto` is offensive-tooling-adjacent: it queries third-party data, downloads remote media, and stores intel on disk. Even fully patched, the local store at `~/.insto/store.db` is not encrypted at rest. A stolen laptop is still a data-loss event. Treat the output directory like any other source of sensitive material. diff --git a/docs/architecture.md b/docs/architecture.md index 3a48f01..e76cd1b 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -5,7 +5,8 @@ Six layers, top to bottom. The rule that holds the design together: each layer t ```text UI: REPL (prompt_toolkit) │ one-shot CLI (argparse) Dispatch: parse → validate → run → render -Commands: commands/{target,profile,media,network,content,interactions,batch,watch,operational,dossier}.py +Commands: commands/{target,profile,media,network,content,interactions,discovery, + direct,saved,places,batch,watch,operational,dossier}.py Service: facade · history · analytics · exporter · watch Backends: OSINTBackend ABC · HikerBackend · AiograpiBackend Models: @dataclass(slots=True) DTOs — Profile, Post, Story, User, Comment, Quota, ... @@ -15,7 +16,7 @@ Models: @dataclass(slots=True) DTOs — Profile, Post, Story, User, Comment, - **Async everywhere.** `httpx` (transitive via `hikerapi`), `asyncio` for fan-out, `asyncio.to_thread` for sqlite calls. - **Backend boundary is a hard wall.** Raw HikerAPI / aiograpi dicts never leave `backends/`. Mappers in `_hiker_map.py` and `_aiograpi_map.py` are the only converters. -- **Lazy backend imports.** `import hikerapi` and `import aiograpi` happen only inside `make_backend(...)`. Import errors stay localized. +- **Lazy backend imports.** `import hikerapi` and `import aiograpi` happen only inside `make_backend(...)`. Missing optional dependencies stay localized and surface with install hints. - **Retry / backoff lives in one place.** `backends/_retry.py` decorates SDK-method calls inside `HikerBackend`; commands never know retries exist. - **CDN streaming through a single helper.** `backends/_cdn.py` is the only code that pulls untrusted bytes off the network. Host allowlist, MIME sniff, byte budget, atomic write — every download passes through it. - **Pagination as `AsyncIterator[T]` + `limit: int | None`.** Every collection method is an async generator. Cursor management lives inside the backend; commands consume one item at a time and stop on `limit`. @@ -85,8 +86,8 @@ Session limits: max 3 active watches, 5-minute floor on the interval, all watche ## Test strategy -- 850+ unit + integration tests, no live API calls in CI. +- 900+ unit + integration tests, no live API calls in CI. - Fixtures: one frozen HikerAPI dict per profile-access state (`public`, `private`, `deleted`, `empty`, `schema_drift`). - `tests/fakes.py:FakeBackend` implements `OSINTBackend` from fixtures with per-method error injection covering every entry of the error taxonomy. - 3 e2e flows under `tests/e2e/`: subprocess one-shot, prompt_toolkit pty REPL session, `/watch` tick with `patch_stdout` capture. -- Strict mypy + ruff format + ruff lint as CI gates. +- Strict mypy + ruff format + ruff lint as CI gates; pytest coverage must stay at or above the current CI floor. diff --git a/docs/backends.md b/docs/backends.md index b616ff3..2f33e3d 100644 --- a/docs/backends.md +++ b/docs/backends.md @@ -13,7 +13,7 @@ The contract lives in `insto/backends/_base.py:OSINTBackend`. Two implementation | Sees private accounts you follow | No | Yes | | Sees DMs / saved feed | No | Read-only Direct threads/messages and saved collections/media exposed; personal feed intentionally not exposed | | Quota visibility | Yes (`/sys/balance`) | No | -| Install footprint | base | `pip install 'insto[aiograpi]'` | +| Install footprint | base | `uv tool install 'insto[aiograpi]'` / `pipx install 'insto[aiograpi]'` | ## Pick a backend @@ -38,7 +38,7 @@ Precedence is **flag > env > toml > default** for every key: | Key | Flag | Env | |---|---|---| -| `backend` | _(no flag yet)_ | `INSTO_BACKEND` | +| `backend` | `--backend` | `INSTO_BACKEND` | | `hikerapi.token` | `--hiker-token` | `HIKERAPI_TOKEN` | | `hikerapi.proxy` | `--proxy` | `HIKERAPI_PROXY` | | `aiograpi.username` | _(no flag)_ | `AIOGRAPI_USERNAME` | @@ -82,11 +82,19 @@ pipx install 'insto[aiograpi]' pip install 'insto[aiograpi]' # in a venv only — see installation.md ``` +For an existing install, inject or reinstall the tool environment: + +```sh +pipx inject insto aiograpi +uv tool install --force 'insto[aiograpi]' +``` + Then run `insto setup`, pick `aiograpi`, paste your Instagram username + password, and (optionally) the TOTP seed for 2FA. What works on aiograpi (>= 0.9.6): -- Every command — `/info`, `/posts`, `/reels`, `/stories`, `/highlights`, `/followers`, `/followings`, `/mutuals`, `/comments`, `/captions`, `/likes`, `/wcommented`, `/hashtags`, `/mentions`, `/locations`, `/tagged`, `/similar`, `/direct`, `/direct-thread`, `/collections`, `/saved`, `/dossier`. +- Most target-scoped commands — `/info`, `/posts`, `/reels`, `/stories`, `/highlights`, `/followers`, `/followings`, `/mutuals`, `/comments`, `/captions`, `/likes`, `/wcommented`, `/hashtags`, `/mentions`, `/locations`, `/tagged`, `/similar`, `/direct`, `/direct-thread`, `/collections`, `/saved`, `/dossier`. +- `/reposts` is hikerapi-only today; Instagram's repost surface is not exposed through aiograpi. - Reads private profiles you follow. - Login is **lazy** — the constructor stores credentials, the actual `client.login()` fires on the first network call. The session is then dumped to `~/.insto/aiograpi.session.json` (mode `0600`); subsequent runs reuse it without re-authenticating. diff --git a/docs/cli-reference.md b/docs/cli-reference.md index 08c6452..e19c6cf 100644 --- a/docs/cli-reference.md +++ b/docs/cli-reference.md @@ -71,7 +71,7 @@ Inline target: `/info instagram`. Active target used otherwise. | `/posts [N]` | Last N feed posts (default 12). | | `/reels [N]` | Last N reels — pulled from feed and filtered (default 10). | | `/tagged [N]` | Posts where the target is tagged (default 10). | -| `/reposts [N]` | Posts the target reposted via IG's repost surface (HikerAPI only). | +| `/reposts [N]` | Posts the target reposted via IG's repost surface (hikerapi only). | | `/audio [N]` | Clips that use a given audio asset id. | | `/postinfo ` | Resolve a media URL / shortcode / pk → full `Post` DTO. No active target needed. | diff --git a/docs/index.md b/docs/index.md index 9204e17..f3cb642 100644 --- a/docs/index.md +++ b/docs/index.md @@ -6,7 +6,7 @@ Interactive Instagram OSINT CLI on the [HikerAPI](https://hikerapi.com/p/6k1q138 ```sh uv tool install insto -insto setup # paste your HikerAPI token +insto setup # choose hikerapi token or aiograpi login insto # REPL with welcome screen insto -c info instagram # one-shot lookup, no REPL ``` @@ -81,4 +81,6 @@ See [Backends](backends.md) for the full breakdown. - [Installation](installation.md) — `uv tool install insto`, dev sources, shell completion. - [Basic usage](basic-usage.md) — REPL walkthrough, one-shot patterns, pipelines. - [CLI reference](cli-reference.md) — every command with flags and examples. +- [Backends](backends.md) — hikerapi vs aiograpi setup, trade-offs, risk. - [Architecture](architecture.md) — how the layers fit together. +- [Troubleshooting](troubleshooting.md) — install, token, backend, and runtime fixes. diff --git a/docs/installation.md b/docs/installation.md index edb9706..7b8a9fb 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -41,10 +41,11 @@ pipx install 'insto[aiograpi]' pip install 'insto[aiograpi]' # in a venv only — see PEP 668 below ``` -If `insto` is already installed through `pipx` and you later switch to `aiograpi`, add the optional dependency to the existing tool venv: +If `insto` is already installed and you later switch to `aiograpi`, update the existing tool environment instead of running the bare installer again: ```sh -pipx inject insto aiograpi +pipx inject insto aiograpi # existing pipx install +uv tool install --force 'insto[aiograpi]' # existing uv tool install ``` After install, `insto setup` will offer `backend (hikerapi | aiograpi)` and prompt for the credentials of the chosen backend. See [Backends](backends.md) for the trade-offs and the account-ban risk. @@ -82,7 +83,7 @@ insto --print-completion bash | sudo tee /etc/bash_completion.d/insto ```sh git clone git@github.com:subzeroid/insto.git cd insto -uv sync --extra dev +uv sync --extra aiograpi uv run insto --help ``` @@ -96,8 +97,10 @@ insto setup Interactive wizard. Writes `~/.insto/config.toml` (mode `0600`) with: +- `backend` — `hikerapi` by default, or `aiograpi` for logged-in access. - `hikerapi.token` — your [HikerAPI](https://hikerapi.com/tokens) access key. - `hikerapi.proxy` (optional) — `http://`, `https://`, or `socks5h://` proxy URL. +- `aiograpi.username`, `aiograpi.password`, `aiograpi.totp_seed` — only when you pick `aiograpi`. - `output_dir` — where downloads and exports land (resolved to absolute). - `db_path` — where the sqlite store lives (default `~/.insto/store.db`). @@ -106,6 +109,6 @@ Token can also live in: - `--hiker-token ` — per-call flag (overrides everything else). - `HIKERAPI_TOKEN` env — overrides the toml file. -Precedence: **flag > env > toml**. Same shape for `--proxy` / `HIKERAPI_PROXY` / `[hikerapi].proxy`. +Precedence: **flag > env > toml**. Same shape for `--proxy` / `HIKERAPI_PROXY` / `[hikerapi].proxy`. Backend selection follows **flag > env > toml > default** via `--backend`, `INSTO_BACKEND`, and `backend = "..."`. `~/.insto/` is created mode `0700`; `store.db` and `config.toml` are `0600`. The setup wizard refuses to write a world-readable file. diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index d7c9a6f..040e45e 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -2,7 +2,7 @@ ## "no HIKERAPI_TOKEN configured" -You ran `insto` without setting up a token. Either: +You are using the default `hikerapi` backend without setting up a token. Either: ```sh insto setup # interactive wizard, writes ~/.insto/config.toml @@ -14,6 +14,28 @@ insto -c info instagram --hiker-token=hk_live_... Precedence is **flag > env > toml**. +If you meant to use aiograpi instead, install the optional dependency and run +setup again: + +```sh +pipx inject insto aiograpi # existing pipx install +uv tool install --force 'insto[aiograpi]' # existing uv tool install +insto setup # choose aiograpi +``` + +## "aiograpi backend requested but the `aiograpi` package is not installed" + +`aiograpi` is optional. Choosing it in `insto setup` writes credentials, but it +does not mutate the already-installed tool environment. Add the extra: + +```sh +pipx inject insto aiograpi # existing pipx install +pipx install 'insto[aiograpi]' # fresh pipx install +uv tool install --force 'insto[aiograpi]' # uv tool install / reinstall +``` + +Then run `insto setup`, pick `aiograpi`, and launch `insto` again. + ## "balance: pending" in the welcome banner The startup `/sys/balance` call hasn't returned yet. By default `insto` waits up to 2 s for this; on a slow network the banner falls back to "balance: pending" and the actual figure shows up the next time you run `/quota`. @@ -31,7 +53,7 @@ What to do: 1. `/health` shows the schema-drift counter for this session. 2. Open an issue with the command you ran and the failing field name. -3. Workarounds typically land in a `0.1.x` patch. +3. Workarounds typically land in the next patch release. ## "filesystem does not support xattr; tagging skipped" diff --git a/marketing/why-instagram-bans-your-osint-account.md b/marketing/why-instagram-bans-your-osint-account.md index 428f546..6e3d366 100644 --- a/marketing/why-instagram-bans-your-osint-account.md +++ b/marketing/why-instagram-bans-your-osint-account.md @@ -102,7 +102,7 @@ Concretely, what changed because of those layers: **Async everywhere.** Backends, facade, commands. Python 3.11+. sqlite via stdlib `sqlite3` wrapped in `asyncio.to_thread`. Strict mypy, -ruff-clean, ~93% coverage. CI gates on all of it. +ruff-clean, 900+ offline tests. CI gates on all of it. **Typed errors, one retry policy.** `with_retry` is the only place that handles `RateLimited.retry_after` sleeps and `Transient` retries. @@ -178,11 +178,11 @@ insto @nasa -c info # one-shot insto # drops into the REPL ``` -The default `hiker` backend needs a HikerAPI token. The free tier -(100 requests after registration) covers basic exploration; paid -tiers scale from there. The `aiograpi` backend is opt-in via the -extra and uses your own IG account — recommended only on a -dedicated burner. +The default `hikerapi` backend needs a HikerAPI token from +. The free tier (100 requests after +registration) covers basic exploration; paid tiers scale from there. +The `aiograpi` backend is opt-in via the `insto[aiograpi]` extra and +uses your own IG account — recommended only on a dedicated burner. ## Honest trade-offs @@ -194,7 +194,7 @@ dedicated burner. `OSINTBackend` ABC is designed so a new provider drops in as a ~300-line module, but the dependency is real today. - **Younger project.** Osintgram has 12k stars and five years of bug - reports against real targets. `insto` has ~93% test coverage and + reports against real targets. `insto` has 900+ offline tests and strict typing, but the long tail of "Instagram returned this weird shape on Tuesday" is shorter. File issues — they're how the schema drift counter pays for itself. diff --git a/uv.lock b/uv.lock index fb1e670..2fd2e7a 100644 --- a/uv.lock +++ b/uv.lock @@ -409,7 +409,7 @@ wheels = [ [[package]] name = "insto" -version = "0.7.12" +version = "0.7.13" source = { editable = "." } dependencies = [ { name = "hikerapi" },