Skip to content

Latest commit

 

History

History
269 lines (207 loc) · 10.2 KB

porting_guide.md

File metadata and controls

269 lines (207 loc) · 10.2 KB

Porting a script into the fetcher contract

A port moves a working collection script, such as one from Paramify's older evidence-fetchers repo, into this repo as a fetcher. Port it as-is: keep what it collects and how, and change only the plumbing around it. You end with a fetcher the runner discovers, runs, and wraps in an envelope. The running example ports a made-up script, sentinelone_exclusions.py, to the fetcher sentinelone_exclusions.

Flow: check the source and target path, port it in six steps, then prove it with a fake-credential smoke test and a real run

You need: the repo installed · the source script, plus any shared module it imports · credentials for a real tenant, for step 7

No script to start from? Write a new fetcher. If the source exports scanner findings (Nessus, Wiz), write an issue-report fetcher instead. Your AI agent can do the port: the create-fetcher skill follows these steps (AI agent).


1. Check the source and the target path

Say in one sentence what the script does, and pick its names: the directory is the short name (exclusions), and name: adds the category (sentinelone_exclusions). A script that loops over projects, regions, or hosts becomes a fanout fetcher, with supports_targets: true (new fetcher's fetcher.yaml). Then check that nothing is in the way:

The path is free, the sentinelone category yaml exists, and the category has no _shared/ module

Copy the commands
test ! -d fetchers/<category>/<short_name> && echo "path OK" || echo "ALREADY EXISTS"
ls fetchers/_categories/<category>.yaml 2>/dev/null && echo "category yaml exists" || echo "create the category yaml"
ls fetchers/<category>/_shared/ 2>/dev/null && echo "category has shared code" || echo "no _shared/"

A new category needs fetchers/_categories/<category>.yaml first; an empty file will do. Porting from evidence-fetchers adds one pre-flight check, which needs access to that private repo: the script must still exist there under fetchers/<category>/.

2. List the env vars it reads

Each env var the script reads becomes a secrets: entry in step 4. Check its imports too: a shared module's env reads count, and the module moves to fetchers/<category>/_shared/ unchanged. This one imports none.

grep lists the script's imports (stdlib and requests only) and its two env reads, SENTINELONE_API_URL and SENTINELONE_API_TOKEN

Copy the commands
grep -nE '^(import|from) ' <source script>
grep -nE 'os\.(environ|getenv)|getenv\(' <source script>

3. Create the directory

Name the directory with the short name only. Copy just the template's fetcher.yaml and fetcher.py; a port like sentinelone/agents ships nothing else. A bash port writes fetcher.sh instead, modeled on okta/authenticators.

mkdir and cp create fetchers/sentinelone/exclusions with fetcher.py and fetcher.yaml

Copy the commands
mkdir fetchers/<category>/<short_name>
cp fetchers/_template/fetcher.yaml fetchers/_template/fetcher.py fetchers/<category>/<short_name>/
ls fetchers/<category>/<short_name>

4. Fill in fetcher.yaml

Fill it in as for a new fetcher, with version: 0.1.0 and one secret per env var from step 2. The template has no evidence_set block, so add one. Copy its values from the source's catalog entry if it has one; if not, make up a stable reference_id. Don't base reference_id on a KSI ID: KSI IDs get renumbered, and changing reference_id orphans the evidence set in every workspace.

name: sentinelone_exclusions
version: 0.1.0
description: Lists SentinelOne path, hash, and certificate exclusions, with counts by type.
category: sentinelone

supports_targets: false

runtime:
  type: python
  entry: fetcher.py

output:
  type: json
  path: sentinelone_exclusions.json

secrets:
  - name: api_token
    env: SENTINELONE_API_TOKEN
  - name: api_url
    env: SENTINELONE_API_URL

evidence_set:
  reference_id: EVD-S1-EXCLUSIONS
  name: SentinelOne Exclusions
  instructions: 'Script: sentinelone/exclusions/fetcher.py. Calls /web/api/v2.1/exclusions (cursor pagination) and counts exclusions by type.'

Check that discovery reads it the way you meant:

paramify list shows sentinelone_exclusions as single-target next to its siblings, and paramify describe shows its two required secrets

Copy the commands
paramify list | grep <category>_
paramify describe <category>_<short_name>

5. Write the entry script

Move the collection code into the entry script unchanged, and rebuild only main() on the template's fetcher.py: read EVIDENCE_DIR instead of an --output-dir flag, log instead of print, and exit 1 with report_failure when collection fails. The example's whole change, with get_exclusions untouched:

--- sentinelone_exclusions.py
+++ fetchers/sentinelone/exclusions/fetcher.py
@@ -3,8 +3,18 @@
 
-import argparse
 import json
+import logging
 import os
+import sys
 from collections import Counter
+from pathlib import Path
 
 import requests
+from dotenv import load_dotenv
+
+SCRIPT_DIR = Path(__file__).resolve().parent
+sys.path.insert(0, str(SCRIPT_DIR.parents[1] / "_lib"))
+
+from fetcher_status import report_failure  # noqa: E402
+
+logger = logging.getLogger("sentinelone_exclusions")
 
@@ -26,6 +36,11 @@ def get_exclusions(base_url, token):
 
-def main():
-    parser = argparse.ArgumentParser()
-    parser.add_argument("--output-dir", default="evidence")
-    args = parser.parse_args()
+def main() -> int:
+    logging.basicConfig(
+        level=os.environ.get("LOG_LEVEL", "INFO"),
+        format="%(asctime)s %(levelname)s %(name)s %(message)s",
+    )
+    load_dotenv()
+
+    output_dir = Path(os.environ.get("EVIDENCE_DIR", "./evidence"))
+    output_dir.mkdir(parents=True, exist_ok=True)
 
@@ -43,10 +58,13 @@ def main():
     except Exception as e:
-        print(f"Error fetching exclusions: {e}")
         result = {"status": "error", "message": str(e), "exclusions": []}
 
-    os.makedirs(args.output_dir, exist_ok=True)
-    path = os.path.join(args.output_dir, "sentinelone_exclusions.json")
-    with open(path, "w") as f:
+    output_path = output_dir / "sentinelone_exclusions.json"
+    with open(output_path, "w") as f:
         json.dump(result, f, indent=2)
-    print(f"Wrote {path}")
+    logger.info("Evidence saved to %s", output_path)
+
+    if result["status"] != "success":
+        report_failure(result["message"])
+        return 1
+    return 0
 
@@ -54,2 +72,2 @@ def main():
 if __name__ == "__main__":
-    main()
+    sys.exit(main())

6. Smoke-test it with fake credentials

Before you touch a real tenant, run it with fake values. The wiring works if it fails at the network: exit 1, with the reason as the last log line.

With a fake token and URL, the fetcher saves its file, logs a DNS failure for fake.example as an ERROR, and exits 1

Copy the command
<ENV_VAR>=fake-token <ANOTHER_ENV_VAR>=https://fake.example \
EVIDENCE_DIR=/tmp/paramify-verify python fetchers/<category>/<short_name>/fetcher.py; echo "exit: $?"

7. Run it end to end

Export the real credentials in the shell you run from, or have your secret manager set them. Then build a manifest for the fetcher:

paramify manifest init, add, and two set-secret calls, each reporting what's still missing, then paramify validate reporting the manifest valid

Copy the commands
paramify manifest init
paramify manifest add <category>_<short_name>
paramify manifest set-secret <category>_<short_name> <secret> <ENV_VAR>   # once per secret
paramify validate manifest.yaml

Run it, then open the evidence file. The runner wraps it in an evidence envelope: metadata should carry your evidence_set, and payload should match what the old script wrote. These pictures ran against a local stand-in for the SentinelOne API, with synthetic data.

paramify run reports OK with exit 0, then paramify evidence shows metadata with the EVD-S1-EXCLUSIONS evidence set and a payload counting three exclusions by type

Copy the commands
paramify run manifest.yaml
paramify evidence evidence/run-<timestamp>/<category>_<short_name>.json

Next, upload the run and write validators for it.


Troubleshooting

Symptom Fix
paramify list fails with schema validation failed The message names the file and the field. Fix fetcher.yaml against step 4.
ModuleNotFoundError in the smoke test Fix the imports: put the _shared/ directory on sys.path, as the commented-out line in the template's fetcher.py does.
Missing required env var, or a KeyError naming one The env var name in fetcher.yaml or your test command doesn't match the script. Re-check step 2.
The smoke test exits 0 with empty data The script catches the failure and reports success, as the example's source did. Exit 1 with report_failure (step 5).
Secret reference ${env:…} could not be resolved The runner reads secrets from its own environment, not from .env. Export them in the shell you run from.
A failed run's metadata.error reads "Evidence saved to …" Nothing reported the reason, so the runner used the end of stderr. Call report_failure after the saved line.

More detail: checking against the contract · following one run