docs: address four skill-doc issues from #50#51
Open
ttokttokttok wants to merge 13 commits into
Open
Conversation
…at note
- Pagination: API and apify-client both return all items by default, not
250K. Verified against apify-client/dist/resource_clients/dataset.js
("Default is all items") and docs.apify.com/api/v2/dataset-items-get
("By default there is no limit").
- Actor ID format: Finding the Right Actor section never said what an ID
looks like. Agents fabricate plausible-but-wrong IDs from memory
(e.g. "apify/google-maps-scraper" when the real top result is
"compass/crawler-google-places"). Adds one-line guidance with examples.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…xception - Generic fallback in Step 1: when both actor-index and dynamic search return nothing usable, fall back to a generic crawler picked by site rendering (cheerio for static HTML, playwright for SPA, camoufox for anti-bot). Previously left undefined, agents would stall or guess. - Quick-answer schema exception in Step 2: Step 2 says always fetch the input schema, Step 3's quick-answer mode says skip preferences for trivial lookups. For obvious inputs (e.g. a username) the schema fetch is unnecessary — exception preserves the PPE cost-gate. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…, not `id` The manifest has both `id` and `name` slug-like fields per entry. `-t` accepts only `name`. Anyone scaling beyond the three documented defaults naturally picks the more-prominent `id` field and hits "Could not find the selected template" — e.g. `-t ts-crawlee-cheerio` errors; `-t project_cheerio_crawler_ts` works. One-line clarification + manifest link. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…fier Per review feedback from @vivek100: the "Exception for quick-answer lookups" callout was too prescriptive — replace it with a single follow-on sentence right after the schema-fetch command. Same net effect (agent may skip the fetch for trivial inputs, still checks pricing for PPE) but reads as a natural qualifier instead of a separate carve-out. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…rative line Replace the follow-on "For trivial inputs..." sentence (added in 516ae13) with a conditional appended directly to the original Step 2 imperative: "Fetch the input schema dynamically, unless you already know the input fields:" Gives the agent a checkable criterion (do I know the fields for this Actor?) rather than a vague hedge ("usually") or a separate exception callout. The PPE pricing reminder is dropped here — gotchas.md already mandates the full cost-estimation protocol, so the inline note was redundant. Net diff against upstream main for this file is now a single line. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Per review feedback from @vivek100: the previous wording ("paginate to bound memory and latency") was generic dev advice that didn't tell the agent what to actually do. Replace with: Don't pull large datasets in one read — pass `limit`/`offset`, or write to a file and process in chunks. (`listItems()` returns all items by default.) Lead with the failure mode (don't pull in one read), give two concrete techniques (limit/offset OR file), keep the factual correction (returns all by default) as a parenthetical. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two-layer eval set, one file per skill:
evals/
├── README.md
├── routing.json cross-skill routing (10 evals)
├── apify-actor-development.json behavioral, type=build|cli (8)
├── apify-sdk-integration.json behavioral, type=discovery|integrate (9)
└── apify-ultimate-scraper.json behavioral, type=selection|run|cost|
schema-and-gotchas|workflow:<name> (89)
Schema per behavioral entry: {prompt, expected, type}. Workflow evals use
`type: "workflow:<workflow-name>"` so a runner can filter all workflows at
once (startsWith) while preserving which workflow a specific entry tests.
Routing schema: {prompt, expected_skill, acceptable_skills, rationale}.
A pick outside expected ∪ acceptable is a fail and points at the description
to fix — eval-first refactors then become small description PRs rather than
rewrites.
116 entries total. README documents the layout, schemas, success criteria
(outcome / process / style / efficiency), and how to run each layer.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The behavioral evals run single-shot — one prompt, no prior turns, no
conversational context. Prompts like "This Actor takes ~20 minutes…" or
"Okay, the run finished — wrap it up for me" reference state ("this Actor",
"the run") that the agent never received in a one-shot eval, so they test
nothing meaningful.
Removed 10 from apify-ultimate-scraper.json and 2 from apify-sdk-integration.json:
- ultimate-scraper: "From the search results, what's the Actor ID I should use?",
"I picked apify/instagram-scraper. Let's go.", "Use this scraper to pull 500
Instagram posts.", "I'm getting empty results from this social-media Actor
and I know the account exists.", "How do I limit this to just 100 items?
I tried `maxResults: 100` but it ignored me.", "Run a large scrape that
will take a while and let me check on it later.", "Give me the results as a
CSV file.", "Okay, the run finished — wrap it up for me.", "Scrape 2,000
results from this pay-per-event Actor.", "Run this PPE Actor for 50,000
items."
- sdk-integration: "This Actor takes ~20 minutes to finish. How should I run
it from my app without blocking?", "Where do I get the scraped data after
the run finishes?"
README updated: counts (116 -> 104) and added a note that all remaining
prompts must stand on their own as single-shot evals.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ed prompts
The earlier deletion removed prompts that only made sense mid-conversation,
which gutted the schema-and-gotchas type (4 -> 0) and thinned the run type
(5 -> 2). The behaviors those tested (schema-fetch-before-run, gotchas check,
cookie/auth diagnosis, field-name variance, async run, retrieve+save,
deliver) are still in the skill and worth testing — they just need
self-contained prompts.
Added:
ultimate-scraper.json (+9):
+4 schema-and-gotchas: schema-before-run, gotchas check, cookie/auth
diagnosis, field-name variance — each names a specific Actor
+3 run: async pattern, CSV save with explicit dataset id, deliver with
explicit run id
+2 cost: PPE cost gate (junglee/Amazon-crawler), high-cost confirmation
(harvestapi/linkedin-profile-scraper)
sdk-integration.json (+2):
+2 integrate: async pattern (apify/website-content-crawler from Node),
result retrieval (after .call() with run object in hand)
Final counts:
routing.json 10
apify-actor-development.json 8
apify-sdk-integration.json 9
apify-ultimate-scraper.json 88
TOTAL 115
Ultimate-scraper type breakdown: selection=5, schema-and-gotchas=4, run=5,
cost=5, workflow:*=69. All new prompts stand on their own as single-shot
evals — they name specific Actor IDs and provide any prior state (run id,
dataset id, pricing model) inline.
README counts updated.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…and effectively also revert the pagination half of the original docs(sdk-integration) commit (`899e198`), restoring the upstream wording. Dropping the pagination change from this PR. On reflection, the 250K-cap factual correction is low-impact (most datasets are well under that, and agent context blows at far smaller sizes regardless), and the agent-centric "write to a file" advice we layered on per review felt off-brand for a doc whose audience is also human app developers. The Actor-ID format note added in `899e198` stays — that one's unambiguously useful and eval-backed. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…lope)
The CLI prints results at the top level (`items`, `.id`, `.status`); only
the api.apify.com/v2 REST API uses the `{ data: { ... } }` wrapper. Agents
were defaulting to `obj.data.items` (the REST shape) when parsing CLI search
output, yielding silently-empty results. Add a global rule plus a literal
example output block so the correct shape is unmissable.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Root-caused from an eval run where the agent picked the wrong Actor, never opened actor-index.md, and burned ~25 calls reverse-engineering an input schema (then wrote a bad apify-client call from memory). apify-ultimate-scraper/SKILL.md - Add "Critical: do not trust internal knowledge" — verify IDs/fields/methods, never write integration code from memory. - Select Actors by source platform, not the verb in the request. - Always read actor-index.md FIRST (it's grouped by platform); define when a task is multi-step and that the workflow guide exists for the handoff. - Fix Step 2: `apify actors info … --input` WITHOUT `--json` prints the schema; `--json` returns the whole Actor object and buries it under taggedBuilds.latest.build.actorDefinition.input. Note the Rule apify#1 exception. apify-sdk-integration/SKILL.md - Get schemas via the CLI (`apify actors info … --input`); do NOT read `.inputSchema` off `client.actor(id).get()` — the Actor object lacks it. - Boundary: not for one-off scrapes — use the CLI via apify-ultimate-scraper. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-file Two fixes root-caused from an eval run where the agent abandoned the CLI for apify-client (then hand-rolled cheerio) after a silent failure. - Rule apify#3: keep `2>/dev/null` for clean JSON, but on empty/unexpected output re-run WITHOUT it and read the error before switching approach. The CLI's errors are specific and name the fix; suppressing them turns a recoverable failure into a tool-switch spiral. - Step 3: `-i` is inline JSON only; pass a file with `--input-file=PATH` (or `-f`). `-i @PATH` is a curl-ism the CLI rejects ("Providing a JSON file path in the --input flag is not supported"). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #50 — see the issue for full context (the eval prompts that surfaced each issue, what we observed, why it happens, and the proposed fix).
Summary
Doc clarifications across three skills so an agent reliably picks the right Actor, reads input schemas the correct way, and doesn't abandon the CLI on an opaque failure — plus a companion evals framework. Root-caused from agent eval runs where the agent chose a suboptimal Actor, skipped
actor-index.md, burned ~25 tool calls reverse-engineering a schema, and (in a later run) bailed from the CLI toapify-client→ hand-rolledcheerioafter a silently-suppressed error.Net skill diff vs
main: 32 insertions, 8 deletions across threeSKILL.mdfiles (mostly additive). Theevals/folder is the remainder.apify-ultimate-scraperactor-index.mdfirst — it's grouped by platform and flags the recommended tier, so a source's native toolkit shows together. Defines when a task is multi-step (one Actor's output must feed another) and that the workflow guide carries the handoff.apify actors info … --inputwithout--jsonprints the schema directly; with--jsonit returns the whole Actor object and buries the schema undertaggedBuilds.latest.build.actorDefinition.input. Documented as the explicit exception to the global "always--json" rule.2>/dev/nullfor clean JSON, but on empty/unexpected output re-run without it and read the error before switching approach. Suppressing the CLI's (specific, fix-naming) errors turns a recoverable failure into a CLI→SDK→hand-rolled spiral.--input-filefor file input —-iis inline JSON only; pass a file with--input-file=PATH(or-f).-i @PATHis a curl-ism the CLI rejects.--jsonis unwrapped (results underitems, nodataenvelope) + a literal example block; generic-crawler fallback rule.apify-sdk-integrationapify actors info … --input); do not read.inputSchemaoffclient.actor(id).get()— the Actor object doesn't include it.apify-ultimate-scraper; reach forapify-clientonly when building an app.apify-actor-developmentapify create -ttakes the manifest'sname, not itsid.Companion:
evals/folderA small two-layer eval framework (
routing.json+ per-skill behavioral files) that surfaced these issues. Kept in its own commits so it can be dropped or factored out if you'd rather not adopt evals in this repo.How to review
🤖 Generated with Claude Code