From 037a42ed73354c8b3eabd9dcc28236152602ef9e Mon Sep 17 00:00:00 2001 From: Marco Vinciguerra <88108002+VinciGit00@users.noreply.github.com> Date: Thu, 25 Jun 2026 09:50:16 +0200 Subject: [PATCH 1/3] ci: run only deterministic unit suites in Test Suite workflow (#1095) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Unit Tests job ran the entire tests/ tree, which includes provider graph tests (tests/graphs/*), integration tests and browser/network suites. Without API keys these fail, and tests/test_chromium.py launches a real headless browser that hangs forever — the last run was killed at GitHub's 6h limit. Narrow CI to the 9 deterministic, mock-based suites (56 tests, ~3s), drop the unused Playwright browser install, and add timeout-minutes: 15 so a future hang can't burn the full runner budget. The excluded suites should be re-added once mocked or gated behind markers + secrets. Co-authored-by: Claude Opus 4.8 (1M context) --- .github/workflows/test-suite.yml | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/.github/workflows/test-suite.yml b/.github/workflows/test-suite.yml index 677332def..28958ba40 100644 --- a/.github/workflows/test-suite.yml +++ b/.github/workflows/test-suite.yml @@ -11,6 +11,8 @@ jobs: unit-tests: name: Unit Tests runs-on: ubuntu-latest + # Hard cap so a hanging test can never burn the full 6h runner budget again. + timeout-minutes: 15 steps: - name: Checkout code @@ -27,8 +29,21 @@ jobs: - name: Install dependencies run: uv sync - - name: Install Playwright browsers - run: uv run playwright install chromium - + # Run only the deterministic, mock-based unit suites. The provider graph + # tests (tests/graphs/*), integration tests (tests/integration/*) and the + # browser/network suites (e.g. tests/test_chromium.py) require real API + # keys or live network/browsers, so in CI they either fail or hang + # indefinitely. They should be re-added here once mocked or gated behind + # markers + secrets. - name: Run unit tests - run: uv run pytest tests/ -m "unit or not integration" + run: > + uv run pytest + tests/test_batch_api.py + tests/test_csv_scraper_multi_graph.py + tests/test_depth_search_graph.py + tests/test_json_scraper_graph.py + tests/test_minimax_models.py + tests/test_scrape_do.py + tests/test_search_graph.py + tests/utils/convert_to_md_test.py + tests/utils/parse_state_keys_test.py From ef3523b1e1c62052b5c0e6b14cebe7933441a7e5 Mon Sep 17 00:00:00 2001 From: Marco Vinciguerra <88108002+VinciGit00@users.noreply.github.com> Date: Thu, 25 Jun 2026 09:50:35 +0200 Subject: [PATCH 2/3] docs(readme): add Open Source vs Managed API comparison (#1091) Add a comparison table and guidance clarifying the difference between the open-source library and the managed cloud API (Python/JS SDKs), so users can pick the right option. Co-authored-by: Claude Opus 4.8 (1M context) --- README.md | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/README.md b/README.md index bb08879ac..974bedc47 100644 --- a/README.md +++ b/README.md @@ -50,6 +50,33 @@ You can find more informations at the following [link](https://scrapegraphai.com - **Low-code Frameworks**: [Pipedream](https://pipedream.com/apps/scrapegraphai), [Bubble](https://bubble.io/plugin/scrapegraphai-1745408893195x213542371433906180), [Zapier](https://zapier.com/apps/scrapegraphai/integrations), [n8n](http://localhost:5001/dashboard), [Dify](https://dify.ai), [Toolhouse](https://app.toolhouse.ai/mcp-servers/scrapegraph_smartscraper) - **MCP server**: [Link](https://smithery.ai/server/@ScrapeGraphAI/scrapegraph-mcp) +## 🆚 Open Source vs Managed API + +ScrapeGraphAI comes in two flavours: **this open-source library**, which you run yourself, and the **managed cloud API** (used via the [Python](https://github.com/ScrapeGraphAI/scrapegraph-py) and [JS/TS](https://github.com/ScrapeGraphAI/scrapegraph-js) SDKs). This table explains the difference so you can pick the right one. + +| | Open Source (`scrapegraphai`) | Managed API (`scrapegraph-py` / `scrapegraph-js`) | +|---|---|---| +| **What it is** | A Python library you run yourself | A hosted cloud service you call via SDK | +| **Where it runs** | Your own infrastructure (self-hosted) | ScrapeGraphAI cloud | +| **LLM** | Bring your own (OpenAI, Groq, Gemini, Azure, local via Ollama) | Managed for you | +| **Browser / JS rendering** | You configure it (Playwright) | Managed (stealth, `auto`/`fast`/`js` modes) | +| **Proxies & anti-bot** | Your responsibility | Included | +| **Scaling & maintenance** | Your responsibility | Fully managed | +| **Cost model** | LLM tokens + your own infra | Pay-as-you-go credits | +| **Auth** | Your own LLM keys | `SGAI_API_KEY` | +| **Capabilities** | Graph pipelines (SmartScraper, Search, Speech, ScriptCreator…) | Scrape, Extract, Search, Crawl, Monitor, History | +| **Setup effort** | More configuration | Minimal — API key + one call | +| **License** | MIT | SDK is MIT; the API service is paid | + +**Choose the open-source library** if you want full control, on-prem/self-hosted data, local LLMs (Ollama), or fine-grained cost tuning — and you're happy to manage browsers, proxies and scaling yourself. + +**Choose the managed API** if you want zero infrastructure, managed JS rendering & anti-bot, built-in **Crawl** and scheduled **Monitor** jobs, and the fastest path to production — billed per credit. + +- Open-source library: https://github.com/ScrapeGraphAI/Scrapegraph-ai +- Python SDK: https://github.com/ScrapeGraphAI/scrapegraph-py +- JS/TS SDK: https://github.com/ScrapeGraphAI/scrapegraph-js +- API docs: https://docs.scrapegraphai.com/introduction + ## 🚀 Quick install The reference page for Scrapegraph-ai is available on the official page of PyPI: [pypi](https://pypi.org/project/scrapegraphai/). From 9ad7e8496ae2f9f399392cab80118d5c74dc41bc Mon Sep 17 00:00:00 2001 From: Marco Vinciguerra <88108002+VinciGit00@users.noreply.github.com> Date: Thu, 25 Jun 2026 09:51:20 +0200 Subject: [PATCH 3/3] Update README.md --- README.md | 52 ++++++++++++++++++++++++++-------------------------- 1 file changed, 26 insertions(+), 26 deletions(-) diff --git a/README.md b/README.md index 974bedc47..491f5c13d 100644 --- a/README.md +++ b/README.md @@ -50,32 +50,6 @@ You can find more informations at the following [link](https://scrapegraphai.com - **Low-code Frameworks**: [Pipedream](https://pipedream.com/apps/scrapegraphai), [Bubble](https://bubble.io/plugin/scrapegraphai-1745408893195x213542371433906180), [Zapier](https://zapier.com/apps/scrapegraphai/integrations), [n8n](http://localhost:5001/dashboard), [Dify](https://dify.ai), [Toolhouse](https://app.toolhouse.ai/mcp-servers/scrapegraph_smartscraper) - **MCP server**: [Link](https://smithery.ai/server/@ScrapeGraphAI/scrapegraph-mcp) -## 🆚 Open Source vs Managed API - -ScrapeGraphAI comes in two flavours: **this open-source library**, which you run yourself, and the **managed cloud API** (used via the [Python](https://github.com/ScrapeGraphAI/scrapegraph-py) and [JS/TS](https://github.com/ScrapeGraphAI/scrapegraph-js) SDKs). This table explains the difference so you can pick the right one. - -| | Open Source (`scrapegraphai`) | Managed API (`scrapegraph-py` / `scrapegraph-js`) | -|---|---|---| -| **What it is** | A Python library you run yourself | A hosted cloud service you call via SDK | -| **Where it runs** | Your own infrastructure (self-hosted) | ScrapeGraphAI cloud | -| **LLM** | Bring your own (OpenAI, Groq, Gemini, Azure, local via Ollama) | Managed for you | -| **Browser / JS rendering** | You configure it (Playwright) | Managed (stealth, `auto`/`fast`/`js` modes) | -| **Proxies & anti-bot** | Your responsibility | Included | -| **Scaling & maintenance** | Your responsibility | Fully managed | -| **Cost model** | LLM tokens + your own infra | Pay-as-you-go credits | -| **Auth** | Your own LLM keys | `SGAI_API_KEY` | -| **Capabilities** | Graph pipelines (SmartScraper, Search, Speech, ScriptCreator…) | Scrape, Extract, Search, Crawl, Monitor, History | -| **Setup effort** | More configuration | Minimal — API key + one call | -| **License** | MIT | SDK is MIT; the API service is paid | - -**Choose the open-source library** if you want full control, on-prem/self-hosted data, local LLMs (Ollama), or fine-grained cost tuning — and you're happy to manage browsers, proxies and scaling yourself. - -**Choose the managed API** if you want zero infrastructure, managed JS rendering & anti-bot, built-in **Crawl** and scheduled **Monitor** jobs, and the fastest path to production — billed per credit. - -- Open-source library: https://github.com/ScrapeGraphAI/Scrapegraph-ai -- Python SDK: https://github.com/ScrapeGraphAI/scrapegraph-py -- JS/TS SDK: https://github.com/ScrapeGraphAI/scrapegraph-js -- API docs: https://docs.scrapegraphai.com/introduction ## 🚀 Quick install @@ -191,6 +165,32 @@ Remember to have [Ollama](https://ollama.com/) installed and download the models [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1sEZBonBMGP44CtO6GQTwAlL0BGJXjtfd?usp=sharing) The documentation for ScrapeGraphAI can be found [here](https://docs.scrapegraphai.com/introduction). +## 🆚 Open Source vs Managed API + +ScrapeGraphAI comes in two flavours: **this open-source library**, which you run yourself, and the **managed cloud API** (used via the [Python](https://github.com/ScrapeGraphAI/scrapegraph-py) and [JS/TS](https://github.com/ScrapeGraphAI/scrapegraph-js) SDKs). This table explains the difference so you can pick the right one. + +| | Open Source (`scrapegraphai`) | Managed API (`scrapegraph-py` / `scrapegraph-js`) | +|---|---|---| +| **What it is** | A Python library you run yourself | A hosted cloud service you call via SDK | +| **Where it runs** | Your own infrastructure (self-hosted) | ScrapeGraphAI cloud | +| **LLM** | Bring your own (OpenAI, Groq, Gemini, Azure, local via Ollama) | Managed for you | +| **Browser / JS rendering** | You configure it (Playwright) | Managed (stealth, `auto`/`fast`/`js` modes) | +| **Proxies & anti-bot** | Your responsibility | Included | +| **Scaling & maintenance** | Your responsibility | Fully managed | +| **Cost model** | LLM tokens + your own infra | Pay-as-you-go credits | +| **Auth** | Your own LLM keys | `SGAI_API_KEY` | +| **Capabilities** | Graph pipelines (SmartScraper, Search, Speech, ScriptCreator…) | Scrape, Extract, Search, Crawl, Monitor, History | +| **Setup effort** | More configuration | Minimal — API key + one call | +| **License** | MIT | SDK is MIT; the API service is paid | + +**Choose the open-source library** if you want full control, on-prem/self-hosted data, local LLMs (Ollama), or fine-grained cost tuning — and you're happy to manage browsers, proxies and scaling yourself. + +**Choose the managed API** if you want zero infrastructure, managed JS rendering & anti-bot, built-in **Crawl** and scheduled **Monitor** jobs, and the fastest path to production — billed per credit. + +- Open-source library: https://github.com/ScrapeGraphAI/Scrapegraph-ai +- Python SDK: https://github.com/ScrapeGraphAI/scrapegraph-py +- JS/TS SDK: https://github.com/ScrapeGraphAI/scrapegraph-js +- API docs: https://docs.scrapegraphai.com/introduction ## 🤝 Contributing