0.15.0 Release

[oltarasenko]

What's Changed

1. Created a simple management Web UI. Try it on localhost:4001
2. Added the possibility of creating spiders with the help of the YML format. Read more here: https://github.com/elixir-crawly/crawly/blob/master/documentation/spiders_in_yml.md
3. Added the possibility to run Crawly (and your scraping projects) without Elixir. Read more here: https://github.com/elixir-crawly/crawly/blob/master/documentation/standalone_crawly.md
4. Added generators for Crawly spiders and configuration files to reduce boilerplate
5. Improved UniqueRequest middleware so that it can store hashes instead of complete URLs (special thanks to @serpent213)
6. Added SameDomainFilter middleware, my favorite, which will probably deprecate the need to rely on the base_url in the future. Again thanks to @serpent213!

---

This discussion was created from the release 0.15.0 Release.

[dogweather]

Hi, after upgrading from 0.14 to 0.15, I get these log messages. I'm thinking they might better both be info? Or even debug? My code appears to still work fine, in spite of the grave-sounding error.

11:36:45.185 [info] Opening/checking dynamic spiders storage
dets: file "dets_simple_storage" not properly closed, repairing ...

11:36:45.213 [error] Could not load spiders: %MatchError{term: {:error, :enoent}}

I looked at the code — it seems like this is when it looks for the non-coding Spider configs?

Would you happen to know I can set the logging level? I'm thinking I'd set it to error and then have both of those logs be info or debug.

[oltarasenko]

Yes. Makes sense. It needs to be moved to debug.

[dogweather]

I'll submit a PR

[dogweather]

#268

[Charlotte-br560]

This looks like some fantastic progress. Excited to try out the management Web UI and the new features for creating spiders with YML. Also, running Crawly without Elixir is a game-changer! Thanks for the hard work, and I'll definitely check out Crawlbase too.