scraping Spiegel archive

[gnmarten]

powerful set of scripts! However, I can't replicate the "Scraping Der Spiegel archive data" part (on Linux)

when running article_crawler.py, no file "dictionary.json" seems to be generated

could you upload that file?

BTW, I had to manually edit article_crawler.py at line 62/63, because it threw an error when pattern matching the filenames
year=tmp.group(1)
month=tmp.group(2)

then the appropriate (yet empty) (sub)directories (years/issue numbers) get created in /data

Thanks in advance for any hints as to how to use the article_extract.py!

[computermacgyver]

Hi @gnmarten , Thanks for flagging this. The dictionary.json file is unnecessary, and I should have removed code referencing that. Sorry for the trouble. I've updated the file but still need to test those revisions and will leave this issue open until I have done so.

[computermacgyver]

I've looked at this a bit further, and there's more that will need to be done as the Der Spiegel website has been redesigned since this code was written.