Similar to the logic in the data registry. https://github.com/open-contracting/data-registry/blob/23767b735e68e52a7705b1c0ea390aa60b15149c/data_registry/process_manager/task/collect.py#L124-L145 We can also add https://scrapy-log-analyzer.readthedocs.io/en/latest/api/index.html#scrapyloganalyzer.ScrapyLogFile.is_complete to indicate whether the crawl was a subset (sample, etc.). cc @yolile Here is the commit from an older PR: https://github.com/open-contracting/notebooks-ocds/commit/b970412cd6ba24f67f5d65dded75ae1f5e2b0992 ---- scrapy-log-analyzer's logparser dependency is GPL. Might need to modify license for relevant notebooks. https://ocp-software-handbook.readthedocs.io/en/latest/python/preferences.html#license-compliance
Similar to the logic in the data registry.
https://github.com/open-contracting/data-registry/blob/23767b735e68e52a7705b1c0ea390aa60b15149c/data_registry/process_manager/task/collect.py#L124-L145
We can also add https://scrapy-log-analyzer.readthedocs.io/en/latest/api/index.html#scrapyloganalyzer.ScrapyLogFile.is_complete to indicate whether the crawl was a subset (sample, etc.).
cc @yolile
Here is the commit from an older PR: b970412
scrapy-log-analyzer's logparser dependency is GPL. Might need to modify license for relevant notebooks.
https://ocp-software-handbook.readthedocs.io/en/latest/python/preferences.html#license-compliance