state_store / checkpoint_dir does not handle invalid JSON (e.g. empty files)

[OEBBleo]

The state_store.py that is responsible for writing the checkpoint_dir under $SPLUNK_HOME/var/lib/splunk/modinputs/misp_*_input does not handle invalid JSON, including empty files.

This functionallity is for continuing partial or chunked ingestions.

When the state file is empty, due to a crash or potentially bad times restart; instead of interpreting this as "we don't have a saved state" and starting over a JSON parsing exception is not being handled.

json.load in state_store.py:get_state().

[Benni0]

When an issue with the state file occurs, it must be fixed manually. Simply returning None would result in a re-ingestion of all data, which could have a high impact.
I will consider to create a state file backup when the input is executed.

[OEBBleo]

Yes returning None will lead to re-ingestion which is the sane thing to do when we lost our state, no?
To make sure that all IOCs are there, the report generaion will de-duplicate everything again then.

Backing up the state file is a good idea.
Just if the backup is corrupt as well, I assume we have to start over.

And actually the corrupted file needs to be deleted. That is missing in my PR.

[Benni0]

I would not recommend to start over, I would rather try to find the latest event which was ingested and create a valid state file with this information instead.
If you have a big MISP instance, a re-ingestion of all MISP Attributes from the last 180d could easily result in an ingest volume of more than 100gb.