Implement regular commits during import

[ericof]

• I signed and returned the Plone Contributor Agreement, and received and accepted an invitation to join a team in the Plone GitHub organization.
• I verified there aren't any other open pull requests for the same change.
• I followed the guidelines in Contributing to Plone.
• I successfully ran code quality checks on my changes locally.
• I successfully ran tests on my changes locally.
• If needed, I added new tests for my changes.
• If needed, I added documentation for my changes.
• I included a change log entry in my commits.

---

Closes #55

---

Comparison

Computer:

• macbook pro m1, 32GB ram

Source data

• ~7GB (25703 content items)
• Data being read from an external ssd via usb3

Destination

• Local instance (no Zeo, no Relstorage) with a freshly created Plone site

Timing

• Before: 2711.06s user 242.56s system 96% cpu 50:58.50 total
• After: 1458.01s user 65.67s system 94% cpu 26:48.19 total

[mister-roboto]

@ericof thanks for creating this Pull Request and helping to improve Plone!

TL;DR: Finish pushing changes, pass all other checks, then paste a comment:

@jenkins-plone-org please run jobs

To ensure that these changes do not break other parts of Plone, the Plone test suite matrix needs to pass, but it takes 30-60 min. Other CI checks are usually much faster and the Plone Jenkins resources are limited, so when done pushing changes and all other checks pass either start all Jenkins PR jobs yourself, or simply add the comment above in this PR to start all the jobs automatically.

Happy hacking!

[jackahl]

Can't really judge this code wise, but

[davisagli]

Please make some way to opt out of incremental commits. Sometimes it's more important to have them, but sometimes I'm testing a migration and I want to be able to undo the entire thing in a single transaction.

[ericof]

@davisagli If you set the environment variable IMPORTER_COMMIT_DISABLE, we will not have the intermediate commits

[fredvd]

What was the reason for adding the intermediate commits, to be able to continue from a part of the import. Or to save/free up memory and write objects to disk? For the latter, there are transaction savepoints, right?

[davisagli]

@fredvd We had a client that was hitting the limit on postgres locks during a large import. I don't think that savepoints would be enough to avoid that.