Performance: limit number of database transactions

[mattboehm]

In an attempt to detect hotspots in the codebase, I ran a a sample program with smiley multiple times, with different parts of the smiley codebase commented out. (Using a LocalPublisher).

Run	Time
Base	8.6s
Comment out insert line in db.trace()	~8.6s
Make all db.trace() calls use the same transaction	2.25s
Comment out transaction and insert in db.trace	1.3s

It appears that using fewer transactions might speed things up by a factor of 2-4. I suspect that even buffering inserts into groups of 5 would go a long way in improving performance. The downside to this approach is added code complexity and the chance that a failed insert causes one to roll back additional records (unless when you catch this, you rollback and apply those inserts in separate transactions.)

If this batching behavior is wanted, it could be implemented in a few ways:

• Always batch
• Batch if an argument is passed into LocalPublisher's constructor
• Subclass LocalPublisher with a BatchingLocalPublisher (or FastLocalPublisher)

I'm happy to give this a shot, but wanted to float the idea out here first to see what others thought.

[dhellmann]

I like the idea of batching up inserts to use fewer transactions. I don't think it needs to be user-tunable, at least to start. Adding an argument to the publisher API would make the LocalPublisher different from the others, or would add an argument to the others that wouldn't really be useful in those cases. If we end up needing to provide a way to control it, we can think about that again.