Integration test topologies assert exactly once, but run with at most once.

[billonahill]

We're seeing occasional failures like this in CI. This issue is that IntegrationTestSpout now decrements the tuplesToAck counter only when tuples are acked (commit link). Previously it decremented that counter even when a fail call was received. What this implies is that our test topologies are dropping tuples and the spout is re-transmitting them. Why the test topologies are dropping tuples is concerning.

A few options we could pursue:

1. Why are tuples failing, causing them to be re-emitted? That shouldn't be happening in our tests
2. We could change IntegrationTestSpout back to decrement upon failure.
3. We could change test assertion to assert atLeastOnce instead of exactlyOnce.

ERROR: Actual result did not match expected result
INFO: Actual result ---------- 
['A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B']
INFO: Expected result ---------- 
['A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B']
ERROR: Checking result failed for 20161113214723_IntegrationTest_MultiSpoutsMultiTasks_861f6ed7-ca34-47c9-9bde-3ba3a2d802e4 topology :: ('Actual result did not match expected result', None)

[objmagic]

constantly fails both Travis CI and internal CI now. We should prioritize this.

[billonahill]

This ticket is blocked on us reaching an agreement for how we should proceed. I suggest we follow option 1 above, because it's unclear to me why tuples should fail in integration tests.

I would like to hear opinions from others though.

[billonahill]

@maosongfu, @kramasamy do you have thoughts re why tuples are being consistently dropped by the integration tests (is this expected?) and if that's something we should fix, or obfuscate by changing to at least once verification?

[billonahill]

Relevant classes:
https://github.com/twitter/heron/blob/master/integration-test/src/java/com/twitter/heron/integration_test/core/IntegrationTestSpout.java#L135
https://github.com/twitter/heron/blob/master/integration-test/src/java/com/twitter/heron/integration_test/core/IntegrationTestBolt.java

[maosongfu]

Process death could lead to tuples dropping and then re-emitting. In earlier integration tests, we consider process death a kind of failure. But seems this assumption is not longer true.

[billonahill]

If a process dies currently and tuples get re-emitted, that would trigger a failure during assertion, since we assert exactly once, not at least once.

These tests fail often though, so if we have frequent process deaths in our integration tests, something is really wrong. Hence my hesitation to change to at least once assertion without first identifying the cause of re-transmits.

[objmagic]

resolved by #1830 #1815