Improve lily task retry logic

[frrist]

Description

When running lily as a distributed indexer and a job fails to complete all tasks successfully, lily reschedules all tasks to be retried. This is inefficient in some cases since some of tasks may have been completed successfully. Lily should instead reschedule only the failed tasks to be retried.

Acceptance criteria

• Only reschedule failed tasks to rety

Where to begin

[kasteph]

closed from #1162

[birdychang]

#1162 will stop all task retries when there's no fatal error. We still need a better solution to retry only fail tasks