[controller] Add metrics and logs to detect number of threads working on the operations of the same store at the same period

[sofiaz11]

Problem Statement

There were issues before that multiple threads in the admin consumption executor were working on the operations from the same store. This can reduce the throughput because for the same store, the operations will be done sequentially. Ideally, each thread should work on different store in the same period. This PR adds metircs and logs to detect if there are multiple threads work on the same store at the same time.

Solution

Code changes

• Added new code behind a config. If so list the config names and their default values in the PR description.
  testNoViolationInflightCounter
  testNoViolationInflightCounterWithCancellationException
  testConcurrentExecutionWithCancellationException
• Introduced new log lines.
• Confirmed if logs need to be rate limited to avoid excessive logging.

Concurrency-Specific Checks

Both reviewer and PR author to verify

• Code has no race conditions or thread safety issues.
• Proper synchronization mechanisms (e.g., synchronized, RWLock) are used where needed.
• No blocking calls inside critical sections that could lead to deadlocks or performance degradation.
• Verified thread-safe collections are used (e.g., ConcurrentHashMap, CopyOnWriteArrayList).
• Validated proper exception handling in multi-threaded code to avoid silent thread termination.

How was this PR tested?

./gradlew services:venice-controller:test --tests  "com.linkedin.venice.controller.kafka.consumer.AdminExecutionTaskTest"

• New unit tests added.
• New integration tests added.
• Modified or extended existing tests.
• Verified backward compatibility (if applicable).

Does this PR introduce any user-facing or breaking changes?

• No. You can skip the rest of this section.
• Yes. Clearly explain the behavior change and its impact.

[Copilot]

Pull request overview

This PR adds observability features to detect when multiple threads in the admin consumption executor are working on operations from the same store concurrently. This is an undesirable situation that reduces throughput since operations for the same store are processed sequentially. The PR introduces metrics and logging to monitor this condition.

Changes:

• Added violation tracking metric to count stores with multiple concurrent processing threads
• Added thread-safe counter tracking for inflight threads per store
• Added comprehensive unit tests for the new tracking functionality

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 7 comments.

File	Description
AdminExecutionTask.java	Adds taskOnStart/taskOnFinish methods to track inflight threads per store and log violations when multiple threads process the same store
AdminConsumptionStats.java	Introduces new violation_stores_count gauge metric and recordViolationStoresCount method
AdminConsumptionTask.java	Initializes inflightThreadsByStore map and passes it to AdminExecutionTask
AdminExecutionTaskTest.java	New comprehensive test suite covering no-violation, violation, and concurrent execution scenarios

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 7 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.