test: Add C++ gRPC cancellation tests to L0_request_cancellation

[yinggeh]

What does the PR do?

Wires the new C++ grpc_cancellation_test gtest (added in the client-side companion PR) into qa/L0_request_cancellation/test.sh as a sibling of the existing Python cancellation suite. Each gtest case is run against a fresh tritonserver and the count of Cancellation notification received for log lines is asserted to match the expected count for that case.

Also temporarily bumps the model's instance_group count to 3 around the TestGrpcAsyncInferMulti case (reverted after) so the three fanned-out requests can execute concurrently; the test cancels two of them while letting the middle one complete naturally and would otherwise serialize on a single CPU instance.

Checklist

• PR title reflects the change and is of format <commit_type>: <Title>
• Changes are described in the pull request.
• Related issues are referenced.
• Populated github labels field
• Added test plan and verified test passes.
• Verified that the PR passes existing CI.
• Verified copyright is correct on all changed files.
• Added succinct git squash message before merging ref.
• All template sections are filled out.
• Optional: Additional screenshots for behavior/output changes with before/after.

Commit Type:

• test

Related PRs:

triton-inference-server/client#896

Where should the reviewer start?

Test plan:

L0_request_cancellation--base

• CI Pipeline ID: 51138710

Caveats:

Background

Triton has had Python-side request cancellation tests since r23.10 but no C++ counterpart, despite the C++ gRPC client gaining matching APIs in the companion client PR. This PR adds the missing test wiring so the two clients' cancellation behavior stays in lock-step on every CI run.

[whoisj]

probably needs a trailing space.

[yinggeh]

Really?

[mudit-eng]

This sentence looks odd. There should be something after 'for'.
Do you have an example the actual log line looks like?

[yinggeh]

Two locations

server/src/grpc/infer_handler.cc

Lines 718 to 722 in 0c063cf

	LOG_VERBOSE(1) << "Cancellation notification received for " << Name()
	<< ", rpc_ok=" << rpc_ok << ", context "
	<< state->context_->unique_id_ << " step "
	<< state->context_->step_ << ", state "
	<< state->unique_id_ << " step " << state->step_;

server/src/grpc/stream_infer_handler.cc

Lines 152 to 156 in 0c063cf

	LOG_VERBOSE(1) << "Cancellation notification received for " << Name()
	<< ", rpc_ok=" << rpc_ok << ", context "
	<< state->context_->unique_id_ << " step "
	<< state->context_->step_ << ", state "
	<< state->unique_id_ << " step " << state->step_;

[mudit-eng]

As J said, there is a trailing space after 'for'.

[mudit-eng]

This change is not going in r26.05.

[yinggeh]

Try to cherry-pick since internal team is waiting for this feature.

[whoisj]

kind of a big overhaul this late into the release, no?

what is the risk assessment? (discuss offline)

[mudit-eng]

This sentence looks odd. There should be something after 'for'.
Do you have an example the actual log line looks like?

[mudit-eng]

is there a check for N >= 3?

[yinggeh]

Can you elaborate? Check N requests, instances or cancellation?

[mudit-eng]

For N concurrent requests to fan out to 3 CPU, shouldn't we have N > 3?

[yinggeh]

In this test, each request execution takes 10 seconds. To avoid backlog in the request queue (reduce overall test time), the model configuration is increased to 3 instances. If N > 3, requests after 3rd will wait in the queue until the first 3 requests have completed execution, which will take 10 seconds.

[yinggeh]

I see what you mean. Here we are testing requests that are cancelled during execution. I can also add a test for in-queue request cancellation.

[mudit-eng]

Thanks. Yes, let's test for in-queue cancellation also.