Improve C++ example

[pentschev]

The C++ example is currently very thin and covers only a minor fraction of the UCXX features (primarily a simple example of tagSend/tagRecv).

Here's an initial list of features that are missing, in a somewhat loosely defined decreasing order of importance:

• CUDA support
• Multi-process example
• Using tagSend/tagRecv callbacks
• Using tagProbe
• Using Active Messages (this should wait for Allow AM receive callback to control data allocation/copy #479)
• Connecting directly to the worker instead of using an endpoint (e.g., use createEndpointFromWorkerAddress)
• Using RMA
• Maybe tagMultiSend/tagMultiRecv (its primary purpose was to use from Python so it's not as well tested and validated)

We should also ensure the example is well-organized, so that it is easy to read and navigate, especially when we add all those new features are added. Therefore we must come up with a design that makes it clear, easy to understand and extend.

[nirandaperera]

@pentschev What did you mean by CUDA support? May be I could start with that. Did you mean to say, add example with cuda buffer sends/ recvs?

[pentschev]

Did you mean to say, add example with cuda buffer sends/ recvs?

Exactly, there are only host-to-host transfers currently. If you have time to start with that, I think would be great to demonstrate both CUDA with proper cudaMalloc* and RMM. I really appreciate your help with that! 😄

[nirandaperera]

CUDA support was added here #526