extract_xgbooost_cluster_env() and xgb.rabit.get_rank() get different rank number

[wulikai1993]

I ran distributed training on k8s.

The rank number was got by extract_xgbooost_cluster_env() as in https://github.com/kubeflow/xgboost-operator/blob/master/config/samples/xgboost-dist/train.py#L29

However, xgb.rabit.get_rank() got another rank number as in https://github.com/kubeflow/xgboost-operator/blob/master/config/samples/xgboost-dist/train.py#L57.

There are two things confusing me:

1. Now that extract_xgbooost_cluster_env() had got the rank number, why usexgb.rabit.get_rank() to get rank number again?
2. Why are the two rank numbers different?

[asahalyft]

Exactly. I also see the same problem today.

(base) asaha-mbp151:maven asaha$ kubectl logs -f xgboost-asaha-rfw4as3le3u-master-0 -n asaha
starting the train job
starting to extract system env
extract the Rabit env from cluster : xgboost-asaha-rfw4as3le3u-master-0, port: 9991, rank: 0, world_size: 3 
start the master node
start listen on 0.0.0.0:9991
###### RabitTracker Setup Finished ######
##### Rabit rank setup with below envs #####
DMLC_NUM_WORKER=3
DMLC_TRACKER_URI=xgboost-asaha-rfw4as3le3u-master-0
DMLC_TRACKER_PORT=9991
DMLC_TASK_ID=0
worker(ip_address=10.46.85.245) connected!
worker(ip_address=10.46.95.126) connected!
##### Rabit rank = 1
@tracker All of 3 nodes getting started
worker(ip_address=10.44.239.26) connected!
Read data from IRIS data source with range from 50 to 100
starting to train xgboost at node with rank 1

@terrytangyuan if you could please comment.

[merlintang]

if you check the code, you can find the second rank is used only on the worker node, this is used to decide the rabbit worker id.

…

On Wed, Mar 17, 2021 at 4:43 AM Anindya Saha ***@***.***> wrote: Exactly. I also see the same problem today. (base) asaha-mbp151:maven asaha$ kubectl logs -f xgboost-asaha-rfw4as3le3u-master-0 -n asaha starting the train job starting to extract system env extract the Rabit env from cluster : xgboost-asaha-rfw4as3le3u-master-0, port: 9991, rank: 0, world_size: 3 start the master node start listen on 0.0.0.0:9991 ###### RabitTracker Setup Finished ###### ##### Rabit rank setup with below envs ##### DMLC_NUM_WORKER=3 DMLC_TRACKER_URI=xgboost-asaha-rfw4as3le3u-master-0 DMLC_TRACKER_PORT=9991 DMLC_TASK_ID=0 worker(ip_address=10.46.85.245) connected! worker(ip_address=10.46.95.126) connected! ##### Rabit rank = 1 @TracKer All of 3 nodes getting started worker(ip_address=10.44.239.26) connected! Read data from IRIS data source with range from 50 to 100 starting to train xgboost at node with rank 1 @terrytangyuan <https://github.com/terrytangyuan> if you could please comment. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#106 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAK5R6JZGN6WZI7UHSSOZLDTD67IJANCNFSM4TZST7MQ> .