Pytorch workers keep crashing if master is not up yet.

[TimZaman]

I've been observing this for a while now, and now I'm confident this consistently happens (for me):

• Create a pytorch job:
  A) If the master is up first, things go well, no pods crash
  B) If the master is not up first, the worker pod keep crashing with below error, until the master pod is up, and then things run fine. See below:

 kubectl logs optimizer-worker-0
2019-01-11 05:43:30,310 INFO     main(rmq_host=rmq.default.svc.cluster.local, rmq_port=5672, batch_size=12)
2019-01-11 05:43:30,310 INFO     init_distribution
Traceback (most recent call last):
  File "optimizer.py", line 459, in <module>
    pretrained_model=args.pretrained_model,
  File "optimizer.py", line 422, in main
    init_distribution()
  File "optimizer.py", line 413, in init_distribution
    torch.distributed.init_process_group(backend=backend)
  File "/root/.local/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 354, in init_process_group
    store, rank, world_size = next(rendezvous(url))
  File "/root/.local/lib/python3.7/site-packages/torch/distributed/rendezvous.py", line 143, in _env_rendezvous_handler
    store = TCPStore(master_addr, master_port, start_daemon)
ValueError: host not found: Name or service not known

The reason this occurs is simply because PyTorch workers require the master to be up to connect to. If they cannot connect to the master, they will die, this is intended behaviour (and nothing to do with the pytorch operator or K8s). However, I would expect the pytorch operator to handle this correctly, and bring the master up before the others.

[johnugeorge]

@TimZaman you are right. Workers will need master to be up before start. It might cause workers to restart few times during startup but it shouldn't cause any issue. Have you seen any side effects because of this issue?

[TimZaman]

Thank for your reponse! Nice work on this KF project, I might try to contribute a bit.

You are right: I don't think for our examples there would be any side effect, other than it's not ideal that they crash, of course (crash reports, logs, lists restarts, wasted cycles). I was assuming this is where the dedicated pytorch operator controller would come into fruition. We cannot look into other people's code, and their workers might do things before the distributed initialisation happens. So ideally we tackle this somehow.

[johnugeorge]

@TimZaman Thanks for your interest and contributions. This issue can be taken care of. I don't think, there is any problem in handling it. I will have a look at it

[johnugeorge]

Adding label /area 0.5.0

[johnugeorge]

@TimZaman Fix is merged

[johnugeorge]

Close this issue?

[Akmazad]

Hi guys,
I'm facing the same problem now and not sure if this is due to other issues in my coding. I'm using pytorch/1.1.0a0-py36, openmpi/3.1.3, cuda/9.0, dist_method="env://" and backend=nccl.

File "/apps/pytorch/1.1.0a0-py36/lib/python3.6/site-packages/torch/distributed/rendezvous.py", line 143, in _env_rendezvous_handler
store = TCPStore(master_addr, master_port, world_size, start_daemon)
ValueError: host not found: Name or service not known

Any suggestion?

[johnugeorge]

Can you create a separate issue?