FLUX LoRA multi-node finetuning?

[AbdullahMu]

is it possible to do mult-inode LoRA finetuing with accelerate? I believe i have made the accelerate config file correctly, and FLUX has been loaded to all the GPUs in all nodes. but after that, i get an error like this:

 torch.distributed.barrier()
  File "/root/SimpleTuner/.venv/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper
    return func(*args, **kwargs)
  File "/root/SimpleTuner/.venv/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4159, in barrier
    work = group.barrier(opts=opts)
torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:2251, internal error - please report this issue to the NCCL developers, NCCL version 2.21.5
ncclInternalError: Internal check failed.
Last error:
Bootstrap : no socket interface found

NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:2251, internal error - please report this issue to the NCCL developers, NCCL version 2.21.5
ncclInternalError: Internal check failed.
Last error:
Bootstrap : no socket interface found
Traceback (most recent call last):
  File "/root/SimpleTuner/train.py", line 50, in <module>
    trainer.init_data_backend()
  File "/root/SimpleTuner/helpers/training/trainer.py", line 670, in init_data_backend
    raise e
  File "/root/SimpleTuner/helpers/training/trainer.py", line 644, in init_data_backend
    configure_multi_databackend(
  File "/root/SimpleTuner/helpers/data_backend/factory.py", line 503, in configure_multi_databackend
    with accelerator.main_process_first():
  File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)
  File "/root/SimpleTuner/.venv/lib/python3.10/site-packages/accelerate/accelerator.py", line 905, in main_process_first
    with self.state.main_process_first():
  File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)
  File "/root/SimpleTuner/.venv/lib/python3.10/site-packages/accelerate/state.py", line 1082, in main_process_first
    with PartialState().main_process_first():
  File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)
  File "/root/SimpleTuner/.venv/lib/python3.10/site-packages/accelerate/state.py", line 495, in main_process_first
    yield from self._goes_first(self.is_main_process)
  File "/root/SimpleTuner/.venv/lib/python3.10/site-packages/accelerate/state.py", line 385, in _goes_first
    self.wait_for_everyone()
  File "/root/SimpleTuner/.venv/lib/python3.10/site-packages/accelerate/state.py", line 374, in wait_for_everyone
    torch.distributed.barrier()
  File "/root/SimpleTuner/.venv/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper
    return func(*args, **kwargs)
  File "/root/SimpleTuner/.venv/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4159, in barrier
    work = group.barrier(opts=opts)
torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:2251, internal error - please report this issue to the NCCL developers, NCCL version 2.21.5
ncclInternalError: Internal check failed.
Last error:
Bootstrap : no socket interface found

[rank0]:[W1214 12:31:47.990544514 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present,  but this warning has only been added since PyTorch 2.4 (function operator())

is there a way to fix this?

[bghira]

check the DISTRIBUTED.md document for a step by step guide. if it fails beyond that, it seems to be a local issue and probably out of scope, so i will convert this to a discussion instead