What config settings do I need to define for training with multiple GPUs?

[wigging]

I would like to use the mattergen-train command to train a model using multiple GPUs. Which command line arguments do I need to define to train the model with multiple GPUs? I found the items listed below in the yaml files provided in mattergen/conf but there is no documentation that explains what any of these settings actually do. And I'm nut sure if there are other settings that I need to define. Some guidance on this would be very helpful.

• devices in mattergen/conf/trainer/default.yml - Is this the number of GPUs?
• num_workers with train, val, test in mattergen/conf/data_module/mp_20.yml - Do each of these need to be set to number of GPUs?
• Are there other settings I need to change for using multiple GPUs?

[danielzuegner]

Hi @wigging,

• devices in mattergen/conf/trainer/default.yml refers to the number of GPUs per node. Set this to the desired number of GPUs.
• We've kept num_workers at 0 for all our experiments because increasing it did not improve speed and sometimes even slowed training down. I recommend to keep it at 0, but feel free to experiment.

Best,

Daniel

[wigging]

I noticed that batch_size is affected by accumulate_grad_batches. Do you have a recommendation on what to set this to for multiple GPUs, the default value is accumulate_grad_batches: 1?

[danielzuegner]

We have the target effective batch size of 512. Depending on the number of GPUs, devices, and gradient accumulation, we're setting the per-GPU batch size to a value that gives us that batch size, see here: https://github.com/microsoft/mattergen/blob/main/mattergen/conf/data_module/alex_mp_20.yaml#L46. For best performance set the smallest accumulation that does not lead to OOM.

[wigging]

Can you explain what zero means here? If train is zero then what does that mean?

num_workers:
  train: 0
  val: 0
  test: 0

[danielzuegner]

See here:

num_workers (int, optional) – how many subprocesses to use for data loading. 0 means that the data will be loaded in the main process. (default: 0)

The different keys represent the respective settings for the train, val, and test dataloaders. 0 means we're effectively turning off multiprocessing for the data loading. But note that we're using pytorch DistributedDataParallel (see config; documentation). This means each GPU will have its own process, so we do benefit from multiple GPUs in the data loading as well (we will load data with one process per GPU in practice).

[wigging]

If num_workers are 0 then I get warnings such as the following:

The dataloader, val_dataloader, does not have many workers which may be a bottleneck. Consider
increasing the value of the `num_workers` argument` (try 128 which is the number of cpus on this
machine) in the `DataLoader` init to improve performance.

It sounds like I should define the number of works based on the number of CPU cores I have available. But this contradicts what you said earlier about setting it to 0 because it might actually slow things down.