Communicator for distributed computing using MPI.
When NCCL is not installed, arrays are moved to CPU, then communicated through MPI, and moved back to original device. When mpi4py is not installed, the communicator errors.
Initialize self. See help(type(self)) for accurate signature.
All reduce operation in-place.
bcast(input[, root, datatype])
Broadcast from root to other nodes.
Gather with variable sizes operation.
Reduce operation in-place.