DiLoCo Bandwidth Simulator

Model & Network Configuration

Model Size

N (number of parameters)

Compute Time per Step

Seconds

Number of Datacenters

M (≥1)

Data Parallel (H=1)

Gradients Data Type

DiLoCo

Gradients Data Type

Synchronization Cadence (H)

Streaming DiLoCo with Overlap

Gradients Data Type

Fragment Period (H)

Fragments (P)

Overlap Delay (τ)

Streaming pattern: H is the period for each fragment. With P fragments, one fragment is sent every 5.0 steps. Fragment size = total_size / P.

Compute Utilization

For all methods, the reduction of gradients or outer gradients is overlapped over the backward pass computation.

Bandwidth Requirements for Target Utilization

Approximate inter-datacenter bandwidth (Gbps) needed to achieve each compute utilization (CU) threshold:

Method	25% CU	50% CU	75% CU	90% CU	95% CU
Data Parallel	-	-	-	-	-
DiLoCo	-	-	-	-	-
Streaming DiLoCo	-	-	-	-	-