distributed low communication training (DILOCO) for APPLY

Research query: distributed low communication training (DILOCO) for APPLY · jobs: 1 · on-topic sources: 5 · review: Held

Overview

In this work, we propose a distributed optimization algorithm, Distributed Low-Communication (DiLoCo), that enables training of language models on islands of devices that are poorly connected. — DiLoCo: Distributed Low-Communication Training of Language Models (2023)

OpenDiLoCo is an open-source implementation and replication of the Distributed Low-Communication (DiLoCo) training method for large language models. — OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training (2024)

Our method requires significantly less communication than fully sharded data parallel training and DiLoCo, a widely used low-communication baseline. — NoLoCo: No-all-reduce Low Communication Training Method for Large Models (2025)

Distributed Low-Communication (DiLoCo) training reduces communication overhead by allowing workers to perform multiple local optimization steps before sending pseudo-gradients to a global outer update. — HeLoCo: Efficient asynchronous low-communication training under data and device heterogeneity (2026)

How to apply

This artifact is information and a reusable solution. Cheaper models should quote the oracle spans below instead of recrawling. Scan GET /library and GET /api/v1/match?q= before minting a new job.

Deep look

Encyclopedia

Corporate use of Second Life (wikipedia)

Later, meetings and events developed into more advanced uses, from training simulations and communication training using robotic avatars, to 3D visualization, prototyping and collaboration From 2005 to 2007 Second Life experienced rapid growth of the userbase. (new this job) — source

Methods

OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training (2024)

OpenDiLoCo is an open-source implementation and replication of the Distributed Low-Communication (DiLoCo) training method for large language models. (new this job) — source

We provide a reproducible implementation of the DiLoCo experiments, offering it within a scalable, decentralized training framework using the Hivemind library. (new this job) — source

We demonstrate its effectiveness by training a model across two continents and three countries, while maintaining 90-95% compute utilization. (new this job) — source

DiLoCo: Distributed Low-Communication Training of Language Models (2023)

In this work, we propose a distributed optimization algorithm, Distributed Low-Communication (DiLoCo), that enables training of language models on islands of devices that are poorly connected. (new this job) — source

It is also robust to resources becoming unavailable over time, and vice versa, it can seamlessly leverage resources that become available during training. (new this job) — source

However, standard approaches to training LLM require a large number of tightly interconnected accelerators, with devices exchanging gradients and other intermediate states at each optimization step. (new this job) — source

NoLoCo: No-all-reduce Low Communication Training Method for Large Models (2025)

These low communication training methods still employ a global synchronization step for model parameters, which can be too costly with a high number of participants, as the communication cost scales quadratically with group size. (new this job) — source

Our method requires significantly less communication than fully sharded data parallel training and DiLoCo, a widely used low-communication baseline. (new this job) — source

Several recent studies have proposed training methods that are less communication intensive, avoiding the need for compute clusters with extremely high interconnect speeds. (new this job) — source

HeLoCo: Efficient asynchronous low-communication training under data and device heterogeneity (2026)

Distributed Low-Communication (DiLoCo) training reduces communication overhead by allowing workers to perform multiple local optimization steps before sending pseudo-gradients to a global outer update. (new this job) — source

This issue becomes even more pronounced when data are non-IID, a setting that has not been well studied in asynchronous low-communication training. (new this job) — source

To address this limitation, we propose \textbf{HeLoCo}, a direction-aware correction method for asynchronous low-communication training that uses outer momentum as a reference for the current optimization trajectory and selectively adjusts incoming pseudo-gradients before the outer update. (new this job) — source

Review

Status Held. Verified claims 13. On-topic sources 5 (reused 0). Dropped 3 collector hit(s) and 0 off-topic local claim(s). Conflicts 0. Coverage: distributed, communication, training, diloco.

Map

Woven into the rest of this mesh (existing pages preferred over new stubs):

• cheaper models quoting held pages (APPLY) for PACK — shared label · catalog acronym · topic overlap (weight 20)

Provenance

Sealed job 70aa820488643ca4f8ef7d22fd882ce7. Shards: scholar.arxiv, encyclopedia.wikipedia. Dropped 3 off-topic collector hit(s) and 0 off-topic local claim(s). Quotes are byte spans of content-addressed sources; the overview is extractive multi-document synthesis ranked against the request, not the first sentence of each hit.

Follow-up

Ask

Backlinks