Pluralis Protocol Learning: decentralized LLM training on consumer hardware
Quote271 words
Explains Pluralis's milestone of training an LLM across consumer devices over the internet: Protocol Models with unextractable weights, fractional ownership by compute contribution, low-rank compression of communication and Proof of Learning verification.
Key claims
An LLM was split and trained across consumer devices over the internet with no loss in speed or performance
Low-rank structure in transformer blocks enables highly compressed node communication
. @PluralisHQ reaches major milestone, demonstrating that efficiently LLM training across consumer-grade hardware via the internet is possible.
This is a game-changer for open-source AI.
TL;DR ↓
- Protocol Learning (PL) merges AI with cryptographic primitives, establishing a new decentralized approach for collaboratively training large foundation models, enabling both open participation and intrinsic monetization via partial ownership.
- Current "open-source" AI relies on large entities releasing costly model weights for free, a practice economically unsustainable at increasing scales. PL resolves this by embedding monetization directly within the model training process, aligning incentives through fractional ownership allocated proportionally to compute contributions.
- Protocol Models (PMs) form the technological core of PL. They are structured so that complete model weights remain cryptographically unextractable by any single participant or group without incurring excessive computational costs.
- Decentralized training typically faces technical barriers due to low-bandwidth internet connections among geographically dispersed nodes. However, recent research demonstrates feasibility by exploiting inherent low-rank structures within transformer blocks, enabling highly compressed communication between nodes, achieving training speeds comparable to centralized datacenters without compromising model quality.
- PL employs cryptographic credentials, necessary for inference, as a mechanism for fractional model ownership. This ensures contributors directly benefit from the model's downstream utility, aligning economic incentives and promoting ongoing community engagement.
- Robust verification protocols (a.k.a. "Proof of Learning") protect against malicious contributions and ensure trustless decentralized training. This guarantees model integrity despite heterogeneous compute environments and varying participant reliability.
- PL facilitates scalable, community-owned, and governed foundation models, unlocking permissionless innovation, broadening public access, and countering concentrated corporate or state control of AI tech.
More details in thread below ↓
QUOTING @Pluralis
We've reached a major milestone in fully decentralized training: for the first time, we've demonstrated that a large language model can be split and trained across consumer devices connected over the internet - with no loss in speed or performance. t.co/mNuhcvS0g2
We've reached a major milestone in fully decentralized training: for the first time, we've demonstrated that a large language model can be split and trained across consumer devices connected over the internet - with no loss in speed or performance. t.co/mNuhcvS0g2