Data availability sampling and light clients: erasure coding explained
Thread1,083 words
Detailed explainer of how DA layers break the link between throughput and verification cost: full versus light nodes, fraud proofs, erasure coding and the data availability sampling process, with Celestia's 2D Reed-Solomon scheme as the example.
Key claims
Celestia uses 2D Reed-Solomon codes with a 1/4 code rate, turning 1MB blocks into 4MB
Light nodes sampling over 25% of the coded block let honest full nodes recover the original
DA networks scale block size with light node count
0/ with data availability layers like @CelestiaOrg, modular blockchains can scale computation while keeping the network trustless & decentralized by breaking the correlation between throughput & verification cost 🚀
a 🧵 on trust-minimized light clients & DAS (0/27) 👇
1/ in most blockchains, nodes need to download full tx data to verify data availability
this is a problem this creates is that when the block size gets increased, the amount of data nodes need to verify increases equally
2/ but what if there was a way to run light nodes alongside full nodes but with the same security guarantees as long as there is at least one honest full node?
would be sick, right?
3/ well, glad to tell you that there is!
data availability sampling (DAS) is a technique that allows light nodes to verify data availability by only downloading a small portion of the block data
4/ this provides security to light nodes so that they can verify invalid blocks (DA & consensus only), and allows a blockchain to scale data availability without equally increasing node requirements
5/ but let's dive in deeper
as @VitalikButerin states in his endgame vision, the future of block production likely centralizes towards pools & specialized producers for scalability purposes, while block verification (keep producers honest) should importantly remain decentralized
6/ this can be achieved through splitting blockchain nodes into full nodes and light clients as suggested above
7/ there are two problems which are relevant in this model:
🔹 block validation (verifying that the computation is correct)
🔹 block availability (verifying all data has been published)
8/ light clients can help address these. But traditional light clients have weaker security assumptions compared to full nodes because they only verify block headers
light clients hence can’t detect if an invalid block is produced by a dishonest majority of block producers
9/ with the before-mentioned data availability sampling, light clients get a security upgrade because they can verify if invalid blocks are produced if the DA layer only does consensus and data availability (which is the case with #Celestia or #Avail)
10/ but let’s quickly take a step back and have a look at the function the different types of nodes have
11/ full nodes require a lot of resources because they download & validate every transaction on the blockchain. Full nodes offer a lot of security guarantees because they can't be fooled into accepting blocks with invalid txs
12/ light nodes on the other hand do not download or validate any of the txs. Instead, they just download & check the block header, while assuming that the block contains valid txs. Hence, light clients offer weaker security guarantees compared to full nodes
13/ nevertheless, bc full nodes are very resource intensive, some network participants will prefer to run a light client. This is especially important from a decentralization point of view (lower barriers of entry)
14/ so to ensure light nodes are not tricked, full nodes can send over a fraud proof to the light clients if a block contains an invalid tx, instead of the light clients checking the all txs themselves (& having to download all the data)
this is the approach @CelestiaOrg takes
15/ in a fraud-proof, full nodes provide light nodes with just enough data so they can autonomously identify invalid txs
16/ ok, so far, so good. But how can light clients ensure all transactional data in a block is released without having to download the entire block, in turn centralizing hardware requirements and thus defeating the purpose of light clients again?
17/ but before we deep dive into the DAS process, let's try to understand the magic of erasure codes ✨👇
18/ Celestia uses a mathematical primitive called erasure coding to perform data availability sampling, where light clients probabilistically determine the entirety of a block has been published by randomly sampling small portions of the block
19/ at a high level, erasure coding adds redundancy to a string of data. The key property is that, if the redundant data is available, the original data can be reconstructed in the event some part of the data gets lost
20/ even more importantly though, it doesn’t matter which part of the data is lost. As long as X% (tolerance threshold) of the data is available, the full original data can be reconstructed
@CelestiaOrg for example uses 2D Reed-Solomon erasure codes with a 1/4 code rate
21/ validators erasure code blocks prior to publishing, turning a 1MB block of data into a 4MB block. Hence, if at least 25% of the txs in a block are available to the network, it can recover the remaining 75%
this makes it very hard for a block producer to hide data unnoticed
22/ but let's have a closer look at the data availability sampling process ✨👇
so, initially a block producer publishes a block header. Subsequently, light nodes will start to request random chunks of data from that block from the block producer
23/ if data has been made available, the producer releases data chunks as light nodes request. Thanks to erasure coding, honest full nodes in the network can recover the original block from the broadcast chunks if light nodes collectively sample >25% of the erasure-coded block
24/ by verifying that all tx data has been made available, light nodes can now fully rely on fraud proofs, knowing that any single honest full node can generate these proofs for them
25/ while, there is a minimum number of light nodes required for this data sampling to work, this allows light nodes to function under almost the same security guarantees as full nodes (if at least one honest full node exists)
26/ if the block producer is malicious and withholds data, the light nodes notice as their sampling test fails and the network can be halted / recovered (ultimately through social consensus if needed)
27/ a powerful feature is also that light client based networks like @CelestiaOrg or @AvailProject can scale with the number of light nodes in the network. The more data sampling light nodes there are, the larger the block size can be while still ensuring security
hope this was more or less digestable
stay tuned for a 🧵 on how @eigencloud's smart contract based approach differs from these specialized DA L1s @CelestiaOrg & @AvailProject
🧵 coming on Monday 🔥 👀
tagging L2/modular chads🫡 👇
@DeFiMinty
@Only1temmy
@Slappjakke
@Louround_
@thesaint_
@0xSalazar
@CryptoShiro_
@ViktorDefi
@stacy_muur
@Flowslikeosmo
@ShivanshuMadan
@0xShivani
@hmalviya9
@daddysether
@bitsplaining
@kunalgoel
@hooeem
@redvelvetzip
@0xDinoEggs
3 more images on X
What came next · since 26 May 2023
Since then
Zero has written 29 more pieces on Celestia since this one.
- What If You Stop Seeing The Forest Because Of All The Trees?On Celestia
- Celestia 'Proof of Governance' proposal: 20x lower inflation and no stakingOn Celestia
- Why TIA is one of the most predatory VC tokens: pricing, unlocks and inflationOn Celestia
- Celestia vs EigenDA: different missions, not direct competitorsOn Celestia