<aside> 💡 This is copied from an internal document drafted in July 2021 amongst a focus on the blockchain’s limited ability to account for ever-increasing storage capacity.
The original document has many comments I have not copied across into public view.
</aside>
@anorth, @nicola, @nikkolas, @luca, @kuba
FIP discussion: https://github.com/filecoin-project/FIPs/issues/119
This document outlines a solution to enabling exponential, or near-enough, growth in the amount of storage committed to the Filecoin network. This proposal has the big picture. Some deeper technical design docs are expected to be necessary to specify the SNARK circuits, new miner actor, and corresponding Lotus miner changes.
The Filecoin network’s capacity to onboard new storage and to maintain proofs of committed storage are limited by blockchain transaction processing throughput. This prevents the network from consistently growing any faster. Miners who are willing to significantly increase their storage in the network are prevented or slowed down when trying to do so. The recently released Hyperdrive network upgrade will raise onboarding capacity to about 500-1000 PiB/day, but we expect this capacity to become saturated.
As onboarding rates increase, the fixed amount of network throughput consumed by maintaining the proofs of already-committed storage will increase. Projections suggest this could become a significant cost to the network eventually. We also bear some risk that throughput may be insufficient to dispute fraudulent proofs, were a large party to attempt them.
Validation of the Filecoin blockchain is subject to a fixed amount of computational work per epoch (including state access), enforced as the block gas limit. It’s also subject to limits on the size of the state tree, though these are less sharp. These limits set a reasonable hardware requirement on being able to validate the chain and thus run a full node.
Many parts of the application logic for onboarding and maintaining storage incur a constant computational and/or state cost per sector. This results in blockchain validation costs that are linear in the rate of growth of storage, and in the total amount of storage committed. Some costs are linear in the number of individual miner actors. These linearities add up to consume the entire computational throughput, thus limiting further onboarding. The total size of the state is also of practical concern, but secondary to the cost of reading or writing that state during processing.
Linearities exist in:
We wish to remove or reduce all such linear costs from the blockchain validation process in order to remove limitations on rate of growth, now and in the long future when power and growth are significantly (exponentially) higher.
The proof aggregation technology of SNARKPack goes a long way toward addressing the linear cost of PoRep and PoSt proof verification. Verifying a proof aggregate is logarithmic in the number of sectors proven. In its current form, however, there remain linear costs associated with providing the public inputs for each sector’s proof.
Our goal is to enable arbitrary amounts of storage to be committed and maintained by miners within a fixed network transaction throughput. Arbitrary amounts here means at least the amounts that could possibly be committed over the next few years, as limited by external constraints like global HDD production.