Interop between IPFS and Filecoin has been a frustrating challenge since the network launched in 2020. We faced a series of hurdles that have prevented us making interop truly seamless, despite lots of ongoing work in this area. In 2021, we addressed the initial hurdle of making Filecoin content discoverable by building a content indexer. The next major hurdle is data transfer — currently, our products use two different transfer stacks. While both stacks are relatively stable now, they are very different and the IPFS stack fails to even consider payments.
Project Thunder is an ambitious, 6 month roadmap to unify our Libp2p transfer stack with support for payments across protocols. Project Thunder has three core components:
Project Thunder is not the end of the interop story — it does not address policy questions around payments (i.e. when and how much to pay for retrievals, who pays, specific mechanisms for payment, etc) or attempt to actually put a wallet in go-ipfs. However, it offers a real, long term solution to the next major hurdle — transfer — and lays the foundation to solve remaining payment questions going forward.
The “two transfer stacks” problem is one we’ve been stuck with for a long time.
We’ve generally settled on a few approaches:
The net result is more than a year of no real progress to actually addressing the problem. Project Thunder is trying to strike a middle ground of actually tackling the problem while not getting bogged down in an interminable grand re-architecting or debate. We’re aiming for a “good enough” default starting point from which we can ship additional improvements as needed.
Because we’re tackling a hard problem, we’re asking for more time that we usually get for individual projects, but we’re also promising to deliver some incremental wins, or when an immediate win is not possible, at least some kind of demonstrable proof we’re on the right track. These wins are outlined in this document.
The current go-bitswap lacks features needed for Project Thunder’s use case. It’s also hard to make changes to and in need of a refactor, especially to support high volume clients and servers. We’ve designed this project to keep the expanded feature set required from go-bitswap small, at least initially, and achievable without a major refactor/rearchitecture. Go-IP stewards may undertake larger refactor efforts in parallel while this project progresses to support long term development. The ask from stewards is to deliver features needed to keep Project Thunder unblocked within the current architecture while larger rearchitecture efforts are underway.
Below is a complete itemized roadmap for Project Thunder. This roadmap has three core milestones, each with specific work items, deliverables, and expected wins. Our intent is that at the end of each milestone, we should be able to assess the overall project success against deliverables and expected wins so far.