The above diagram illustrates how an IPFS Gateway currently retrieves data from Filecoin. There are essentially three “hops” that connect Filecoin data to IPFS Gateways. First, Gateway infrastructure (BIRD/Nginx) routes HTTP traffic to a Kubo (formerly go-ipfs) node. Then, Kubo sends Bitswap requests to the IPFS network. In the process, some of those Bitswap requests MAY go to autoretrieve. Autoretrieve may serve them from a local cache, or if it doesn’t have the content, will query the Filecoin Indexer and MAY make a retrieval deal with a Filecoin Storage provider.
We propose to remove autoretrieve from the flow by adding Bitswap support to SPs. In this flow, Gateway routes traffic to Kubo, then Kubo sends bitswap requests to the network. It connects with SPs by finding them directly through the indexer with ReFrame, then fetches content through Bitswap. This dramatically simplifies the process. At the same time, this removes autoretrieve which is a primary source for metrics for the time being. Instrumentation is key to measuring retrieval success and insuring we actually create real improvements for users.
As part of analyzing these flows, we would like to be able to track the overall health of requests in this entire pipeline, and be able to trace a given request we initiate through any point in the pipeline so that we can pinpoint failures, and performance bottlenecks.
Here is how we will measure performance in this flow.
Our primary group we’ll use to gather metrics on SP Bitswap Retrieval is Slingshot Evergreen. We will use the incentives of Slingshot Evergreen to push these SPs to: