Context: To enable a medium term solution for Filecoin retrieval, it's important to understand the breadth of users who may want to retrieve data from Filecoin. There are currently three broad categories of retrieval users
The first category of retrieval users are what we define as "aggregators", who provide a simplified Filecoin storage/retrieval experience for an end user.
In general, for aggregators, the expectation is that retrieval is fast. Most of the aggregators today are using IPFS for the retrieval story.
web3.storage users are generally web3 application developers who want to leverage a simple interface to use our stack of storage solutions (IPFS, Filecoin) for use cases like off chain storage, serverless applications, or simple content addressing. The product currently uses IPFS for data retrieval
For application development, the expectation for retrieval is that latency and throughput are similar to Amazon S3 + Cloudfront solutions.
nft.storage users are generally consumers who want to easily store their nft assets (art files, etc.) on a persistent storage layer. This product has a large user base with around 8,900 end users, and supports a pinning service API for easy retrieval via IPFS.
Due to the consumer facing nature of the product, the users expect fast retrieval
Estuary does not have as specific of a use case as the other aggregators, and aim to be a more general product that clients can use to more easily store their datasets on Filecoin. Estuary uses IPFS for storage and retrieval, but does periodic garbage collection, so if a client wants to retrieve data which is no longer pinned on the node, Estuary will retrieve the data from Filecoin.
Since the use cases for customers on Estuary varies more drastically, there isn't a clear cut requirement for fast retrieval. For example, there are archival datasets uploaded via Estuary, and in this case data does not need to be quickly retrieved.
Note: As of October 2021 there are 9,316,195 files sealed on the Filecoin Network, 32,095 successful deals on chain. 81.89 TiB of pinned data as a IPFS node, and 269.44 TiB total storage sealed on chain.
Textile bidbot is currently designed to be a storage auction system that can help facilitate storage provider selection for clients. To learn more about bidbot, you can view these resources: