Update: re-ran this experiment using various commits of Kubo before the release candidate. See Update 30th Sep 2022 - testing intermediate commits below

This is a summary of an experiment that was run in Thunderdome. Thunderdome is our tool for comparing performance of different versions, configurations or deployments of IPFS gateways. More information here.

This experiment was designed to compare performance of Kubo 0.16.0-rc1 against v0.15.0 and v0.14.0 to identify any potential regressions or improvements.

Two variants of the experiment were run: 10 and 20 requests per second.

Key results

• At 10rps there was very little difference in performance between the versions.
• At 20rps some instances showed a noticeable drop in performance but notably the v0.16 release candidates did not seem to be affected.
• P99 TTFB for the all versions was around ~7.5s at 10rps but there was a big difference at 20rps:
  • v0.16.0-rc1 recorded TTFB ~9.7s
  • v0.15.0 was ~24.1s
  • v0.14.0 was ~22.8s

From past experiments It appears that Kubo can very easily get into a pathologically poor performing state. This is characterized by elevated heap and goroutine count and many timeouts which causes TTFB to trend towards 30s (the client timeout used in the experiments).

Higher request rates trigger this condition more frequently.

Kubo v0.16.0-rc1 seems to have avoided this state so far. This might mean that the underlying cause of the problem has been fixed or it might be that it is simply less likely to hit the bug.

Either way it looks like the current release candidate performs as well as previous versions but has much more predictable performance at higher scale.

Details

The experimental setup:

• three instances of each version of Kubo (9 instances in total)
• all instances have the same configuration
• all instances are sent the same requests at the same rate
• requests are filtered to only allow path style requests (i.e. requests beginning with /ipfs or /ipns)
• experiments ran for at least 24 hours each