criterion performance measurements

overview

want to understand this report?

1/200000

lower bound estimate upper bound
Mean execution time 0.9344322188441125 0.9402231844012109 0.9471978886668053
Standard deviation 4.8471936527075904e-2 5.645122707349507e-2 6.687323584970133e-2

Outlying measurements have severe (0.8037756848581197%) effect on estimated standard deviation.

2/100000

lower bound estimate upper bound
Mean execution time 0.4527658009274966 0.4579128042602704 0.463966074759182
Standard deviation 4.310817509263909e-2 4.9323826454858895e-2 5.9122420015937684e-2

Outlying measurements have severe (0.929422904580316%) effect on estimated standard deviation.

4/50000

lower bound estimate upper bound
Mean execution time 0.2555680076027082 0.26021319704693224 0.2655602844302025
Standard deviation 3.929697251162303e-2 4.409223095937771e-2 4.998546938909939e-2

Outlying measurements have severe (0.9716148059578221%) effect on estimated standard deviation.

6/33333

lower bound estimate upper bound
Mean execution time 0.18934379495622386 0.1932392827415632 0.19772559560777414
Standard deviation 3.215416467445737e-2 3.679332178029179e-2 4.376381279562756e-2

Outlying measurements have severe (0.9786202393370247%) effect on estimated standard deviation.

8/25000

lower bound estimate upper bound
Mean execution time 0.15694220937730538 0.15984081504187653 0.16319903689068227
Standard deviation 2.4214594700951726e-2 2.758783775241642e-2 3.3028428897342886e-2

Outlying measurements have severe (0.9716769455863762%) effect on estimated standard deviation.

12/16666

lower bound estimate upper bound
Mean execution time 0.11418552555085884 0.11663955924353665 0.1196033104006615
Standard deviation 2.035690374890195e-2 2.373716245258639e-2 2.8738462548757098e-2

Outlying measurements have severe (0.9821173309406289%) effect on estimated standard deviation.

16/12500

lower bound estimate upper bound
Mean execution time 0.1247751593335635 0.12749035594306057 0.13073369977316923
Standard deviation 2.268955819157438e-2 2.6233372161670383e-2 3.0817349477064986e-2

Outlying measurements have severe (0.982143881882991%) effect on estimated standard deviation.

32/6250

lower bound estimate upper bound
Mean execution time 0.11415525354387031 0.1167322396977908 0.11987733838400906
Standard deviation 2.155133768776211e-2 2.5079415933096195e-2 2.9691630526102283e-2

Outlying measurements have severe (0.9822422772023917%) effect on estimated standard deviation.

64/3125

lower bound estimate upper bound
Mean execution time 0.116662181987779 0.11940398372651802 0.12275166508994168
Standard deviation 2.2799545854937247e-2 2.661242926614585e-2 3.309889086991555e-2

Outlying measurements have severe (0.9856459427828538%) effect on estimated standard deviation.

128/1562

lower bound estimate upper bound
Mean execution time 0.12513572769802478 0.1280850942675438 0.1317087824885216
Standard deviation 2.4806836172372792e-2 2.882375225955644e-2 3.595637780485698e-2

Outlying measurements have severe (0.9856654320978487%) effect on estimated standard deviation.

understanding this report

In this report, each function benchmarked by criterion is assigned a section of its own. In each section, we display two charts, each with an x axis that represents measured execution time. These charts are active; if you hover your mouse over data points and annotations, you will see more details.

Under the charts is a small table displaying the mean and standard deviation of the measurements. We use a statistical technique called the bootstrap to provide confidence intervals on our estimates of these values. The bootstrap-derived upper and lower bounds on the mean and standard deviation let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)

A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.