Home

modsecurity-performance wiki

Sample results

2017-02-28 (1)

• nginx: 1.11.10
• ModSecurity: v3/dev/parser (8b8fd84)
• ModSecurity-nginx: v3/dev/parser (9f6d3a7)
• environment: 2-core VirtualBox VM on MBP A1502 (early 2015, 2-core i5 2.9GHz)

Summary for /modsec-off, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10      39445.56      46228.25      44283.87     43476.616     2184.7312
 latency (ms)
x  10           1.1          1.37          1.21         1.222   0.091627264

Summary for /modsec-light, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10       8071.25      11953.68      10634.85     10374.126     1200.2281
 latency (ms)
x  10          4.23          7.82             5         5.358     1.2109941

Summary for /modsec-full, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10         238.8        259.02        246.88       247.418     5.9640549
 latency (ms)
x  10        208.11        230.52        217.76       218.968     6.9131273

2017-02-28 (2)

• nginx: 1.11.10
• ModSecurity: v3/master (3a41308)
• ModSecurity-nginx: master (134bd36)
• environment: 2-core VirtualBox VM on MBP A1502 (early 2015, 2-core i5 2.9GHz)

Summary for /modsec-off, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10       38136.4      47561.79      44300.42     43351.954     2743.1755
 latency (ms)
x  10          1.07           1.4          1.19         1.206   0.099911072

Summary for /modsec-light, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10      10120.76       12979.9      12727.41     12290.594     891.67524
 latency (ms)
x  10          3.88          5.02          3.98         4.128    0.34726871

Summary for /modsec-full, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10        334.64        370.86        363.77       356.733     13.667706
 latency (ms)
x  10        142.49        158.27        147.04       148.176     5.8598297

2017-03-05

• nginx: 1.11.10
• ModSecurity: v3/dev/parser (8b8fd84)
• ModSecurity-nginx: v3/dev/parser (9f6d3a7)
• environment: 12-core KVM/libvirt VM on bare-metal server (Intel Xeon E5645 2.4GHz, 24 cores total)
• configuration details:
• nginx: worker_processes 6; worker_cpu_affinity 111111000000;
• wrk: taskset -c 0-5 wrk -t6 -c600 -d30s (10 iterations)

Summary for /modsec-off, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10      87538.41      97062.44      93506.49      92832.83     2717.0839
 latency (ms)
x  10          6.22          6.88          6.49         6.508    0.21054427

Summary for /modsec-light, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10      27984.07      31588.76      31013.95     30484.634     1151.9494
 latency (ms)
x  10         18.98         21.57         20.06         20.01    0.81266366

Summary for /modsec-full, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10         164.5        251.93        181.79       191.226     28.760167
 latency (ms)
x  10        197.41        429.26        330.67       308.639     73.228166

The numbers for /modsec-off and /modsec-light with multi-worker nginx setup are significantly better than in single-worker mode, but /modsec-full does not show any difference. In the process of investigation it turned out that disabling audit log (by setting SecAuditEngine Off) greatly improves overall performance with OWASP CRS v3.0.0 loaded:

Summary for /modsec-full, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10        577.99        864.63        854.69       823.902      87.17359
 latency (ms)
x  10        633.42        730.81        646.95       654.834     28.487218

Further investigation showed that in case of SecAuditEngine RelevantOnly only one CPU core (among the set of cores nginx is using according to worker_cpu_affinity) is 100% busy:

With SecAuditEngine Off, all cores that are being used by nginx are constantly busy:

2017-03-07

• nginx: 1.11.10
• ModSecurity: v3/master (53485c7)
• ModSecurity-nginx: master (5175214)
• environment: 2-core VirtualBox VM on MBP A1502 (early 2015, 2-core i5 2.9GHz)

Summary for /modsec-off, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10      44040.08       56882.7      53699.65     51600.859     4057.3947
 latency (ms)
x  10          0.87          1.14          0.99         0.977   0.084859361

Summary for /modsec-light, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10      10568.21      13532.37      12921.28      12336.21     1207.9385
 latency (ms)
x  10          3.71          4.74           3.9         4.104    0.42893149

Summary for /modsec-full, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10        238.79        259.41        253.79        250.74     7.4913906
 latency (ms)
x  10        207.47        221.61        213.96       213.635     5.7969556

2017-03-10

• nginx: 1.11.10
• ModSecurity: v3/dev/speedup (d9fabea)
• ModSecurity-nginx: master (5175214)
• environment: 2-core VirtualBox VM on MBP A1502 (early 2015, 2-core i5 2.9GHz)

Summary for /modsec-off, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10      48151.25      54508.49      52135.16     51337.515     2114.2494
 latency (ms)
x  10          0.92          1.14          0.99         0.996   0.069633964

Summary for /modsec-light, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10      10456.84       13941.7      13011.89      12441.49     1375.0929
 latency (ms)
x  10          3.59           4.8          3.97         4.078    0.48276518

Summary for /modsec-full, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10        270.44        284.01        278.54       278.824     4.1302763
 latency (ms)
x  10        187.65        198.52        191.41       191.247     3.5242337

2017-04-11 (1)

• nginx: 1.11.10
• ModSecurity: v3/master (b58f713)
• ModSecurity-nginx: master (3de175b)
• environment: 2-core VirtualBox VM on MBP A1502 (early 2015, 2-core i5 2.9GHz)

Summary for /modsec-off, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10      46879.99      58925.23      53077.84     53435.713     3596.6361
 latency (ms)
x  10          0.85        843.78          0.96        85.236      266.5252

Summary for /modsec-light, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10      10060.67      13613.14      12955.81     12213.674     1312.7484
 latency (ms)
x  10          3.67          5.16          4.02         4.164     0.5084661

Summary for /modsec-full, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10        271.63        295.32        290.55       287.362     7.2825313
 latency (ms)
x  10        179.23        192.48        183.71       185.278     4.2224369

2017-04-11 (2)

• nginx: 1.11.10
• ModSecurity: v3/master (b58f713)
• ModSecurity-nginx: master (3de175b)
• environment: 12-core KVM/libvirt VM on bare-metal server (Intel Xeon E5645 2.4GHz, 24 cores total)
• configuration details:
• nginx: worker_processes 6; worker_cpu_affinity 111111000000;
• wrk: taskset -c 0-5 wrk -t6 -c600 -d30s (10 iterations)

Summary for /modsec-off, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10      97613.95     101573.29      99853.65     99354.804     1370.9853
 latency (ms)
x  10          5.94          6.85          6.11         6.145     0.2596258

Summary for /modsec-light, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10      23101.87      31886.19      31184.36     30033.129     2761.2698
 latency (ms)
x  10         18.89         26.73            20        21.017     2.5738603

Summary for /modsec-full, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10        931.68        946.17        939.08       939.032     4.0696869
 latency (ms)
x  10        580.29         626.9        600.27       598.622     14.275303

With the recent sources all the nginx workers are busy while benchmarking /modsec-full location (previously only one worker process reached to 100% CPU usage).

Also, turning SecAuditEngine Off does not affect latency && rps anymore.

2017-08-21

• OS updated to Ubuntu 17.04 "zesty" (kernel 4.10.0-30-generic #34-Ubuntu)
• nginx: 1.13.4
• ModSecurity: v3/master (8d6209f)
• ModSecurity-nginx: master (abbf2c4)
• environment: 12-core KVM/libvirt VM on bare-metal server (Intel Xeon E5-2660 2.20GHz, 32 cores total)
• configuration details:
• nginx: worker_processes 6; worker_cpu_affinity 111111000000;
• wrk: taskset -c 0-5 wrk -t6 -c600 -d30s (10 iterations)

Summary for /modsec-off, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10      53343.88     103336.29     100394.62     94998.443     14838.037
 latency (ms)
x  10          5.81         11.35          6.12          6.58     1.6827689

Summary for /modsec-light, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10      20488.98      29468.81       28974.4     28141.574     2706.7983
 latency (ms)
x  10         20.44         29.72         21.36         22.29     2.7457927

Summary for /modsec-full, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10        813.95         965.7        940.75       931.136     42.406455
 latency (ms)
x  10        541.48        627.62        605.03       599.103     23.770327

2017-08-28

• nginx: 1.13.4
• ModSecurity: v3/master (04f7009)
• ModSecurity-nginx: master (abbf2c4)
• environment: 12-core KVM/libvirt VM on bare-metal server (Intel Xeon E5-2660 2.20GHz, 32 cores total)
• configuration details:
• nginx: worker_processes 6; worker_cpu_affinity 111111000000;
• wrk: taskset -c 0-5 wrk -t6 -c600 -d30s (10 iterations)

Summary for /modsec-off, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10      59232.06      98088.52      94674.82     90202.254     11188.852
 latency (ms)
x  10          6.18         10.15          6.44         6.799     1.1892336

Summary for /modsec-light, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10      25687.36      26564.29      26299.24     26197.744     294.37041
 latency (ms)
x  10         22.76         25.53         25.08        24.543    0.91399064

Summary for /modsec-full, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10        988.93       1037.52       1008.77       1010.23     12.988515
 latency (ms)
x  10        575.26        604.29        589.26       590.084     8.6321379

2017-12-11

• nginx: 1.13.7
• ModSecurity: v3/master (81e1cdc)
• ModSecurity-nginx: master (a2a5858)
• environment: 12-core KVM/libvirt VM on bare-metal server (Intel Xeon E5-2660 2.20GHz, 32 cores total)
• configuration details:
• nginx: worker_processes 6; worker_cpu_affinity 111111000000;
• wrk: taskset -c 0-5 wrk -t6 -c600 -d30s (10 iterations)

Summary for /modsec-off, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10      98673.71     109369.41     107954.84     106506.79     3085.2164
 latency (ms)
x  10          5.48          6.08          5.59         5.666    0.18530455

Summary for /modsec-light, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10      27851.53      29340.27      28257.97     28321.345     475.51244
 latency (ms)
x  10          20.8         22.41         22.26        21.944     0.5227959

Summary for /modsec-full, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10        720.59        731.77        726.96       726.013     4.0179377
 latency (ms)
x  10        425.48        743.31        604.91       606.949     117.04856

2018-01-10, v3.0.0 performance degradation investigation

• nginx: 1.13.7
• ModSecurity: revisions 10c4f9b, 9e9db08, fa7973a, 2988c5b, 63bef3d (v3/master); d9d702f (v3/dev/performance)
• ModSecurity-nginx: https://github.com/SpiderLabs/ModSecurity-nginx/releases/tag/v1.0.0
• environment: 16-core KVM/libvirt VM on bare-metal server (Intel Xeon E5-2660 2.20GHz, 32 cores total)
• configuration details:
• nginx: worker_processes 8; worker_cpu_affinity 1111111100000000;
• wrk: taskset -c 0-7 wrk -t8 -c800 -d30s (3 iterations, averages from ministat for /modsec-full location)

test@vagrant:~$ ./batchbench.sh stats
;rps_avg,latency_avg,workers_utime_avg,revision,date,commit_log
701.53,785.33,23801.33,10c4f9b1b2476f71159fa5569d9238001760404c,2017-08-19 10:21:57 +0300,add a test for macro expansion in @rx
703.76,806.01,23810.33,9e9db08b874fe7c1200aafd95fe6bccd41148ae5,2017-08-19 11:16:54 +0300,add @rx macro expansion test to list in Makefile
32.25,1510.65,19958.33,fa7973a4ef99b0d91122d16ffee51744288d037f,2017-10-06 20:32:40 +0000,Removes a regex optimization added at #1536
40.11,1553.15,23940.00,2988c5bb07c4a5ad434855413f20fec11008c818,2017-10-06 20:35:09 +0000,CHANGES: add info about #1536
38.72,1540.82,23961.33,63bef3d142b2ae25ed42d344c40729fb5f3d552e,2017-10-03 20:50:02 +0000,Support to JSON stuff on serial logging
834.01,701.07,23766.33,d9d702f401c870bf399d8cd5bc4ae212c7d52195,2018-01-03 09:49:20 -0300,Fix the debuglogs for the regression tests

"Bad" changeset: https://github.com/SpiderLabs/ModSecurity/commit/fa7973a4ef99b0d91122d16ffee51744288d037f

2018-02-21 (1)

• nginx: 1.13.9
• ModSecurity: v3/master (2b052b0) after merging things from v3/dev/performance
• ModSecurity-nginx: master (c0ae166)
• environment: 16-core KVM/libvirt VM on bare-metal server (Intel Xeon E5-2660 2.20GHz, 32 cores total)
• configuration details:
• nginx: worker_processes 8; worker_cpu_affinity 1111111100000000;
• wrk: taskset -c 0-7 wrk -t8 -c800 -d30s (10 iterations)

Summary for /modsec-off, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10     122166.89     130836.04     127696.53     127721.72     2512.4105
 latency (ms)
x  10          6.13          6.55          6.28         6.293    0.11954079

Summary for /modsec-light, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10      21701.01      22558.73      22176.23     22137.719     263.26924
 latency (ms)
x  10         35.47         37.07         36.41         36.27    0.46714261

Summary for /modsec-full, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10       1417.27       1470.06       1444.13      1442.311      18.23003
 latency (ms)
x  10        510.99        556.53        529.76       530.937     12.394951

2018-02-21 (2)

• re-run after this change: perfrun: align request URI and headers with test/benchmark/benchmark.cc (58429f5), to make benchmark parameters in line with official sources and batchperfrun.sh
• nginx: 1.13.9
• ModSecurity: v3/master (2b052b0) after merging things from v3/dev/performance
• ModSecurity-nginx: master (c0ae166)
• environment: 16-core KVM/libvirt VM on bare-metal server (Intel Xeon E5-2660 2.20GHz, 32 cores total)
• configuration details:
• nginx: worker_processes 8; worker_cpu_affinity 1111111100000000;
• wrk: taskset -c 0-7 wrk -t8 -c800 -d30s (10 iterations)

Summary for /modsec-off, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10     111668.95     115903.34     113320.77     113303.27     1325.0454
 latency (ms)
x  10          6.97          7.23          7.06         7.076   0.082758014

Summary for /modsec-light, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10      15669.93      16356.81      16205.53     16158.662      198.1997
 latency (ms)
x  10         49.02          51.4         49.52        49.699    0.71657286

Summary for /modsec-full, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10        763.27        798.54        776.69       777.669     11.918996
 latency (ms)
x  10        875.06        916.94        905.78       898.856     14.756683

2018-03-21

• nginx: 1.13.10
• ModSecurity: v3/master (480a2f8)
• ModSecurity-nginx: master (c0ae166)
• environment: 16-core KVM/libvirt VM on bare-metal server (Intel Xeon E5-2660 2.20GHz, 32 cores total)
• configuration details:
• nginx: worker_processes 8; worker_cpu_affinity 1111111100000000;
• wrk: taskset -c 0-7 wrk -t8 -c800 -d30s (10 iterations)

Summary for /modsec-off, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10     104478.12     110897.35     109413.86     108292.21     2436.9943
 latency (ms)
x  10          7.23          7.67          7.37          7.41    0.17682383

Summary for /modsec-light, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10      15501.77      16428.82      16224.92     16124.139     283.24234
 latency (ms)
x  10         48.93         51.57         49.55        49.784    0.83446856

Summary for /modsec-full, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10        763.65        782.83        769.76       772.489     6.8919687
 latency (ms)
x  10        814.86        921.28        907.02        897.04     30.261893

2018-09-12

• nginx: 1.15.2
• ModSecurity: v3/master (c2bc695)
• ModSecurity-nginx: master (4b50399)
• environment: 16-core KVM/libvirt VM on bare-metal server (Intel Xeon E5-2660 2.20GHz, 32 cores total)
• configuration details:
• nginx: worker_processes 8; worker_cpu_affinity 1111111100000000;
• wrk: taskset -c 0-7 wrk -t8 -c800 -d30s (10 iterations)

Summary for /modsec-off, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10     107957.01     116518.73     114973.68     113465.84      2668.353
 latency (ms)
x  10          6.85          7.42          7.09         7.058    0.17254951

Summary for /modsec-light, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10      20629.51      21555.77      21271.15     21241.834     313.33944
 latency (ms)
x  10         37.34         38.96         37.87        37.907    0.50035654

Summary for /modsec-full, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10        823.14        839.74         828.6       830.392     5.2695304
 latency (ms)
x  10        842.06        861.77        853.57       852.918     6.2198139

2018-10-25

• nginx: 1.15.2
• ModSecurity: v3/master (973c1f1)
• ModSecurity-nginx: master (56cfa4e)
• environment: 16-core KVM/libvirt VM on bare-metal server (Intel Xeon E5-2660 2.20GHz, 32 cores total)
• configuration details:
• nginx: worker_processes 8; worker_cpu_affinity 1111111100000000;
• wrk: taskset -c 0-7 wrk -t8 -c800 -d30s (10 iterations)

Summary for /modsec-off, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10     102468.93     107361.52     105511.47     104881.01     1593.9929
 latency (ms)
x  10          7.46           7.8          7.65         7.639    0.11957796

Summary for /modsec-light, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10      20382.72      20983.25       20843.1     20726.095     218.84551
 latency (ms)
x  10         38.27         39.58         38.76        38.768    0.42012696

Summary for /modsec-full, RPS (count):
    N           Min           Max        Median           Avg        Stddev
x  10       1805.67       1820.32       1813.08      1812.302     5.1526343
 latency (ms)
x  10        418.25        430.86        422.75       422.814     3.4299537