1# American Fuzzy Lop plus plus (AFL++) 2 3## benchmarking 4 5This directory contains benchmarking tools that allow you to compare one machine 6with another in terms of raw ability to execute a fuzzing target repeatedly. 7 8To achieve this, we use a sample program ("test-instr.c") where each path is 9equally likely, supply it a single seed, and tell AFL to exit after one run of 10deterministic mutations against that seed. 11 12**Note that this is not a real-world scenario!** 13Because the target does basically nothing this is rather a stress test on 14Kernel I/O / context switching. 15For this reason you will not see a difference if you run the multicore test 16with 20 or 40 threads - or even see the performance decline the more threads 17(`-f` parameter) you use. In a real-world scenario you can expect to gain 18exec/s until 40-60 threads (if you have that many available on your CPU). 19 20Usage example: 21 22``` 23cd aflplusplus/benchmark 24python3 benchmark.py 25 [*] Ready, starting benchmark... 26 [*] Compiling the test-instr-persist-shmem fuzzing harness for the benchmark to use. 27 [*] singlecore test-instr-persist-shmem run 1 of 2, execs/s: 124883.62 28 [*] singlecore test-instr-persist-shmem run 2 of 2, execs/s: 126704.93 29 [*] Average execs/sec for this test across all runs was: 125794.28 30 [*] Using 16 fuzzers for multicore fuzzing (use --fuzzers to override). 31 [*] multicore test-instr-persist-shmem run 1 of 2, execs/s: 1179822.66 32 [*] multicore test-instr-persist-shmem run 2 of 2, execs/s: 1175584.09 33 [*] Average execs/sec for this test across all runs was: 1177703.38 34 [*] Results have been written to the benchmark-results.jsonl file. 35 [*] Results have been written to the COMPARISON.md file. 36``` 37 38By default, the script will use a number of parallel fuzzers equal to your 39available CPUs/threads (change with `--fuzzers`), and will perform each test 40three times and average the result (change with `--runs`). 41 42The script will use multicore fuzzing instead of singlecore by default (change 43with `--mode singlecore`) and use a persistent-mode shared memory harness for 44optimal speed (change with `--target test-instr`). 45 46Feel free to submit the resulting line for your CPU added to the COMPARISON.md 47and benchmark-results.jsonl files back to AFL++ in a pull request. 48 49Each run writes results to [benchmark-results.jsonl](benchmark-results.jsonl) 50in [JSON Lines](https://jsonlines.org/) format, ready to be pulled in to other 51tools such as [jq -cs](https://jqlang.github.io/jq/) or 52[pandas](https://pandas.pydata.org/) for analysis. 53 54## Data analysis 55 56There is sample data in [benchmark-results.jsonl](benchmark-results.jsonl), and 57a Jupyter notebook for exploring the results and suggesting their meaning at 58[benchmark.ipynb](benchmark.ipynb). 59 60