Update main benchmark script

[stavros11]

Makes the following changes to the script that runs the main Qibo benchmarks (QFT, variational circuit, etc.):

• Disable the possibility to pass nqubits lists. Now each run for different nqubits has to be executed seperately. This is clearer in terms of memory usage.
• Improve logging by using simple dicts that are dumped to text files using json. It is also possible to update an existing text file of logs with new logs.
• Update docs and example tests accordingly.

I think this script is sufficient to run QFT and variational circuit benchmarks using single device or multi-GPU configurations. We may want to extend the list of available benchmark circuits with more options.

Example bash script for CPU and GPU benchmarks:

FILENAME="logs.dat"
export CUDA_VISIBLE_DEVICES="0"
for NQUBITS in 25 26 27 28 29 30
do
  python main.py --type qft --nqubits $NQUBITS --filename $FILENAME
  python main.py --type variational --nqubits $NQUBITS --filename $FILENAME
done
export CUDA_VISIBLE_DEVICES=""
for NQUBITS in 25 26 27 28 29 30
do
  python main.py --type qft --nqubits $NQUBITS --filename $FILENAME
  python main.py --type variational --nqubits $NQUBITS --filename $FILENAME
done

Example bash script for multi-GPU configurations:

FILENAME="logs.dat"
export CUDA_VISIBLE_DEVICES="0,1,2,3"
python main.py --type qft --filename $FILENAME --nqubits 31 --device /CPU:0 --accelerators 1/GPU:0,1/GPU:1
python main.py --type qft --filename $FILENAME --nqubits 32 --device /CPU:0 --accelerators 1/GPU:0,1/GPU:1,1/GPU:2,1/GPU:3
python main.py --type qft --filename $FILENAME --nqubits 33 --device /CPU:0 --accelerators 2/GPU:0,2/GPU:1,2/GPU:2,2/GPU:3

Optimal accelerator configurations for each nqubits depend on the GPU memory. It is also possible to switch to single precision using --precision single.

I will try to add an additional benchmark script based on pytest-benchmarks to check if the code is simpler and easier to use.

[codecov]

Codecov Report

Merging #375 (5eb964a) into master (6642e00) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##            master      #375   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           78        78           
  Lines        12733     12733           
=========================================
  Hits         12733     12733           

Flag	Coverage Δ
unittests	100.00% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6642e00...5eb964a. Read the comment docs.

[scarrazza]

I would mention that the number in the front corresponds to the reuse?

[scarrazza]

@stavros11 thanks for this, looks good.

Let me know if pytest-benchmarks works for you, if so, we can write the "script" directly there, otherwise we can push a bash script with some basic setup for the dgx.

[stavros11]

Let me know if pytest-benchmarks works for you, if so, we can write the "script" directly there, otherwise we can push a bash script with some basic setup for the dgx.

Thanks for the review. I fixed the documentation point and pushed an example bash script with the DGX specs in mind. The script contains benchmarks for CPU, GPU and multi-GPU for QFT and variational circuit in single and double precision but different configurations can be tested by adding/removing the corresponding lines.

I am checking pytest-benchmark. I like that it has a default mechanism that logs everything (even machine specs) and that if I understand correctly it executes repeated benchmarks and averages to get the final time, so it should be a bit more accurate. I am just not sure if it can have the flexibility of measuring circuit creation, execution and measurement time seperately like I do in the main.py script. Regardless, I will try to add some pytest benchmarks in a different PR so that both options are available.

[scarrazza]

Thank you, looks good to me. I have tested successfully the script here in system running Qibo 0.1.3.