A dataset of Compiler-Introduced-Security-bugs (CISB) with reproduction materials. These CISBs are manually collected from the GCC/Clang bugzilla and Linux kernel through an empirical study.
See our paper (to appear) for more information on the CISB taxonomy and collection methodology.
Our data is stored in CISB-dataset/dataset. More details here.
We provide the following reproduction materials:
- test code for all the reproducted CISB;
- an automatic tool to test whether one CISB is triggered with pre-defined oracles.
More details here
We provide a Dockerfile that automates the setup process for our artifact. With this Dockerfile, users can easily download the dataset and evaluation materials, as well as install all the necessary software requirements in one step.
For running one of the mitigation evaluation experiments that requires SPEC CPU 2006, it is recommended to mount the host directory containing SPEC CPU 2006 to a specific directory (/cisb_docker/CISB-dataset/spec/cpu2006) in the Docker container. Here are the instructions to build and run a Docker container with this:
- Make sure you have Docker installed on your system.
- Download the SPEC CPU 2006 benchmark and extract it to a directory on your host machine.
- Navigate to the directory where you have the Dockerfile and run the following command to build the Docker image:
cd path/to/Dockerfile
docker build -t cisb_docker .
- Once the build is complete, run the following command to start a container:
docker run -itd -v /path/to/cpu2006:/cisb_docker/CISB-dataset/spec/cpu2006 --privileged cisb_docker
As an alternative, you can also place SPEC CPU 2006 anywhere you like within the Docker container. In that case, you will need to set the environment variable before running the experiment in the container.
export SEPC_CPU_2006_PATH='path/to/cpu2006'
All of our experiments can be done through a script.
Execute the Python script to obtain the statistics of CISBs in our dataset. The result should be in line with the data in Figure 2 and Figure 3 of the paper.
python3 statistic.py -e cisb-statistics
- Review a list of bugs where the prevention performed by programmers failed. This list can be obtained by executing a script. The expected result is those CISBs exist.
python3 statistic.py -e human-mitigation
- Run a script to obtain statistics on the effectiveness of compiler mitigations. The output results should be in line with the data shown in Table 6 of the paper. We also provide a guide to measure the effectiveness of each strategy separately.
python3 statistic.py -e mitigation-effectiveness
- Run two script to measure the overhead of different compiler prevention strategies using the SPEC CPU 2006 benchmark. First, run the script to lauch all the SPEC CPU 2006 tests. It takes 62 hours to finish all the tests. You might need to set up your SPEC CPU 2006 before that.
# python3 spec/config/test_all.py
Second, run a script to obtain the statistics of the overhead of tested mitigations
# python3 statistic.py -e mitigation-overhead
The output results should be in line with the data shown in Table 6 of the paper. We also provide a guide to measure the overhead of each strategy separately.
- Execute the script to obtain the statistics of CISBs that can theoretically be prevented by automatic prevention works. The result should be in line with the data in Figure 7 of the paper.
python3 statistic.py -e target-cisb
- Check the lists of CISBs we summarized and shown in the script. These bugs should be within the scope of the corresponding prevention work.