Coverage-guided fuzz testing (ULTIMATE)

Coverage-guided fuzz testing sends random inputs to an instrumented version of your application in an effort to cause unexpected behavior. Such behavior indicates a bug that you should address. GitLab allows you to add coverage-guided fuzz testing to your pipelines. This helps you discover bugs and potential security issues that other QA processes may miss.

We recommend that you use fuzz testing in addition to the other security scanners in GitLab Secure and your own test processes. If you're using GitLab CI/CD, you can run your coverage-guided fuzz testing as part your CI/CD workflow.

Coverage-guided fuzz testing process

The fuzz testing process:

1. Compiles the target application.
2. Runs the instrumented application, using the gitlab-cov-fuzz tool.
3. Parses and analyzes the exception information output by the fuzzer.
4. Downloads the corpus and crash events from previous pipelines.
5. Outputs the parsed crash events and data to the gl-coverage-fuzzing-report.json file.

The results of the coverage-guided fuzz testing are available in the CI/CD pipeline.

Supported fuzzing engines and languages

You can use the following fuzzing engines to test the specified languages.

Configuration

To enable coverage-guided fuzz testing, edit the .gitlab-ci.yml file:

1. Add the fuzz stage to the list of stages.

2. If your application is not written in Go, provide a Docker image using the matching fuzzing engine. For example:

   image: python:latest

3. Include the Coverage-Fuzzing.gitlab-ci.yml template provided as part of your GitLab installation.

4. Customize the my_fuzz_target job to meet your requirements.

Example extract of coverage-guided fuzzing configuration

stages:
  - fuzz

include:
  - template: Coverage-Fuzzing.gitlab-ci.yml

my_fuzz_target:
  extends: .fuzz_base
  script:
    # Build your fuzz target binary in these steps, then run it with gitlab-cov-fuzz
    # See our example repos for how you could do this with any of our supported languages
    - ./gitlab-cov-fuzz run --regression=$REGRESSION -- <your fuzz target>

The Coverage-Fuzzing template includes the hidden job .fuzz_base, which you must extend for each of your fuzzing targets. Each fuzzing target must have a separate job. For example, the go-fuzzing-example project contains one job that extends .fuzz_base for its single fuzzing target.

Note that the hidden job .fuzz_base uses several YAML keys that you must not override in your own job. If you include these keys in your own job, you must copy their original content:

• before_script
• artifacts
• rules

Available CI/CD variables

Use the following variables to configure coverage-guided fuzz testing in your CI/CD pipeline.

Seed corpus

Files in the seed corpus must be updated manually. They are not updated or overwritten by the coverage-guide fuzz testing job.

Output

Each fuzzing step outputs these artifacts:

• gl-coverage-fuzzing-report.json: A report containing details of the coverage-guided fuzz testing and its results.
• artifacts.zip: This file contains two directories:
  • corpus: Contains all test cases generated by the current and all previous jobs.
  • crashes: Contains all crash events the current job found and those not fixed in previous jobs.

You can download the JSON report file from the CI/CD pipelines page. For more information, see Downloading artifacts.

Coverage-guided fuzz testing report

Introduced in GitLab 13.3 as an Alpha feature.

For detailed information about the gl-coverage-fuzzing-report.json file's format, read the schema.

Example coverage-guided fuzzing report:

{
  "version": "v1.0.8",
  "regression": false,
  "exit_code": -1,
  "vulnerabilities": [
    {
      "category": "coverage_fuzzing",
      "message": "Heap-buffer-overflow\nREAD 1",
      "description": "Heap-buffer-overflow\nREAD 1",
      "severity": "Critical",
      "stacktrace_snippet": "INFO: Seed: 3415817494\nINFO: Loaded 1 modules   (7 inline 8-bit counters): 7 [0x10eee2470, 0x10eee2477), \nINFO: Loaded 1 PC tables (7 PCs): 7 [0x10eee2478,0x10eee24e8), \nINFO:        5 files found in corpus\nINFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes\nINFO: seed corpus: files: 5 min: 1b max: 4b total: 14b rss: 26Mb\n#6\tINITED cov: 7 ft: 7 corp: 5/14b exec/s: 0 rss: 26Mb\n=================================================================\n==43405==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x602000001573 at pc 0x00010eea205a bp 0x7ffee0d5e090 sp 0x7ffee0d5e088\nREAD of size 1 at 0x602000001573 thread T0\n    #0 0x10eea2059 in FuzzMe(unsigned char const*, unsigned long) fuzz_me.cc:9\n    #1 0x10eea20ba in LLVMFuzzerTestOneInput fuzz_me.cc:13\n    #2 0x10eebe020 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) FuzzerLoop.cpp:556\n    #3 0x10eebd765 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool*) FuzzerLoop.cpp:470\n    #4 0x10eebf966 in fuzzer::Fuzzer::MutateAndTestOne() FuzzerLoop.cpp:698\n    #5 0x10eec0665 in fuzzer::Fuzzer::Loop(std::__1::vector\u003cfuzzer::SizedFile, fuzzer::fuzzer_allocator\u003cfuzzer::SizedFile\u003e \u003e\u0026) FuzzerLoop.cpp:830\n    #6 0x10eead0cd in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) FuzzerDriver.cpp:829\n    #7 0x10eedaf82 in main FuzzerMain.cpp:19\n    #8 0x7fff684fecc8 in start+0x0 (libdyld.dylib:x86_64+0x1acc8)\n\n0x602000001573 is located 0 bytes to the right of 3-byte region [0x602000001570,0x602000001573)\nallocated by thread T0 here:\n    #0 0x10ef92cfd in wrap__Znam+0x7d (libclang_rt.asan_osx_dynamic.dylib:x86_64+0x50cfd)\n    #1 0x10eebdf31 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) FuzzerLoop.cpp:541\n    #2 0x10eebd765 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool*) FuzzerLoop.cpp:470\n    #3 0x10eebf966 in fuzzer::Fuzzer::MutateAndTestOne() FuzzerLoop.cpp:698\n    #4 0x10eec0665 in fuzzer::Fuzzer::Loop(std::__1::vector\u003cfuzzer::SizedFile, fuzzer::fuzzer_allocator\u003cfuzzer::SizedFile\u003e \u003e\u0026) FuzzerLoop.cpp:830\n    #5 0x10eead0cd in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) FuzzerDriver.cpp:829\n    #6 0x10eedaf82 in main FuzzerMain.cpp:19\n    #7 0x7fff684fecc8 in start+0x0 (libdyld.dylib:x86_64+0x1acc8)\n\nSUMMARY: AddressSanitizer: heap-buffer-overflow fuzz_me.cc:9 in FuzzMe(unsigned char const*, unsigned long)\nShadow bytes around the buggy address:\n  0x1c0400000250: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa\n  0x1c0400000260: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa\n  0x1c0400000270: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa\n  0x1c0400000280: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa\n  0x1c0400000290: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa\n=\u003e0x1c04000002a0: fa fa fd fa fa fa fd fa fa fa fd fa fa fa[03]fa\n  0x1c04000002b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa\n  0x1c04000002c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa\n  0x1c04000002d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa\n  0x1c04000002e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa\n  0x1c04000002f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa\nShadow byte legend (one shadow byte represents 8 application bytes):\n  Addressable:           00\n  Partially addressable: 01 02 03 04 05 06 07 \n  Heap left redzone:       fa\n  Freed heap region:       fd\n  Stack left redzone:      f1\n  Stack mid redzone:       f2\n  Stack right redzone:     f3\n  Stack after return:      f5\n  Stack use after scope:   f8\n  Global redzone:          f9\n  Global init order:       f6\n  Poisoned by user:        f7\n  Container overflow:      fc\n  Array cookie:            ac\n  Intra object redzone:    bb\n  ASan internal:           fe\n  Left alloca redzone:     ca\n  Right alloca redzone:    cb\n  Shadow gap:              cc\n==43405==ABORTING\nMS: 1 EraseBytes-; base unit: de3a753d4f1def197604865d76dba888d6aefc71\n0x46,0x55,0x5a,\nFUZ\nartifact_prefix='./crashes/'; Test unit written to ./crashes/crash-0eb8e4ed029b774d80f2b66408203801cb982a60\nBase64: RlVa\nstat::number_of_executed_units: 122\nstat::average_exec_per_sec:     0\nstat::new_units_added:          0\nstat::slowest_unit_time_sec:    0\nstat::peak_rss_mb:              28",
      "scanner": {
        "id": "libFuzzer",
        "name": "libFuzzer"
      },
      "location": {
        "crash_address": "0x602000001573",
        "crash_state": "FuzzMe\nstart\nstart+0x0\n\n",
        "crash_type": "Heap-buffer-overflow\nREAD 1"
      },
      "tool": "libFuzzer"
    }
  ]
}

Duration of coverage-guided fuzz testing

The available durations for coverage-guided fuzz testing are: 10 minutes (default) and 60 minutes.

• 10-minute duration: Recommended for the default branch.
• 60-minute duration: Recommended for the development branch and merge requests. The longer duration provides greater coverage. In the COVFUZZ_ADDITIONAL_ARGS variable set the value --regression=true.

For a complete example, read the Go coverage-guided fuzzing example.

Continuous coverage-guided fuzz testing

It's also possible to run the coverage-guided fuzzing jobs longer and without blocking your main pipeline. This configuration uses the GitLab parent-child pipelines.

The suggested workflow in this scenario is to have long-running, asynchronous fuzzing jobs on the main or development branch, and short synchronous fuzzing jobs on all other branches and MRs. This balances the needs of completing the per-commit pipeline complete quickly, while also giving the fuzzer a large amount of time to fully explore and test the app. Long-running fuzzing jobs are usually necessary for the coverage-guided fuzzer to find deeper bugs in your codebase.

The following is an extract of the .gitlab-ci.yml file for this workflow. For the full example, see the Go fuzzing example's repository:

sync_fuzzing:
  variables:
    COVFUZZ_ADDITIONAL_ARGS: '-max_total_time=300'
  trigger:
    include: .covfuzz-ci.yml
    strategy: depend
  rules:
    - if: $CI_COMMIT_BRANCH != 'continuous_fuzzing' && $CI_PIPELINE_SOURCE != 'merge_request_event'

async_fuzzing:
  variables:
    COVFUZZ_ADDITIONAL_ARGS: '-max_total_time=3600'
  trigger:
    include: .covfuzz-ci.yml
  rules:
    - if: $CI_COMMIT_BRANCH == 'continuous_fuzzing' && $CI_PIPELINE_SOURCE != 'merge_request_event'

This creates two jobs:

1. sync_fuzzing: Runs all your fuzz targets for a short period of time in a blocking configuration. This finds simple bugs and allows you to be confident that your MRs aren't introducing new bugs or causing old bugs to reappear.
2. async_fuzzing: Runs on your branch and finds deep bugs in your code without blocking your development cycle and MRs.

The covfuzz-ci.yml is the same as that in the original synchronous example.

Offline environment

To use coverage fuzzing in an offline environment:

1. Clone gitlab-cov-fuzz to a private repository that your offline GitLab instance can access.

2. For each fuzzing step, set COVFUZZ_URL_PREFIX to ${NEW_URL_GITLAB_COV_FUZ}/-/raw, where NEW_URL_GITLAB_COV_FUZ is the URL of the private gitlab-cov-fuzz clone that you set up in the first step.

Interacting with the vulnerabilities

After a vulnerability is found, you can address it. The merge request widget lists the vulnerability and contains a button for downloading the fuzzing artifacts. By clicking one of the detected vulnerabilities, you can see its details.

You can also view the vulnerability from the Security Dashboard, which shows an overview of all the security vulnerabilities in your groups, projects, and pipelines.

Clicking the vulnerability opens a modal that provides additional information about the vulnerability:

• Status: The vulnerability's status. As with any type of vulnerability, a coverage fuzzing vulnerability can be Detected, Confirmed, Dismissed, or Resolved.
• Project: The project in which the vulnerability exists.
• Crash type: The type of crash or weakness in the code. This typically maps to a CWE.
• Crash state: A normalized version of the stack trace, containing the last three functions of the crash (without random addresses).
• Stack trace snippet: The last few lines of the stack trace, which shows details about the crash.
• Identifier: The vulnerability's identifier. This maps to either a CVE or CWE.
• Severity: The vulnerability's severity. This can be Critical, High, Medium, Low, Info, or Unknown.
• Scanner: The scanner that detected the vulnerability (for example, Coverage Fuzzing).
• Scanner Provider: The engine that did the scan. For Coverage Fuzzing, this can be any of the engines listed in Supported fuzzing engines and languages.