Cluster Image Scanning (ULTIMATE)

Introduced in GitLab 14.1.

WARNING: This analyzer is in Alpha and is unstable. The JSON report and CI/CD configuration may be subject to change or breakage across GitLab releases.

Your Kubernetes cluster may run workloads based on images that the Container Security analyzer didn't scan. These images may therefore contain known vulnerabilities. By including an extra job in your pipeline that scans for those security risks and displays them in the vulnerability report, you can use GitLab to audit your Kubernetes workloads and environments.

GitLab provides integration with open-source tools for vulnerability analysis in Kubernetes clusters:

• Starboard

To integrate GitLab with security scanners other than those listed here, see Security scanner integration.

You can use cluster image scanning through the following methods:

• The cluster image scanning analyzer
• The GitLab Agent

Use the cluster image scanning analyzer

You can use the cluster image scanning analyzer to run cluster image scanning with GitLab CI/CD. To enable the cluster image scanning analyzer, include the CI job in your existing .gitlab-ci.yml file.

Prerequisites

To enable cluster image scanning in your pipeline, you need the following:

• Cluster Image Scanning runs in the test stage, which is available by default. If you redefine the stages in the .gitlab-ci.yml file, the test stage is required.
• GitLab Runner with the docker or kubernetes executor.
• Docker 18.09.03 or later installed on the same computer as the runner. If you're using the shared runners on GitLab.com, then this is already the case.
• Starboard Operator installed and configured in your cluster.
• The configuration for accessing your Kubernetes cluster stored in the CIS_KUBECONFIG configuration variable with the type set to File (see Configuring the cluster).

Configuring the cluster

1. Create a new service account.

   To properly fetch vulnerabilities from the cluster and to limit analyzer access to the workload, you must create a new service account with the cluster role limited to get, list, and watch vulnerabilityreports in the Kubernetes cluster:

   kubectl apply -f https://gitlab.com/gitlab-org/security-products/analyzers/cluster-image-scanning/-/raw/main/gitlab-vulnerability-viewer-service-account.yaml

2. Obtain the Kubernetes API URL.

   Get the API URL by running this command:

   API_URL=$(kubectl cluster-info | grep -E 'Kubernetes master|Kubernetes control plane' | awk '/http/ {print $NF}')

3. Obtain the CA certificate:

   1. List the secrets with kubectl get secrets. One should have a name similar to default-token-xxxxx. Copy that token name for use below.

   2. Run this command to get the certificate:

      CA_CERTIFICATE=$(kubectl get secret <secret name> -o jsonpath="{['data']['ca\.crt']}")

4. Obtain the service account token:

   TOKEN=$(kubectl -n kube-system get secret $(kubectl -n kube-system get secret | grep gitlab-vulnerability-viewer | awk '{print $1}') -o jsonpath="{.data.token}" | base64 --decode)

5. Generate the value for the CIS_KUBECONFIG variable. Copy the printed value from the output:

   echo "
   ---
   apiVersion: v1
   kind: Config
   clusters:
   - name: gitlab-vulnerabilities-viewer
     cluster:
       server: $API_URL
       certificate-authority-data: $CA_CERTIFICATE
   contexts:
   - name: gitlab-vulnerabilities-viewer
     context:
       cluster: gitlab-vulnerabilities-viewer
       namespace: default
       user: gitlab-vulnerabilities-viewer
   current-context: gitlab-vulnerabilities-viewer
   users:
   - name: gitlab-vulnerabilities-viewer
     user:
       token: $TOKEN
   "

6. Set the CI/CD variable:

   1. Navigate to your project's Settings > CI/CD.

   2. Expand the Variables section.

   3. Select Add variable and fill in the details:

      • Key: CIS_KUBECONFIG.
      • Value: generated value
      • Type: File

WARNING: The CIS_KUBECONFIG variable is accessible by all jobs executed for your project. Mark the Protect variable flag to export this variable to pipelines running on protected branches and tags only. You can apply additional protection to your cluster by restricting service account access to a single namespace, and configuring Starboard Operator to install in restricted mode.

Configuration

To include the Cluster-Image-Scanning.gitlab-ci.yml template (GitLab 14.1 and later), add the following to your .gitlab-ci.yml file:

include:
  - template: Security/Cluster-Image-Scanning.gitlab-ci.yml

The included template:

• Creates a cluster_image_scanning job in your CI/CD pipeline.
• Connects to your Kubernetes cluster with credentials provided in the CIS_KUBECONFIG variable and fetches vulnerabilities found by Starboard Operator.

GitLab saves the results as a Cluster Image Scanning report artifact that you can download and analyze later. When downloading, you always receive the most recent artifact.

Customize the cluster image scanning settings

You can customize how GitLab scans your cluster. For example, to restrict the analyzer to get results for only a certain workload, use the variables parameter in your .gitlab-ci.yml to set CI/CD variables. The variables you set in your .gitlab-ci.yml overwrite those in Cluster-Image-Scanning.gitlab-ci.yml.

CI/CD variables for cluster image scanning

You can configure analyzers by using the following CI/CD variables:

Override the cluster image scanning template

If you want to override the job definition (for example, to change properties like variables), you must declare and override a job after the template inclusion, and then specify any additional keys.

This example sets CIS_RESOURCE_NAME to nginx:

include:
  - template: Security/Cluster-Image-Scanning.gitlab-ci.yml

cluster_image_scanning:
  variables:
    CIS_RESOURCE_NAME: nginx

Connect with Kubernetes cluster associated to the project

If you want to connect to the Kubernetes cluster associated with the project and run Cluster Image Scanning jobs without configuring the CIS_KUBECONFIG variable, you must extend cluster_image_scanning and specify the environment you want to scan.

This example configures the cluster_image_scanning job to scan the Kubernetes cluster connected with the staging environment:

cluster_image_scanning:
  environment:
    name: staging
    action: prepare

Reports JSON format

The cluster image scanning tool emits a JSON report file. For more information, see the schema for this report.

Here's an example cluster image scanning report:

{
  "version": "14.0.2",
  "scan": {
    "scanner": {
      "id": "starboard_trivy",
      "name": "Trivy (using Starboard Operator)",
      "url": "https://github.com/aquasecurity/starboard",
      "vendor": {
        "name": "GitLab"
      },
      "version": "0.16.0"
    },
    "start_time": "2021-04-28T12:47:00Z",
    "end_time": "2021-04-28T12:47:00Z",
    "type": "cluster_image_scanning",
    "status": "success"
  },
  "vulnerabilities": [
    {
      "id": "c15f22205ee842184c2d55f1a207b3708283353f85083d66c34379c709b0ac9d",
      "category": "cluster_image_scanning",
      "message": "CVE-2011-3374 in apt",
      "description": "",
      "cve": "library/nginx:1.18:apt:CVE-2011-3374",
      "severity": "Low",
      "confidence": "Unknown",
      "solution": "Upgrade apt from 1.8.2.2",
      "scanner": {
        "id": "starboard_trivy",
        "name": "Trivy (using Starboard Operator)"
      },
      "location": {
        "dependency": {
          "package": {
            "name": "apt"
          },
          "version": "1.8.2.2"
        },
        "operating_system": "library/nginx:1.18",
        "image": "index.docker.io/library/nginx:1.18"
      },
      "identifiers": [
        {
          "type": "cve",
          "name": "CVE-2011-3374",
          "value": "CVE-2011-3374",
          "url": "https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2011-3374"
        }
      ],
      "links": [
        "https://avd.aquasec.com/nvd/cve-2011-3374"
      ]
    }
  ]
}

Cluster image scanning with the GitLab Agent

You can use the GitLab Agent to scan images from within your Kubernetes cluster and record the vulnerabilities in GitLab.

Prerequisites

• Starboard Operator installed and configured in your cluster.
• GitLab Agent set up in GitLab, installed in your cluster, and configured using a configuration repository.

Configuration

The Agent runs the cluster image scanning once the cluster_image_scanning directive is added to your Agent's configuration repository.

Security Dashboard

The Security Dashboard shows you an overview of all the security vulnerabilities in your groups, projects, and pipelines.

Interacting with the vulnerabilities

After a vulnerability is found, you can address it.

Troubleshooting

Getting warning message gl-cluster-image-scanning-report.json: no matching files

For information on this error, see the general Application Security troubleshooting section.