Skip to content

GitLab

GitLab Agent for Kubernetes (FREE)

  • Introduced in GitLab 13.4.
  • Support for grpcs introduced in GitLab 13.6.
  • Introduced in GitLab 13.10, KAS became available on GitLab.com under wss://kas.gitlab.com through an Early Adopter Program.
  • Introduced in GitLab 13.11, the GitLab Agent became available to every project on GitLab.com.
  • Moved from GitLab Premium to GitLab Free in 14.5.
  • Renamed from "GitLab Kubernetes Agent" to "GitLab Agent for Kubernetes" in GitLab 14.6.

The GitLab Agent for Kubernetes ("Agent", for short) is an active in-cluster component for connecting Kubernetes clusters to GitLab safely to support cloud-native deployment, management, and monitoring.

The Agent is installed into the cluster through code, providing you with a fast, safe, stable, and scalable solution.

With GitOps, you can manage containerized clusters and applications from a Git repository that:

  • Is the single source of truth of your system.
  • Is the single place where you operate your system.
  • Is a single resource to monitor your system.

By combining GitLab, Kubernetes, and GitOps, it results in a robust infrastructure:

  • GitLab as the GitOps operator.
  • Kubernetes as the automation and convergence system.
  • GitLab CI/CD as the Continuous Integration and Continuous Deployment engine.

Beyond that, you can use all the features offered by GitLab as the all-in-one DevOps platform for your product and your team.

Agent's features

By using the Agent, you can:

See the Agent roadmap to track its development.

To contribute to the Agent, see the Agent's development documentation.

Agent's GitOps workflow (PREMIUM)

The Agent uses multiple GitLab projects to provide a flexible workflow that can suit various needs. This diagram shows these repositories and the main actors involved in a deployment:

sequenceDiagram
  participant D as Developer
  participant A as Application code repository
  participant M as Manifest repository
  participant K as GitLab Agent
  participant C as Agent configuration repository
  loop Regularly
    K-->>C: Grab the configuration
  end
  D->>+A: Pushing code changes
  A->>M: Updating manifest
  loop Regularly
    K-->>M: Watching changes
    M-->>K: Pulling and applying changes
  end

For more details, refer to our architecture documentation in the Agent project.

Install the Agent in your cluster

See how to install the Agent in your cluster.

GitOps deployments (PREMIUM)

To perform GitOps deployments with the Agent, you need:

  • A properly-configured Kubernetes cluster where the Agent is running.
  • A configuration repository that contains a config.yaml file, which tells the Agent the repositories to synchronize with the cluster.
  • A manifest repository that contains manifest files. Any changes to manifest files are applied to the cluster.

You can use a single GitLab project or different projects for the Agent configuration and manifest files, as follows:

  • Single GitLab project (recommended): When you use a single repository to hold both the manifest and the configuration files, these projects can be either private or public.
  • Two GitLab projects: When you use two different GitLab projects (one for manifest files and another for configuration files), the manifests project must be public, while the configuration project can be either private or public.

Support for separated private manifest and configuration repositories is tracked in this issue.

Kubernetes Network Security Alerts (ULTIMATE)

The GitLab Agent also provides an integration with Cilium. This integration provides a simple way to generate network policy-related alerts and to surface those alerts in GitLab.

There are several components that work in concert for the Agent to generate the alerts:

The setup process follows the same Agent's installation steps, with the following differences:

  • When you define a configuration repository, you must do so with Cilium settings.
  • You do not need to specify the gitops configuration section.

Remove an agent

You can remove an agent using the GitLab UI or through the GraphQL API.

Remove an agent through the GitLab UI

Introduced in GitLab 14.7.

To remove an agent from the UI:

  1. Go to your agent's configuration repository.
  2. From your project's sidebar, select Infrastructure > Kubernetes clusters.
  3. Select your agent from the table, and then in the Options column, click the vertical ellipsis ({ellipsis_v}) button and select Delete agent.

Remove an agent with the GitLab GraphQL API

  1. Get the <cluster-agent-token-id> from a query in the interactive GraphQL explorer. For GitLab.com, go to https://gitlab.com/-/graphql-explorer to open GraphQL Explorer. For self-managed GitLab instances, go to https://gitlab.example.com/-/graphql-explorer, replacing gitlab.example.com with your own instance's URL.

    query{
  project(fullPath: "<full-path-to-agent-configuration-project>") {
    clusterAgent(name: "<agent-name>") {
      id
      tokens {
        edges {
          node {
            id
          }
        }
      }
    }
  }
}

  2. Remove an agent record with GraphQL by deleting the clusterAgentToken.

    mutation deleteAgent {
  clusterAgentDelete(input: { id: "<cluster-agent-id>" } ) {
    errors
  }
}

mutation deleteToken {
  clusterAgentTokenDelete(input: { id: "<cluster-agent-token-id>" }) {
    errors
  }
}

  3. Verify whether the removal occurred successfully. If the output in the Pod logs includes unauthenticated, it means that the agent was successfully removed:

    {
    "level": "warn",
    "time": "2021-04-29T23:44:07.598Z",
    "msg": "GetConfiguration.Recv failed",
    "error": "rpc error: code = Unauthenticated desc = unauthenticated"
}

  4. Delete the Agent in your cluster:

    kubectl delete -n gitlab-kubernetes-agent -f ./resources.yml

Migrating to the GitLab Agent from the legacy certificate-based integration

Find out how to migrate to the GitLab Agent for Kubernetes from the certificate-based integration depending on the features you use.

Troubleshooting

If you face any issues while using the Agent, read the service logs with the following command:

kubectl logs -f -l=app=gitlab-kubernetes-agent -n gitlab-kubernetes-agent

GitLab administrators can additionally view the GitLab Agent Server logs.

Agent logs

Transport: Error while dialing failed to WebSocket dial

{
  "level": "warn",
  "time": "2020-11-04T10:14:39.368Z",
  "msg": "GetConfiguration failed",
  "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing failed to WebSocket dial: failed to send handshake request: Get \\\"https://gitlab-kas:443/-/kubernetes-agent\\\": dial tcp: lookup gitlab-kas on 10.60.0.10:53: no such host\""
}

This error is shown if there are some connectivity issues between the address specified as kas-address, and your Agent pod. To fix it, make sure that you specified the kas-address correctly.

{
  "level": "error",
  "time": "2021-06-25T21:15:45.335Z",
  "msg": "Reverse tunnel",
  "mod_name": "reverse_tunnel",
  "error": "Connect(): rpc error: code = Unavailable desc = connection error: desc= \"transport: Error while dialing failed to WebSocket dial: expected handshake response status code 101 but got 301\""
}

This error occurs if the kas-address doesn't include a trailing slash. To fix it, make sure that the wss or ws URL ends with a trailing slash, such as wss://GitLab.host.tld:443/-/kubernetes-agent/ or ws://GitLab.host.tld:80/-/kubernetes-agent/.

ValidationError(Deployment.metadata)

{
  "level": "info",
  "time": "2020-10-30T08:56:54.329Z",
  "msg": "Synced",
  "project_id": "root/kas-manifest001",
  "resource_key": "apps/Deployment/kas-test001/nginx-deployment",
  "sync_result": "error validating data: [ValidationError(Deployment.metadata): unknown field \"replicas\" in io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta, ValidationError(Deployment.metadata): unknown field \"selector\" in io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta, ValidationError(Deployment.metadata): unknown field \"template\" in io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta]"
}

This error is shown if a manifest file is malformed, and Kubernetes can't create specified objects. Make sure that your manifest files are valid. You may try using them to create objects in Kubernetes directly for more troubleshooting.

Error while dialing failed to WebSocket dial: failed to send handshake request

{
  "level": "warn",
  "time": "2020-10-30T09:50:51.173Z",
  "msg": "GetConfiguration failed",
  "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing failed to WebSocket dial: failed to send handshake request: Get \\\"https://GitLabhost.tld:443/-/kubernetes-agent\\\": net/http: HTTP/1.x transport connection broken: malformed HTTP response \\\"\\\\x00\\\\x00\\\\x06\\\\x04\\\\x00\\\\x00\\\\x00\\\\x00\\\\x00\\\\x00\\\\x05\\\\x00\\\\x00@\\\\x00\\\"\""
}

This error is shown if you configured wss as kas-address on the agent side, but KAS on the server side is not available via wss. To fix it, make sure the same schemes are configured on both sides.

It's not possible to set the grpc scheme due to the issue It is not possible to configure KAS to work with grpc without directly editing GitLab KAS deployment. To use grpc while the issue is in progress, directly edit the deployment with the kubectl edit deployment gitlab-kas command, and change --listen-websocket=true to --listen-websocket=false. After running that command, you should be able to use grpc://gitlab-kas.<YOUR-NAMESPACE>:8150.

Decompressor is not installed for grpc-encoding

{
  "level": "warn",
  "time": "2020-11-05T05:25:46.916Z",
  "msg": "GetConfiguration.Recv failed",
  "error": "rpc error: code = Unimplemented desc = grpc: Decompressor is not installed for grpc-encoding \"gzip\""
}

This error is shown if the version of the agent is newer that the version of KAS. To fix it, make sure that both agentk and KAS use the same versions.

Certificate signed by unknown authority

{
  "level": "error",
  "time": "2021-02-25T07:22:37.158Z",
  "msg": "Reverse tunnel",
  "mod_name": "reverse_tunnel",
  "error": "Connect(): rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing failed to WebSocket dial: failed to send handshake request: Get \\\"https://GitLabhost.tld:443/-/kubernetes-agent/\\\": x509: certificate signed by unknown authority\""
}

This error is shown if your GitLab instance is using a certificate signed by an internal CA that is unknown to the agent. One approach to fixing it is to present the CA certificate file to the agent via a Kubernetes configmap and mount the file in the agent /etc/ssl/certs directory from where it will be picked up automatically.

For example, if your internal CA certificate is myCA.pem:

kubectl -n gitlab-kubernetes-agent create configmap ca-pemstore --from-file=myCA.pem

Then in resources.yml:

    spec:
      serviceAccountName: gitlab-kubernetes-agent
      containers:
      - name: agent
        image: "registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/agentk:<version>"
        args:
        - --token-file=/config/token
        - --kas-address
        - wss://kas.host.tld:443 # replace this line with the line below if using Omnibus GitLab or GitLab.com.
        # - wss://gitlab.host.tld:443/-/kubernetes-agent/
        # - wss://kas.gitlab.com # for GitLab.com users, use this KAS.
        # - grpc://host.docker.internal:8150 # use this attribute when connecting from Docker.
        volumeMounts:
        - name: token-volume
          mountPath: /config
        - name: ca-pemstore-volume
          mountPath: /etc/ssl/certs/myCA.pem
          subPath: myCA.pem
      volumes:
      - name: token-volume
        secret:
          secretName: gitlab-kubernetes-agent-token
      - name: ca-pemstore-volume
        configMap:
          name: ca-pemstore
          items:
          - key: myCA.pem
            path: myCA.pem

Alternatively, you can mount the certificate file at a different location and include it using the --ca-cert-file agent parameter:

      containers:
      - name: agent
        image: "registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/agentk:<version>"
        args:
        - --ca-cert-file=/tmp/myCA.pem
        - --token-file=/config/token
        - --kas-address
        - wss://kas.host.tld:443 # replace this line with the line below if using Omnibus GitLab or GitLab.com.
        # - wss://gitlab.host.tld:443/-/kubernetes-agent/
        # - wss://kas.gitlab.com # for GitLab.com users, use this KAS.
        # - grpc://host.docker.internal:8150 # use this attribute when connecting from Docker.
        volumeMounts:
        - name: token-volume
          mountPath: /config
        - name: ca-pemstore-volume
          mountPath: /tmp/myCA.pem
          subPath: myCA.pem