1. Overview

This guide demonstrates how to scan images emitted from the OpenShift (OSE) build process. You can pass or fail the build process based on the severity of the vulnerability and compliance issues uncovered by the Twistlock scanner called twistcli. The scanner produces actionable data that application developers can use to resolve issues. This setup leverages the postCommit build hook to scan images before they’re pushed to the registry.

You can also use postCommit hooks with OpenShift Jenkins pipeline builds because pipeline builds integrate with the OpenShift BuildConfig process. Examples of OpenShift Jenkins Pipeline builds can be found here.

2. Background

Every release of Twistlock ships with a Twistlock Jenkins plugin called twistlock-jenkins-plugin.hpi. The Twistlock Jenkins plugin works with containerized local on-premises Jenkins and Cloudbees Jenkins. This plugin is designed to be integrated into Jenkins Pipeline, Freestyle and Maven build workflows. The Twistlock Jenkins plugin requires that the docker socket be mounted in the Jenkins container. The docker socket is used when scanning an image that is being built by the Jenkins workflow.

The version of Jenkins included with OpenShift is specially modified for the OpenShift Container Platform.

In addition to standard Jenkins Pipeline Syntax, the OpenShift Jenkins image provides the OpenShift Domain Specific Language (DSL) (through the OpenShift Jenkins Client Plug-in), which aims to provide a readable, concise, comprehensive, and fluent syntax for rich interactions with an OpenShift API server, allowing for even more control over the build, deployment, and promotion of applications on your OpenShift cluster.

The Twistlock Jenkins plugin does not understand the OpenShift Domain Specific Language nor does the OpenShift Jenkins Master and Slave containers mount the docker socket. Therefore, the standard configuration of the Twistlock Jenkins plugin is not applicable to the OpenShift Jenkins. But the OpenShift Jenkins leverages the OpenShift BuildConfig process as part of the Jenkins build process. Twistlock can scan images built via the OpenShift BuildConfig process via the postCommit build hook using the twistcli utility.

The postCommit field of a BuildConfig object executes commands inside a temporary container that is running the build output image. The hook is executed immediately after the last layer of the image has been committed and before the image is pushed to a registry.

The hook fails if the script or command returns a non-zero exit code or if starting the temporary container fails. When the hook fails it marks the build as failed and the image is not pushed to a registry. The reason for failing can be inspected by looking at the build logs.

The twistcli scan of the image during the postCommit phase returns either success (exit 0) or error (exit 1) upon exit. The OpenShift build process will either continue or stop based upon these exit codes. You can configure the twistcli scan to pass/fail builds passed upon the vulnerabilities discovered in the image and configuration compliance.

3. Twistlock Console configuration

The Twistlock scanning application is called twistcli. The twistcli scanner must authenticate to the Twistlock Console’s API to assess vulnerability and compliance data. Since the CI User account is the least privileged account we will pass the credentials as part of the script in this example. The CI User account does not have the rights to authenticate to the Twistlock Console’s user interface. The scanner itself accesses Console’s API via the twistlock-console.twislock.svc OSE internal DNS FQDN. You can use OpenShift secrets to protect the username/password of the CI User account. See Appendix: OpenShift secret for the Twistlock credentials.

Prerequisites

Twistlock Console is fully operational. If not, see the Twistlock OpenShift deployment guide.

Procedure

Create a least privileged CI User account.

  1. Log into Twistlock Console.

  2. Go to Manage > Authentication > Users.

  3. Click Add user.

    1. Username: bob_the_builder

    2. Password: twistlock2019

    3. Role: CI User

If using SAML or Active Directory/OpenLDAP integration this account must be of type Basic

4. Create and configure an OpenShift build

Modify the BuildConfig to scan an image, then build an image and review the results.

Procedure

  1. Log into OpenShift using oc.

  2. Create a new build. This example build creates an image named pfox-ose-build-scan-demo from the registry.access.redhat.com/rhel7 base image.

    $ oc new-build \
      -D $'FROM registry.access.redhat.com/rhel7' \
      --to=pfox-ose-build-scan-demo
  3. In the OpenShift Console, go to Project > Builds > Builds. If you are using a Jenkins Pipeline, go to Project > Builds > Pipelines.

  4. Click on the pfox-ose-build-scan-demo build.

  5. In the Actions drop-down list, select Edit YAML.

  6. In the spec: postCommit node, add the following script, modifying it as necessary for your environment. The >- characters are a block chomping indicator that strips the line feeds from the multi-line script string. It’s one of the ways to write multi-line strings in YAML files. For a line-by-line breakdown of how the script works, see Post commit hook script.

    postCommit:
      script: >-
        curl -k -ssl -u "bob_the_builder:twistlock2019"
        https://twistlock-console.twistlock.svc:8083/api/v1/util/twistcli -o twistcli &&
        chmod +x ./twistcli &&
        ./twistcli images scan
        --containerized
        --user bob_the_builder
        --password twistlock2019
        --address https://twistlock-console.twistlock.svc:8083
        --vulnerability-threshold high
        --details
        $OPENSHIFT_BUILD_NAME
    openshift build twistcli postCommit
  7. Build a new image.

    $ oc start-build pfox-ose-build-scan-demo
  8. Monitor the build logs.

    $ oc logs -f bc/pfox-ose-build-scan-demo

Results

Since twistcli is configured with --vulnerability-threshold high and --details, the output contains detailed information for vulnerabilities in the image.

openshift build twistcli scan1

The pass/fail status of the twistcli scan is printed at the end of the scan.

openshift build twistcli scan2

If you want less verbosity from the scanner, remove the --details option from the postCommit script. To push the image to registry regardless of the scanner’s findings, don’t set any thresholds by removing the --vulnerability-threshold high option.

postCommit:
  script: >-
    curl -k -ssl -u "bob_the_builder:twistlock2019"
    https://twistlock-console.twistlock.svc:8083/api/v1/util/twistcli -o twistcli &&
    chmod +x ./twistcli &&
    ./twistcli images scan
    --containerized
    --user bob_the_builder
    --password twistlock2019
    --address https://twistlock-console.twistlock.svc:8083
    --details
    $OPENSHIFT_BUILD_NAME

With the updated postCommit script, the scanner provides just a summary report:

openshift build twistcli scan3

In the OpenShift Console, the build information and twistcli scan output is displayed.

openshift build twistcli ose build

The scan results can be reviewed in Twistlock Console under Monitor > Vulnerabilities > Twistcli Scans.

openshift build twistcli ose twistcli scans

Click on the image to drill down into the detailsClick on the image to drill down into the details.

openshift build twistcli ose twistcli scans1

5. Post commit hook script

The postCommit script runs the Twistlock scanner on the image just built. This section describes how the script works.

postCommit:
  script: >-
    curl -k -ssl -u "bob_the_builder:twistlock2019" (1)
    https://twistlock-console.twistlock.svc:8083/api/v1/util/twistcli -o twistcli &&
    chmod +x ./twistcli && (2)
    ./twistcli images scan (3)
    --containerized
    --user bob_the_builder
    --password twistlock2019
    --address https://twistlock-console.twistlock.svc:8083
    --vulnerability-threshold high
    --details
    $OPENSHIFT_BUILD_NAME (4)
1 Pulls the twistcli binary from Twistlock Console API. This guarantees that the Twistlock Console and twistcli versions are synchronized. Note that if the image being built does not contain curl, then add the twistcli application to the image itself.
2 Makes the twistcli binary executable.
3 Scans the image within the running container. The postCommit field of a BuildConfig object executes commands inside a temporary container that is running the build output image.
4 Name of the image being scanned based upon the build’s environment variable.

The following options control how the scan runs: See twistcli images scan --help for additional flags and details.

  • --containerized — Run the scan from within a container.

  • --vulnerability-threshold high — Minimum vulnerability threshold for failing the build on vulnerability checks.

  • --details — Show all vulnerability details.

Twistcli returns an exit code of 1 if there are any high severity vulnerabilities in the image. An exit code of 1 notifies the OSE start-build process that the postCommit task has failed and that the process should stop before the image is pushed to the registry.

6. Appendix: OpenShift secret for the Twistlock credentials

Create an OpenShift generic secret to protect your CI User credentials. These credentials are presented as environment variables to the script run in the postCommit stage. More information about providing credentials to a BuildConfig, see OpenShift’s docs on input secrets.

The OpenShift build process will create environment variables containing the Twistlock CI User account’s username:password in the resulting image.

Procedure

  1. Log into OpenShift using oc, and go to the project where you run builds.

  2. Create files for the username and password.

    $ echo -n 'bob_the_builder' > ./username.txt
    $ echo -n 'twistlock2019' > ./password.txt
  3. Create an OpenShift generic secret with the username and password.

    $ oc create secret generic twistlock-scan \
      --from-file=username=./username.txt \
      --from-file=password=./password.txt
  4. Grant the builder service account access to the secret.

    $ oc secrets link builder twistlock-scan
  5. Create a new build. This example build creates an image named pfox-ose-build-scan-demo from the registry.access.redhat.com/rhel7 base image. Associate the twistlock-scan secret with the build.

    $ oc new-build \
      -D $'FROM registry.access.redhat.com/rhel7' \
      --to=pfox-ose-build-scan-demo \
      --build-secret twistlock-scan
  6. In the OpenShift Console, go to Project > Builds > Builds. If you are using a Jenkins Pipeline, go to Project > Builds > Pipelines.

  7. Click on the pfox-ose-build-scan-demo build.

  8. In the Actions drop-down list, select Edit YAML.

  9. Add in the postCommit script as instructed above.

  10. Modify the spec:source and spec:strategy nodes accordingly.

        source:
          dockerfile: FROM registry.access.redhat.com/dotnet-beta/dotnet-20-rhel7
          secrets:
            - secret:
                name: twistlock-scan
          sourceSecret:
            name: twistlock-scan
          type: Dockerfile
        strategy:
          dockerStrategy:
            env:
              - name: twistlock_scan_username
                valueFrom:
                  secretKeyRef:
                    key: username
                    name: twistlock-scan
              - name: twistlock_scan_password
                valueFrom:
                  secretKeyRef:
                    key: password
                    name: twistlock-scan
    openshift build twistcli secrets