1. Overview

Twistlock High Availability is designed specifically for customers running Console outside of a cluster manager. If you’re running Console in a cluster, such as Kubernetes, Twistlock HA should not be used. Instead, utilize the native availability features of your cluster manager.

When you deploy Console on multiple hosts, Twistlock HA ensures that one Console is always available, even if a host fails. Twistlock HA creates a high availability clusters from redundant hosts running Console, and automatically manages cluster membership, leader election, and failover.

Twistlock HA offers the following features:

  • Automatic database replication and failover.

  • Automatic Console failover.

Defender can continue to operate autonomously, even if communication with Console is temporarily severed. Therefore, it might not be necessary to deploy Twistlock HA in your environment.
Keep in mind that HA and data backup are two separate issues. Even if you run Twistlock in HA mode, you should still routinely backup and archive Twistlock to some place outside of your production environment. Twistlock supports disaster recovery and a method to quickly restore Console from a backup. By default, backups are taken every 24 hours, and can be written to durable storage, such as S3 buckets exposed as volumes. To learn more about backups, see Disaster recovery.
Twistlock HA is not compatible with Projects. If you want to enable Projects, consider using a cluster manager rather than High Availability to oversee the uptime of the Console container.

2. Architecture

The Twistlock HA cluster is set up in an active/passive configuration, where each host is a fully redundant node. Only one node at a time, the primary node, provides services. Redundant (secondary) nodes are only brought online when the primary node fails.

The following diagram shows a setup with three Console nodes organized into an HA cluster. Console nodes run behind a load balancer. At the top, there are three nodes running Defender. Defenders connect to the load balancer to access Console services.

tw ha 706614

Each Console has a local instance of the database that runs inside the Console container. Under the covers, we are using standard MongoDB clustering technology to manage data replication and leader election. When a node is primary, Console opens all Twistlock ports (management port, web socket) to provide services. All other nodes close their Twistlock ports and appear offline. In the background, secondary nodes periodically query the cluster to sync data and verify state. If the primary node fails, the remaining members elect a new primary, and Console on the new primary opens all Twistlock ports to offer continued access to Console services.

2.1. Load balancer

Before you deploy Twistlock HA, you must set up a load balancer. Numerous options are possible, including Amazon Elastic Load Balancer (ELB), nginx, and Round Robin DNS. If you deploy a load balancer, such as ELB and nginx, it will act as a reverse proxy that routes traffic to the primary node. Configure your load balancer to periodically health check the nodes in the cluster with Twistlock’s ping API:

$ curl -u admin:Password https://console-ip:8083/api/v1/_ping

Most load balancing setups distribute traffic to all nodes. With Twistlock HA, however, the two secondary nodes appear offline, and the load balancer sends all traffic to the primary node. In this setup, the logic for routing traffic resides in the load balancer.

An alternative setup is Round Robin DNS. Round Robin DNS responds to DNS requests with a list of IP addresses for all nodes in the cluster. Defender (or your browser, if you are accessing the Console UI) tries all IP addresses until it successfully connects to the primary node. In this case, the logic for routing traffic to the primary node resides in the client.

3. Setting up Twistlock HA

This procedure shows you how to set up an HA cluster with three nodes. Adding cluster nodes is a simple process that can be done entirely from the Console UI.

Don’t use Twistlock HA if you’re running Console inside a cluster, such as Kubernetes, Docker Swarm, or Amazon ECS. In such cases, use the native capabilities of the cluster manager to ensure Console is always available.
Although it is possible to design a custom service that runs multiple instances of Console in an active/active configuration, it is not tested or supported configuration.

Prerequisites

  • Twistlock only supports three nodes in an HA cluster setup. One primary Console and two secondary Consoles.

  • Port 27017 must be available on all cluster nodes. This is a MongoDB requirement.

  • All nodes must be on the same local, non-routed network.

  • You have set up a load balancer.

Procedure

  1. Install Primary Console

    1. Download the latest Twistlock release, and unpack the tarball.

    2. Open twistlock.cfg for editing.

    3. In the section for High availability settings, set HIGH_AVAILABILITY_ENABLED to true.

      #############################################
      #      High availability settings
      #############################################
      HIGH_AVAILABILITY_ENABLED=true
      HIGH_AVAILABILITY_STATE=PRIMARY
      HIGH_AVAILABILITY_PORT=8086
    4. Install Twistlock Console.

  2. Install a secondary Console

    1. Open the primary Console. In a browser, go to <CONSOLE_PRIMARY_NODE>:8443, and login as admin.

    2. Go to Manage > System > High Availability.

      You should see your current host set as the primary Console.

      You cannot see the High Availability tab in the Console UI unless you have enabled HIGH_AVAILABILITY_ENABLED and installed Console with your updated twistlock.cfg file.
    3. Copy the script to install a secondary Console.

    4. On a different host, designated as secondary, run the script you just copied.

      Console is installed on your secondary host.

      This is the only step required to install Console on a secondary node. This script installs Console on the host, and joins the cluster. Do not run the procedure documented in the Install Twistlock article on any secondary cluster nodes.

      In your primary Console, you will the secondary host waiting to join the cluster in the REQUEST_JOIN state.

    5. Follow the steps a-d to install another secondary Console.

      You now have a total of three Consoles. One primary Console and two secondary Consoles.

    6. When you see the REQUEST_JOIN, state for both secondary Consoles, click APPLY. After a few moments, the state changes to SECONDARY.

      Do not click APPLY until you see both secondary Consoles appear in the primary Console with the REQUEST_JOIN state.
  3. Test the cluster by taking the primary Console offline.

    1. Shut down the host running the primary Console

    2. Open Console in a browser.

      The secondary console will not become active until the primary console is disabled / disconnected. You will not be able to log in to other consoles in HA cluster as long as your primary console is up and running.

      Your load balancer should detect the failure and reroute traffic to the new primary in the cluster. The Console UI should open in your browser window:

    3. Go to Configure > SYSTEM > HIGH AVAILABILITY.

      You will see that the state of the nodes has changed. The previous primary node is now marked as unavailable, and another node has been promoted to primary.

4. Upgrading an HA cluster

To upgrade an HA cluster, first remove all secondary nodes. Then upgrade the primary node, and redeploy Twistlock to the secondary nodes. For more information, see Upgrade Twistlock