This is the manual for tlp-cluster, a provisioning tool for Apache Cassandra designed for developers looking to benchmark and test Apache Cassandra. It assists with builds and starting instances on AWS.

If you are looking for a tool to aid in benchmarking these clusters please see the companion project tlp-stress.

If you’re looking for tools to help manage Cassandra in production environments please see the Reaper project and cstar

Prerequisites

There are some AWS resources tlp-cluster expects to exist in us-west-2 (support for other regions will come soon):

  • An AWS access key and secret. tlp-cluster uses Terraform to create and destroy instances.

  • A security group that allows inbound SSH traffic, and all machines in that group can talk to each other. This requirement will be lifted soon (see Issue #42).

Installation

The cluster provisioning tool is somewhat self explanatory (hopefully).

It launches Cassandra clusters in AWS. In addition to Cassandra, the tool can also launch stress nodes and a prometheus node to gather all the metrics.

The tool itself is written in Kotlin, leveraging Docker containers to avoid too many external dependencies.

To get started, add the bin directory of tlp-cluster to your $PATH. For example:

export PATH="$PATH:/path/to/tlp-cluster/bin"
cd /path/to/tlp-cluster
./gradlew assemble

Setup

We currently have a dependency on shell scripts meaning you’ll need a Mac or Linux box to use this tool.

If you’ve never used to the tool before, the first time you run a command you’ll be asked to supply some information, which will generate a configuration file which will be placed in your $HOME/.tlp-cluster/profiles/default/settings.yaml.

We currently only support the ubuntu 16 ami in us-west-2. We’d love a pull request to improve this!

Running the command without any arguments will print out the usage:

tlp-cluster

You’ll see the help. It looks like this:

Usage: tlp-cluster [options] [command] [command options]
  Options:
    --help, -h
      Shows this help.
      Default: false
  Commands:
    init      Initialize this directory for tlp-cluster
      Usage: init [options] Client, Ticket, Purpose
        Options:
          --ami
            AMI
            Default: ami-51537029
          --cassandra, -c
            Number of Cassandra instances
            Default: 3
          --instance
            Instance Type
            Default: c5d.2xlarge
          --monitoring, -m
            Enable monitoring (beta)
            Default: false
          --region
            Region
            Default: us-west-2
          --stress, -s
            Number of stress instances
            Default: 0
          --up
            Start instances automatically
            Default: false

    up      Starts instances
      Usage: up [options]
        Options:
          --auto-approve, -a, --yes
            Auto approve changes
            Default: false

    start      Start cassandra on all nodes via service command
      Usage: start [options]
        Options:
          --all, -a
            Start all services on all instances. This overrides all other
            options
            Default: false
          --monitoring, -m
            Start services on monitoring instances
            Default: false

    stop      Stop cassandra on all nodes via service command
      Usage: stop [options]
        Options:
          --all, -a
            Start all services on all instances. This overrides all other
            options
            Default: false
          --monitoring, -m
            Start services on monitoring instances
            Default: false

    install      Install Everything
      Usage: install

    down      Shut down a cluster
      Usage: down [options]
        Options:
          --auto-approve, -a, --yes
            Auto approve changes
            Default: false

    build      Create a custom named Cassandra build from a working directory.
      Usage: build [options] Path to build
        Options:
          -n
            Name of build

    ls      List available builds
      Usage: ls

    use      Use a Cassandra build
      Usage: use [options]
        Options:
          --config, -c
            Configuration settings to change in the cassandra.yaml file
            specified in the format key:value,...
            Default: []

    clean      null
      Usage: clean

    hosts      null
      Usage: hosts


Done

Initialize a Cluster

The tool uses the current working directory as a project, of sorts. To get started, run the following, substituting your customer / project, ticket and purpose.

tlp-cluster init CLIENT TICKET PURPOSE

Where:

  • CLIENT - Name of the customer, client, or project associated with the work you are doing with tlp-cluster.

  • TICKET - Jira or github ticket number associated with the work you are doing with tlp-cluster.

  • PURPOSE - Reason why you are creating the cluster.

This will initialize the current directory with a terraform.tf.json. You can open this up in an editor. Here you can change the number of nodes in the cluster, as well as configure the number of stress nodes you want. You can also change the instance type. Generally speaking though, you shouldn’t have to do this. If you find yourself doing it often, please submit an issue describing your requirements and we’ll work with you to solve the problem.

Certain instances types may not work with the ami that’s hard coded at the moment, we’re looking to fix / improve this.

Launch Instances

Launch your instances with the following:

tlp-cluster up

Terraform will eventually ask you to type yes and fire up your instances. Optionally you can pass --yes to the -up command and you won’t be prompted.

Installing Cassandra

The Easy Way - Use a Released Build

The easiest path forward to getting a cluster up and running is the following:

tlp-cluster use 3.11.4
tlp-cluster install
tlp-cluster start

Simply replace 3.11.4 with the release version.

The Hard Way - Use a custom Build

To install Cassandra on your instances, you will need to follow these steps:

  1. Build the version you need and give it a build name (optional)

  2. Tell tlp-cluster to use the custom build

The first step is optional because you may already have a build in the ~/.tlp-cluster/build directory that you want to use.

If you have no builds you will need to run the following:

tlp-cluster build -n BUILD_NAME /path/to/repo

Where:

  • BUILD_NAME - Name you want to give the build e.g. my-build-cass-4.0.

  • /path/to/repo - Full path to clone of the Cassandra repository.

If you already have a build that you would like to use you can run the following:

tlp-cluster use BUILD_NAME

This will copy the binaries and configuration files to the provisioning/cassandra directory in your tlp-cluster repository. The provisioning directory contains a number of files that can be used to set up your instances. Being realistic, since we do so much non-standard work (EBS vs instance store, LVM vs FS directly on a device, caches, etc) we need the ability to run arbitrary commands. This isn’t a great use case for puppet / chef / salt / ansible (yet), so we are just using easy to modify scripts for now.

If you want to install other binaries or perform other operations during provisioning of the instances, you can add them to the provisioning/cassandra directory. Note that any new scripts you add should be prefixed with a number which is used to determine the order they are executed by the install.sh script.

To provision the instances run the following:

tlp-cluster install

Where:

  • SSH_KEY_PATH - Is the full path to the private key from the key pair used when creating the instances.

This will push the contents of the provisioning/cassandra directory up to each of the instances you have created and install Cassandra on them.