This is the manual for tlp-cluster, a provisioning tool for Apache Cassandra designed for developers looking to benchmark and test Apache Cassandra. It assists with builds and starting instances on AWS.
If you are looking for a tool to aid in benchmarking these clusters please see the companion project tlp-stress.
An AWS access key and secret. tlp-cluster uses Terraform to create and destroy instances. You will be prompted for these the first time you start tlp-cluster.
The access key needs permissions to create an S3 bucket as well as create SSH keys. Separate keys are used by default for security reasons.
The easiest way to get started is to use one of our prebuilt packages.
Installing a Package
The easiest way to get started is to use your favorite package manager.
The current version is 0.5.
$ echo "deb https://dl.bintray.com/thelastpickle/tlp-tools-deb weezy main" | sudo tee -a /etc/apt/sources.list $ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 2895100917357435 $ sudo apt update $ sudo apt install tlp-cluster
You’ll need the bintray repo set up on your machine. Create this
[bintraybintray-thelastpickle-tlp-tools-rpm] name=bintray-thelastpickle-tlp-tools-rpm baseurl=https://dl.bintray.com/thelastpickle/tlp-tools-rpm gpgcheck=0 repo_gpgcheck=0 enabled=1
Then run the following to install:
$ yum install tlp-cluster
If you’re using mac, for now you’ll need to grab our tarball using:
$ curl -L -O "https://dl.bintray.com/thelastpickle/tlp-tools-tarball/tlp-cluster-0.5.tar" $ tar -xzf tlp-cluster-0.5.tar
To get started, add the bin directory of tlp-cluster to your $PATH. For example:
export PATH="$PATH:/path/to/tlp-cluster/bin" cd /path/to/tlp-cluster ./gradlew assemble
We currently have a dependency on shell scripts meaning you’ll need a Mac or Linux box to use this tool.
If you’ve never used to the tool before, the first time you run a command you’ll be asked to supply some information, which will generate a configuration file which will be placed in your
|We currently only support the ubuntu 16 ami in us-west-2. We’d love a pull request to improve this!|
Running the command without any arguments will print out the usage:
You’ll see the help. It looks like this:
Usage: tlp-cluster [options] [command] [command options] Options: --help, -h Shows this help. Default: false Commands: init Initialize this directory for tlp-cluster Usage: init [options] Client, Ticket, Purpose Options: --ami AMI Default: ami-51537029 --cassandra, -c Number of Cassandra instances Default: 3 --instance Instance Type Default: c5d.2xlarge --monitoring, -m Enable monitoring (beta) Default: false --region Region Default: us-west-2 --stress, -s Number of stress instances Default: 0 --up Start instances automatically Default: false up Starts instances Usage: up [options] Options: --auto-approve, -a, --yes Auto approve changes Default: false start Start cassandra on all nodes via service command Usage: start [options] Options: --all, -a Start all services on all instances. This overrides all other options Default: false --monitoring, -m Start services on monitoring instances Default: false stop Stop cassandra on all nodes via service command Usage: stop [options] Options: --all, -a Start all services on all instances. This overrides all other options Default: false --monitoring, -m Start services on monitoring instances Default: false install Install Everything Usage: install down Shut down a cluster Usage: down [options] Options: --auto-approve, -a, --yes Auto approve changes Default: false build Create a custom named Cassandra build from a working directory. Usage: build [options] Path to build Options: -n Name of build ls List available builds Usage: ls use Use a Cassandra build Usage: use [options] Options: --config, -c Configuration settings to change in the cassandra.yaml file specified in the format key:value,... Default:  clean null Usage: clean hosts null Usage: hosts Done
Initialize a Cluster
The tool uses the current working directory as a project, of sorts. To get started, run the following, substituting your customer / project, ticket and purpose.
tlp-cluster init CLIENT TICKET PURPOSE
CLIENT- Name of the customer, client, or project associated with the work you are doing with
TICKET- Jira or github ticket number associated with the work you are doing with
PURPOSE- Reason why you are creating the cluster.
This will initialize the current directory with a terraform.tf.json. You can open this up in an editor. Here you can change the number of nodes in the cluster, as well as configure the number of stress nodes you want. You can also change the instance type. Generally speaking though, you shouldn’t have to do this. If you find yourself doing it often, please submit an issue describing your requirements and we’ll work with you to solve the problem.
Certain instances types may not work with the ami that’s hard coded at the moment, we’re looking to fix / improve this.
Launch your instances with the following:
Terraform will eventually ask you to type
yes and fire up your instances. Optionally you can pass
--yes to the
-up command and you won’t be prompted.
tlp-cluster will create a file,
env.sh, which has helpful aliases and bash functions that will help you run your cluster. Run the following:
This will set up SSH, SCP, SFTP, and rsync to use a local sshConfig file, as well as some other helpful aliases.
SSH alises for all Cassandra nodes are automatically created as
ssh is not required. For example:
c0 nodetool status
In addition, the following are defined:
c-all executes a command on every node in the cluster sequentially.
c-collect-artifacts will collect metrics, nodetool output and system information into the artifacts directory. It takes a name as a parameter. This is useful when doing performance testing to capture the state of the system at a given moment.
c-start Starts cassandra on all nodes.
c-restart Restarts cassandra on all nodes. Not a graceful operation. To test true rolling restarts we recommend using cstar.
nodetool statuson cassandra0.
nodetool tpstatson all nodes.
The Easy Way - Use a Released Build
The easiest path forward to getting a cluster up and running is the following:
tlp-cluster use 3.11.4 tlp-cluster install tlp-cluster start
Simply replace 3.11.4 with the release version.
The Hard Way - Use a custom Build
To install Cassandra on your instances, you will need to follow these steps:
Build the version you need and give it a build name (optional)
Tell tlp-cluster to use the custom build
The first step is optional because you may already have a build in the
~/.tlp-cluster/build directory that you want to use.
If you have no builds you will need to run the following:
tlp-cluster build -n BUILD_NAME /path/to/repo
BUILD_NAME- Name you want to give the build e.g. my-build-cass-4.0.
/path/to/repo- Full path to clone of the Cassandra repository.
If you already have a build that you would like to use you can run the following:
tlp-cluster use BUILD_NAME
This will copy the binaries and configuration files to the
provisioning/cassandra directory in your
tlp-cluster repository. The
provisioning directory contains a number of files that can be used to set up your instances. Being realistic, since we do so much non-standard work (EBS vs instance store, LVM vs FS directly on a device, caches, etc) we need the ability to run arbitrary commands. This isn’t a great use case for puppet / chef / salt / ansible (yet), so we are just using easy to modify scripts for now.
If you want to install other binaries or perform other operations during provisioning of the instances, you can add them to the
provisioning/cassandra directory. Note that any new scripts you add should be prefixed with a number which is used to determine the order they are executed by the
To provision the instances run the following:
SSH_KEY_PATH- Is the full path to the private key from the key pair used when creating the instances.
This will push the contents of the
provisioning/cassandra directory up to each of the instances you have created and install Cassandra on them.
|This documentation is very rough, and needs improvment before the 1.0 release.|
The dashboards are built using a tool called jsonnet, with a library called grafonnet. The jsonnet files are here:
if you want to do anything in the dashboards, you’d need to run
You can regenerate the dashboards using the following:
Optionally you can add the
-t flag and Gradle will watch for changes, and rebuild the dashboards when it detects changes.
They get output here:
There’s a docker-compose file,
docker-compose-monitoring-dev.yml, which has all the necessary configuration to start a full environment designed to make editing dashboards easier. You can start it with:
This will start Cassandra, tlp-stress, Prometheus and Grafana.
The following ports are open:
Grafana web interface
Prometheus Web Interface
Cassandra Native Protocol (cql)
tlp-stress prometheus port
Cassandra prometheus port
the normal ports are all mapped for you so you can reach prometheus on 9090 and grafana on 3000