Use FireCamp to Deploy a Cassandra Container Cluster on AWS

Cassandra+AWS.png

Introduction

Apache Cassandra (https://cassandra.apache.org) is one of the most popular open source NoSQL data storage system that are used globally. It is a peer-to-peer model distributed system, which has been adopted by many high performance cloud scale systems for data services, with the key benefit of high availability with no single point of failure.

    CloudStax FireCamp platform (https://github.com/cloudstax/firecamp) aims to help stateful services like Cassandra to be containerized easily and simplify the operational management overhead of Cassandra cluster, including quick deployment, elastic scaling, smooth upgrade and automated failover.

    This blog provides an instruction on how AWS users could deploy a Cassandra container cluster with FireCamp platform. There are three phases to make this deployment:

    • Phase 1: Prepare AWS VPC environment;
    • Phase 2: Deploy FireCamp cluster on AWS;
    • Phase 3: Use FireCamp CLI to create and manage a multi-node Cassandra service;

    Suggestion: The following Amazon AWS cloud services are used for the deployment, we suggest you get familiar with these services before reading: Amazon Elastic Compute Cloud (EC2),  Elastic Container Service (ECS),  Virtual Private Cloud (VPC),  Route 53,  Elastic Block Store (EBS),  AWS Auto Scaling,  AWS CloudFormation.

    Phase 1: Preparing AWS VPC environment

    We need to have a Amazon Virtual Private Cloud (Amazon VPC) planned as an isolated and clean environment to run Cassandra services and applications. The basic Cassandra topology concepts need to match into AWS scopes as follows:

    • Cassandra data center -> AWS Region
    • Cassandra rack -> AWS Availability Zone
    • Cassandra node -> AWS container instance that run Cassandra docker image

    The following figure illustrates the VPC network topology we are going to create.

    awsvpc.png

    Highlights for above VPC environment:

    • The VPC utilizes three Availability Zones (AZ) from one AWS region. This means we will create three "racks" to run Cassandra nodes for high availability. In case one zone is down entirely, the overall service is still functioning as the failing node in one zone can be automatically failover to another healthy zone.
    • For the best security practices, each AZ is configured with one private subnet for Cassandra nodes. This ensures Cassandra nodes do not expose direct access from Internet.
    • Each AZ is configured with one NAT gateway, to allow Cassandra nodes access Internet for pulling docker images from public repository.
    • An VPC endpoint is created to allow applications/services access other Amazon services without leaving AWS network. In our case, we need Amazon Simple Storage Service (Amazon S3) can be accessed through VPC endpoint.
    • A bastion host is deployed in the public subnet to allow SSH login from Internet. This host is used as FireCamp administrative node, where we can run FireCamp CLI commands to manage the FireCamp cluster.

    Step by Step: Creating VPC from CloudFormation Template

    Amazon provides a scalable VPC Quickstart (https://aws.amazon.com/quickstart/architecture/vpc/) which defines a pre-cooked VPC CloudFormation template to create the VPC environment we need. We will create the VPC with this pre-cooked CloudFormation template file.

    1. Login to your AWS account and go to AWS management console.

    2. Select the region where you want to deploy the VPC from top bar. Also make sure a key pair is created or imported for this region which is necessary to access EC2 hosts.

    3. Select “CloudFormation” to open up CloudFormation service page.

    4. Click “Create new stack” button to start deploying a new stack.

    5. In the Select Template page, select “Specify an Amazon S3 template URL” in the Choose a template area, and paste the following URL, and then click “Next” button: http://cloudstax.s3.amazonaws.com/aws/vpc/latest/templates/aws-vpc.template.

    6. In the Specify Details page, provide proper input for the parameters, and then click “Next” button. Some notes:

      • Define a meaningful stack name, e.g. “my-cassandra-vpc”.

      • Select all three Availability Zones (AZ) in the region, and the “Number of Availability Zones” value must be 3 to match the selected zone number.

      • Create private subnets” set to "true".

      • Based on the AZ number, make sure the CIDR of the public subnets and private subnets are properly defined. You can simply use the default value if no special concerns on the IP counts for each subnet.

      • Create additional private subnets with dedicated network ACLs” set to "false".

      • In NAT Instance section, Keypair and NAT instance type settings are only if the selected region does not support NAT gateway;

    7. In the Options page, leave all the default settings if there is no special concern, and then click “Next” button.

    8. In the Reviews page, double check every setting for the VPC. If all set, click “Create” button. This triggers the VPC creation process, and it takes about 5 minutes to get VPC created.

    Step by Step: Creating Bastion hosts in VPC for FireCamp administration

    Amazon provides a Linux Bastion Quickstart (https://aws.amazon.com/quickstart/architecture/linux-bastion/) which defines a pre-cooked linux bastion CloudFormation template to provision the bastion host in our created VPC.

    1. Login to your AWS account and go to management console.

    2. Select the region from the top bar on where you deploy the VPC.

    3. Select “CloudFormation” to open up CloudFormation service page.

    4. Click “Create new stack” button to start deploying a new stack.

    5. In the Select Template page, select “Specify an Amazon S3 template URL” in the Choose a template area, and paste the following URL, and then click “Next” button: http://cloudstax.s3.amazonaws.com/linux/bastion/latest/templates/linux-bastion.template.

    6. In the Specify Details page, provide proper input for the parameters, and then click “Next” button. Some notes:

      • Give a meaningful stack name, e.g. “bastion-host”.

      • In VPC ID field, select the VPC we created previously.

      • In Public Subnet 1/2 ID fields, select the public subnets created within the VPC.

      • In Allowed Bastion External Access CIDR field, provide the SSH client host IP address range that will login to the Bastion host. This should be the laptop/PC that you will use to access AWS network.

      • For the rest parameters, leave for the default values if no special concern.

    7. In the Options page, leave all the default settings if there is no special concern, and then click “Next” button.

    8. In the Reviews page, double check every setting. If all set, click “Create” button. This triggers the bastion host creation process, and it takes about 5 minutes to finish.

    By now, we have two stacks (VPC and Bastion) created in the selected AWS region.

    屏幕快照 2018-05-13 上午1.19.37.png

    Phase 2: Deploying FireCamp cluster into the VPC

    With the VPC environment ready, we can deploy FireCamp cluster now. FireCamp cluster comprises a configurable number of AWS EC2 hosts that can be used to run docker instances. FireCamp Management Service is a separate docker instance deployed on one EC2 host from FireCamp managed EC2 hosts.

    The following figure illustrates the FireCamp cluster to be deployed on top of the VPC created previously.

    firecamp-cluster.png

    We created a FireCamp CloudFormation template to facilitate the deployment of the EC2 hosts and FireCamp management service. There are also some advanced features designed in FireCamp CloudFormation template, to make sure we get a stable, scalable, secure cluster to further deploy Cassandra service.

    Auto-Scaling and Failover

    FireCamp template enables Amazon Auto Scaling to support Cassandra docker instance auto-scaling and failover. To enable this, the selected AWS region needs to support Auto Scaling capability.

    Security

    FireCamp template enables multiple security groups for different tier access permissions, as show in below table.

    Security Group Name Purpose
    ServiceSecurityGroup The security group that all Cassandra hosts are added to, the group only allows the following access paths:
    1. Hosts in BastionSecurityGroup to SSH to the cluster hosts;
    2. Hosts in BastionSecurityGroup to access FireCamp management service via FireCamp CLI command;
    3. Real application hosts in AppSecurityGroup to access Cassandra service through Cassandra API;
    InternalAccessSecurityGroup The security group that all Cassandra hosts are added to, for Internal communication between nodes within the Cassandra cluster.
    AppAccessSecurityGroup The security group that real application hosts need to be added to, in order to get access to Cassandra API.
    BastionSecurityGroup The security group for hosts who need to do FireCamp administration.
    Note:The BastionSecurityGroup has been created along side the Bastion host provision. Bastion host is added to this group already.

    Step by Step: Deploying FireCamp cluster from CloudFormation Template

    We use the pre-cooked FireCamp CloudFormation template to deploy FireCamp cluster.

    1. Login to your AWS account and go to management console.

    2. Select the region from the top bar on where you deploy the VPC.

    3. Select CloudFormation” to open up CloudFormation service page.

    4. Click “Create new stack” button to start deploying a new stack.

    5. In the Select Template page, select “Specify an Amazon S3 template URL” in the Choose a template area, and paste the following URL, and then click “Next” button: http://cloudstax.s3.amazonaws.com/firecamp/releases/latest/templates/firecamp.template.

    6. In the Specify Details page, provide proper input for the parameters, and then click “Next” button. Notes:

      • Define a meaningful stack name, e.g. “my-firecamp”.

      • Select all three availability zones in the region, and the “Number of Availability Zones” value must be "3" to match the selected zone number.

      • In VPC ID field, select the VPC we created previously.

      • In Private Subnet 1/2/3 fields, select private subnet IDs that are created within the VPC.

      • In Bastion Security Group field, select the Bastion Security group ID created previously.

      • In Container Platform field, select “ecs” as we will use Amazon ECS as the container cluster management framework.

      • In Cluster Name field, define a name for the FireCamp cluster to be created, e.g. “casdb”. The name will be used later in the FireCamp CLI.

      • In CloudStax FireCamp Release Version field, select “latest” version.

      • In Number of Container Worker Nodes field, provide the number of EC2 nodes to be provisioned for the Cassandra cluster, we will use “3”, to have one EC2 node in each availability zone.

      • In Node Instance Type field, select the proper node tier that meet the performance needs to run Cassandra service. Here we start with “t2.xlarge”.

    7. In the Options page, leave all the default settings if there is no special concern, and then click “Next” button.

    8. In the Reviews page, double check every setting for FireCamp cluster setup. If all set, click “Create” button. This triggers the FireCamp cluster creation process, and be a little patient as it takes about 30 minutes to finish.

    Now, we have a running FireCamp cluster, and we can move the next phase to deploy Cassandra service.

    屏幕快照 2018-05-13 上午2.27.08.png

    Phase 3: Using FireCamp CLI to manage Cassandra service

    FireCamp cluster is deployed, and it exposes an CLI interface to serve all the management works. We use FireCamp CLI to create and manage Cassandra service.

    Get FireCamp CLI

    To use FireCamp CLI, we need to retrieve FireCamp CLI executable file onto Bastion host.

    • Find out the Bastion host IP address from AWS management console, and SSH login to Bastion host using your PC.
    # ssh -i <keypair.pem> ec2-user@<bastion_host_ip>
    • Download the FireCamp CLI tgz that matches your FireCamp cluster version deployed (in our case, this is "latest" version) and extract the file.
    # wget https://s3.amazonaws.com/cloudstax/firecamp/releases/latest/packages/firecamp-service-cli.tgz
    # tar zxf ./firecamp-service-cli.tgz

    Create Cassandra Service

    Let's now create a three node Cassandra service within the created FireCamp cluster. The following figure illustrates the Cassandra container services to be created in the topology.

    firecamp-cluster-cassandra.png

    To create new Cassandra service, use the FireCamp CLI command as follows:

    # ./firecamp-service-cli  -region=ap-southeast-1 -cluster=casdb -op=create-service -service-type=cassandra -service-name=cas1 -replicas=3 -volume-size=50 -journal-volume-size=8 -cas-heap-size=4096 -jmx-user=jmxuser -jmx-passwd=changeme

    Common CLI command parameters:

    • -cluster : the FireCamp cluster name defined during FireCamp CloudFormation stack creation. In our example, this is “casdb”.
    • -op: the CLI operation code. The op code “create-service” is for creating a service.
    • -service-type: the service type to be created. In this case, the service type is “cassandra”.
    • -service-name: the Cassandra service name. This gives a unique name of the created service, as FireCamp can manage multiple services in the same cluster.

    Cassandra service specific parameters:

    • -replicas: The replica number of Cassandra instances. In this case, we use replica number 3, one per AZ.
    • -volume-size: The data EBS volume size in GB unit.
    • -journal-volume-size: The journal EBS volume size in GB unit.
    • -cas-heap-size: The JVM heap size setting to run Cassandra service in MB unit.
    • -jmx-user: Cassandra management user name.
    • -jmx-passwd: Cassandra manager user password.

    With this command executed successfully, we get a Cassandra service up running, and AWS ECS manage console shows the container task status as follows.

    屏幕快照 2018-05-14 下午5.37.05.png

    Check Cassandra Service Status

    Three nodes Cassandra service is created, each node is assigned a domain name by FireCamp that can be used for access. The domain name is in format <service-name>-$#.<cluster-name>-firecamp.com. For instance, the service name in our case is "cas1", and the cluster name is "casdb", so the domain names are: cas1-0.casdb-firecamp.com, cas1-1.casdb-firecamp.com, cas1-2.casdb-firecamp.com

    We could add Bastion host to the AppAccessSecurityGroup, and then install Cassandra 3.11 on the Bastion host (refer to Cassandra document on how to install Cassandra on Linux) to use nodetool and cqlsh command. Run the following command to check the cluster status, node information, etc.

    # nodetool -h cas1-0.casdb-firecamp.com -u jmxuser -pw changeme status
    # nodetool -h cas1-1.casdb-firecamp.com -u jmxuser -pw changeme info

    Stop Cassandra Service

    To stop the Cassandra service, use the FireCamp CLI command as follows:

    # ./firecamp-service-cli -region=ap-southeast-1 -cluster=casdb -op=stop-service -service-name=cas1

    Summary

    In this blog, we went through a full process on how to use FireCamp to deploy Cassandra container cluster on AWS. We leverages AWS Scalable VPC Quickstart and Linux Bastion Quickstart to prepare the environment, and then uses FireCamp CloudFormation template to deploy FireCamp cluster. After all these are done in about 30 minutes time, we use the FireCamp CLI to create and manage Cassandra service.

    We also published a Cassandra dedicated Quickstart project, where an all-in-one CloudFormation template is defined to help deploying an Cassandra cluster from scratch to creating VPC, FireCamp cluster and then deploy Cassandra nodes, with just one click of launching the template file. Please also have a try here: https://aws.amazon.com/quickstart/architecture/cloudstax-nosql-db-for-cassandra/.