Standalone Deployment on External Cassandra Server

Cassandra is a highly-scalable, eventually consistent, distributed, structured key-value store which acts as the default, persistence storage for WSO2 Message Broker. The Message Broker can also be configured to connect to an external Cassandra server instead of the in-built Cassandra server, which is not recommended when processing large volumes of queues in a typical production environment.

WSO2 recommends the following on an actual, production deployment:

Using an external server or cluster such as Cassandra or Zookeeper.
Have the external server on a separate machine or a dedicated VM instance and point to it from the Message Broker.

This section provides instructions to install, setup and point an external Cassandra server to a standalone WSO2 Message Broker instance.

Installation Prerequisites

Cassandra requires the most stable version of Java 1.6 for Sun JVM. (u19 or higher is recommended).

Download Cassandra Kit

Download links for the latest, stable Apache Cassandra release can be found on : (http://cassandra.apache.org/download).
Users of Debian or Debian-based derivatives can install the latest stable release in package form. For instructions, refer to http://wiki.apache.org/cassandra/DebianPackaging.
Users of RPM-based distributions can get packages from Datastax: http://www.datastax.com/dev/blog/announcing-rpms-cassandra.
To build Cassandra from source, refer to http://wiki.apache.org/cassandra/HowToBuild.
For more details on miscellaneous builds, refer to 'Cassandra versions and builds' page: http://wiki.apache.org/cassandra/VersionsAndBuilds.

Edit Configuration Files

Cassandra configuration files can be found in $CASSANDRA_HOME/conf directory of binary and source distributions. If you have installed Cassandra from RPM packages, configuration files will be placed into /etc/cassandra/conf. Cassandra configurations can be done in the following files.

Edit cassandra.yaml

The default cassandra.yaml contains reasonable settings for single node operation. There are few settings that need to be verified:

By default, there are paths defined for data_file_directories, commitlog_directory, and saved_caches_directory. Ensure that the paths exist for data_file_directories, commitlog_directory, and saved_caches_directory. If not correct the paths to a valid location on your deployment environment.
Verify storage_port and rpc_port do not conflict with other services on your computer. By default, Cassandra uses 7000 for storage_port, and 9160 for rpc_port. The storage_port must be identical between Cassandra nodes in a cluster. Cassandra client applications use rpc_port to connect to Cassandra.
It is recommended to change the default cluster_name to avoid conflicts with existing clusters.
initial_token can be left blank, but setting to 0 is recommended if you are configuring your first node.
If the external cassandra server is not hosted in the same host which MB runs it is needed to change the rpc_address with the IP address of the host in which the Cassandra server running.

Edit log4j-server.properties

$CARBON_HOME /repository/conf/log4j.properties file contains a path for the log file as required.

# Edit the next line to point to your logs directory
log4j.appender.R.File=/var/log/cassandra/system.log

Edit cassandra-env.sh

By default, Cassandra allocates memory based on the physical memory capacity of your system. For example, it will allocate 1GB heap on 2GB system, and 2GB heap on 8GB system. To specify Cassandra heap size, remove leading pound sign(#) on the following lines and specify memory size for them.

#MAX_HEAP_SIZE="4G"
#HEAP_NEWSIZE="800M"

If you are not familiar with Java GC, 1/4 of MAX_HEAP_SIZE may be a good start point for HEAP_NEWSIZE.

Although Cassandra can be run on smaller footprint for test drive, it needs more Giga Bytes of heap for production. For example, to assign 128 MB as maximum, edit the lines as follows.

MAX_HEAP_SIZE="128M"
HEAP_NEWSIZE="32M"

It is recommend to use at least "2048M" for MAX_HEAP_SIZE. In case of OutOfMemory exceptions (http://wiki.apache.org/cassandra/OutOfMemory) or massive GCs with this configuration, increase these values.

Note

It is not recommended to start production services with small heap configurations.

Start Cassandra

Cassandra can be started by invoking bin/cassandra -f from the command line. The -f option ensures that the service starts in the foreground and logs gratuitously to standard-out. If there are no messages with "errors", or words such as "fatal" or that look like a Java stack traces, then you've succeeded. After starting the Cassandra server, it starts listening for thrift clients as follows.

....
....
....
INFO 12:18:29,140 Listening for thrift clients...

Press "Ctrl+ C" to stop Cassandra. If you start up Cassandra without "-f" option, it runs in the background, which requires killing the process to stop.

Instruct WSO2 MB to Use External Cassandra Server

Edit $CARBON_HOME/repository/conf/advanced/qpid-config.xml file to instruct WSO2 MB server to use external Cassandra Server. For that you need to un-comment the line <externalCassandraServerRequired> and make the value of that entry to true.

<!--Are we running an External Cassandra server ? true|false-->
    <externalCassandraServerRequired>true</externalCassandraServerRequired>

After editing the configuration, it will look like bellow.

<clustering>
    <enabled>false</enabled>       
    <OnceInOrderSupportEnabled>false</OnceInOrderSupportEnabled>
    <!--Are we running an External Cassandra server ? true|false-->
    <externalCassandraServerRequired>true</externalCassandraServerRequired>
    <!--Are we running an External Zookeeper server ? true|false -->
    <!--externalZookeeperServerRequired>false</externalZookeeperServerRequired-->

Point WSO2 MB to Cassandra

Edit $CARBON_HOME/repository/conf/advanced/qpid-virtualhosts.xml file to point to Cassandra server setup. If your cassandra server is hosted in the same host which MB runs, you do not need to make any changes in the connection string since by default it is as "<connectionString>localhost:9160</connectionString>". If your cassandra server running in different host , eg: 10.100.3.234 server, you need to change the connection string accordingly. Eg: "<connectionString>10.100.3.234:9160</connectionString>"

 <virtualhost>
        <name>carbon</name>
        <carbon>
           <store>
                <class>org.wso2.andes.server.store.CassandraMessageStore</class>
                <username>admin</username>
                <password>admin</password>
                <cluster>ClusterOne</cluster>
                <idGenerator>org.wso2.andes.server.cluster.coordination.TimeStampBasedMessageIdGenerator</idGenerator>
                <connectionString>localhost:9160</connectionString>
            </store>
...