Cassandra Tuning

As instructed before, Apache Cassandra should be running in a dedicated server.

Memory allocated for Cassandra is as follows:

In cassandra-env.sh

MAX_HEAP_SIZE="4G"

HEAP_NEWSIZE="800M"

In general HEAP_NEWSIZE parameter should be 1/4 of the MAX_HEAP_SIZE.

Override these to set the amount of memory to allocate to the JVM at start-up. For production use, you may wish to adjust this for your environment. MAX_HEAP_SIZE is the total amount of memory dedicated to the Java heap; HEAP_NEWSIZE refers to the size of the young generation. Both MAX_HEAP_SIZE and HEAP_NEWSIZE should either be set or not (if you set one, you must set the other).

The main trade-off for the young generation is that the larger it is, the longer the GC pause times will be. The shorter it is, the more expensive GC will be (usually). The example HEAP_NEWSIZE assumes a modern 8-core+ machine for decent pause times. If in doubt, and if you do not particularly want to tweak, go with 100 MB per physical CPU core.

In cassandra.yaml we have the following setting for memtable:

flush_largest_memtables_at: 0.4

reduce_cache_sizes_at: 0.85

reduce_cache_capacity_to: 0.6

Consider reducing the flush_largest_memtables_at value to around 0.45 under massive write load. The following configurations are identified as good for large messages. There are some other tuning parameters listed in Cassandra Tuned Up Configurations.

Virtual nodes

VNodes can be configured in the cassandra.yaml file. For more information on virtual nodes, see Virtual nodes in Cassandra 1.2.

For more information on configurations, see Configuring virtual nodes.

Token generation

If you are not planning to use virtual nodes, use generated tokens for each node to achieve a balanced cluster.

For Murmur3Partitioner, the following formula can be used to generate tokens:

python -c 'print [str(((2**64 / number_of_tokens) * i) - 2**63) for i in range(number_of_tokens)]'

Enable Java Native Access (JNA)

Download the JNA jar and copy to $CASSANDRA_HOME/lib or put in to CLASSPATH. JNA enables Cassandra to do native calls efficiently.

Enable G1GC garbage collector for Java7

Cassandra default configuration expects to have the default CMS garbage collector but G1GC with Java7 performs better than CMS. The parameters below will depend on the available resources.

Select the number of threads according to the number of cores in the node.

# GC tuning options
#JVM_OPTS="$JVM
_OPTS -XX:+UseParNewGC"
#JVM_OPTS="$JVM_OPTS -XX:+UseConcMarkSweepGC"
#JVM_OPTS="$JVM_OPTS -XX:+CMSParallelRemarkEnabled"
#JVM_OPTS="$JVM_OPTS -XX:SurvivorRatio=8"
#JVM_OPTS="$JVM_OPTS -XX:MaxTenuringThreshold=1"
#JVM_OPTS="$JVM_OPTS -XX:CMSInitiatingOccupancyFraction=75"
#JVM_OPTS="$JVM_OPTS -XX:+UseCMSInitiatingOccupancyOnly"
#JVM_OPTS="$JVM_OPTS -XX:+UseTLAB"

 #WSO2
JVM_OPTS="-XX:+UseG1GC -XX:G1ReservePercent=30 -XX:ParallelGCThreads=8 -XX:ConcGCThreads=4"

Cassandra node related system configurations

Cassandra needs more system resources than a normal user program. Therefore, upgrade the system limits using limit.conf.

/etc/security/limits.conf

* soft nofile 32768
* hard nofile 32768
root soft nofile 32768
root hard nofile 32768
* soft memlock unlimited
* hard memlock unlimited
root soft memlock unlimited
root hard memlock unlimited
* soft as unlimited
* hard as unlimited
root soft as unlimited
root hard as unlimited

Increase the virtual memory areas:

sysctl -w vm.max_map_count=131072

Disable SWAP

Running Cassandra without swap memory helps to improve performances.

sudo swapoff --all

Synchronize the clock with NTP

Synchronize the OS system clock with NTP in each node.

ntpdate 0.pool.ntp.org

A sample configuration for large messages

Change the memory allocations of the JVM In cassandra-env.sh file to the following values. (Server has 8GB memory)

MAX_HEAP_SIZE="6G"

HEAP_NEWSIZE="1500M"

Adjust the memory table flushing threshold In cassandra.yaml as below:

flush_largest_memtables_at: 0.5

Change the commit log related parameters to overcome the OOM issues observed with larger message sizes.
commitlog_total_space_in_mb: 16
commitlog_segment_size_in_mb: 16