...
DAS indexer is implemented using Apache Lucene which is a full text search library. Users can index records and search for records later via Lucene queries. Events received by DAS are converted to a list of records and inserted into FileSystem based queues. These queues are created in <DAS_HOME>/repository/data/index_staging_queues directory. With a background thread, these queues are consumed and records are indexed. The indexed data is stored in the <DAS_HOME>/repository/data/index_data directory. The DAS index consists of smaller indexes known as shards. A shard can be accessed nly by one index writer at a given time. (Index writer is the lucene class visible to outside world which is used to write lucene documents to a file system based index).Therefore having multiple shards can increase the write throughput (however, the write throughput can be limited by Disk IO operations). By default, DAS is configured to have six shards and one replica.
...
By default, a single DAS server is configured to have six shards, and one replica for each shard (i.e., a total of twelve shards). Therefore, in a minimum HA setup, each DAS server contains three shards and replicas of the other three shards. This is communicated between the nodes via Hazlecast messages. Even Even if one server is shut down, the second server has replicas of the three shards that were there in the node that was shut down. This allows the high availability to be maintained.
The number of replicas and the number of shards are defined in the analytics-config.xml. By changing these properties we can change the number of shards and replicas, but once they are defined and the server is started, we cannot change them. Events are not needed to be published to both nodes. Replication happens through hazelcast messages. records are sent to other nodes via hazelcast.
The following example further explains how shards are managed in a clustered deployment.
...