Configuring Database and File System State Persistence

This section explains how to prevent the loss of data that can result from a system failure by persisting the state of WSO2 SP periodically either into a database system or into the file system.

Prerequisites

Before configuring database persistence, the following prerequisites must be completed.

One or more Siddhi Applications must be running in the WSO2 SP server.
A working RDBMS instance that can be used for data persistence must exist.
The requirements of the datasource must be already defined.
Database persistence involves updating the databases connected to WSO2 Stream Processor with the latest information relating to the events that are being processed by WSO2 SP at a given time.

Configuring database system persistence

The supported databases are H2, MySQL, Postgres, MSSQL and Oracle. The relevant jdbc driver jar should be downloaded and added to the <SP_HOME>/lib directory to prior to using database system persistence.

To configure periodic data persistence, update the <SP_HOME>/conf/worker/deployment.yaml file under state.persistence as follows:

Parameter	Purpose	Required Value
`enabled`	This enables data persistence.	`true`
`intervalInMin`	The time interval in minutes that defines the interval in which state of Siddhi applications should be persisted	1
`revisionsToKeep`	The number of revisions to keep in the system. When a new persist takes place, the old revisions are removed.	3
`persistenceStore`	The persistence store.	`org.wso2.carbon.stream.processor.core.persistence.DBPersistenceStore`
`config > datasource`	The datasource to be used in persisting the state. The provided datasource should be properly defined in the deployment.yaml. For detailed instructions of how to configure a datasource, see Configuring Datasources.	WSO2_PERSISTENCE_DB (Datasource with this name should be defined in wso2.datasources)
`config > table`	The table that should be created and used for the persisting of the state.	PERSISTENCE_TABLE

The following is a sample segment of the required configurations in the <SP_HOME>/conf/worker/deployment.yaml file to configure file system persistence.

Sample deployment.yaml segment

state.persistence:
  enabled: true
  intervalInMin: 1
  revisionsToKeep: 3
  persistenceStore: org.wso2.carbon.stream.processor.core.persistence.DBPersistenceStore
  config:
    datasource: <DATASOURCE NAME>   # A datasource with this name should be defined in wso2.datasources namespace
    table: <TABLE NAME>

Configuring file system persistence

This section explains how to persist the states of Siddhi applications during a required time interval in the file system in order to maintain back-ups. To configure state persistence, update the <SP_HOME>/conf/worker/deployment.yaml file under state.persistence as follows:

Parameter	Purpose	Required Value
`enabled`	This enables data persistence.	`true`
`intervalInMin`	The time interval in minutes that defines the interval in which state of Siddhi applications should be persisted	`1`
`revisionsToKeep`	The number of revisions to keep in the system. When a new persist takes place, the old revisions are removed.	`3`
`persistenceStore`	The persistence store.	org.wso2.carbon.stream.processor.core.persistence.FileSystemPersistenceStore
`config > location`	A fully qualified folder location to where the revision files should be persisted.	siddhi-app-persistence

The following is a sample segment of the required configurations in the <SP_HOME>/conf/worker/deployment.yaml file to configure file system persistence.

Sample deployment.yaml segment

state.persistence:
  enabled: true
  intervalInMin: 1
  revisionsToKeep: 2
  persistenceStore: org.wso2.carbon.stream.processor.core.persistence.FileSystemPersistenceStore
  config:
    location: siddhi-app-persistence