Monitoring Spark Performance

The following sections explain how the performance of Spark can be monitored via JMX.

Setting up JMX to monitor Spark

To enable Spark monitoring via JMX, follow the steps below:

1. Stop the DAS server(s). For detailed instructions, see Running the Product.

2. Create a file named metrics.properties, include the following configuration in it and save in in the <DAS_HOME>/repository/conf/analytics/spark directory.

   *.sink.jmx.class=org.apache.spark.metrics.sink.JmxSink

3. In the <DAS_HOME>/repository/conf/analytics/spark/spark-defaults.conf file, add a reference to the metrics.properties  file you created in the previous step as shown below.

   spark.metrics.conf /<Product_Home>/repository/conf/analytics/spark/metrics.properties

4. Add the following configuration under the #Network in the <DAS_HOME>/repository/conf/analytics/spark/spark-defaults.conf file.

   spark.driver.extraJavaOptions -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8090 -Dcom.sun.management.jmxremote.rmi.port=8091 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=localhost

5. Start the DAS server(s).

Viewing information in the J Console

Once you have done the required configurations to enable Spark monitoring via JMX as instructed in Setting up JMX to monitor Spark, you can view information relating to Spark performance as described in the WSO2 Product Administration Guide - JMX-Based Monitoring.

If your DAS deployment is a cluster, you can view information for the following additional process:

• org.apache.spark.executor.CoarseGrainedExecutorBackend: This displays information related to the Spark worker JVM.