...
Monitoring via Spark UIs
| Note |
|---|
Before you enable Spark UIs, note that they are only supported with HTTP, and the Environment tab of Application UI may display security sensitive information such as user IDs and keystore passwords. the keystore password depending on your Spark Configuration. If you do not want to expose such information, it is recommended to disable these UIsyou can disable Spark UIs as explained in Disabling Spark UI. |
Apache Spark provides a set of user interfaces (UI) that allow you to monitor and troubleshoot the issues in a Spark cluster. This section helps you to understand the information accessed from these UIs.
...
This tab displays detailed information about the SQL queries of the selected Spark application.
Spark issues in a production envirionment
The following are three issues that may occur when you work with Spark in a multi node DAS cluster:
| Info |
|---|
The following issues only occur when the DAS cluster is running in RedHat Linux environments. |
- The DAS nodes consuming too much CPU processing power.
- DAS nodes running out of memory.
- Too many log directories being created in the
<DAS_HOME>/workdirectory.
All of the above issues can be created as a result of he symbolic link not being correctly resolved in the operating system. To address this, you are required to update the <DAS_HOME>/bin/wso2server.sh file with the following entry so that the <DAS_HOME> is exported. Export CARBON_HOME=<symbolic link