...
The following information is displayed for each Spark worker in its web UI under Running Executors.
| Info |
|---|
It is recommended to run only one executor per DAS worker. If you observe any memory or Spark execution time issues for that executor, you can increase the amount of memory and the number of CPU cores allocated for that executor. For more information about configuring Spark executors, see Spark Configurations - Executor configurations. |
| Column | Description |
|---|---|
| ExecutorID | The ID of the executor to which the information applies. |
| Cores | The number of cores used by the executor. |
| State | The current status of the executor. |
| Memory | The memory used by the executor. |
| Job Details | This displays the following:
|
| Logs | This lists the IDs of logs generated for the Spark worker. To view a specific log, click on the relevant ID. |
...
This tab displays detailed information about the SQL queries of the selected Spark application.
Spark issues in a production envirionment
The following are three issues that may occur when you work with Spark in a multi node DAS cluster:
| Info |
|---|
The following issues only occur when the DAS cluster is running in RedHat Linux environments. |
- The DAS nodes consuming too much CPU processing power.
- DAS nodes running out of memory.
- Too many log directories being created in the
<DAS_HOME>/workdirectory.
All of the above issues can be created as a result of he symbolic link not being correctly resolved in the operating system. To address this, you are required to update the <DAS_HOME>/bin/wso2server.sh file with the following entry so that the <DAS_HOME> is exported. Export CARBON_HOME=<symbolic link