Versions Compared

Old Version 28

changes.mady.by.user Former user

Saved on

compared with

New Version Current

changes.mady.by.user Former user

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

This section covers the configurations required to use Apache Spark with WSO2 DAS.

...

If you want to add additional jars, you can add them to the SPARK_CLASSPATH in the <DAS_HOME>/bin/external-spark-classpath.conf  file in a UNIX environment.

...

Following are the Carbon related configurations that are used for Apache Spark. These configurations are shipped with the product by default in the <DAS_home>/repository/conf/analytics/spark/spark-defaults.conf file.

PropertyDefault ValueDescription
carbon.spark.master local

The Spark master has three possible states as follows:

  • local: This starts Spark in the local mode. e.g, carbon.spark.master local or carbon.spark.master local[2]
  • client : This mode results in the DAS acting as a client for an external Spark cluster. e.g., carbon.spark.master spark://<host name>:<port>. For more details on setting up WSO2 DAS and Apache Spark in this mode, see Connecting a DAS Instance to an Existing External Apache Spark Cluster.
  • cluster: This mode results in the DAS creating its own Spark cluster using Carbon Clustering. When Spark runs in this mode, it is required to specify a value for the carbon.spark.master.count property. e.g., carbon.spark.master local AND carbon.spark.master.count  <number of redundant masters>
carbon.spark.master.count1

The maximum number of masters allowed at a given time when DAS creates its own Spark cluster.

Info

This property is applicable only when the Spark master runs in the cluster mode.

carbon.das.symbolic.link

Anchor
symbolic link
symbolic link

This links to your DAS home by default.

The symbolic link for the jar files in the Spark class path.

In a clustered DAS deployment, the directory path for the Spark Class path is different for each node depending on the location of the <DAS_HOME>. The symbolic link redirects the Spark Driver Application to the relevant directory for each node when it creates the Spark class path. The symbolic link should be located in the same path for each <DAS_HOME>.

Info
  • The symbolic link is not specified by default. When it is not specified, the jar files are added in the DAS home.
  • In a multi node DAS cluster that runs in a RedHat Linux environment, you also need to update the <DAS_HOME>/bin/wso2server.sh file with the following entry so that the <DAS_HOME> is exported. This is because the symbolic link may not be resolved correctly in this operating system.

    Export CARBON_HOME=<symbolic link>

Spark related configurations

...

Application configurations

PropertyDefault Value

spark.app.name

CarbonAnalytics

spark.driver.cores
1
spark.driver.memory 512m
spark.executor.memory 512m

Spark UI configurations

PropertyDefault Value

spark.ui.port

CarbonAnalytics

spark.history.ui.port 18080

Compression and serialization configurations

PropertyDefault Value

spark.serializer

org.apache.spark.serializer.KryoSerializer

spark.kryoserializer.buffer
 256k
spark.kryoserializer.buffer.max
 256m

Networking configurations

PropertyDefault Value

spark.blockManager.port

12000

spark.broadcast.port12500
spark.driver.port13000
spark.executor.port13500
spark.fileserver.port14000
spark.replClassServer.port14500

Scheduling configurations

PropertyDefault Value

spark.scheduler.mode

FAIR

Info

In addition to having FAIR as the value for the spark.scheduler.mode property in the spark-defaults.conf file, it is required to have a pool configuration with the schedulingMode parameter set to FAIR in the <DAS_HOME>/repository/conf/analytics/spark/fairscheduler.xml file as shown in the configuration below. This is because Apache Spark by default uses a scheduler pool which runs scheduler tasks in a First-In-First-Out (FIFO) order.

Code Block
languagexml
<?xml version="1.0"?>

<allocations>
    <pool name="carbon-pool">
        <schedulingMode>FAIR</schedulingMode>
        <weight>1000</weight>
        <minShare>1</minShare>
    </pool>
    <pool name="test">
        <schedulingMode>FIFO</schedulingMode>
        <weight>1</weight>
        <minShare>0</minShare>
    </pool>
</allocations>

 


Standalone cluster configurations

PropertyDefault Value

spark.deploy.recoveryMode

CUSTOM

spark.deploy.recoveryMode.factory
org.wso2.carbon.analytics.spark.core.deploy.AnalyticsRecoveryModeFactory

Master configurations

PropertyDefault Value

spark.master.port

7077

spark.master.rest.port6066
spark.master.webui.port8081

Worker configurations

PropertyDefault Value

spark.worker.cores

1

spark.worker.memory1g
spark.worker.dirwork
spark.worker.port11000
spark.worker.webui.port11500

Executor configurations
Anchor
executor
executor

It is recommended to run only one executor per DAS worker. If you observe any memory or Spark executor time issues for this executor, you can increase the amount of memory and the number of CPU cores allocated to it.

PropertyDefault ValueDescription
spark.executor.cores1The number of cores allocated to the Spark executors that are running in the DAS node. All the availble CPU cores of the worker are allocated to the executor(s) by default.
spark.executor.memory1gThe amount of CPU memory allocated to the spark executor(s).

Optional Spark related configurations

...

Code Block
languagetext
spark.executor.logs.rolling.strategy  size
spark.executor.logs.rolling.maxSize  10000000
spark.executor.logs.rolling.maxRetainedFiles 10
PropertyDescription
spark.executor.logs.rolling.strategy
This indicates the strategy used to control the amount of logs saved in the <DAS_HOME>/work directory. In the above configuration, property value size indicates that the amount of logs that are allowed to be kept is restricted based on the size.
spark.executor.logs.rolling.maxSize
The maximum size (in bytes) allowed for logs saved in the <DAS_HOME>/work  directory at any given time. Older log files are deleted when new logs are generated so that the specified maximum size is not exceeded.
spark.executor.logs.rolling.maxRetainedFiles
The maximum number of log files allowed to be kept in the <DAS_HOME>/work  directory at any given time. Older log files are deleted when new logs are generated so that the specified maximum number of files is not exceeded. In the above configuration, this property is overruled by the spark.executor.logs.rolling.maxSize property because the value specified for the spark.executor.logs.rolling.strategy is size. If the maximum size specified for logs is reached, older logs are deleted even if the maximum number of files specified is not yet reached.