Versions Compared

Old Version 27

changes.mady.by.user Former user

Saved on

compared with

New Version 28

changes.mady.by.user Former user

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

This section covers the configurations required to use Apache Spark with WSO2 DAS.

...

If you want to add additional jars, you can add them to the SPARK_CLASSPATH in the <DAS_HOME>/bin/external-spark-classpath.conf  file in a UNIX environment.

Tip

Each path should have a separate line.

Info

When WSO2 DAS connects with an external Spark cluster, the distribution of DAS is copied to each node in the Spark cluster. This allows each Spark node to access the jars it needs to work with WSO2 DAS.

Carbon related configurations 

...

PropertyDefault ValueDescription
carbon.spark.master local

The Spark master has three possible states as follows:

  • local: This starts Spark in the local mode. e.g, carbon.spark.master local or carbon.spark.master local[2]
  • client : This mode results in the DAS acting as a client for an external Spark cluster. e.g., carbon.spark.master spark://<host name>:<port>
  • cluster: This mode results in the DAS creating its own Spark cluster using Carbon Clustering. When Spark runs in this mode, it is required to specify a value for the carbon.spark.master.count property. e.g., carbon.spark.master local AND carbon.spark.master.count  <number of redundant masters>
carbon.spark.master.count 1

The maximum number of masters allowed at a given time when DAS creates its own Spark cluster.

Info

This property is applicable only when the Spark master runs in the cluster mode.

carbon.das.symbolic.link
Anchor
symbolic link
symbolic link
This links to your DAS home by default.

The symbolic link for the jar files in the Spark class path.

In a clustered DAS deployment, the directory path for the Spark Class path is different for each node depending on the location of the <DAS_HOME>. The symbolic link redirects the Spark Driver Application to the relevant directory for each node when it creates the Spark class path. The symbolic link should be located in the same path for each <DAS_HOME>.

Info

The symbolic link is not specified by default. When it is not specified, the jar files are added in the DAS home.

...