The following deployment patterns are supported for WSO2 ML.
...
By default, WSO2 ML runs with an inbuilt Apache Spark instance. However, when working with big data, you can handle those large data sets in a distributed environment through WSO2 ML. You can carry out data pre-processing and model building processes on an Apache Spark cluster to share the workload between the nodes of the cluster. Using a Spark cluster optimizes the performance and reduces the time consumed to build and train a machine learning model for a large data set.
Follow the steps below to run the ML jobs by connecting WSO2 ML to an external Apache Spark cluster.
Info |
---|
|
Press Ctrl+C keys to shutdown the WSO2 ML server. For more information on shutting down WSO2 ML server, see Running the Product
.In the
<ML_HOME>/repository/conf/etc/spark-config.xml
file, enter the Spark master URL as the value of the<
spark.master>
property as shown in the example below. You can find the Spark Master URL in the Apache Spark Web UI as shown belowTip .
Code Block language xml <property name="spark.master">{SPARK_MASTER_URL}</property>
- Create a directory named <SPARK_HOME>/ml/ and copy the following jar files into it. These jar files can be found in the
<ML_HOME>/repository/components/plugins
directory.org.wso2.carbon.ml.core_1.0.2.jar
org.wso2.carbon.ml.commons_1.0.2.jar
org.wso2.carbon.ml.database_1.0.2.jar
kryo_2.24.0.wso2v1.jar
Create a file named
spark-env.sh
in the<SPARK_HOME>/conf/
directory and add the following entries.Code Block language text SPARK_MASTER_IP=127.0.0.1 SPARK_CLASSPATH={SPARK_HOME}/ml/org.wso2.carbon.ml.core_1.0.2.jar:{SPARK_HOME}/ml/org.wso2.carbon.ml.commons_1.0.2.jar:{SPARK_HOME}/ml/org.wso2.carbon.ml.database_1.0.2.jar:{SPARK_HOME}/ml/kryo_2.24.0.wso2v1.jar
Restart the external Spark cluster using the following commands:
Code Block {SPARK_HOME}$ ./sbin/stop-all.sh {SPARK_HOME}$ ./sbin/start-all.sh
In the
<ML_HOME>/repository/conf/etc/spark-config.xml
file, enter the Spark master URL as the value of the<
spark.master>
property as shown in the example below.Tip You can find the Spark Master URL in the Apache Spark Web UI as shown below.
Code Block language xml <property name="spark.master">{SPARK_MASTER_URL}</property>
Restart the WSO2 ML server. For more information on restarting WSO2 ML server, see Running the Product.
...