Creating Spark User Defined Aggregate Functions

Apache Spark UDAFs (User Defined Aggregate Functions) allow you to implement customized aggregate operations on Spark rows. Custom UDAFs can be written and added to DAS if the required functionality does not already exist in Spark.

In addition to the definition of custom Spark UDAFs, WSO2 DAS also provides an abstraction layer for generic Spark UDAF functionality that be used to introduce custom UDAFs to the DAS/Spark runtime without changes to the server configuration.

The following query is an example of a custom UDAF named geometricMean.

SELECT geometricMean(price) as meanPrice FROM products;

Apache Spark 1.6.2, and by extension WSO2 DAS 3.1.0, do not support primitive data type returns. Therefore, all the methods in a POJO class should return the wrapper class of the corresponding primitive data type.
Apache Spark 1.6.2 also does not differentiate between algebraic and non-algebraic UDAFs. Therefore, all the methods in the parent class need to be implemented for all the UDAFs. e.g., Mathematical operations such as median do not support merge operations, but they are still required.
Method overloading for UDAFs is not supported. Different UDAFs should have different method names for the expected behaviour. e.g., a UDAF with the same functionality for String and Numeric types need to be defined as two separate UDAF implementations, where each handles a different type.

For a list of return types supported for Apache Spark, see Spark SQL and DataFrames and Datasets Guide - Data Types.

There are two patterns that can be followed to create a custom UDAF. Click on the relevant tab based on the pattern you want to follow.