deequ | library built on top of Apache Spark

I had to do the inevitable and add this to my build.sbt:

You can't use Deeque version 2.0.0 with Spark 3.0 because it's binary incompatible due of the changes in the Spark's internals. With Spark 3.0 you need to use version 1.2.2-spark-3.0

Copy the contents of $HADOOP_HOME/etc/hadoop from the Hadoop/YARN master node to the local host and set the HADOOP_CONF_DIR environment variable to point to the directory.

• Launching Spark on YARN

Ensure that HADOOP_CONF_DIR or YARN_CONF_DIR points to the directory which contains the (client side) configuration files for the Hadoop cluster. These configs are used to write to HDFS and connect to the YARN ResourceManager. The configuration contained in this directory will be distributed to the YARN cluster so that all containers used by the application use the same configuration.

hasUniqueness takes a function that accepts an in/float parameter and returns a boolean :

Creates a constraint that asserts any uniqueness in a single or combined set of key columns. Uniqueness is the fraction of unique values of a column(s) values that occur exactly once.

Here's an example of usage :

You can use this Spark SQL expression :

check_status is the overal status for the Check group you run. It depends on the CheckLevel and the constraint status. If you look at the code :

Check has a method satisfies which can take a column expression as condition parameter.

To check whether column1 is between -20%column2 and +20%column2, you can use expression like:

|column1 - column2| < 0.20*column2

or column1 between 0.80*column2 and 1.20*column2:

Based on what I understand from your question, you could create the function that does the common logic and you could call the same function from different places. You could have multiple parameters for your function based on different values that you have for your different work flows.

For any beginner who might be stuck here:

You will need to have an IDE (I used IntelliJ IDEA). Steps to follow:

1. Create a scala project - put down all dependencies you need, in the build.sbt file.
2. Create a package (say 'pkg') and under it create a scala object (say 'obj').
3. Define a main method in your scala object and write your logic.
4. Process the project to form a single .jar file. (use IDE tools or run 'sbt package' in your project directory)
5. Submit using the following command