Package com.logicalclocks.hsfs.spark.engine.profile

Class ColumnProfiler

java.lang.Object

com.logicalclocks.hsfs.spark.engine.profile.ColumnProfiler

public class ColumnProfiler
extends Object

Replacement for Deequ's ColumnProfilerRunner producing JSON wire-compatible with Deequ 2.0.7-spark-3.5 for all keys except kll.

KLL gating divergence from Deequ

Deequ always emits kll for numeric columns regardless of withKLLProfiling(); the toggle is a no-op in 2.0.7-spark-3.5. This implementation emits kll only when kll=true, aligning with the Phase-1 API contract. The kll=false path produces smaller profiles. The golden-parity test (task #6) accounts for this known divergence.

Entropy computation

Shannon entropy is derived from the exact per-value frequency distribution via groupBy(col).count() per column. For numeric columns where all values are unique this equals ln(exactNumDistinctValues), but the groupBy is required for correctness when duplicates exist.

Uniqueness formula

uniqueness = singletons / nonNull: Deequ's exact definition (fraction of values appearing exactly once). The singleton count comes from the same per-value frequency pass as entropy, so no additional Spark job is paid for it. (An earlier shortcut, (2 * exactDistinct - nonNull) / nonNull, is only equivalent when no value occurs more than twice and undercounts otherwise.)

stdDev

Uses Spark's stddev_pop() (population standard deviation, dividing by n). Deequ's StandardDeviation metric also uses population stddev; verified against the baseline.

Constructor Summary

Constructors

Constructor	Description
ColumnProfiler()

Method Summary

Modifier and Type	Method	Description
String	profile(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> df, List<String> restrictToColumns, boolean correlation, boolean histogram, int histogramBins, boolean exactUniqueness, boolean kll)	Profiles the given dataframe and returns a JSON string matching the Deequ wire format.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

ColumnProfiler

public ColumnProfiler()

Method Details

profile

public String profile(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> df,
 List<String> restrictToColumns,
 boolean correlation,
 boolean histogram,
 int histogramBins,
 boolean exactUniqueness,
 boolean kll)

Profiles the given dataframe and returns a JSON string matching the Deequ wire format.

Parameters:

df - source dataframe

restrictToColumns - columns to profile; null or empty means all columns

correlation - whether to compute pairwise Pearson correlations for numeric columns

histogram - whether to compute histogram bins

histogramBins - number of histogram bins (used only when histogram=true)

exactUniqueness - whether to compute exact distinct counts via countDistinct

kll - whether to compute KLL sketches and derived percentiles for numeric columns

Returns:

JSON string with top-level {"columns": [...]}