Package com.logicalclocks.hsfs.spark.engine.profile

Class KllAggregator

java.lang.Object
com.logicalclocks.hsfs.spark.engine.profile.KllAggregator
public class KllAggregator extends Object
Builds a KllDoublesSketch over a numeric column using a distributed mapPartitions approach.

Avoids Spark's Aggregator pattern and its Kryo serialisation requirements. Each partition independently builds a sketch, serialises it to byte[] and ships it to the driver, where sketches are merged. This is the recommended pattern for pure-Java aggregators that cannot implement java.io.Serializable.

Only finite values are fed to the sketch: callers derive bin edges from its min/max, and a column with nothing finite yields an empty sketch, which most operations reject.

K=2048 matches the Deequ baseline's effective sketch resolution (Deequ also used K=2048). Normalised rank error is ~0.13%, tight enough for extreme-quantile monitoring on wide-range integer columns where K=200 showed ≥3% tail error. Larger K = more memory per sketch; 2048 is the established trade-off.

  • Constructor Summary

    Constructors
    Constructor
    Description
    KllAggregator()
     

  • Method Summary

    Modifier and Type
    Method
    Description
    byte[]
    computeSketch(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> df, String columnName)
    Builds a merged KllDoublesSketch over the given numeric column and returns its byte representation.
    static org.apache.datasketches.kll.KllDoublesSketch
    heapify(byte[] bytes)
    Deserialises bytes produced by KllDoublesSketch.toByteArray() back into a live sketch.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • KllAggregator

      public KllAggregator()

  • Method Details

    • computeSketch

      public byte[] computeSketch(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> df, String columnName)
      Builds a merged KllDoublesSketch over the given numeric column and returns its byte representation.
      Parameters:
      df - source dataframe
      columnName - numeric column (must be castable to double)
      Returns:
      serialised sketch bytes from KllDoublesSketch.toByteArray()

    • heapify

      public static org.apache.datasketches.kll.KllDoublesSketch heapify(byte[] bytes)
      Deserialises bytes produced by KllDoublesSketch.toByteArray() back into a live sketch.