Package com.logicalclocks.hsfs.spark.engine.profile

Class KllMerger

java.lang.Object
com.logicalclocks.hsfs.spark.engine.profile.KllMerger
public final class KllMerger extends Object
Merges per-batch native-format KLL sketches into a single aggregate sketch, then derives a 100-element percentiles vector and a histogram-from-CDF. The Phase-2 unlock for rolling-window reference distributions on HUDI/DELTA feature groups.

Called from the Python SDK via Py4J: 

 engine.get_instance()._jvm
   .com.logicalclocks.hsfs.spark.engine.profile.KllMerger
   .merge(baseB64List, histogramBins)

Input: list of base64-encoded KllDoublesSketch.toByteArray() blobs, each emitted by ProfileJsonSerializer when kll=true on a per-batch profile run. Empty input returns a JSON object with empty arrays and no kll field.

Output JSON shape (stable contract, consumed by the Python merge driver): 

 {
   "percentiles": [p01, p02, ..., p99],
   "histogram":   [{"value":"X.XX to Y.YY","count":..,"ratio":..}, ...],
   "kll":         "<base64 of merged sketch bytes>",
   "kllFormat":   "datasketches-native-v1",
   "n":           <total weight>,
   "min":         <double>,
   "max":         <double>
 }

Histogram counts are approximated from the merged sketch's CDF; exact per-bin counts are not recoverable from a KLL structure. The error is bounded by the KLL rank error (~0.13% with K=2048) — acceptable for PSI/KL/JS/Hellinger on 10-20 bin histograms.

  • Method Details

    • merge

      public static String merge(List<String> base64Sketches, int histogramBins)
      Merge per-batch KLL sketches into a single sketch and derive monitoring artifacts.
      Parameters:
      base64Sketches - list of base64-encoded sketch bytes (one per batch)
      histogramBins - number of equi-width bins for the approximated histogram
      Returns:
      JSON string with percentiles, histogram, merged sketch bytes, and format tag