Skip to content

hsfs.transformation_statistics #

[source] FeatureTransformationStatistics #

Data class that contains all the statistics parameters that can be used for transformations inside a custom transformation function.

[source] feature_name property #

feature_name: str

Name of the feature.

[source] count property #

count: int | None

Number of values.

[source] completeness property #

completeness: float | None

Fraction of non-null values in a column.

[source] num_non_null_values property #

num_non_null_values: int | None

Number of non-null values.

[source] num_null_values property #

num_null_values: int | None

Number of null values.

[source] approx_num_distinct_values property #

approx_num_distinct_values: int | None

Approximate number of distinct values.

[source] min property #

min: float | None

Minimum value.

[source] max property #

max: float | None

Maximum value.

[source] sum property #

sum: float | None

Sum of all feature values.

[source] mean property #

mean: float | None

Mean value.

[source] stddev property #

stddev: float | None

Standard deviation of the feature values.

[source] percentiles property #

percentiles: Mapping[str, float] | None

Percentiles.

[source] distinctness property #

distinctness: float | None

Fraction of distinct values of a feature over the number of all its values. Distinct values occur at least once.

Example

[a, a, b] contains two distinct values a and b, so distinctness is 2/3.

[source] entropy property #

entropy: float | None

Entropy is a measure of the level of information contained in an event (feature value) when considering all possible events (all feature values).

Entropy is estimated using observed value counts as the negative sum of (value_count/total_count) * log(value_count/total_count).

Example

[a, b, b, c, c] has three distinct values with counts [1, 2, 2].

Entropy is then (-1/5*log(1/5)-2/5*log(2/5)-2/5*log(2/5)) = 1.055.

[source] uniqueness property #

uniqueness: float | None

Fraction of unique values over the number of all values of a column. Unique values occur exactly once.

Example

[a, a, b] contains one unique value b, so uniqueness is 1/3.

[source] exact_num_distinct_values property #

exact_num_distinct_values: int | None

Exact number of distinct values.

[source] correlations property #

correlations: dict | None

Correlations of feature values.

[source] histogram property #

histogram: dict | None

Histogram of feature values.

[source] kll property #

kll: dict | None

KLL of feature values.

[source] unique_values property #

unique_values: dict | None

Number of Unique Values.

[source] TransformationStatistics #

Class that stores feature transformation statistics of all features that require training dataset statistics in a transformation function.

All statistics for a feature is initially initialized with null values and will be populated with values when training dataset is created for the soe.

PARAMETER DESCRIPTION
*features

str. The features for which training dataset statistics need to be computed.

TYPE: str DEFAULT: ()

Example 
# Defining transformation statistics
transformation_statistics = TransformationStatistics("feature1", "feature2")

# Accessing feature transformation statistics for a specific feature
feature_transformation_statistics_feature1 = transformation_statistics.feature1