Skip to content

Feature Validation with the Hopsworks Feature Store#

Correct feature data is essential for developing accurate machine learning models. Raw data being ingested into the feature store maybe suffer from incorrect or corrupt values, may need to be validated against certain features depending on the domain. For example, a feature representing the customer's age should not be a negative number and should always have a value set.

HSFS provides an API to define expectations on data being inserted into feature groups and also view results over time of these expectations in the form of feature validations.

Feature validations can therefore be easily integrated with existing feature ingestion pipelines. HSFS utilizes the Deequ open source library and support is currently being added for working with the Great Expectations.

Below we describe the different API components of the hsfs feature validation API and we walk you through Feature validation is part of the HSFS Java/Scala and Python API for working with Feature Groups. Users work with the abstractions:

Rule definitions#

A set of pre-defined and immutable rule definitions (RuleDefiniton) that are unique by name and are used for creating validation rules (Rule) and expectations (Expectation) applied on a dataframe that is inserted into a Feature Group.

Retrieval#

[source]

get_rules#

Connection.get_rules()

Get a rule with a certain name or all rules available for data validation.

[source]

get_rule#

Connection.get_rule(name)

Get a rule with a certain name or all rules available for data validation.

For example, to get all available rule definitions in hsfs:

import hsfs
connection = hsfs.connection()
rules = connection.get_rules()

import com.logicalclocks.hsfs._
val connection = HopsworksConnection.builder().build();
val rules = connection.getRules()

and to get a rule definition by name:

import hsfs
connection = hsfs.connection()
rules = connection.get_rules()

import com.logicalclocks.hsfs._
val connection = HopsworksConnection.builder().build();
val rules = connection.getRules()

Properties#

[source]

RuleDefinition#

hsfs.ruledefinition.RuleDefinition(
    name,
    predicate,
    accepted_type,
    description,
    href=None,
    expand=None,
    items=None,
    count=None,
    type=None,
)

Metadata object representing the validation rule that is used by feature group expectations.

The set of rule definitions, for example "has max", "has avg" is provided by hsfs and cannot be modified.

[source]

accepted_type#

The type of the feature, one of "Null", "Fractional", "Integral", "Boolean", "String".

[source]

description#

[source]

name#

Name of the rule definition. Unique across all features stores.

[source]

predicate#

Predicate of the rule definition, one of "VALUE", "LEGAL_VALUES", "ACCEPTED_TYPE", "PATTERN".

Rules#

Used as part of expectations that are applied on ingested features. Rule names correspond to the names of the rule definitions (see section above) and you can set the severity level and the actual values that the feature should respect.

Setting rules values#

Rules applied on numerical features except for a specific value they can also accept a range of values, for example if you need a feature to be ingested if its minimum value is below zero, then you can set min(0) and max(0) but if you want the minimum to fall within a range of 0 and 1000 then you need to set min(0) and max(1000). See section Expectations below for a detailed example.

Properties#

[source]

Rule#

hsfs.rule.Rule(
    name,
    level,
    min=None,
    max=None,
    value=None,
    pattern=None,
    accepted_type=None,
    legal_values=None,
    href=None,
    expand=None,
    items=None,
    count=None,
    type=None,
)

Metadata object representing the validation rule that is used by feature group expectations.

This class is made for hsfs internal use only.

[source]

accepted_type#

Data type accepted for a feature. Applicable only to the HAS_DATATYPE rule.

[source]

List of legal values a feature should be found int. feature.Applicable only to IS_CONTAINED_IN rule.

[source]

level#

Severity level of a rule, one of "WARNING" or "ERROR".

[source]

max#

The upper bound of the value range this feature should fall into.

[source]

min#

The lower bound of the value range this feature should fall into.

[source]

name#

Name of the rule as found in rule definitions.

[source]

pattern#

Pattern to check for a feature's pattern compliance. Applicable only to the HAS_PATTERN rule.

[source]

value#

The upper bound of the value range this feature should fall into.

Expectations#

A set of rule instances that are applied on a set of features. Expectations are created at the feature store level and can be attached to multiple feature groups. An expectation contains a list of featues it is applied to. If the feature group the expectation is attached to, does not contain all the expectations features, the expectation will not be met.

Creation#

Create an expectation with two rules for ensuring the min and max of the features are valid:

expectation_sales = fs.create_expectation("sales",
                                           description="min and max sales limits",
                                           features=["salary", "commission"],
                                           rules=[Rule(name="HAS_MIN", level="WARNING", min=0), Rule(name="HAS_MAX", level="ERROR", max=1000000)])
expectation_sales.save()

// Create an expectation for the "salary" and "commissio" features so that their min value is "10" and their max is "100"
val expectationSales = (fs.createExpectation()
                          .rules(Seq(
                              Rule.createRule(RuleName.HAS_MIN).min(0).level(Level.WARNING).build(), //Set rule by name
                              Rule.createRule(ruleMax).max(1000000).level(Level.ERROR).build())) //Set rule by passing the RuleDefinition metadata
                          .name("sales")
                          .description("min and max sales limits")
                          .features(Seq("salary", "commission"))
                          .build())
expectationSales.save()

[source]

create_expectation#

FeatureStore.create_expectation(name, description="", features=[], rules=[])

Create an expectation metadata object.

Lazy

This method is lazy and does not persist the expectation in the feature store on its own. To materialize the expectation and save call the save() method of the expectation metadata object.

Arguments

  • name str: Name of the training dataset to create.
  • description Optional[str]: A string describing the expectation that can describe its business logic and applications within the feature store.
  • features Optional[List[str]]: The features this expectation is applied on.
  • rules Optional[List[hsfs.rule.Rule]]: The validation rules this expectation will apply to the its features.

Returns:

Expectation: The expectation metadata object.

Retrieval#

[source]

get_expectation#

FeatureStore.get_expectation(name)

Get an expectation entity from the feature store.

Getting an expectation from the Feature Store means getting its metadata handle so you can subsequently add features and/or rules and save it which will overwrite the previous instance.

Arguments

  • name str: Name of the training dataset to get.

Returns

Expectation: The expectation metadata object.

Raises

  • RestAPIError: If unable to retrieve the expectation from the feature store.

[source]

get_expectations#

FeatureStore.get_expectations()

Get all expectation entities from the feature store.

Getting expectations from the Feature Store means getting their metadata handles so you can subsequently add features and/or rules and save it which will overwrite the previous instance.

Returns

Expectation: The expectation metadata object.

Raises

  • RestAPIError: If unable to retrieve the expectations from the feature store.

Properties#

[source]

Expectation#

hsfs.expectation.Expectation(
    name,
    features,
    rules,
    description=None,
    featurestore_id=None,
    href=None,
    expand=None,
    items=None,
    count=None,
    type=None,
)

Metadata object representing an feature validation expectation in the Feature Store.

[source]

description#

Description of the expectation.

[source]

features#

List of features this expectation is applied to.

[source]

name#

Name of the expectation, unique per feature store (project).

[source]

rules#

List of rules applied to the features of the expectation.

Methods#

[source]

save#

Expectation.save()

Persist the expectation metadata object to the feature store.

Validations#

The results of expectations against the ingested dataframe are assigned a validation time and are persisted within the Feature Store. Users can then retrieve validation results by validation time and by commit time for time-travel enabled feature groups.

Validation Type#

Feature Validation is disabled by default, by having the validation_type feature group attribute set to NONE. The list of allowed validation types are:

  • STRICT: Data validation is performed and feature group is updated only if validation status is "Success"
  • WARNING: Data validation is performed and feature group is updated only if validation status is "Warning" or lower
  • ALL: Data validation is performed and feature group is updated only if validation status is "Failure" or lower
  • NONE: Data validation not performed on feature group

For example, to update the validation type to all:

fg.validation_type = "ALL"

import com.logicalclocks.hsfs.metadata.validation._
fg.updateValidationType(ValidationType.ALL)

Validate#

You can also apply the expectations on a dataframe without inserting data, that can be helpful for debugging. [source]

validate#

FeatureGroup.validate(dataframe)

Run validation based on the attached expectations

Arguments

  • dataframe pyspark.sql.DataFrame: The PySpark dataframe to run the data validation expecations against.

Returns

FeatureGroupValidation. The feature group validation metadata object.

Retrieval#

[source]

get_validations#

FeatureGroup.get_validations(validation_time=None, commit_time=None)

Get feature group data validation results based on the attached expectations

Arguments

  • validation_time: The data validation time, when the data validation started. commit_time: The commit time of a time travel enabled feature group.

Returns

FeatureGroupValidation. The feature group validation metadata object.

Properties#

[source]

ValidationResult#

hsfs.validation_result.ValidationResult(
    status,
    message,
    value,
    features,
    rule,
    href=None,
    expand=None,
    items=None,
    count=None,
    type=None,
)

Metadata object representing the validation result of a single rule of an expectation result of a Feature Group.

[source]

features#

Feature of the validation result on which the rule was applied.

[source]

message#

Message describing the outcome of applying the rule against the feature.

[source]

rule#

Feature of the validation result on which the rule was applied.

[source]

status#

[source]

value#

The computed value of the feature according to the rule.