Skip to content

Expectation Suite#

[source]

ExpectationSuite#

hsfs.expectation_suite.ExpectationSuite(
    expectation_suite_name,
    expectations,
    meta,
    id=None,
    data_asset_type=None,
    ge_cloud_id=None,
    run_validation=True,
    validation_ingestion_policy="ALWAYS",
    feature_store_id=None,
    feature_group_id=None,
    href=None,
    expand=None,
    items=None,
    count=None,
    type=None,
    created=None,
)

Metadata object representing an feature validation expectation in the Feature Store.

Creation with Great Expectations#

import great_expectations as ge

expectation_suite = ge.core.ExpectationSuite(
    "new_expectation_suite",
    expectations=[
        ge.core.ExpectationConfiguration(
            expectation_type="expect_column_max_to_be_between",
            kwargs={
                "column": "feature",
                "min_value": -1,
                "max_value": 1
            }
        )
    ]
)

Attach to Feature Group#

[source]

save_expectation_suite#

FeatureGroup.save_expectation_suite(
    expectation_suite, run_validation=True, validation_ingestion_policy="ALWAYS", overwrite=False
)

Attach an expectation suite to a feature group and saves it for future use. If an expectation suite is already attached, it is replaced. Note that the provided expectation suite is modified inplace to include expectationId fields.

Example

# connect to the Feature Store
fs = ...

# get the Feature Group instance
fg = fs.get_or_create_feature_group(...)

fg.save_expectation_suite(expectation_suite, run_validation=True)

Arguments

  • expectation_suite Union[hsfs.expectation_suite.ExpectationSuite, great_expectations.core.expectation_suite.ExpectationSuite]: The expectation suite to attach to the Feature Group.
  • overwrite bool: If an Expectation Suite is already attached, overwrite it. The new suite will have its own validation history, but former reports are preserved.
  • run_validation bool: Set whether the expectation_suite will run on ingestion
  • validation_ingestion_policy str: Set the policy for ingestion to the Feature Group.
    • "STRICT" only allows DataFrame passing validation to be inserted into Feature Group.
    • "ALWAYS" always insert the DataFrame to the Feature Group, irrespective of overall validation result.

Raises

hsfs.client.exceptions.RestAPIError.

Single Expectation API#

An API to edit the expectation list based on Great Expectations API.

[source]

add_expectation#

ExpectationSuite.add_expectation(expectation, ge_type=True)

Append an expectation to the local suite or in the backend if attached to a Feature Group.

Example

# check if the minimum value of specific column is within a range of 0 and 1
expectation_suite.add_expectation(
    ge.core.ExpectationConfiguration(
        expectation_type="expect_column_min_to_be_between",
        kwargs={
            "column": "foo_id",
            "min_value": 0,
            "max_value": 1
        }
    )
)

# check if the length of specific column value is within a range of 3 and 10
expectation_suite.add_expectation(
    ge.core.ExpectationConfiguration(
        expectation_type="expect_column_value_lengths_to_be_between",
        kwargs={
            "column": "bar_name",
            "min_value": 3,
            "max_value": 10
        }
    )
)

Arguments

  • expectation Union[hsfs.ge_expectation.GeExpectation, great_expectations.core.expectation_configuration.ExpectationConfiguration]: The new expectation object.
  • ge_type bool: Whether to return native Great Expectations object or Hopsworks abstraction, defaults to True.

Returns

The new expectation attached to the Feature Group.

Raises

hsfs.client.exceptions.RestAPIError hsfs.client.exceptions.FeatureStoreException

[source]

replace_expectation#

ExpectationSuite.replace_expectation(expectation, ge_type=True)

Update an expectation from the suite locally or from the backend if attached to a Feature Group.

Example

updated_expectation = expectation_suite.replace_expectation(new_expectation_object)

Arguments

  • expectation Union[hsfs.ge_expectation.GeExpectation, great_expectations.core.expectation_configuration.ExpectationConfiguration]: The updated expectation object. The meta field should contain an expectationId field.
  • ge_type bool: Whether to return native Great Expectations object or Hopsworks abstraction, defaults to True.

Returns

The updated expectation attached to the Feature Group.

Raises

hsfs.client.exceptions.RestAPIError hsfs.client.exceptions.FeatureStoreException

[source]

remove_expectation#

ExpectationSuite.remove_expectation(expectation_id=None)

Remove an expectation from the suite locally and from the backend if attached to a Feature Group.

Example

expectation_suite.remove_expectation(expectation_id=123)

Arguments

  • expectation_id Optional[int]: Id of the expectation to remove. The expectation will be deleted both locally and from the backend.

Raises

hsfs.client.exceptions.RestAPIError hsfs.client.exceptions.FeatureStoreException

Properties#

[source]

data_asset_type#

Data asset type of the expectation suite, not used by backend.

[source]

expectation_suite_name#

Name of the expectation suite.

[source]

expectations#

List of expectations to run at validation.

[source]

ge_cloud_id#

ge_cloud_id of the expectation suite, not used by backend.

[source]

ge_cloud_id#

ge_cloud_id of the expectation suite, not used by backend.

[source]

id#

Id of the expectation suite, set by backend.

[source]

meta#

Meta field of the expectation suite to store additional informations.

[source]

run_validation#

Boolean to determine whether or not the expectation suite shoudl run on ingestion.

[source]

validation_ingestion_policy#

Whether to ingest a df based on the validation result.

"STRICT" : ingest df only if all expectations succeed, "ALWAYS" : always ingest df, even if one or more expectations fail

Methods#

[source]

add_expectation#

ExpectationSuite.add_expectation(expectation, ge_type=True)

Append an expectation to the local suite or in the backend if attached to a Feature Group.

Example

# check if the minimum value of specific column is within a range of 0 and 1
expectation_suite.add_expectation(
    ge.core.ExpectationConfiguration(
        expectation_type="expect_column_min_to_be_between",
        kwargs={
            "column": "foo_id",
            "min_value": 0,
            "max_value": 1
        }
    )
)

# check if the length of specific column value is within a range of 3 and 10
expectation_suite.add_expectation(
    ge.core.ExpectationConfiguration(
        expectation_type="expect_column_value_lengths_to_be_between",
        kwargs={
            "column": "bar_name",
            "min_value": 3,
            "max_value": 10
        }
    )
)

Arguments

  • expectation Union[hsfs.ge_expectation.GeExpectation, great_expectations.core.expectation_configuration.ExpectationConfiguration]: The new expectation object.
  • ge_type bool: Whether to return native Great Expectations object or Hopsworks abstraction, defaults to True.

Returns

The new expectation attached to the Feature Group.

Raises

hsfs.client.exceptions.RestAPIError hsfs.client.exceptions.FeatureStoreException

[source]

from_ge_type#

ExpectationSuite.from_ge_type(
    ge_expectation_suite,
    run_validation=True,
    validation_ingestion_policy="ALWAYS",
    id=None,
    feature_store_id=None,
    feature_group_id=None,
)

Used to create a Hopsworks Expectation Suite instance from a great_expectations instance.

Arguments

  • ge_expectation_suite great_expectations.core.expectation_suite.ExpectationSuite: great_expectations.ExpectationSuite The great_expectations ExpectationSuite instance to convert to a Hopsworks ExpectationSuite.
  • run_validation bool: bool Whether to run validation on inserts when the expectation suite is attached.
  • validation_ingestion_policy str: str The validation ingestion policy to use when the expectation suite is attached. Defaults to "ALWAYS". Options are "STRICT" or "ALWAYS".
  • id Optional[int]: int The id of the expectation suite in Hopsworks. If not provided, a new expectation suite will be created.
  • feature_store_id Optional[int]: int The id of the feature store of the feature group to which the expectation suite belongs.
  • feature_group_id Optional[int]: int The id of the feature group to which the expectation suite belongs.

Returns

Hopsworks Expectation Suite instance.

[source]

from_response_json#

ExpectationSuite.from_response_json(json_dict)

[source]

get_expectation#

ExpectationSuite.get_expectation(expectation_id, ge_type=True)

Fetch expectation with expectation_id from the backend.

Example

# connect to the Feature Store
fs = ...

# get the Feature Group instance
fg = fs.get_or_create_feature_group(...)

expectation_suite = fg.get_expectation_suite()
selected_expectation = expectation_suite.get_expectation(expectation_id=123)

Arguments

  • expectation_id int: Id of the expectation to fetch from the backend.
  • ge_type bool: Whether to return native Great Expectations object or Hopsworks abstraction, defaults to True.

Returns

The expectation with expectation_id registered in the backend.

Raises

hsfs.client.exceptions.RestAPIError hsfs.client.exceptions.FeatureStoreException

[source]

json#

ExpectationSuite.json()

[source]

remove_expectation#

ExpectationSuite.remove_expectation(expectation_id=None)

Remove an expectation from the suite locally and from the backend if attached to a Feature Group.

Example

expectation_suite.remove_expectation(expectation_id=123)

Arguments

  • expectation_id Optional[int]: Id of the expectation to remove. The expectation will be deleted both locally and from the backend.

Raises

hsfs.client.exceptions.RestAPIError hsfs.client.exceptions.FeatureStoreException

[source]

replace_expectation#

ExpectationSuite.replace_expectation(expectation, ge_type=True)

Update an expectation from the suite locally or from the backend if attached to a Feature Group.

Example

updated_expectation = expectation_suite.replace_expectation(new_expectation_object)

Arguments

  • expectation Union[hsfs.ge_expectation.GeExpectation, great_expectations.core.expectation_configuration.ExpectationConfiguration]: The updated expectation object. The meta field should contain an expectationId field.
  • ge_type bool: Whether to return native Great Expectations object or Hopsworks abstraction, defaults to True.

Returns

The updated expectation attached to the Feature Group.

Raises

hsfs.client.exceptions.RestAPIError hsfs.client.exceptions.FeatureStoreException

[source]

to_dict#

ExpectationSuite.to_dict()

[source]

to_ge_type#

ExpectationSuite.to_ge_type()

[source]

to_json_dict#

ExpectationSuite.to_json_dict(decamelize=False)