Skip to content

hopsworks.core.semantic_layer_api #

SemanticLayerApi #

The semantic projects of a Hopsworks project: metric definitions compiled from YAML and queried as metrics.

A semantic project reads dbt or MetricFlow semantic YAML from a HopsFS directory, usually a dbt project in a Hopsworks git repository. Its semantic models are feature groups, and its queries run on Trino as the caller, so members see only the data they can already read. Each SemanticProject compiles, lists its metrics and dimensions, and turns metric queries into SQL or rows.

Register and query a dbt project
import hopsworks

project = hopsworks.login()
sl = project.get_semantic_layer_api()

sales = sl.create_project("sales", "Jupyter/analytics/dbt")
sales.compile()
df = sales.query(metrics=["revenue"], group_by=["metric_time__month"])

create_project #

create_project(
    name: str,
    root_path: str,
    git_repository_id: int | None = None,
    format: Literal[
        "AUTO", "DBT", "NATIVE", "MANIFEST_JSON"
    ] = "AUTO",
    deployment_mode: Literal["FOLLOW", "PINNED"] = "FOLLOW",
    tags: dict[str, Any] | None = None,
) -> semantic_layer.SemanticProject

Register a directory of semantic definitions, which starts its first compile in the background.

PARAMETER DESCRIPTION
name

Name of the semantic project, unique in the Hopsworks project, lowercase letters, digits and underscores starting with a letter.

TYPE: str

root_path

Directory of the definitions, relative to the project root, under /hopsfs/ in a Hopsworks terminal, or an absolute /Projects/<project>/... path. For a dbt project it is the directory holding dbt_project.yml.

TYPE: str

git_repository_id

Hopsworks git repository holding root_path, or None to detect it from the path.

TYPE: int | None DEFAULT: None

format

DBT for dbt project YAML, NATIVE for MetricFlow YAML, MANIFEST_JSON for a semantic_manifest.json, or AUTO to detect it from the files.

TYPE: Literal['AUTO', 'DBT', 'NATIVE', 'MANIFEST_JSON'] DEFAULT: 'AUTO'

deployment_mode

FOLLOW to serve every new valid build, or PINNED to serve a build until another is promoted.

TYPE: Literal['FOLLOW', 'PINNED'] DEFAULT: 'FOLLOW'

tags

Schematised tags to attach in the same request, tag schema name to value, such as {"ownership": {"owner": "finance"}}. A cluster with mandatory semantic project tags refuses a project without them, so pass them here.

TYPE: dict[str, Any] | None DEFAULT: None

RETURNS DESCRIPTION
semantic_layer.SemanticProject

The semantic project.

RAISES DESCRIPTION
hopsworks.client.exceptions.RestAPIError

If the name is taken or invalid, the path is outside the project, a tag is unknown, invalid or a missing mandatory one, or the backend encounters another error.

create_time_spine #

create_time_spine(
    start: str | datetime.date,
    end: str | datetime.date,
    granularity: Literal[
        "second",
        "minute",
        "hour",
        "day",
        "week",
        "month",
        "quarter",
        "year",
    ] = "day",
    custom_granularities: Mapping[
        str, str | Callable[[Any], Any]
    ]
    | None = None,
) -> FeatureGroup

Write the time spine that cumulative metrics, offsets, join_to_timespine and custom granularities join to.

The spine is the offline feature group time_spine_<granularity>, version 1, with one row per period from start through end. Its primary key date_<granularity> holds the start of each period: a date for a day or a coarser grain, a timestamp below a day. A semantic project's time spine model at that grain binds the feature group by this name, unless its config.meta.hopsworks.feature_group names another. When the feature group exists the rows are inserted into it, so a wider range extends the spine, as long as the custom granularities are the ones it already has. The spine holds one row per period, so a fine grain over a long range is large: ten years are 3,653 days but 315 million seconds.

A daily spine with a fiscal quarter and a fiscal year
sl = project.get_semantic_layer_api()
sl.create_time_spine(
    "2020-01-01",
    "2030-12-31",
    custom_granularities={
        "fiscal_quarter": "Q-JAN",
        "fiscal_year": lambda day: f"FY{day.year + (day.month > 1)}",
    },
)
PARAMETER DESCRIPTION
start

First day or instant of the spine, as an ISO-8601 string, a date or a datetime; it is moved back to the start of its period.

TYPE: str | datetime.date

end

Last day or instant of the spine, included.

TYPE: str | datetime.date

granularity

Grain of the spine; weeks start on Monday.

TYPE: Literal['second', 'minute', 'hour', 'day', 'week', 'month', 'quarter', 'year'] DEFAULT: 'day'

custom_granularities

One column per custom granularity the time spine model declares, column name to definition. A string is a pandas period frequency, such as "Q-JAN" for the quarters of a fiscal year that ends in January, and each row holds the start of its period at that frequency, typed like date_<granularity>. A callable takes each row's date_<granularity> value and returns that row's value of the column.

TYPE: Mapping[str, str | Callable[[Any], Any]] | None DEFAULT: None

RETURNS DESCRIPTION
FeatureGroup

The feature group, with the rows written.

RAISES DESCRIPTION
ValueError

For a granularity not listed, a start or end that is not a date, an end before start, or a string that is not a pandas period frequency.

hopsworks.client.exceptions.RestAPIError

If the backend fails to create or write the feature group, for example when an existing spine has other columns.

delete_project #

delete_project(name: str) -> None

Delete a semantic project and its builds; the definitions in HopsFS are left in place.

PARAMETER DESCRIPTION
name

Name of the semantic project.

TYPE: str

RAISES DESCRIPTION
hopsworks.client.exceptions.SemanticLayerException

If the Hopsworks project has no semantic project with that name.

hopsworks.client.exceptions.RestAPIError

If the backend encounters an error when handling the request.

get_project #

get_project(
    name: str,
) -> semantic_layer.SemanticProject | None

A semantic project by name.

PARAMETER DESCRIPTION
name

Name of the semantic project.

TYPE: str

RETURNS DESCRIPTION
semantic_layer.SemanticProject | None

The semantic project, or None if the Hopsworks project has none with that name.

RAISES DESCRIPTION
hopsworks.client.exceptions.RestAPIError

If the backend encounters an error when handling the request.

get_projects #

get_projects() -> list[semantic_layer.SemanticProject]

Every semantic project of the Hopsworks project.

RETURNS DESCRIPTION
list[semantic_layer.SemanticProject]

The semantic projects, each with its newest build.

RAISES DESCRIPTION
hopsworks.client.exceptions.RestAPIError

If the backend encounters an error when handling the request.