Semantic Layer Overview#
The Hopsworks Semantic Layer reads metric definitions written in YAML over feature groups, and answers metric queries with SQL it generates for Trino. A query names metrics and the dimensions to group them by; the semantic layer writes the joins and the aggregations.
The service#
The semantic layer is a Hopsworks service with two parts:
- the control plane, in Hopsworks: semantic projects, their builds, the definitions editor, tags, metric exports and the query history;
- the semantic gateway, a Deployment of its own, which compiles the definitions, plans every metric query and runs it on Trino as the caller.
| The gateway serves | To | Guide |
|---|---|---|
| REST, JSON, Arrow and CSV | The Python client, the hops CLI, the query workbench in the UI, the agent tools, Superset, Excel's From Web | Query metrics, Excel and ODBC |
GraphQL, dbt's Semantic Layer API: metadata and compileSql | dbt-sl-sdk's metadata calls, GraphQL clients | Connect dbt clients |
| Arrow Flight SQL, dbt's JDBC dialect | dbt-sl-sdk queries, ADBC, the Arrow Flight SQL JDBC driver | Connect dbt clients |
| Arrow Flight SQL, tables | Superset, DBeaver, JDBC and ODBC tools, Excel, dbt's Tableau and Power BI connectors | Connect BI tools, Excel and ODBC |
Around the queries:
| Feature | Guide |
|---|---|
| A repeated query answered from the gateway's memory | Result cache |
| A saved query written into a feature group by a scheduled job, and queries answered from it | Metric exports |
| Every query recorded, for its members and Data Owners | Query history |
| The YAML edited in the UI, validated as you type | Edit definitions in the UI |
| Dashboards from metrics in Superset | Dashboards in Superset |
| The semantic layer tools of the Hopsworks MCP server | Agent tools |
| Turning it on and configuring it | Semantic Layer administration |
Semantic projects#
A semantic project is a folder of definitions in the project's file system, registered under a name. The folder is usually a dbt project in a git repository, holding semantic models, dimensions, entities, measures and metrics. The name is unique in the Hopsworks project, and is made of lowercase letters, digits and underscores, starting with a letter.
| Setting | Values |
|---|---|
| Root path | The folder of the definitions; for a dbt project, the folder that holds dbt_project.yml |
| Format | AUTO (default), DBT, NATIVE or MANIFEST_JSON |
| Deployment mode | FOLLOW (default) or PINNED, see Serving modes |
| Git repository | The Hopsworks git repository that holds the root path, filled in from the path |
| Owner | The Data owner its export jobs run as: the member who registers it, until a Data owner transfers it, see The owner of a semantic project |
DBT reads dbt project YAML, NATIVE reads MetricFlow YAML, and MANIFEST_JSON reads a dbt semantic_manifest.json. With AUTO, a dbt_project.yml at the root selects DBT, otherwise a semantic_manifest.json selects MANIFEST_JSON, and anything else is read as NATIVE.
Deleting a semantic project deletes its builds and leaves the YAML files in place.
Builds#
A compile reads the definitions under the root path, checks them against the feature groups, and records the result as a build. A build does not change once its compile ends, and has one of four statuses:
| Status | Meaning |
|---|---|
COMPILING | The compile is running. |
VALID | The definitions compiled without errors, so the build can answer queries. |
INVALID | The definitions have errors, each reported at its file and line. |
FAILED | The compile could not run. |
A build records the git commit it was read at, when the root path is inside a Hopsworks git repository. Compiling files and feature groups that have not changed returns their existing build. A semantic project keeps its 20 newest builds and the build that answers queries; an administrator changes the number with hopsworks.variables.semantic_layer_builds_retained in the Helm values, see Cluster variables.
Metrics and group-bys#
A semantic model reads one feature group. A measure aggregates a column or an expression of a semantic model, and a metric is a named calculation over measures: a simple metric, a ratio of two metrics, an expression over other metrics, a cumulative metric over a time window, or a conversion metric between two events, see Metric types.
A query groups metrics by group-bys:
- dimensions, categorical or time, named
<entity>__<dimension>, such ascustomer__country; - entities, the keys that join semantic models;
metric_time, the time axis of every metric, taken from the time dimension its measures aggregate over.
A time group-by takes a grain suffix, such as metric_time__month, or a custom granularity of the time spine, a feature group with one row per period. The semantic layer joins feature groups along their entities, so revenue from orders can be grouped by customer__country from customers. A saved query is a metric query defined by name in the YAML, which every client runs by name.
Serving modes#
The build that answers queries is the semantic project's current build. An INVALID or FAILED build never becomes current: queries keep the last valid build and report that it is stale.
| Mode | Current build | Suits |
|---|---|---|
FOLLOW | Each new valid build, as soon as it compiles. | Your own clone, on a feature branch. |
PINNED | The last build promoted; a new valid build waits for a promote. | A production repository that only CI or an owner pulls. |
A promote works in both modes. In FOLLOW mode a promote of an older valid build rolls back until the next valid compile. Compile and builds has the commands.
How a query runs#
- A client sends the query to the semantic gateway, with the caller's own credential: an API key, or the session of the UI.
- The gateway generates the SQL for the query from the current build.
- The SQL runs on Trino as the caller's own Trino user,
<project>__<user>, so a query reads only the data that member can read; Who a query runs as lists the answers that are not computed for the caller. - The rows come back as an Arrow stream or as JSON.
The Python client, the hops CLI, the UI and the agent tools all query through the gateway, from inside and outside the cluster. dbt Semantic Layer clients read metadata over GraphQL and run queries over Arrow Flight SQL, see Connect dbt clients, and BI tools read it as tables, see Connect BI tools. A saved query's exports write its rows into feature groups on a schedule, and the gateway answers the saved query from them while they hold its exact rows, see Metric exports. The gateway records every query it runs in the query history. Superset charts query through the gateway too, over Arrow Flight SQL on one database connection of each member per Hopsworks project, see Who can see what. The semantic layer reads the offline feature store.
Who a query runs as#
A metric query a member sends over REST, Arrow Flight SQL or the BI tables runs on Trino as that member's own Trino user, <project>__<user>, under the access and the Trino rules that apply to that member. That covers the Python client, the hops CLI, the query workbench in the UI, dbt-sl-sdk, ADBC, JDBC, ODBC, BI tools and the charts you build on your own Superset connection. An agent tool queries as the user the MCP server is logged in as. GraphQL runs no query: it answers metadata and the SQL of a query.
These answers are not computed for the member who asks:
| Answer | Computed |
|---|---|
| A chart on a dataset of another member's Superset connection, such as a chart of a dashboard shared with you | As the member whose connection the dataset is on, usually the dashboard's author, see Who can see what. |
| A run of a metric export | As the semantic project's owner. |
| A query answered from an export | As the owner, when the export last ran. The read of the export's feature group runs on Trino as the member who asks, and an export answers only while no Trino row filter, column mask or denied column applies to its sources or its feature group, see When an export answers. |
| A query answered from the result cache | Earlier, by the same member's own query over the same feature group commits and Trino rules; nothing runs, see Result cache. |
The names, labels, descriptions and definitions of metrics, dimensions and saved queries come from the build, without Trino, for every member who may read the semantic project.
Prerequisites#
- An administrator turns the semantic layer on with the Helm value
global._hopsworks.semanticLayer.enabled: true, which installs the semantic gateway and adds Semantic Layer to the project sidebar, see Semantic Layer administration. While it is off, the semantic layer endpoints answer 404SEMANTIC_LAYER_DISABLED. - Data owners and data scientists of a project create, compile, promote and delete semantic projects, edit their definitions and tags, and query their metrics. Observers list and read semantic projects and their definitions; Trino gives an Observer no access to the project's feature groups, so their metric queries fail on Trino. Feature store restricted members are refused every semantic layer request with 403
SEMANTIC_LAYER_FORBIDDEN, since a semantic project reads every feature of a feature group. - Only data owners run metric exports by hand, and an export's job runs as the semantic project's owner, who must be a data owner.
- Roles and Permissions lists each action with the roles that may take it and the custom role permission it needs.
- An API key needs the
FEATURESTOREscope for semantic projects and builds. Everything the gateway answers, from listing metrics to running a query, also needs theTRINOscope, since the gateway connects to Trino as the key's user.