Result Cache#
Each replica of the semantic gateway keeps the results of metric queries in memory, and answers a repeated query from there instead of running it on Trino. A cached result is never served after a feature group it read has a new commit, or once the project can no longer read that feature group, within the bounds under Freshness.
What is cached#
The cache holds the results of the queries that run on Trino, from every surface of the gateway:
| Surface | Cached |
|---|---|
| REST | query, as JSON and as Arrow, and dimension-values: the Python client, the hops CLI, the query workbench in the UI and the agent tools. |
| Arrow Flight SQL | semantic_layer.query(...) and semantic_layer.dimension_values(...): dbt-sl-sdk queries, ADBC and JDBC, and SQL over the BI tables, which Superset charts send. |
Listings of metrics, dimensions, entities and saved queries, and the SQL of a query, over REST, Arrow Flight SQL and GraphQL, come from the build without running anything, so they are not cached.
A result is cached under a key made of:
- the surface, so a JSON and an Arrow answer to one query are two entries;
- the build that planned the query, so a newly served build starts with no entries;
- the request as the surface received it, and any SQL around a dbt JDBC call;
- the row cap of the answer;
- the caller's Trino user,
<project>__<user>, so two users never share an entry; - the revision of the cluster's Trino access rules, so a row filter, column mask or denied column that is added, changed or removed starts with no entries;
- the time of the latest commit of every feature group the query reads, the time spine included;
- for an answer read from an export, the commit of the export's feature group.
Each replica keeps its own cache: a repeated query that reaches another replica runs on Trino. A replica keeps up to 256 MiB of results, each for up to 300 seconds, and drops the least recently used first when it is full.
What invalidates a result#
Before it answers from the cache, the gateway asks Hopsworks, with the caller's credential, for the revision of the Trino access rules, and for each feature group the build reads: the time of its latest commit, and whether the project can still read it.
| Change | Effect |
|---|---|
A write that records a commit on a feature group, such as fg.insert(df) | The commit time changes the key, so the next query runs on Trino. |
| A newly served build, after a compile or a promote | The build id changes the key. |
| A feature group deleted, or no longer shared with the project | The query runs on Trino, which refuses it or reads what the project can still read. |
| A Trino row filter, column mask or denied column added, changed or removed | The rules revision changes the key, so the next query runs on Trino under the new rules. |
| A member removed from the project | The member's queries are refused. |
| 300 seconds since the result was cached | The entry is dropped. |
Freshness#
A replica reuses Hopsworks' answer on commits and access for 30 seconds per caller, semantic project and build, and a caller's session for 60 seconds.
| After | A cached result can still be served for up to |
|---|---|
| A new commit on a feature group the query reads | 30 seconds |
| A share revoked, or a feature group deleted | 30 seconds |
| A Trino access rule changed | 30 seconds |
| A member removed, or a role changed | 60 seconds |
A query that runs on Trino reads the data and the access of that moment, as without a cache.
What is never cached#
- A query over a feature group shared with the project with only some of its features: the shared features can change without a commit, so Hopsworks reports it as not readable for the cache, and every such query runs on Trino.
- A query over a feature group without commits, such as one without time travel or an external feature group: it has no commit time to key on.
- A result larger than 8 MiB: it streams to the client and is not kept.
- A query while Hopsworks cannot answer for commits, access or the Trino rules revision: it runs on Trino.
- The runs of metric exports, which always read the data as it is.
A JSON answer of the REST query endpoint says in its X-Semantic-Cache header whether the cache answered it:
| Value | Meaning |
|---|---|
hit | The gateway pod answered from its cache. |
miss | The query ran on Trino, and the pod keeps its result when it fits in the cache. |
bypass | The query ran on Trino without the cache: "cache": false, or a feature group whose version Hopsworks could not tell. |
Each gateway pod has its own cache, so with several pods a repeated query can be a miss on a pod that has not run it yet.
Skip the cache#
Each surface takes an option that runs the query on Trino and leaves the cache as it is:
| Client | Option |
|---|---|
| Python | sales.query(..., read_cache=False) |
| CLI | hops semantic query sales --metrics revenue --no-cache |
| REST | "cache": false in the query or dimension-values body |
| dbt's JDBC dialect | {{ semantic_layer.query(..., read_cache=False) }} |
| dbt-sl-sdk | client.query(..., read_cache=False) |
For administrators#
semantic-gateway value | Default | Meaning |
|---|---|---|
cache.maxBytes | 268435456 | The results a replica keeps, in bytes. |
cache.ttlSeconds | 300 | How long a result is served from the cache. |
cache.entryMaxBytes | 8388608 | The largest result the cache keeps. |
cache.dataVersionSeconds | 30 | How long a replica reuses Hopsworks' answer on commits, access and the Trino rules revision, which bounds how long a cached result outlives a commit, a revoked share or a changed rule. |
The gateway's metrics port reports the cache:
| Metric | Meaning |
|---|---|
semantic_gateway_cache_hits_total | Queries answered from the cache. |
semantic_gateway_cache_misses_total | Cacheable queries that ran on Trino. |
semantic_gateway_cache_evictions_total | Results dropped to make room. |
semantic_gateway_cache_bytes | The bytes the cache holds. |
semantic_gateway_data_version_lookups_total | Questions to Hopsworks on commits and access. |