## Lineage
Hopsworks tracks the lineage (or provenance) of ML assets automatically for you.
The lineage chain runs end to end: data source, feature group, feature view, training data, model, deployment.
This is what makes governance answerable: from a biased model you can trace back to the exact feature groups and data sources that fed it.
You can see what features are used in which feature view or training dataset, and what training dataset was used to train a given model.
For assets that are managed outside of Hopsworks, there is support for the explicit definition of lineage dependencies.
--8<-- "concepts/projects/search/provenance-lineage.html"
The lineage of an asset is shown in the Hopsworks UI, here the feature groups behind a feature view and the training data and models derived from it.
================================================================================
# CI/CD
Source: https://docs.hopsworks.ai/latest/concepts/projects/cicd/
# CI/CD Support
You can setup traditional development, staging, and production environment in Hopsworks using Projects.
A project enables you provide access control for the different environments - just like a GitHub repository, owners of projects can add and remove members of projects and assign different roles to project members - the "data owner" role can write to feature store, while a "data scientist" can only read from the feature store and create training data.
## Dev, Staging, Prod
You can create dev, staging, and prod projects - either on the same cluster, but mostly commonly, with production on its own cluster:
--8<-- "concepts/projects/cicd/dev-staging-prod.html"
## Versioning
Automated promotion across dev, staging, and prod relies on every ML asset being versioned.
Hopsworks versions feature groups, feature views, training data, and models, while deployments stay mutable behind the deployment API.
See [Versioning](../fs/feature_group/versioning.md) for what is versioned and how.
## Pytest for feature logic and feature pipeline tests
Pytest and Great Expectations can be used for testing feature pipelines.
Pytest is used to test feature logic and for end-to-end feature pipeline tests, while Great Expectations is used for data validation tests.
Here, we can see how a feature pipeline test uses sample data to compute features and validate they have been written successfully, first to a development feature store, and then they can be pushed to a staging feature store, before finally being promoted to production.
--8<-- "concepts/projects/cicd/pytest-feature-logic.html"
================================================================================
# Model Training
Source: https://docs.hopsworks.ai/latest/concepts/mlops/training/
# Model Training
A training pipeline is a program that orchestrates the training of a machine learning model, reading features and labels from the feature store as training data.
Hopsworks supports running model training pipelines on any Python environment, whether on an external Python client or on a Hopsworks cluster.
The outputs of a training pipeline are typically experiment results, including logs, and possibly a trained model.
You can plugin your own experimentation tracking platform or model registry, or you can use Hopsworks.
A training pipeline typically runs five steps: select a feature view and a training dataset version, train the model, evaluate it, validate it, and register it in the model registry if it passes.
## Evaluation and validation
Model evaluation and model validation are not the same thing.
Evaluation measures the model's performance on a held-out test set, using metrics such as accuracy or AUC.
Validation is a pass/fail gate: the model is run against evaluation data, including bias slices of the holdout built with feature-view filters and training helper columns (a column such as gender used to slice results but dropped before training), and only a model that passes is registered.
The output of validation is a model validation scorecard, and it is what decides whether the model reaches the registry.
## Training Pipelines on Hopsworks
If you train models with Hopsworks, you can setup CI/CD pipelines as shown below, where the experiments are tracked by Hopsworks, and any model created is published to a model registry.
Each project has its own private model registry, so when you are working in a development project, you typically publish models to your project's private development registry, and if all model validation tests pass, and the model performance is good enough, the same training pipeline can be submitted via a CI/CD pipeline (e.g., GitHub push request) to a staging project, and the same procedure can be repeated to push the training pipeline to a production project.
--8<-- "concepts/mlops/training/training-pipelines-on-hopsworks.html"
Hopsworks [Model Registry](registry.md) and [Model Serving](serving.md) capabilities can then be used to build a batch or online prediction service using the model.
================================================================================
# Model Registry
Source: https://docs.hopsworks.ai/latest/concepts/mlops/registry/
# Model Registry
Hopsworks Model Registry is designed with specific support for KServe and MLOps, through versioning.
It enables developers to publish, test, monitor, govern and share models for collaboration with other teams.
The model registry is where developers publish their models during the experimentation phase.
The model registry can also be used to share models with the team and stakeholders.
Like other project-based multi-tenant services in Hopsworks, a model registry is private to a project.
That means you can easily add a development, staging, and production model registry to a cluster, and implement CI/CD processes for transitioning a model from development to staging to production.
The model registry for KServe's capability are shown in the diagram below:
--8<-- "concepts/mlops/registry/model-registry.html"
The model registry centralizes model management, enabling models to be securely accessed and governed.
Models are more than just the model itself - the registry also stores sample data for testing, configuration information, provenance information, environment variables, links to the code used to generate the model, the model version, and tags/descriptions).
When you save a model, you can also save model metrics with the model, enabling users to understand, for example, performance of the model on test (or unseen) data.
## Model Package
A ML model consists of a number of different components in a model package:
- Model Input/Output Schema
- Model artifacts
- Model version information
- Model format (based on the ML framework used to train the model - e.g., .pkl or .tb files)
You can also optionally include in your packaged model:
- Sample data (used to test the model in KServe)
- The source notebook/program/experiment used to create the model
================================================================================
# Model Serving
Source: https://docs.hopsworks.ai/latest/concepts/mlops/serving/
# Model Serving
In Hopsworks, you can easily deploy models from the model registry using [KServe](https://kserve.github.io/website/latest/), the standard open-source framework for model serving on Kubernetes.
You rarely deploy just a model.
What you deploy is an online inference pipeline, of which the model is one part, alongside feature retrieval, transformations, and logging.
You can deploy models programmatically using [`Model.deploy`][hsml.model.Model.deploy] or via the UI.
A KServe model deployment can include the following components:
**`Predictor (KServe component)`**
: A predictor runs a model server (Python, TensorFlow Serving, or vLLM) that loads a trained model, handles inference requests and returns predictions.
**`Transformer (KServe component)`**
: A ^^pre-processing^^ and ^^post-processing^^ component that can transform model inputs before predictions are made, and predictions before these are delivered back to the client.
Not available for vLLM deployments.
**`Inference Logger`**
: Hopsworks logs inputs and outputs of transformers and predictors to a ^^Kafka topic^^ that is part of the same project as the model.
This is for storing inference requests and responses for later consumption and analysis, and is separate from the feature logging that powers [Model Monitoring](model_monitoring.md).
Not available for vLLM deployments.
**`Inference Batcher`**
: Inference requests can be batched to improve throughput (at the cost of slightly higher latency).
**`Istio Model Endpoint`**
: You can publish a model over REST(HTTP) or gRPC using a Hopsworks API key, accessible via **path-based routing** through Istio.
API keys have scopes to ensure the principle of least privilege access control to resources managed by Hopsworks.
For more details on path-based routing of requests through Istio, see [REST API Guide](../../user_guides/mlops/serving/rest-api.md).
!!! warning "Host-based routing"
The Istio Model Endpoint supports host-based routing for inference requests; however, this approach is considered legacy.
Path-based routing is recommended for new deployments.
Models deployed on KServe in Hopsworks can be easily integrated with the Hopsworks Feature Store using either a Transformer or Predictor Python script, that builds the predictor's input feature vector using the application input and pre-computed features from the Feature Store.
--8<-- "concepts/mlops/serving/model-serving.html"
## Deployment API
The deployment API is the interface to the online inference pipeline that clients send prediction requests to.
It is the deployment API, not the model signature, that clients should version against.
The model signature (the input and output schema of the model) changes whenever you retrain with a different set of features, so coupling clients to it turns every model update into a breaking change.
The deployment API is a stable contract that can stay the same across model versions.
A client request to the deployment API carries two kinds of parameter:
- **serving keys**: the entity IDs used to retrieve pre-computed features from the feature store.
- **request parameters**: values known only at request time, sent in the request and used to build the feature vector or to compute on-demand features.
Because clients depend on it, a deployment API should carry an SLO, typically a p99 latency target for online predictions.
## Testing model deployments
Two release-safety mechanisms are often confused, because they test different things.
A blue/green test tests the correctness and performance of the model deployment directly, running the new deployment alongside the old one so clients can be switched over with no risk.
An A/B test does not test the deployment; it tests the model's effect on the application, measured against an application KPI, to decide whether the new model actually makes the product better.
!!! info "Model Serving Guide"
More information can be found in the [Model Serving guide](../../user_guides/mlops/serving/index.md).
!!! tip "Python deployments"
For deploying Python scripts without a model artifact, see the [Python Deployments](../../user_guides/projects/python-deployment/python-deployment.md) page.
================================================================================
# Model Monitoring
Source: https://docs.hopsworks.ai/latest/concepts/mlops/model_monitoring/
# Model Monitoring
Model monitoring lets you track how a deployed model behaves in production by comparing the data it serves against the data it was trained on.
When a model runs in production, the statistical properties of its inputs and predictions can drift away from those of the training data.
This degrades model quality silently, without any error being raised.
Model monitoring detects this drift early so you can decide whether to retrain the model.
## How it works
Model monitoring builds on two existing Hopsworks capabilities:
- **Feature logging**: a model deployment logs the features it serves and its predictions to the feature view's logging feature group through the Feature View logging APIs.
See the [Feature Logging guide](../../user_guides/fs/feature_view/feature_logging.md).
- **Feature monitoring**: Hopsworks computes statistics over windows of feature data and compares them against a reference, optionally raising alerts on significant drift.
See the [Feature Monitoring concept](../fs/feature_group/feature_monitoring.md).
??? note "Log untransformed and transformed features"
Log both the untransformed and the transformed feature values.
Untransformed features drive feature monitoring and debugging, since drift is easiest to read on the raw values.
Transformed features drive model monitoring and SHAP explainability, since those are the values the model actually sees.
!!! info "Feature logging vs. the inference logger"
Hopsworks provides two separate inference logging mechanisms.
The [inference logger](../../user_guides/mlops/serving/inference-logger.md) stores the model inputs and predictions from inference requests and responses into Kafka, for later consumption and analysis.
[Feature logging](../../user_guides/fs/feature_view/feature_logging.md) supports more fine-grained logging of inference logs and features, enabling feature monitoring and model monitoring.
Model monitoring relies on feature logging, not on the inference logger.
A model monitoring configuration is a feature monitoring configuration over the logging feature group, filtered to a single model and version.
The detection window covers the recently served inference data, and the reference defaults to the training dataset version that was used to train that model.
By comparing the two, on a scalar metric or on the whole feature distribution, Hopsworks detects feature drift over time.
This comparison detects drift, not skew: offline-online feature skew is a difference in the transformation code between the offline and inference pipelines, so it is invisible to a distribution comparison and is prevented, not monitored.
Feature drift is one kind of drift among several.
Concept drift, in particular, is not detected by comparing distributions: you detect it by comparing the actual outcomes against the model's past predictions, once those outcomes are known.
--8<-- "concepts/mlops/model_monitoring/how-it-works.html"
## Where to configure it
Because monitoring is anchored on the feature view that backs the model, you can configure model monitoring from whichever entity is most convenient:
- a **model deployment**, when operating a model in production.
- a **model** in the model registry.
- a **feature view**, when working directly with the feature data.
All three resolve to the same underlying configuration.
!!! info "Model Monitoring Guide"
More information can be found in the [Model Monitoring guide](../../user_guides/mlops/model_monitoring/index.md).
================================================================================
# Vector Index
Source: https://docs.hopsworks.ai/latest/concepts/mlops/opensearch/
# Vector Index
A vector index stores embeddings so you can retrieve the items most similar to a query vector, the retrieval half of a recommender or a RAG system.
In Hopsworks, a vector index is a property of an online-enabled feature group: a feature group with an embedding column can be indexed for similarity search, alongside its online and offline stores.
The vector index is backed by OpenSearch, included as a multi-tenant service in projects.
OpenSearch provides the index through its k-NN plugin, which supports several engines for embedding indexes.
Hopsworks creates its indexes on the FAISS engine, which is the default from Hopsworks 5.1.
Earlier releases used the nmslib engine, which OpenSearch has deprecated and which does not accept the filter that Hopsworks 5.1 and later send inside the nearest-neighbor query, so the upgrade to Hopsworks 5.2 recreates those indexes on FAISS.
The [OpenSearch upgrade guide][upgrading-opensearch] describes what that upgrade involves.
Through Hopsworks, OpenSearch also provides enterprise capabilities, including authentication and access control to indexes (an index can be private to a Hopsworks project), filtering, scalability, high availability, and disaster recovery support.
To learn how OpenSearch powers vector similarity search in Hopsworks, you can see [this guide](../../user_guides/fs/vector_similarity_search.md).
--8<-- "concepts/mlops/opensearch/vector-index.html"
================================================================================
# Development Inside Hopsworks
Source: https://docs.hopsworks.ai/latest/concepts/dev/inside/
# Development Inside Hopsworks
Hopsworks provides a complete self-service development environment for feature engineering and model training.
You can develop programs as Jupyter notebooks or jobs, customize the bundled FTI (feature, training and inference pipeline) python environments, you can manage your source code with Git, and you can orchestrate jobs with Airflow.
A browser terminal runs inside the project with the Hopsworks CLI and coding agents preinstalled, and the Wizard uses it to build a system end to end from a few choices.
--8<-- "concepts/dev/inside/development-inside-hopsworks.html"
## Jupyter Notebooks
Hopsworks provides a Jupyter notebook development environment for programs written in Python, Spark, and SparkSQL.
You can also develop in your IDE (PyCharm, IntelliJ, etc), test locally, and then run your programs as Jobs in Hopsworks.
Jupyter notebooks can also be run as Jobs.
## Source Code Control
Hopsworks provides source code control support using Git (GitHub, GitLab or BitBucket).
You can securely check out code into your project and commit and push updates to your code to your source code repository.
## FTI Pipeline Environments
Hopsworks postulates that building ML systems following the FTI pipeline architecture is best practice.
This architecture consists of three independently developed and operated ML pipelines:
- Feature pipeline: takes as input raw data that it transforms into features (and labels)
- Training pipeline: takes as input features (and labels) and outputs a trained model
- Inference pipeline: takes new feature data and a trained model and makes predictions
In order to facilitate the development of these pipelines Hopsworks bundles several python environments containing necessary dependencies.
Each of these environments may then also be customized further by cloning it and installing additional dependencies from PyPi, Wheel files, GitHub repos or a custom Dockerfile.
Internal compute such as Jobs and Jupyter is run in one of these environments and changes are applied transparently when you install new libraries using our APIs.
That is, there is no need to write a Dockerfile, users install libraries directly in one or more of the environments.
You can setup custom development and production environments by creating separate projects or creating multiple clones of an environment within the same project.
## Jobs
In Hopsworks, a Job is a schedulable program that is allocated compute and memory resources.
You can run a Job in Hopsworks:
- From the UI
- Programmatically with the Hopsworks SDK (Python, Java) or REST API
- From Airflow programs (either inside our outside Hopsworks)
- From your IDE using a plugin ([PyCharm/IntelliJ plugin](https://plugins.jetbrains.com/plugin/15537-hopsworks))
## Orchestration
Airflow comes out-of-the box with Hopsworks, but you can also use an external Airflow cluster (with the Hopsworks Job operator) if you have one.
Airflow can be used to schedule the execution of Jobs, individually or as part of Airflow DAGs.
## Terminal { #inside-terminal }
Every project has a browser terminal: a shell running in a pod under your project user, with your HopsFS home mounted, and `hops`, `git`, Claude Code and Codex preinstalled and already connected to the project.
It is the fastest way to work with a project from inside Hopsworks, and the seat the Wizard drives.
See the [Terminal guide][terminal] and the [Hopsworks CLI guide][hopsworks-cli].
## Wizard { #inside-wizard }
The Wizard asks what you want to build, where the data comes from and what to predict, then writes a kickoff prompt and hands it to Claude in the terminal, which builds the feature pipeline, the model and the dashboard with `hops`.
See the [Wizard guide][wizard].
================================================================================
# Development Outside Hopsworks
Source: https://docs.hopsworks.ai/latest/concepts/dev/outside/
# Development Outside Hopsworks
You can write programs that use Hopsworks in any [Python, Spark, or PySpark environment](../../user_guides/integrations/index.md).
Hopsworks also supports running SQL queries to compute features in external data warehouses.
The Feature Store can also be queried with SQL.
There is REST API for Hopsworks that can be used with a valid API key, generated in Hopsworks.
However, it is often easier to develop your programs against the Hopsworks SDK, available in Python and Java/Scala, which covers the feature store, the model registry and model serving.
The same library ships the `hops` command line, so a shell, a CI pipeline or a coding agent on your machine can read and write the project with the same API key; see the [Hopsworks CLI guide][hopsworks-cli].
--8<-- "concepts/dev/outside/development-outside-hopsworks.html"
================================================================================
# BI Tools
Source: https://docs.hopsworks.ai/latest/concepts/mlops/bi_tools/
# BI Tools
Feature groups have well-defined schemas and live in two stores, so any BI tool that speaks SQL can analyze features and build reports on them.
- The offline store is queried through the [Query Engine](../../user_guides/projects/trino/query_engine.md), Trino, with one catalog per table format (`delta`, `iceberg`, `hudi`).
Any tool with a Trino connector (JDBC or ODBC) can read it.
- The online store, RonDB, is queried over the MySQL protocol, so any tool with a MySQL connector can read the latest feature values.
Hopsworks bundles [Apache Superset](https://superset.apache.org/) as a project service, already connected to the project's Trino catalogs.
Dashboards live inside the project and follow its access control.
- :material-console:{ .lg .middle } **Start here**
---
Install the client and authenticate once.
Every guide in this section runs from the same session.
```bash
uv venv && source .venv/bin/activate
uv pip install "hopsworks[python]"
hops setup
```
[Client installation](client_installation/index.md) · [Create a project](projects/project/create_project.md) · [Create a feature group](fs/feature_group/create.md)
:material-database:{ .hops-role-ico } Feature Store
{ .hops-role-cap }
- [Feature groups](fs/feature_group/index.md)
Write features from a DataFrame, validate them, keep statistics.
- [Feature views](fs/feature_view/index.md)
Read training data, batch data and online feature vectors.
- [Data sources](fs/data_source/index.md)
Connect warehouses, object stores and databases as inputs.
- [Feature monitoring](fs/feature_monitoring/index.md)
Watch statistics over time and compare them to a reference.
- [Transformations and integrations](fs/transformation_functions.md)
Model-independent transformations, compute engines, external clients.
:material-rocket-launch-outline:{ .hops-role-ico } MLOps
{ .hops-role-cap }
- [Model registry](mlops/registry/index.md)
Register models with metrics, schema and evaluation artifacts.
- [Model serving](mlops/serving/index.md)
Deploy a model with a predictor, transformer, logging and autoscaling.
- [Model monitoring](mlops/model_monitoring/index.md)
Compare inference data against training data on a schedule.
- [Agents](agents/index.md)
Run agent tasks as jobs or serve interactive agents.
:material-folder-outline:{ .hops-role-ico } Projects and compute
{ .hops-role-cap }
- [Projects](projects/index.md)
Sign in, create a project, manage members, secrets, keys and alerts.
- [Compute](compute/index.md)
Jupyter, the terminal, jobs, Airflow and Python environments.
- [Analytics](analytics/index.md)
SQL over the offline store with Trino, dashboards in Superset.
:material-wrench-outline:{ .hops-role-ico } Platform
{ .hops-role-cap }
- [Clients](client_installation/index.md)
Python and Java libraries for your own environment, and the `hops` CLI.
- [Setup and administration](../setup_installation/index.md)
Install on a cloud or on-prem, manage users and operations.
- [Migration 3.x to 4.0](migration/40_migration.md)
What changed and how to move.
- :material-database-plus-outline:{ .lg .middle } **Start here**
---
Create a feature group from a DataFrame and insert it.
Everything else in this section builds on a feature group that exists.
```python
fg = fs.get_or_create_feature_group(
name="transactions",
version=1,
primary_key=["tid"],
event_time="datetime",
online_enabled=True,
)
fg.insert(df)
```
[Create a feature group](feature_group/create.md) · [Create a feature view](feature_view/overview.md) · [Training data](feature_view/training-data.md)
:material-database-import-outline:{ .hops-role-ico } Write
{ .hops-role-cap }
- [Data sources](data_source/index.md)
Register warehouses, object stores and databases to read from and write to.
- [Feature groups](feature_group/index.md)
Create, insert, evolve the schema, set time to live, deprecate.
- [External and spine groups](feature_group/create_external.md)
Point at data that stays where it is, or supply keys and labels without storing them.
- [Ingest with dltHub](feature_group/ingest_with_dlthub.md)
Load from hundreds of sources through dlt pipelines.
:material-check-decagram-outline:{ .hops-role-ico } Trust
{ .hops-role-cap }
- [Statistics](feature_group/statistics.md)
Descriptive statistics on every insert, configurable per group.
- [Data validation](feature_group/data_validation.md)
Great Expectations suites run on insert, with a policy on failure.
- [Feature monitoring](feature_monitoring/index.md)
Scheduled statistics and drift detection against a reference window.
- [Notifications and observability](feature_group/notification.md)
Change notifications and online ingestion status.
:material-database-export-outline:{ .hops-role-ico } Read
{ .hops-role-cap }
- [Feature views](feature_view/index.md)
Select features across groups and read them the same way for training and inference.
- [Training data](feature_view/training-data.md)
Materialise splits as files or read them straight into memory.
- [Batch and online reads](feature_view/batch-data.md)
Batch inference data by time range, single vectors from the online store.
- [Feature server](feature_view/feature-server.md)
Serve feature vectors over REST without the Python client.
:material-function-variant:{ .hops-role-ico } Transform and run
{ .hops-role-cap }
- [Transformation functions](transformation_functions.md)
Model-independent functions applied on write, model-dependent on read.
- [Compute engines](compute_engines.md)
Which operations run on Python, Spark or Flink.
- [Client integrations](../integrations/index.md)
Databricks, SageMaker, EMR, Azure ML, Flink, Beam and more.
- [Vector similarity search](vector_similarity_search.md)
Embeddings in a feature group, nearest-neighbour queries.
- :simple-snowflake:{ .lg .middle style="color:#29B5E8" } **Snowflake**
---
Query Snowflake databases and tables using SQL.
[:octicons-arrow-right-24: Configure](creation/snowflake.md)
- :simple-apachekafka:{ .lg .middle } **Kafka**
---
Read from a Kafka cluster into a Spark Structured Streaming Dataframe.
[:octicons-arrow-right-24: Configure](creation/kafka.md)
- :simple-sap:{ .lg .middle style="color:#0FAAFF" } **SAP HANA**
---
Query SAP HANA tenant databases using SQL.
[:octicons-arrow-right-24: Configure][data-source-sap-hana]
- :material-server-network:{ .lg .middle style="color:var(--hops-accent-text)" } **Hopsworks**
---
Read and ingest feature groups from another Hopsworks cluster.
[:octicons-arrow-right-24: Configure][data-source-hopsworks]
- :material-database:{ .lg .middle style="color:var(--hops-accent-text)" } **JDBC**
---
Connect to any JDBC compatible database and query it using SQL.
[:octicons-arrow-right-24: Configure](creation/jdbc.md)
- :material-api:{ .lg .middle style="color:var(--hops-accent-text)" } **REST API**
---
Connect to external HTTP APIs with configurable headers and authentication.
[:octicons-arrow-right-24: Configure](creation/rest_api.md)
- :material-chart-box-outline:{ .lg .middle style="color:var(--hops-accent-text)" } **CRM, Sales & Analytics**
---
Connect to supported CRM, sales, and analytics platforms.
[:octicons-arrow-right-24: Configure](creation/crm_sales_analytics.md)
- :material-folder-network-outline:{ .lg .middle style="color:var(--hops-accent-text)" } **HopsFS**
---
Connect and read from directories of Hopsworks' internal file system.
[:octicons-arrow-right-24: Configure](creation/hopsfs.md)
## AWS
For AWS the following storage systems are supported:
- :fontawesome-brands-aws:{ .lg .middle style="color:#FF9900" } **S3**
---
Read file-based storage in S3 such as parquet or CSV.
[:octicons-arrow-right-24: Configure](creation/s3.md)
- :fontawesome-brands-aws:{ .lg .middle style="color:#FF9900" } **AWS Glue**
---
Integrate with the Glue Data Catalog over S3, for Iceberg, Delta, Hudi and plain files.
[:octicons-arrow-right-24: Configure](creation/glue.md)
- :fontawesome-brands-aws:{ .lg .middle style="color:#FF9900" } **Redshift**
---
Query Redshift databases and tables using SQL.
[:octicons-arrow-right-24: Configure](creation/redshift.md)
- :fontawesome-brands-aws:{ .lg .middle style="color:#FF9900" } **RDS (SQL)**
---
Query the Amazon Relational Database Service using SQL.
[:octicons-arrow-right-24: Configure](creation/sql.md)
## Azure
For Azure the following storage systems are supported:
- :material-microsoft-azure:{ .lg .middle style="color:#0078D4" } **ADLS**
---
Read file-based storage in ADLS such as parquet or CSV.
[:octicons-arrow-right-24: Configure](creation/adls.md)
## GCP
For GCP the following storage systems are supported:
- :simple-googlebigquery:{ .lg .middle style="color:#4285F4" } **BigQuery**
---
Query BigQuery databases and tables using SQL.
[:octicons-arrow-right-24: Configure](creation/bigquery.md)
- :simple-googlecloudstorage:{ .lg .middle style="color:#4285F4" } **GCS**
---
Read file-based storage in Google Cloud Storage such as parquet or CSV.
[:octicons-arrow-right-24: Configure](creation/gcs.md)
## Databricks (AWS only)
For Databricks **on AWS** the following storage systems are supported:
- :simple-databricks:{ .lg .middle style="color:#FF3621" } **Unity Catalog**
---
Browse catalogs, schemas, and Delta tables, and mount them as external feature groups.
[:octicons-arrow-right-24: Configure](creation/unity_catalog.md)
Databricks on Azure and Databricks on GCP are not supported yet. See the [Unity Catalog guide](creation/unity_catalog.md) for the specific reasons and the status of follow-up work.
## Next Steps
Move on to the [Configuration and Creation Guides](creation/jdbc.md) to learn how to set up a data source.
================================================================================
# JDBC
Source: https://docs.hopsworks.ai/latest/user_guides/fs/data_source/creation/jdbc/
# How-To set up a JDBC Data Source
## Introduction
JDBC is an API provided by many database systems.
Using JDBC connections one can query and update data in a database, usually oriented towards relational databases.
Examples of databases you can connect to using JDBC are MySQL, Postgres, Oracle, DB2, MongoDB or Microsoft SQLServer.
In this guide, you will configure a Data Source in Hopsworks to save all the authentication information needed in order to set up a JDBC connection to your database of choice.
When you're finished, you'll be able to query the database using Spark through Hopsworks APIs.
!!! note
Currently, it is only possible to create data sources in the Hopsworks UI.
You cannot create a data source programmatically.
## Prerequisites
Before you begin this guide you'll need to retrieve the following information from your JDBC compatible database:
- **JDBC Connection URL:** Consult the documentation of your target database to determine the correct JDBC URL and parameters.
As an example, for MySQL the URL could be:
```plaintext
jdbc:mysql://10.0.2.15:3306/[databaseName]?useSSL=false&allowPublicKeyRetrieval=true
```
- **Username and Password:** Typically, you will need to add username and password in your JDBC URL or as key/value parameters.
So make sure you have retrieved a username and password with the suitable permissions for the database and table you want to query.
## Creation in the UI
### Step 1: Set up new Data Source
Head to the `Data Sources` view on Hopsworks (1) and click `New data source` (2).
The `Add a data source` catalog lists the available sources, grouped under `Object storage`, `Data warehouse`, `Database`, `Streaming` and `API & SaaS`.
Pick the `JDBC` card to open the creation form.
