How-To set up an SQL Data Source#

Introduction#

The SQL Data Source connects Hopsworks to a Relational Database Service. Supported database types are MySQL, PostgreSQL, Oracle, and Microsoft SQL Server (including Azure SQL Database). Using this connector, you can query and update data in your relational database from Hopsworks.

In this guide, you will configure a Data Source in Hopsworks to securely store the authentication information needed to set up a connection to your database instance. When you're finished, you'll be able to query your SQL database using Hopsworks APIs.

Prerequisites#

Before you begin, ensure you have the following information from your database instance:

• Host: The endpoint for your database instance.

  Example from AWS: 1. Go to the AWS Console → Aurora and RDS 2. Click on your DB instance. 3. Under Connectivity & security, you'll find the endpoint, e.g.: mydb.abcdefg1234.us-west-2.rds.amazonaws.com

• Database: The name of the database to connect to. For Oracle, this is the service name (e.g. ORCL or a TNS alias). For SQL Server, this is the database the login connects to by default; every database the login can access can still be browsed.

• Port: The port to connect to (e.g. 3306 for MySQL, 5432 for PostgreSQL, 1521 for Oracle, 1433 for SQL Server).

• Username and Password: A username and password with the necessary permissions to access the required tables.

Optional: Oracle Wallet for mTLS Authentication#

If your Oracle database requires mutual TLS (mTLS) authentication, which is common with Oracle Autonomous Database and Oracle Cloud, you will also need:

• Wallet file: A .zip file containing the wallet credentials (e.g. cwallet.sso, tnsnames.ora, sqlnet.ora).
• Wallet password: The password for the wallet, if using a PKCS12 wallet (ewallet.p12). Auto-login wallets (cwallet.sso) do not require a password.

Creation in the UI#

Step 1: Set up a new Data Source#

Head to the Data Sources view on Hopsworks (1) and click New data source (2). The Add a data source catalog lists the available sources, grouped under Object storage, Data warehouse, Database, Streaming and API & SaaS. Pick your database's card under Database, such as MySQL, PostgreSQL, Oracle or SQL Server, to open the creation form.

Step 2: Enter SQL Settings#

Enter the details for your database. Start by giving the connector a name and an optional description.

1. The form opens for the database whose card you picked in the gallery (MySQL, PostgreSQL, Oracle, SQL Server and the others), which sets the database type. Click Change source to pick a different one.
2. Enter the host endpoint. Leave it empty when using an Oracle wallet: the wallet supplies the connection details, and the database field names the TNS alias to use.
3. Enter the database name (service name for Oracle).
4. Specify the port.
5. For Oracle, choose under Credentials whether they are Stored with the data source (the default) or Provided by each user. With Provided by each user the username, password and wallet fields disappear and every member adds their own later, see provided credentials.
6. For Stored with the data source, provide the username and password.
7. For Stored with the data source on a database behind mTLS, upload the wallet zip file and provide the wallet password (if required).
8. Click on "Save Credentials".

Oracle-Specific Notes#

The generic read, external feature group, and training data workflows are covered in the usage guide for data sources. The following notes apply only to Oracle.

JDBC driver on the Spark classpath#

The Oracle JDBC driver JAR (e.g. ojdbc11.jar) must be available on the Spark classpath. Upload it via the Jupyter configuration or Job configuration in Additional Jars. The MySQL, PostgreSQL and SQL Server drivers are included in Hopsworks by default.

Spark JDBC limitations#

Python engine#

The Python engine reads Oracle via the Hopsworks Arrow Flight service, which handles the database connection server-side. No JDBC driver or wallet files are needed on the client, and the Spark JDBC limitations above do not apply.

SQL Server-Specific Notes#

Host and port#

The Host is a host name or an IPv4 address. For a named instance, enter the server's host name and the port the instance listens on, not host\instance: the JDBC driver would connect to the port and the Python engine to the instance, which can be different servers. For an IPv6-only server, use a host name that resolves to it, because the JDBC driver does not accept an IPv6 address in its URL.

Databases, schemas and tables#

SQL Server names a table database.schema.table. When you browse a SQL Server data source, the database is the top level and the schema (often dbo) is the group, so an external feature group over a browsed table reads [database].[schema].[table]. A hand-written query can use the same three-part name, which lets one data source read any database on the server that the login can access. Write such a query as a plain SELECT: Hopsworks reads it as a derived table, where SQL Server does not accept a WITH clause.

Azure SQL Database does not support three-part names across databases. Create one data source per Azure SQL database, with that database in the Database field.

Encryption and certificates#

Every connection is encrypted, and the server certificate is validated against the public certificate authorities, including its host name. For a server with a self-signed certificate, such as a default SQL Server installation, add the argument trustServerCertificate with the value true. This applies to Spark, the query engine, the Python engine and DLTHub ingestion alike. Spark and the query engine use Microsoft's JDBC driver, so other JDBC connection properties can be added as arguments the same way; the Python engine and DLTHub ingestion read only trustServerCertificate. The argument encrypt=false therefore turns off encryption for Spark and the query engine only: the Python engine and DLTHub ingestion always encrypt. The arguments cannot replace the host, port or database (serverName, portNumber, databaseName); the data source's own fields set those.

Provided credentials for Oracle#

A data source has a credentials mode. The Data owner chooses it when creating the data source, and it cannot be changed afterwards, because a switch would invalidate every member's setup at once.

PROVIDED is available for the Oracle database type of the SQL data source. A provided data source stores the host or the wallet alias, the port and the service name, and nothing else. Creating one with a username, a password or a wallet is refused with INVALID_CREDENTIALS_MODE_ARGS (270339). The Python client raises the error itself, before sending the request, when a new provided SqlConnector carries any of them; each member adds credentials with set_credentials() once the data source exists. Reads run with the credentials of the member who makes them, so a Data owner who creates a provided data source adds their own credentials before browsing its tables or mounting feature groups; the UI opens the dialog for them right after the data source is saved.

hops datasource create sql oracle_sales --database-type ORACLE --host db.example.com --port 1521 --database ORCL --credentials-mode provided

Adding your credentials#

Each member adds their credentials once per data source. Data owners and Data scientists can add credentials; Feature store restricted members and Observers cannot. For a data source in a feature store shared with your project, you add them in the project that owns the data source, see feature groups from a shared feature store. Hopsworks tests them by opening a connection and running SELECT 1 FROM DUAL, and saves nothing when the database rejects them. Validate and save are rate limited to 5 calls per minute per user and data source; a call over the limit returns HTTP 429 with DATA_SOURCE_CREDENTIALS_RATE_LIMITED (270343).

In the UI, open Data Sources, find the data source in the list and pick My credentials from its row actions. The Credentials column of the list shows Shared, or Provided with Yours: set, Yours: missing or Yours: incomplete. Yours: incomplete means your binding references an account entry that no longer exists, see where your credentials are stored. In the dialog, type a username and a password, or pick an env var and a secret you already keep in your account. For a database behind mTLS, upload the wallet zip and give its password. The upload is staged as a new file, see replacing a wallet. Click Validate, then Save. Save stays disabled until Validate has passed for the values in the dialog, and editing a field disables it again. Remove my credentials in the same dialog deletes your binding, your wallet file and the account entries saved under their default names.

From Python, the same operations are methods on the SQL connector:

import hopsworks


project = hopsworks.login()
fs = project.get_feature_store()
sc = fs.get_data_source("oracle_sales").storage_connector

if sc.validate_credentials(user="SCOTT", password="..."):
    sc.set_credentials(user="SCOTT", password="...")

# a database behind mTLS also takes a wallet, uploaded to a new file in your home
wallet = f"{project.home_path}/.datasources/oracle_sales/wallet-7f3a9c.zip"
sc.set_credentials(
    user="SCOTT", password="...", wallet_path=wallet, wallet_password="..."
)

# reuse a secret you already keep in your account
sc.set_credentials(user="SCOTT", password_secret="my_oracle_pwd")

sc.get_credentials()
sc.delete_credentials()

set_credentials raises a FeatureStoreException carrying the database's own error when the credentials are rejected. get_credentials returns the names of the account entries your binding references, never their values.

From the CLI, hops datasource credentials has set, validate, show and delete. --password - and --wallet-password - read the value from stdin, one of them per call, and without either flag the HOPSWORKS_DS_CREDENTIALS_PASSWORD and HOPSWORKS_DS_CREDENTIALS_WALLET_PASSWORD environment variables are read. --wallet takes a local zip and stages it as a new wallet file for you.

HOPSWORKS_DS_CREDENTIALS_WALLET_PASSWORD=... hops datasource credentials set oracle_sales --user SCOTT --password - --wallet ./Wallet_ORCL.zip
hops datasource credentials show oracle_sales --json

{
  "status": "VALID",
  "username_env_var": "DS_ORACLE_SALES_67_USER",
  "password_secret_name": "ds_oracle_sales_67_password",
  "wallet_path": "/Projects/sales/Users/scott/.datasources/oracle_sales/wallet-7f3a9c.zip",
  "wallet_password_secret_name": "ds_oracle_sales_67_wallet_password",
  "validated_at": "2026-10-07T09:12:44Z"
}

hops datasource list shows the mode in its CREDENTIALS column: shared, provided: yours set, provided: missing or provided: incomplete.

Where your credentials are stored#

<NAME> is the data source name upper-cased with every character outside A-Z, 0-9 and _ replaced by _; <name> is the same in lower case. The feature store id is part of the default names because your account entries are not scoped to a project, so two data sources with the same name in different feature stores would otherwise share one entry. The names are editable in the dialog and through the user_env_var, password_secret and wallet_password_secret arguments, so an existing entry can be reused instead of creating one. Your home directory is readable by you alone, so the wallet is too.

The data source keeps a reference to these entries, not a copy. Fetching or listing a provided data source returns the status of your binding and the names of your env var and secrets, never your username or password. The Python client reads the values from your own account entries when it needs them. Deleting the secret or the env var from your account leaves the binding INCOMPLETE, and the data source refuses to read until you set your credentials again. Removing your credentials deletes the binding, the wallet file, and the env var and secrets that carry the default names above. The same deletion happens for a member who is removed from the project, and for every member when the data source is deleted. Entries saved under names of your own stay in your account, since you may use them elsewhere.

Replacing a wallet#

Every wallet upload, from the dialog or with hops datasource credentials set --wallet, is staged as a new file, <home>/.datasources/<name>/wallet-<random>.zip, so it never overwrites the wallet your binding points at. A wallet that fails validation, or a save the database rejects, leaves your binding and its wallet unchanged. When the save succeeds, the binding points at the new file and Hopsworks deletes the wallet it pointed at before. From Python, upload the wallet under a name of your own that is not in use, and pass that path as wallet_path.

Runtime variables#

Every runtime you start in the project (a job, a Jupyter server, a terminal, an app or a Ray cluster) receives your credentials for each provided data source you have set them for. A job run receives them through a Kubernetes Secret created for that run, so the values are not in the pod spec that other project members can read. Code inside the runtime reads the same variables either way:

import os


user = os.environ["HOPS_DS_ORACLE_SALES_USER"]
password = os.environ["HOPS_DS_ORACLE_SALES_PASSWORD"]
wallet = os.environ.get("HOPS_DS_ORACLE_SALES_WALLET_PATH")

Model deployments do not receive these variables. A deployment is shared by the project and serves every caller with the same environment, so one member's credentials would answer everyone's requests.

The HOPS_ prefix is reserved for the platform, so an account env var cannot shadow these. Two provided data sources in one feature store cannot map to the same <NAME>; the second one is refused at creation. The Hopsworks client reads these variables itself inside a runtime, so a read() through the data source needs no further configuration there. It uses them only for the data source whose id is in HOPS_DS_<NAME>_CONNECTOR_ID. A data source with the same name in another feature store, such as one shared with the project, maps to the same variable names, so for it the client reads your values from your own account entries.

The /hopsfs path in HOPS_DS_<NAME>_WALLET_PATH is not mounted in Spark. When that path does not exist, the client downloads your wallet from HopsFS to the Spark driver instead, so a Spark read needs no further configuration either.

Access to mounted tables#

Only Data owners mount external feature groups, and a Data owner's database user may read tables that another member's may not. Hopsworks therefore keeps, per member and per external feature group on a provided data source, whether that member can read the backing table or query:

A check runs on every feature group of the data source in these cases:

• For you, when you set or remove your credentials.
• For every member with credentials, when a Data owner mounts a new feature group.
• For a member who joins the project.
• Every 10 minutes, in a reconcile pass that matches the stored states with the current members and credentials and queues a check for any event that was missed.

Checks run in the background in batches, and each one may take up to a minute, so a state usually settles within a few minutes. The state is shown in the feature group catalog and in the feature group's Data Source submenu. It informs you and does not gate reads: a read in NO_ACCESS still goes to Oracle and fails there with the database error.

Test Connection#

Test Connection, on the Data Source submenu of a feature group and on greyed-out feature groups in the catalog, runs the check for you now, as you, with a limit of one row, and stores the result. It takes up to a minute. Use it after a privilege was granted on the database side, since Hopsworks learns about grants only through the events above and this button. On a feature group whose data source has shared credentials it tests the shared credentials and stores nothing.

Trino catalogs#

A provided Oracle data source can back a Trino catalog, and each query through the catalog runs as the member who sends it. Such a catalog is always a private catalog of the member who creates it, named _<username>__<name> and available in every project they belong to. Each member who wants to query the data source through Trino creates their own, as a Data Owner or a Data Scientist. A request for a project catalog from a provided data source is refused with CREDENTIALS_MODE_NOT_SUPPORTED (270337), and a catalog on a provided data source cannot be shared, so only its owner queries it. The catalog stores no login. Its properties name the Trino extra credentials that carry the username and password with each query:

user-credential-name=hops_ds_<data source id>_user
password-credential-name=hops_ds_<data source id>_password
oracle.connection-pool.enabled=false

The connection pool is off because a pooled connection keeps the login of the member who opened it. Metadata caching is refused for the same reason: a cached table list would show one member what another member's database user can see.

Which clients send the extra credentials:

A client that is not listed fetches your values from GET /hopsworks-api/api/project/<project id>/trino/extra-credentials, with an API key that has the TRINO scope, and sends each entry as a Trino extra credential, for example with the CLI's --extra-credential hops_ds_<data source id>_user=<username> or the JDBC extraCredentials property. A member who has not set credentials for the data source has nothing to send, and the Oracle login fails.

Creating the catalog requires you to have set your own credentials for the data source. For a database behind mTLS, the catalog's wallet (TNS_ADMIN) is built from your wallet into a mountable secret of your account; it only secures the transport, and every query still logs in with your username and password. The wallet follows your credentials: it is rebuilt when you save a new wallet, and deleted when you remove your credentials, when you leave the project (your credentials go with it), when the data source is deleted, or when you delete your last catalog that uses it. The catalog itself stays, like any private catalog, until you delete it. It reads only in projects that can read the data source's feature store, since only there are your extra credentials for it sent, and without your credentials it no longer reads at all. Test connection on the catalog runs with your credentials. The catalog becomes queryable after the query engine restarts, as described in When the catalog goes live.

Next Steps#

Move on to the usage guide for data sources to see how you can use your newly created SQL connector. You can also make the database queryable from the query engine by adding a Trino catalog derived from this data source.