hopsworks.core #
ExecutionPodLog #
The live log of an execution's pod, or of one table's pod in a multi-table ingestion.
Returned by Execution.get_pod_logs. Unlike the archived stdout/stderr of Execution.download_logs, this reads the running pod directly, so it reflects a table that is still in progress.
Returned by
FeatureColumnMapping #
Returned by
FullLoadConfig #
Returned by
LoadingConfig #
Returned by
SinkJobConfiguration #
Returned by
file_columns property writable #
SQL Server varbinary(max) or image columns stored as files in HopsFS, besides the FILESTREAM columns.
Each entry is a column name, or a dict with source_column and optionally path_feature and size_feature. The feature group holds each file's full path in the path feature and its length in bytes in the size feature. Incremental ingestion with file columns needs the MERGE write mode.
TableIngestionTarget #
A single source table -> feature group of a multi-table ingestion job.
Each target is copied by its own worker pod and can override the job-level ingestion settings; unset fields fall back to the job-level defaults.
| PARAMETER | DESCRIPTION |
|---|---|
feature_group | The (saved) feature group to ingest into. Either this or TYPE: |
feature_group_id | The id of the feature group, when the object is not at hand. TYPE: |
enabled | Whether this table is ingested. A disabled target stays part of the job but is skipped for the run. Defaults to TYPE: |
loading_config | Per-target loading strategy (full / incremental). TYPE: |
column_mappings | Per-target source column -> feature name mappings. TYPE: |
transform_script_path | Per-target transformation script path. TYPE: |
write_mode | Per-target write mode ( TYPE: |
batch_size | Per-target write batch size. TYPE: |
sql_source_fetch_chunk_size | Per-target source fetch chunk size for SQL sources. TYPE: |
sql_table_num_partitions | Per-target number of read partitions for SQL sources. TYPE: |
max_upload_batch_size_mb | Per-target maximum upload batch size in MB. TYPE: |
source_read_workers | Per-target number of source read workers. TYPE: |
data_processing_workers | Per-target number of data processing workers. TYPE: |
resource_config | Per-target worker pod resources, e.g. TYPE: |
endpoint_config | Per-target REST endpoint configuration. TYPE: |
file_columns | SQL Server |
files_directory | The project directory this table's files are written to. TYPE: |
Returned by
MultiTableIngestionJob #
A multi-table ingestion job assembled before it is created on the server.
Collect the feature groups to ingest, either by attaching the job to them with sink_job= when creating each feature group, or by calling MultiTableIngestionJob.add_target directly, then call MultiTableIngestionJob.save to create one job that copies every collected table. Targets are only sent to the server on save, so a partial or failed set of feature groups never leaves a half-built job behind.
Obtain one from DataSource.new_ingestion_job.
Returned by
job property #
job: job.Job | None
The created job, or None until MultiTableIngestionJob.save has run.
add_target #
add_target(
feature_group: fg.FeatureGroup | None = None,
*,
feature_group_id: int | None = None,
**overrides,
) -> MultiTableIngestionJob
Add a feature group as a target of this ingestion job.
A target already added for the same feature group is replaced, so adding a feature group twice (for example on a re-run) never duplicates it. Any keyword override accepted by TableIngestionTarget applies to this target only.
| PARAMETER | DESCRIPTION |
|---|---|
feature_group | The feature group to ingest into. TYPE: |
feature_group_id | The id of the feature group, as an alternative to passing the object. TYPE: |
| RETURNS | DESCRIPTION |
|---|---|
MultiTableIngestionJob | This ingestion job, so calls can be chained. |
run #
save #
save() -> job.Job
Create the ingestion job on the server from the collected targets.
Sends every collected target in a single request, so the job is created atomically. Calling save again updates the job with the current set of targets.
| RETURNS | DESCRIPTION |
|---|---|
job.Job | The created ingestion job. |
| RAISES | DESCRIPTION |
|---|---|
ValueError | If no targets have been collected yet. |
set_table_enabled #
set_table_enabled(
feature_group: fg.FeatureGroup | None = None,
*,
feature_group_id: int | None = None,
enabled: bool = True,
) -> MultiTableIngestionJob
Enable or disable one table of this ingestion job.
A disabled table stays part of the job but is skipped on runs. This takes effect on the next run; if the job has already been saved it is re-saved with the change, keeping the table's existing column mappings and overrides intact. To stop a table that is currently running, use Execution.stop_table instead.
| PARAMETER | DESCRIPTION |
|---|---|
feature_group | The target feature group to toggle. TYPE: |
feature_group_id | The id of the target feature group, as an alternative to passing the object. TYPE: |
enabled | Whether the table is ingested on runs. TYPE: |
| RETURNS | DESCRIPTION |
|---|---|
MultiTableIngestionJob | This ingestion job, so calls can be chained. |
| RAISES | DESCRIPTION |
|---|---|
ValueError | If no target has been added for the given feature group. |