Skip to content

hopsworks.core #

ExecutionPodLog #

The live log of an execution's pod, or of one table's pod in a multi-table ingestion.

Returned by Execution.get_pod_logs. Unlike the archived stdout/stderr of Execution.download_logs, this reads the running pod directly, so it reflects a table that is still in progress.

Returned by

available property #

available: bool | None

Whether the log content is available.

container_name property #

container_name: str | None

Name of the container the log was read from.

log property #

log: str | None

The tail of the pod log, or None when it is not available.

message property #

message: str | None

A human-readable explanation when the log is not available.

pod_name property #

pod_name: str | None

Name of the pod the log was read from.

status property #

status: str | None

Why the log is or is not available.

One of AVAILABLE, WAITING_FOR_POD, MULTIPLE_ACTIVE_PODS, UNSUPPORTED_WORKLOAD, or LOGS_UNAVAILABLE.

FeatureColumnMapping #

Returned by

    FullLoadConfig #

    Returned by

      LoadingConfig #

      Returned by

        SinkJobConfiguration #

        Returned by

        TableIngestionTarget #

        A single source table -> feature group of a multi-table ingestion job.

        Each target is copied by its own worker pod and can override the job-level ingestion settings; unset fields fall back to the job-level defaults.

        PARAMETER DESCRIPTION
        feature_group

        The (saved) feature group to ingest into. Either this or feature_group_id must be provided.

        TYPE: Any DEFAULT: None

        feature_group_id

        The id of the feature group, when the object is not at hand.

        TYPE: int | None DEFAULT: None

        enabled

        Whether this table is ingested. A disabled target stays part of the job but is skipped for the run. Defaults to True.

        TYPE: bool DEFAULT: True

        loading_config

        Per-target loading strategy (full / incremental).

        TYPE: LoadingConfig | dict | None DEFAULT: None

        column_mappings

        Per-target source column -> feature name mappings.

        TYPE: Sequence[FeatureColumnMapping | dict] | None DEFAULT: None

        transform_script_path

        Per-target transformation script path.

        TYPE: str | None DEFAULT: None

        write_mode

        Per-target write mode (APPEND / MERGE).

        TYPE: WriteMode | str | None DEFAULT: None

        batch_size

        Per-target write batch size.

        TYPE: int | None DEFAULT: None

        sql_source_fetch_chunk_size

        Per-target source fetch chunk size for SQL sources.

        TYPE: int | None DEFAULT: None

        sql_table_num_partitions

        Per-target number of read partitions for SQL sources.

        TYPE: int | None DEFAULT: None

        max_upload_batch_size_mb

        Per-target maximum upload batch size in MB.

        TYPE: int | None DEFAULT: None

        source_read_workers

        Per-target number of source read workers.

        TYPE: int | None DEFAULT: None

        data_processing_workers

        Per-target number of data processing workers.

        TYPE: int | None DEFAULT: None

        resource_config

        Per-target worker pod resources, e.g. {"cores": 2, "memory": 4096}.

        TYPE: dict | None DEFAULT: None

        endpoint_config

        Per-target REST endpoint configuration.

        TYPE: dict | RestEndpointConfig | None DEFAULT: None

        Returned by

        MultiTableIngestionJob #

        A multi-table ingestion job assembled before it is created on the server.

        Collect the feature groups to ingest, either by attaching the job to them with sink_job= when creating each feature group, or by calling MultiTableIngestionJob.add_target directly, then call MultiTableIngestionJob.save to create one job that copies every collected table. Targets are only sent to the server on save, so a partial or failed set of feature groups never leaves a half-built job behind.

        Obtain one from DataSource.new_ingestion_job.

        Returned by

        job property #

        job: job.Job | None

        The created job, or None until MultiTableIngestionJob.save has run.

        name property #

        name: str

        Name of the ingestion job.

        targets property #

        targets: list[TableIngestionTarget]

        The feature-group targets collected so far.

        add_target #

        add_target(
    feature_group: fg.FeatureGroup | None = None,
    *,
    feature_group_id: int | None = None,
    **overrides,
) -> MultiTableIngestionJob

        Add a feature group as a target of this ingestion job.

        A target already added for the same feature group is replaced, so adding a feature group twice (for example on a re-run) never duplicates it. Any keyword override accepted by TableIngestionTarget applies to this target only.

        PARAMETER DESCRIPTION
        feature_group

        The feature group to ingest into.

        TYPE: fg.FeatureGroup | None DEFAULT: None

        feature_group_id

        The id of the feature group, as an alternative to passing the object.

        TYPE: int | None DEFAULT: None

        RETURNS DESCRIPTION
        MultiTableIngestionJob

        This ingestion job, so calls can be chained.

        run #

        run(await_termination: bool = False) -> Execution

        Run the ingestion job, creating it first if it has not been saved.

        PARAMETER DESCRIPTION
        await_termination

        Whether to block until the run finishes.

        TYPE: bool DEFAULT: False

        RETURNS DESCRIPTION
        Execution

        The started execution.

        save #

        save() -> job.Job

        Create the ingestion job on the server from the collected targets.

        Sends every collected target in a single request, so the job is created atomically. Calling save again updates the job with the current set of targets.

        RETURNS DESCRIPTION
        job.Job

        The created ingestion job.
        RAISES DESCRIPTION
        ValueError

        If no targets have been collected yet.

        set_table_enabled #

        set_table_enabled(
    feature_group: fg.FeatureGroup | None = None,
    *,
    feature_group_id: int | None = None,
    enabled: bool = True,
) -> MultiTableIngestionJob

        Enable or disable one table of this ingestion job.

        A disabled table stays part of the job but is skipped on runs. This takes effect on the next run; if the job has already been saved it is re-saved with the change, keeping the table's existing column mappings and overrides intact. To stop a table that is currently running, use Execution.stop_table instead.

        PARAMETER DESCRIPTION
        feature_group

        The target feature group to toggle.

        TYPE: fg.FeatureGroup | None DEFAULT: None

        feature_group_id

        The id of the target feature group, as an alternative to passing the object.

        TYPE: int | None DEFAULT: None

        enabled

        Whether the table is ingested on runs.

        TYPE: bool DEFAULT: True

        RETURNS DESCRIPTION
        MultiTableIngestionJob

        This ingestion job, so calls can be chained.
        RAISES DESCRIPTION
        ValueError

        If no target has been added for the given feature group.