hopsworks.core.job_api #

[source] JobApi #

For backwards compatibility hopsworks.core.job_api.JobApi is still available as hopsworks.core.job_api.JobsApi, hsfs.core.job_api.JobApi. The use of these aliases is discouraged as they are to be deprecated.

Returned by

Project.get_job_api

[source] create_job #

create_job(name: str, config: dict) -> job.Job

Create a new job or update an existing one.

import hopsworks

project = hopsworks.login()

job_api = project.get_job_api()

spark_config = job_api.get_configuration("PYSPARK")

spark_config['appPath'] = "/Resources/my_app.py"

job = job_api.create_job("my_spark_job", spark_config)

PARAMETER	DESCRIPTION
`name`	Name of the job. TYPE: `str`
`config`	Configuration of the job. TYPE: `dict`

RETURNS	DESCRIPTION
`job.Job`	The created job.

RAISES	DESCRIPTION
`hopsworks.client.exceptions.RestAPIError`	If the backend encounters an error when handling the request.

[source] get_job #

get_job(name: str) -> job.Job | None

Get a job.

PARAMETER	DESCRIPTION
`name`	Name of the job. TYPE: `str`

RETURNS	DESCRIPTION
`job.Job \| None`	The Job object or `None` if it does not exist.

RAISES	DESCRIPTION
`hopsworks.client.exceptions.RestAPIError`	If the backend encounters an error when handling the request.

[source] get_jobs #

get_jobs() -> list[job.Job]

Get all jobs.

RETURNS	DESCRIPTION
`list[job.Job]`	List of all jobs.

RAISES	DESCRIPTION
`hopsworks.client.exceptions.RestAPIError`	If the backend encounters an error when handling the request.

[source] get_configuration #

get_configuration(
    type: Literal[
        "SPARK", "PYSPARK", "PYTHON", "DOCKER", "FLINK"
    ],
) -> dict

Get configuration for the specific job type.

PARAMETER	DESCRIPTION
`type`	The job type to retrieve the configuration of. TYPE: `Literal['SPARK', 'PYSPARK', 'PYTHON', 'DOCKER', 'FLINK']`

RETURNS	DESCRIPTION
`dict`	The default job configuration for the specific job type.

RAISES	DESCRIPTION
`hopsworks.client.exceptions.RestAPIError`	If the backend encounters an error when handling the request.