Skip to content

Jobs API#

Handle#

[source]

get_jobs_api#

Project.get_jobs_api()

Get the jobs api for the project.

Returns

JobsApi: The Jobs Api handle

Configuration#

[source]

get_configuration#

JobsApi.get_configuration(type)

Get configuration for the specific job type.

Arguments

  • type str: Type of the job. Currently, supported types include: SPARK, PYSPARK, PYTHON, DOCKER, FLINK.

Returns

dict: Default job configuration

Raises

  • RestAPIError: If unable to get the job configuration

Creation#

[source]

create_job#

JobsApi.create_job(name, config)

Create a new job or update an existing one.

import hopsworks

project = hopsworks.login()

jobs_api = project.get_jobs_api()

spark_config = jobs_api.get_configuration("PYSPARK")

spark_config['appPath'] = "/Resources/my_app.py"

job = jobs_api.create_job("my_spark_job", spark_config)
Arguments

  • name str: Name of the job.
  • config dict: Configuration of the job.

Returns

Job: The Job object

Raises

  • RestAPIError: If unable to create the job

Retrieval#

[source]

get_job#

JobsApi.get_job(name)

Get a job.

Arguments

  • name str: Name of the job.

Returns

Job: The Job object

Raises

  • RestAPIError: If unable to get the job

[source]

get_jobs#

JobsApi.get_jobs()

Get all jobs.

Returns

List[Job]: List of Job objects

Raises

  • RestAPIError: If unable to get the jobs

Properties#

[source]

config#

Configuration for the job

[source]

creation_time#

Date of creation for the job

[source]

creator#

Creator of the job

[source]

id#

Id of the job

[source]

job_schedule#

Return the Job schedule

[source]

job_type#

Type of the job

[source]

name#

Name of the job

Methods#

[source]

delete#

Job.delete()

Delete the job

Potentially dangerous operation

This operation deletes the job and all executions.

Raises

RestAPIError.

[source]

get_executions#

Job.get_executions()

Retrieves all executions for the job.

Returns

List[Execution]

Raises

RestAPIError in case the backend fails to retrieve executions.

[source]

get_url#

Job.get_url()

[source]

run#

Job.run(args=None, await_termination=None)

Run the job, with the option of passing runtime arguments.

Example of a blocking execution and downloading logs once execution is finished.

# Run the job
execution = job.run(await_termination=True)

# True if job executed successfully
print(execution.success)

# Download logs
out_log_path, err_log_path = execution.download_logs()
Arguments

  • args Optional[str]: optional runtime arguments for the job
  • await_termination Optional[bool]: if True wait until termination is complete

Returns

Execution. The execution object for the submitted run.

[source]

save#

Job.save()

Save the job.

This function should be called after changing a property such as the job configuration to save it persistently.

job.config['appPath'] = "Resources/my_app.py"
job.save()
Returns

Job. The updated job object.

[source]

schedule#

Job.schedule(cron_expression, start_time=None, end_time=None)

Schedule the execution of the job.

If a schedule for this job already exists, the method updates it.

# Schedule the job
job.schedule(
    cron_expression="0 */5 * ? * * *",
    start_time=datetime.datetime.now(tz=timezone.utc)
)

# Retrieve the next execution time
print(job.job_schedule.next_execution_date_time)

Arguments

  • cron_expression str: The quartz cron expression
  • start_time Optional[datetime.datetime]: The schedule start time in UTC. If None, the current time is used. The start_time can be a value in the past.
  • end_time Optional[datetime.datetime]: The schedule end time in UTC. If None, the schedule will continue running indefinitely. The end_time can be a value in the past.

Returns

JobSchedule. The schedule of the job

[source]

unschedule#

Job.unschedule()

Unschedule the exceution of a Job