Job

The job module provides classes that run transforms on a remote server.

If a JobBuilder is attached to a Public API Project, the transforms will run through jobs, with writting/submitting of pbs files.

class osekit.job.job.Job(script_path: Path, script_args: dict | None = None, config: JobConfig | None = None, name: str = 'osekit_transform', output_folder: Path | None = None)

Job that concerns a specific transform.

Initialize a Job.

Parameters

script_path: Path

Path to the script file the job must run.

script_args: dict | None

Additional arguments to pass to the script file.

config: JobConfig | None

Optional configuration to pass to the server request.

name: str

Name of the job.

output_folder: Path | None

Folder in which the output files (.out and .err) will be written.

get_arg_string() → str

Build a string representation of the job’s arguments.

property info: dict

Information about the job.

property job_id: str | None

Job ID.

property mem: str

Maximum amount of physical memory used by the job.

property name: str

Name of the job.

property nb_nodes: int

Number of nodes on which the job runs.

property ncpus: int

Number of total cores used per node.

property ngpus: int | None

Number of total GPU used per node.

property output_folder: Path | None

Folder in which the output files (.out and .err) will be written.

property path: Path | None

Path of the job file.

property script_args: dict

Additional arguments to pass to the script file.

property script_path: Path

Path to the script file the job must run.

property status: JobStatus

Status of the job.

UNPREPARED: The job file hasn’t been written yet. PREPARED: The job file has been written but not submitted. QUEUED: The job has been queued. RUNNING: The job is currently running. SUSPENDED: The job has been suspended or is held. COMPLETED: The job is exiting or has been completed.

property venv_name: str

Name of the conda virtual environment in which the job is running.

property walltime: Timedelta

Maximum amount of real time during which the job can be running.

property walltime_str: str

String representation of the walltime.

Job Config

class osekit.job.config.JobConfig(nb_nodes: int = 1, ncpus: int = 2, ngpus: int | None = None, mem: str = '8gb', walltime: str | Timedelta = '01:00:00', venv_name: str = 'osekit')

Configuration of the computing resources allowed for a job.

Parameters

nb_nodes: int

Number of nodes on which the job runs.

ncpus: int

Number of total cores used per node.

ngpus: int | None

Number of total GPU used per node.

mem: str

Maximum amount of physical memory used by the job.

walltime: str | Timedelta

Maximum amount of real time during which the job can be running.

venv_name: str

Name (or path) of the conda virtual environment in which the job is running.

Job Scheduler

class osekit.job.scheduler.pbs.Pbs(queue: Literal['omp', 'mpi'] = 'omp')

Abstract class representing a PBS job scheduler.

Initialize the PBS scheduler.

property queue: str

Queue in which the job will be submitted.

Job Builder

class osekit.job.builder.JobBuilder(config: JobConfig | None = None, scheduler: Scheduler | None = None)

Class that should be attached to a Public API Project for working with jobs.

If a Project has a JobBuilder, it will run its transforms through jobs using the specified scheduler.

Initialize a JobBuilder instance.

Parameters

config: JobConfig

Config of the jobs built by this job builder.

scheduler: Scheduler

Scheduler used to format, write and submit jobs.

create_job(script_path: Path, script_args: dict | None = None, name: str = 'osekit_transform', output_folder: Path | None = None) → None

Create a new Job instance.

Parameters

script_path: Path

Path to the script file the job must run.

script_args: dict | None

Additional arguments to pass to the script file.

name: str

Name of the job.

output_folder: Path | None

Folder in which the output files (.out and .err) will be written.

create_jobs(nb_tasks: int, script_path: Path, script_args: dict, output_folder: Path, job_name: str = 'osekit_transform', nb_jobs: int = 1) → None

Create the jobs corresponding to each batch.

Parameters

nb_tasks:

The number of tasks that are distributed across nb_jobs jobs.

script_path: Path

Path to the export script.

script_args: dict

Arguments passed to the export script.

job_name: str

Name of the job. If there are multiple batches, each batch will be suffixed with “_{index}”.

output_folder: Path

Folder in which the job output log files are saved.

nb_jobs: int

Number of batches used to run the transform. Each batch will run in a separate job.

property jobs: list[Job]

Return the jobs created by this job builder.

submit(dependencies: dict[Job, dict[str, str | Job | list[str | Job]]] | None = None) → None

Submit all prepared jobs to the scheduler system.

Parameters

dependencies: dict[Job, dict[str, str | Job | list[str | Job]]] | None

Optional mapping of the jobs dependencies.

For each key (which is a job), the value is a dictionary explaining its dependencies.

Such dictionary follows the following format: The keys of the dictionary are the dependency types, that are proper to the scheduler. The values are the other jobs (or their ID) job depends on with the given dependency type.

If None, the jobs are submitted without any dependency.