Job

The job module provides classes that run analyses on a remote server.

If a JobBuilder is attached to a Public API Dataset, the analyses will run through jobs, with writting/submitting of pbs files.

class osekit.utils.job.Job(script_path: Path, script_args: dict | None = None, config: JobConfig | None = None, name: str = 'osekit_analysis', output_folder: Path | None = None)

Job that concerns a specific analysis.

Initialize a Job.

Parameters

script_path: Path

Path to the script file the job must run.

script_args: dict | None

Additional arguments to pass to the script file.

config: JobConfig | None

Optional configuration to pass to the server request.

name: str

Name of the job.

output_folder: Path | None

Folder in which the output files (.out and .err) will be written.

property job_id: str

Job ID.

property job_info: dict | None

Information about the job as returned by a qstat request.

property mem: str

Maximum amount of physical memory used by the job.

property name: str

Name of the job.

property nb_nodes: int

Number of nodes on which the job runs.

property ncpus: int

Number of total cores used per node.

property output_folder: Path | None

Folder in which the output files (.out and .err) will be written.

property path: Path | None

Path of the job file.

progress() → None

Bring the job to the next state.

property queue: Literal['omp', 'mpi']

Queue in which the job will be submitted.

property script_args: dict

Additional arguments to pass to the script file.

property script_path: Path

Path to the script file the job must run.

property status: JobStatus

Status of the job.

UNPREPARED: The job file hasn’t been written yet. PREPARED: The job file has been written but not submitted. QUEUED: The job has been queued. RUNNING: The job is currently running. COMPLETED: The job has been completed.

submit_pbs(dependency: Job | list[Job] | str | list[str] | None = None) → None

Submit the pbs file of the job to a pbs queueing system.

Parameters

dependency: Job | list[Job] | str | None

Job dependency. Can be: - A Job instance: will wait for that job to complete successfully - A list[Job]: will wait for all jobs to complete successfully - A str: job ID (e.g., "12345.datarmor") or dependency specification - None: no dependency

update_info() → None

Request info about the job and update it.

update_status() → JobStatus

Request info about the job and update its status.

Returns

JobStatus:

The updated status of the job.

property venv_activate_script: str

Bash script used for activating the conda virtual environment.

property venv_name: str

Name of the conda virtual environment in which the job is running.

property walltime: Timedelta

Maximum amount of real time during which the job can be running.

property walltime_str: str

String representation of the walltime.

write_pbs(path: Path) → None

Write a pbs file matching the job.

Parameters

path: Path

Path of the pbs file to write.

class osekit.utils.job.JobBuilder(config: JobConfig = <class 'osekit.utils.job.JobConfig'>)

Class that should be attached to a Public API Dataset for working with jobs.

If a Dataset has a JobBuilder, it will use it to run analyses through jobs.

Initialize a JobBuilder instance.

Parameters

config: JobConfig

Config of the jobs built by this job builder.

create_job(script_path: Path, script_args: dict | None = None, name: str = 'osekit_analysis', output_folder: Path | None = None) → None

Create a new Job instance.

Parameters

script_path: Path

Path to the script file the job must run.

script_args: dict | None

Additional arguments to pass to the script file.

name: str

Name of the job.

output_folder: Path | None

Folder in which the output files (.out and .err) will be written.

submit_pbs(dependencies: dict[str, Job | list[Job]] | None = None) → None

Submit all prepared jobs to the pbs queueing system.

Parameters

dependencies: dict[str, Job | list[Job]] | None

Optional dictionary mapping job names to their dependencies. Example: {"job2": job1, "job3": [job1, job2]}

class osekit.utils.job.JobConfig(nb_nodes: int = 1, ncpus: int = 2, mem: str = '8gb', walltime: str | Timedelta = '01:00:00', venv_name: str = 'osmose', queue: Literal['omp', 'mpi'] = 'omp')

Config used for creating a job.

Parameters

nb_nodes: int

Number of nodes on which the job runs.

ncpus: int

Number of total cores used per node.

mem: str

Maximum amount of physical memory used by the job.

walltime: str | Timedelta

Maximum amount of real time during which the job can be running.

venv_name: str

Name (or path) of the conda virtual environment in which the job is running.

queue: Literal[“omp”, “mpi”]

Queue in which the job will be submitted.