xyzpy.gen.growing

Start and monitor crop batch processes.

Attributes

_THREAD_ENV_VARS
_MEMORY_RE

Classes

_BatchTask	One crop batch to run.
_BatchCompletion	The result of one batch process.
_ActiveBatch
_SubprocessRunner	Start and monitor batch processes with resource limits.

Functions

_parse_resource_ids(raw)	Parse resource IDs as integers.
_parse_memory(raw)	Parse a memory size such as "100G" or "512mb" as bytes.
_format_memory(nbytes)	Format a number of bytes with a unit, such as "1.5G".
_process_tree(pid)	Return a process and all its descendants, read from Linux /proc.
_memory_usage(pids)	Return the total resident memory of some processes, in bytes.
_acquire_affinity(resource_id, args, env)	Add the command that selects one CPU core.
_acquire_gpu(resource_id, args, env)	Select one GPU for the process.

Module Contents

xyzpy.gen.growing._THREAD_ENV_VARS = ('OMP_NUM_THREADS', 'MKL_NUM_THREADS', 'OPENBLAS_NUM_THREADS', 'VECLIB_MAXIMUM_THREADS',...

xyzpy.gen.growing._parse_resource_ids(raw)

Parse resource IDs as integers.

xyzpy.gen.growing._MEMORY_RE

xyzpy.gen.growing._parse_memory(raw)

Parse a memory size such as "100G" or "512mb" as bytes.

Units are powers of 1024. A plain number is bytes.

xyzpy.gen.growing._format_memory(nbytes)

Format a number of bytes with a unit, such as "1.5G".

xyzpy.gen.growing._process_tree(pid)

Return a process and all its descendants, read from Linux /proc.

xyzpy.gen.growing._memory_usage(pids)

Return the total resident memory of some processes, in bytes.

xyzpy.gen.growing._acquire_affinity(resource_id, args, env)

Add the command that selects one CPU core.

xyzpy.gen.growing._acquire_gpu(resource_id, args, env)

Select one GPU for the process.

class xyzpy.gen.growing._BatchTask

One crop batch to run.

crop_name: str

parent_dir: pathlib.Path

batch_id: int

property key

property crop_dir

property batch_file

property result_file

property log_file

class xyzpy.gen.growing._BatchCompletion

The result of one batch process.

task: _BatchTask

success: bool

returncode: int

message: str

log_path: pathlib.Path | None

class xyzpy.gen.growing._ActiveBatch

task: _BatchTask

process: subprocess.Popen

output_file: object

log_path: pathlib.Path | None

started: float

gpu: int | None

affinity: int | None

max_memory: int | None

memory_used: int | None = None

class xyzpy.gen.growing._SubprocessRunner(*, num_workers=1, num_threads=1, gpus=None, affinities=None, max_memory=None, log=False, verbosity_grow=0, append_logs=False, raise_errors=True)

Start and monitor batch processes with resource limits.

active

append_logs = False

raise_errors = True

configure(*, num_workers, num_threads, gpus, affinities, max_memory, log, verbosity_grow)

Set limits for new processes.

_free_slots(name)

can_submit()

submit(task)

Start one batch with the current settings.

poll()

Return the batches that have finished.

terminate()

Stop all active processes and close their output files.