xyzpy.gen.cropping¶

Attributes¶

`BTCH_NM`
`RSLT_NM`
`BTCH_RE`
`RSLT_RE`
`FNCT_NM`
`INFO_NM`
`_SGE_HEADER`
`_SGE_ARRAY_HEADER`
`_PBS_HEADER`
`_PBS_ARRAY_HEADER`
`_SLURM_HEADER`
`_SLURM_ARRAY_HEADER`
`_ARRAY_TASK_IDS`
`_BASE`
`_CLUSTER_GROW_ARRAY_SCRIPT`
`_CLUSTER_GROW_ARRAY_INDEX_SCRIPT`
`_BASE_CLUSTER_GROW_SINGLE`
`_BASE_CLUSTER_SCRIPT_END`

Classes¶

`Crop`	Encapsulates all the details describing a single 'crop', that is,
`Sower`	Class for sowing a 'crop' of batched combos to then 'grow' (on any
`Reaper`	Class that acts as a stateful function to retrieve already sown and

Functions¶

`write_to_disk`(obj, fname)	Write a pickle and replace the target atomically.
`read_from_disk`(fname)
`get_picklelib`([picklelib])
`to_pickle`(obj[, picklelib])
`from_pickle`(s[, picklelib])
`parse_crop_details`(fn, crop_name, crop_parent)	Work out how to structure the sowed data.
`parse_fn_farmer`(fn, farmer)
`calc_clean_up_default_res`(crop, clean_up, allow_incomplete)	Logic for choosing whether to automatically clean up a crop, and what,
`check_ready_to_reap`(crop, allow_incomplete, wait)
`load_crops`([directory])	Automatically load all the crops found in the current directory.
`grow`(batch_number[, crop, fn, num_workers, check_mpi, ...])	Automatically process a batch of cases into results. Should be run in an
`gen_cluster_script`(crop, scheduler[, batch_ids, mode, ...])	Generate a cluster script to grow a Crop.
`grow_cluster`(crop, scheduler[, batch_ids, hours, ...])	Automagically submit SGE, PBS, or slurm jobs to grow all missing
`gen_qsub_script`(crop[, batch_ids, scheduler])	Generate a qsub script to grow a Crop. Deprecated in favor of
`qsub_grow`(crop[, batch_ids, scheduler])	Automagically submit SGE or PBS jobs to grow all missing results.
`clean_slurm_outputs`(crop, job[, directory, ...])	Check and tidy the output files of a slurm array job growing `crop`.
`manage_slurm_outputs`(crop, job[, directory, wait_time])	Repeatedly check the output files of a slurm array job growing

Module Contents¶

xyzpy.gen.cropping.BTCH_NM = 'xyz-batch-{}.jbdmp'¶

xyzpy.gen.cropping.RSLT_NM = 'xyz-result-{}.jbdmp'¶

xyzpy.gen.cropping.BTCH_RE¶

xyzpy.gen.cropping.RSLT_RE¶

xyzpy.gen.cropping.FNCT_NM = 'xyz-function.clpkl'¶

xyzpy.gen.cropping.INFO_NM = 'xyz-settings.jbdmp'¶

xyzpy.gen.cropping.write_to_disk(obj, fname)[source]¶: Write a pickle and replace the target atomically.

xyzpy.gen.cropping.read_from_disk(fname)[source]¶

xyzpy.gen.cropping.get_picklelib(picklelib='cloudpickle')[source]¶

xyzpy.gen.cropping.to_pickle(obj, picklelib='cloudpickle')[source]¶

xyzpy.gen.cropping.from_pickle(s, picklelib='cloudpickle')[source]¶

xyzpy.gen.cropping.parse_crop_details(fn, crop_name, crop_parent)[source]¶

Work out how to structure the sowed data.

Parameters:

fn (callable, optional) – Function to infer name crop_name from, if not given.
crop_name (str, optional) – Specific name to give this set of runs.
crop_parent (str, optional) – Specific directory to put the “.xyz-{crop_name}/” folder in with all the cases and results.

Returns:

crop_location (str) – Full path to the crop-folder.
crop_name (str) – Name of the crop.
crop_parent (str) – Parent folder of the crop.

xyzpy.gen.cropping.parse_fn_farmer(fn, farmer)[source]¶

xyzpy.gen.cropping.calc_clean_up_default_res(crop, clean_up, allow_incomplete)[source]¶: Logic for choosing whether to automatically clean up a crop, and what, if any, the default all-nan result should be.

xyzpy.gen.cropping.check_ready_to_reap(crop, allow_incomplete, wait)[source]¶

class xyzpy.gen.cropping.Crop(*, fn=None, name=None, parent_dir=None, save_fn=None, batchsize=None, num_batches=None, shuffle=False, farmer=None, autoload=True)[source]¶

Encapsulates all the details describing a single ‘crop’, that is, its location, name, and batch size/number. Also allows tracking of crop’s progress, and experimentally, automatic submission of workers to grid engine to complete un-grown cases. Can also be instantiated directly from a Runner or Harvester or Crop instance.

Parameters:

fn (callable, optional) – Target function - Crop name will be inferred from this if not given explicitly. If given, Sower will also default to saving a version of fn to disk for cropping.grow to use.
name (str, optional) – Custom name for this set of runs - must be given if fn is not.
parent_dir (str, optional) – If given, alternative directory to put the “.xyz-{name}/” folder in with all the cases and results.
save_fn (bool, optional) – Whether to save the function to disk for cropping.grow to use. Will default to True if fn is given.
batchsize (int, optional) – How many cases to group into a single batch per worker. By default, batchsize=1. Cannot be specified if num_batches is.
num_batches (int, optional) – How many total batches to aim for, cannot be specified if batchsize is.
farmer ({xyzpy.Runner, xyzpy.Harvester, xyzpy.Sampler}, optional) – A Runner, Harvester or Sampler, instance, from which the fn can be inferred and which can also allow the Crop to reap itself straight to a dataset or dataframe.
autoload (bool, optional) – If True, check for the existence of a Crop written to disk with the same location, and if found, load it.

See also

Runner.Crop, Harvester.Crop, Sampler.Crop

name = None¶

parent_dir = None¶

save_fn = None¶

batchsize = None¶

num_batches = None¶

shuffle = False¶

_batch_remainder = None¶

_all_nan_result = None¶

_num_sown_batches = -1¶

_num_results = -1¶

property runner¶

choose_batch_settings(*, combos=None, cases=None)[source]¶: Work out how to divide all cases into batches, i.e. ensure that batchsize * num_batches >= num_cases.

ensure_dirs_exists()[source]¶: Make sure the directory structure for this crop exists.

save_info(combos=None, cases=None, fn_args=None)[source]¶: Save information about the sowed cases.

load_info()[source]¶: Load the full settings from disk.

load_batch(batch_number)[source]¶: Load a specific batch from disk.

load_result(batch_number)[source]¶: Load a specific result from disk.

save_result(batch_number, result)[source]¶: Save a specific result to disk.

_sync_info_from_disk(only_missing=True)[source]¶: Load information about the saved cases.

save_function_to_disk()[source]¶: Save the base function to disk using cloudpickle

load_function()[source]¶: Load the saved function from disk, and try to re-insert it back into Harvester or Runner if present.

prepare(combos=None, cases=None, fn_args=None)[source]¶: Write information about this crop and the supplied combos to disk. Typically done at start of sow, not when Crop instantiated.

is_prepared()[source]¶: Return True when the settings and any saved function exist.

calc_progress()[source]¶: Calculate how much progressed has been made in growing the batches.

is_ready_to_reap()[source]¶: Have all batches been grown?

completed_results() → tuple[int, ...][source]¶: Return tuple of batches which have been grown already.

missing_results() → tuple[int, ...][source]¶: Return tuple of batches which haven’t been grown yet.

delete_all()[source]¶: Delete the crop directory and all its contents, and reset any loaded information on this Crop object.

handle_existing(action='ask', msg=None, e=None, overwrite=False)[source]¶

Handle an already prepared crop.

Parameters:

action ({'ask', 'reap', 'delete', 'skip', 'raise'}) – What to do with the existing crop. If 'ask' (default), interactively prompt the user. Otherwise, execute the specified action directly.
msg (str, optional) – Message to display when prompting.
e (Exception, optional) – Exception to re-raise if action is 'raise'.
overwrite (bool, optional) – Whether to overwrite existing data when reaping.

property all_nan_result¶: Get a stand-in result for cases which are missing still.

__str__()[source]¶

__repr__()[source]¶

parse_constants(constants=None)[source]¶

sow_combos(combos, cases=None, constants=None, shuffle=False, verbosity=1, desc='Sow', batchsize=None, num_batches=None)[source]¶

Sow combos to disk to be later grown, potentially in batches. Note if you have already sown this Crop, as long as the number of batches hasn’t changed (e.g. you have just tweaked the function or a constant argument), you can safely resow and only the batches will be overwritten, i.e. the results will remain.

Parameters:

combos (dict_like[str, iterable]) – The combinations to sow for all or some function arguments.
cases (iterable or mappings, optional) – Optionally provide a sequence of individual cases to sow for some or all function arguments.
constants (mapping, optional) – Provide additional constant function values to use when sowing.
shuffle (bool or int, optional) – If given, sow the combos in a random order (using random.seed and random.shuffle), which can be helpful for distributing resources when not all cases are computationally equal.
verbosity (int, optional) – How much information to show when sowing. 0: no output, 1: progress bar, 2: progress bar with each setting being sown.
desc (str, optional) – Description to show in the progress bar when sowing.
batchsize (int, optional) – If specified, set a new batchsize for the crop.
num_batches (int, optional) – If specified, set a new num_batches for the crop.

sow_cases(fn_args, cases, combos=None, constants=None, verbosity=1, batchsize=None, num_batches=None)[source]¶

Sow cases to disk to be later grown, potentially in batches.

Parameters:

fn_args (iterable[str] or str) – The names and order of the function arguments, can be None if each case is supplied as a dict.
cases (iterable or mappings, optional) – Sequence of individual cases to sow for all or some function arguments.
combos (dict_like[str, iterable]) – Combinations to sow for some or all function arguments.
constants (mapping, optional) – Provide additional constant function values to use when sowing.
verbosity (int, optional) – How much information to show when sowing. 0: no output, 1: progress bar, 2: progress bar with each setting being sown.
batchsize (int, optional) – If specified, set a new batchsize for the crop.
num_batches (int, optional) – If specified, set a new num_batches for the crop.

sow_samples(n, combos=None, constants=None, verbosity=1)[source]¶: Sow n samples to disk.

grow_subprocess(batch_ids=None, num_workers=None, num_threads=None, gpus=None, affinities=None, max_memory=None, raise_errors=False, log=False, append_logs=False, debugging=False, min_wait=1e-06, max_wait=0.1, verbosity=1, verbosity_grow=0, desc='Grow')[source]¶

Grow each selected batch in a new process.

A new process applies thread, GPU, CPU and memory settings before numeric libraries load. It also isolates batch memory. Process startup adds overhead.

Parameters:

batch_ids (int or sequence of int, optional) – Batches to grow. The default is all missing batches.
num_workers (int, optional) – Maximum number of batches to run at once. The default is 1.
num_threads (int, optional) – Thread limit for each process. The default is 1.
gpus (int, str, or sequence of int, optional) – GPU IDs for CUDA_VISIBLE_DEVICES. Each process gets one entry. The pool also limits concurrency. Repeat an ID to let more than one process use that GPU, e.g. 0,0,1,1.
affinities (int, str, or sequence of int, optional) – CPU core IDs for taskset. Each process gets one entry. The pool also limits concurrency.
max_memory (int or str, optional) – Memory limit for each process. This should be a number of bytes or a size such as "100G" or "512MB". Units are powers of 1024. The resident memory of each process and its children is checked at every poll, and the process is killed if it goes over the limit. Memory allocated quickly can overshoot the limit between checks. Linux only.
raise_errors (bool, optional) – Stop new batches and raise after active batches finish.
log (bool, optional) – Save stdout and stderr to logs/batch-{batch_id}.log. The default is False. Without a log, stdout is discarded and stderr is shown only after an error.
append_logs (bool, optional) – Add to existing log files, with a header for each attempt, rather than replacing them.
debugging (bool, optional) – Set the logging level to debug in each process.
min_wait (float, optional) – Minimum polling interval in seconds.
max_wait (float, optional) – Maximum polling interval in seconds.
verbosity (int, optional) – Output level. 0 hides output. 1 shows progress. 2 also shows each setting.
verbosity_grow (int, optional) – Output level inside each batch.
desc (str, optional) – Progress label.

grow(batch_ids=None, subprocess='auto', num_workers=None, num_threads=None, gpus=None, affinities=None, max_memory=None, raise_errors=False, debugging=False, verbosity=1, verbosity_grow=0, log=False, desc='Grow', **combo_runner_opts)[source]¶

Grow selected batches here or in child processes.

Parameters:

batch_ids (int or sequence of ints, optional) – Batches to grow. The default is all missing batches.
subprocess ("auto" or bool, optional) – Use a new process for each batch. "auto" does this when num_threads, gpus, affinities, max_memory or log is set. See Crop.grow_subprocess().
num_workers (int, optional) – Maximum number of batches to run at once. Child-process mode uses 1 by default. In-process mode passes this value to combo_runner_core.
num_threads (int, optional) – Thread limit for each child process. This sets standard environment variables before numeric libraries load. With "auto", this enables child-process mode. It cannot be used with subprocess=False.
gpus (int, str, or sequence of int, optional) – GPU IDs for CUDA_VISIBLE_DEVICES. Each child process gets one entry. The pool also limits concurrency. Repeat an ID to let more than one process use that GPU. With "auto", this enables child-process mode. It cannot be used with subprocess=False.
affinities (int, str, or sequence of int, optional) – CPU core IDs for taskset. Each child process gets one entry. The pool also limits concurrency. With "auto", this enables child-process mode. It cannot be used with subprocess=False.
max_memory (int or str, optional) – Memory limit for each child process, such as "100G". See Crop.grow_subprocess(). With "auto", this enables child-process mode. It cannot be used with subprocess=False.
raise_errors (bool, optional) – Raise after a batch fails.
debugging (bool, optional) – Set the logging level to debug.
verbosity (int, optional) – Output level. 0 hides output. 1 shows progress. 2 also shows each setting.
verbosity_grow (int, optional) – Output level inside each batch.
log (bool, optional) – Save child-process output to log files. With "auto", this enables child-process mode. It cannot be used with subprocess=False.
desc (str, optional) – Progress label.
**combo_runner_opts – Extra options for Crop.grow_subprocess() in child-process mode, or combo_runner_core in in-process mode.

grow_missing(**combo_runner_opts)[source]¶: Grow any missing results using this process.

reap_combos(wait=False, clean_up=None, allow_incomplete=False, verbosity=1, desc='Reap')[source]¶

Reap already sown and grown results from this crop.

Parameters:

wait (bool, optional) – Whether to wait for results to appear. If false (default) all results need to be in place before the reap.
clean_up (bool, optional) – Whether to delete all the batch files once the results have been gathered. If left as None this will be automatically set to not allow_incomplete.
allow_incomplete (bool, optional) – Allow only partially completed crop results to be reaped, incomplete results will all be filled-in as nan.
verbosity (int, optional) – How much information to show when reaping. 0: no output, 1: progress bar, 2: progress bar with each setting being reaped.
desc (str, optional) – Description to show in the progress bar when reaping.

Returns:

results – ‘N-dimensional’ tuple containing the results.

Return type:

nested tuple

reap_combos_to_ds(var_names=None, var_dims=None, var_coords=None, constants=None, attrs=None, parse=True, wait=False, clean_up=None, allow_incomplete=False, to_df=False, verbosity=1, desc='Reap')[source]¶

Reap a function over sowed combinations and output to a Dataset.

Parameters:

var_names (str, sequence of strings, or None) – Variable name(s) of the output(s) of fn, set to None if fn outputs data already labeled in a Dataset or DataArray.
var_dims (sequence of either strings or string sequences, optional) – ‘Internal’ names of dimensions for each variable, the values for each dimension should be contained as a mapping in either var_coords (not needed by fn) or constants (needed by fn).
var_coords (mapping, optional) – Mapping of extra coords the output variables may depend on.
constants (mapping, optional) – Arguments to fn which are not iterated over, these will be recorded either as attributes or coordinates if they are named in var_dims.
resources (mapping, optional) – Like constants but they will not be recorded.
attrs (mapping, optional) – Any extra attributes to store.
wait (bool, optional) – Whether to wait for results to appear. If false (default) all results need to be in place before the reap.
clean_up (bool, optional) – Whether to delete all the batch files once the results have been gathered. If left as None this will be automatically set to not allow_incomplete.
allow_incomplete (bool, optional) – Allow only partially completed crop results to be reaped, incomplete results will all be filled-in as nan.
to_df (bool, optional) – Whether to reap to a xarray.Dataset or a pandas.DataFrame.
verbosity (int, optional) – How much information to show when reaping. 0: no output, 1: progress bar, 2: progress bar with each setting being reaped.
desc (str, optional) – Description to show in the progress bar when reaping.

Returns:

Multidimensional labeled dataset containing all the results.

Return type:

xarray.Dataset or pandas.Dataframe

reap_runner(runner, wait=False, clean_up=None, allow_incomplete=False, to_df=False, verbosity=1, desc='Reap', **kwargs)[source]¶: Build a Dataset or DataFrame from this Crop and a Runner.

reap_harvest(harvester, wait=False, sync=True, overwrite=None, clean_up=None, allow_incomplete=False, verbosity=1, desc='Reap', **kwargs)[source]¶: Reap a Crop over sowed combos and merge with the dataset defined by a Harvester.

reap_samples(sampler, wait=False, sync=True, clean_up=None, allow_incomplete=False, verbosity=1, desc='Reap', **kwargs)[source]¶: Reap a Crop over sowed combos and merge with the dataframe defined by a Sampler.

reap(wait=False, sync=True, overwrite=None, clean_up=None, allow_incomplete=False, verbosity=1, desc='Reap')[source]¶

Reap sown and grown combos from disk. Return a dataset if a runner or harvester is set, otherwise, the raw nested tuple.

Parameters:

wait (bool, optional) – Whether to wait for results to appear. If false (default) all results need to be in place before the reap.
sync (bool, optional) – Immediately sync the new dataset with the on-disk full dataset or dataframe if a harvester or sampler is used.
overwrite (bool, optional) – How to compare data when syncing to on-disk dataset. If None, (default) merge as long as no conflicts. True: overwrite with the new data. False, discard any new conflicting data.
clean_up (bool, optional) – Whether to delete all the batch files once the results have been gathered. If left as None this will be automatically set to not allow_incomplete.
allow_incomplete (bool, optional) – Allow only partially completed crop results to be reaped, incomplete results will all be filled-in as nan.
verbosity (int, optional) – How much information to show when reaping. 0: no output, 1: progress bar, 2: progress bar with each setting being reaped.
desc (str, optional) – Description to show in the progress bar when reaping.

Return type:

nested tuple or xarray.Dataset

check_bad(delete_bad=True)[source]¶

Check that the result dumps are not bad -> sometimes length does not match the batch. Optionally delete these so that they can be re-grown.

Parameters:: delete_bad (bool) – Delete bad results as they are come across.
Returns:: bad_ids – The bad batch numbers.
Return type:: tuple

_get_fn()[source]¶

_set_fn(fn)[source]¶

_del_fn()[source]¶

fn¶

property num_sown_batches¶: Total number of batches to be run/grown.

property num_results¶

xyzpy.gen.cropping.load_crops(directory='.')[source]¶

Automatically load all the crops found in the current directory.

Parameters:: directory (str, optional) – Which directory to load the crops from, defaults to ‘.’ - the current.
Returns:: Mapping of the crop name to the Crop.
Return type:: dict[str, Crop]

class xyzpy.gen.cropping.Sower(crop)[source]¶

Class for sowing a ‘crop’ of batched combos to then ‘grow’ (on any number of workers sharing the filesystem) and then reap.

crop¶

_batch_cases = []¶

_counter = 0¶

_batch_counter = 0¶

save_batch()[source]¶: Save the current batch of cases to disk and start the next batch.

__enter__()[source]¶

__call__(**kwargs)[source]¶

__exit__(exception_type, exception_value, traceback)[source]¶

xyzpy.gen.cropping.grow(batch_number, crop=None, fn=None, num_workers=None, check_mpi=True, verbosity=2, debugging=False, raise_errors=True)[source]¶

Automatically process a batch of cases into results. Should be run in an “.xyz-{fn_name}” folder, or crop should be specified.

Parameters:

batch_number (int) – Which batch to ‘grow’ into a set of results.
crop (xyzpy.Crop) – Description of where and how to store the cases and results.
fn (callable, optional) – If specified, the function used to generate the results, otherwise the function will be loaded from disk.
num_workers (int, optional) – If specified, grow using a pool of this many workers. This uses joblib.externals.loky to spawn processes.
check_mpi (bool, optional) – Whether to check if the process is rank 0 and only save results if so - allows mpi functions to be simply used. Defaults to true, this should only be turned off if e.g. a pool of workers is being used to run different grow instances.
verbosity ({0, 1, 2}, optional) – How much information to show.
debugging (bool, optional) – Set logging level to DEBUG.
raise_errors (bool, optional) – Whether to raise errors that occur during the computation. If growing many batches in parallel, it can be useful to set this to False so a single error doesn’t crash the whole process.

class xyzpy.gen.cropping.Reaper(crop, num_batches, wait=False, default_result=None)[source]¶

Class that acts as a stateful function to retrieve already sown and grow results.

crop¶

results¶

__call__(**kwargs)[source]¶

check_finished()[source]¶

xyzpy.gen.cropping._SGE_HEADER = Multiline-String¶

Show Value

"""#!/bin/bash -l
#$ -S /bin/bash
#$ -N {name}
#$ -l h_rt={hours}:{minutes}:{seconds},mem={gigabytes}G
#$ -l tmpfs={temp_gigabytes}G
#$ -wd {output_directory}
#$ -pe {pe} {num_procs}
{header_options}"""

xyzpy.gen.cropping._SGE_ARRAY_HEADER = Multiline-String¶

Show Value

"""#$ -t 1-{num_tasks}
"""

xyzpy.gen.cropping._PBS_HEADER = Multiline-String¶

Show Value

"""#!/bin/bash -l
#PBS -N {name}
#PBS -lselect={num_nodes}:ncpus={num_procs}:mem={gigabytes}gb
#PBS -lwalltime={hours:02}:{minutes:02}:{seconds:02}
{header_options}"""

xyzpy.gen.cropping._PBS_ARRAY_HEADER = Multiline-String¶

Show Value

"""#PBS -J 1-{num_tasks}
"""

xyzpy.gen.cropping._SLURM_HEADER = Multiline-String¶

Show Value

"""#!/bin/bash -l
#SBATCH --job-name={name}
#SBATCH --time={hours:02}:{minutes:02}:{seconds:02}
{header_options}"""

xyzpy.gen.cropping._SLURM_ARRAY_HEADER = Multiline-String¶

Show Value

"""#SBATCH --array=1-{num_tasks}
"""

xyzpy.gen.cropping._ARRAY_TASK_IDS¶

xyzpy.gen.cropping._BASE = Multiline-String¶

Show Value

"""echo 'XYZPY script starting...'
cd {working_directory}
{thread_exports}{shell_setup}
read -r -d '' SCRIPT << 'EOM' || true
{setup}
import os
from xyzpy.gen.cropping import grow, Crop
if __name__ == '__main__':
    crop = Crop(name={name!r}, parent_dir={parent_dir!r})
    print('Growing:', repr(crop))
    grow_kwargs = dict(
{grow_kwargs}    )
"""

xyzpy.gen.cropping._CLUSTER_GROW_ARRAY_SCRIPT = Multiline-String¶

Show Value

"""    batch_id = {task_id}
    print('xyzpy: growing batch', batch_id)
    crop.grow(batch_id, **grow_kwargs)
"""

xyzpy.gen.cropping._CLUSTER_GROW_ARRAY_INDEX_SCRIPT = Multiline-String¶

Show Value

"""    batch_ids = {batch_ids}
    batch_id = batch_ids[{task_id} - 1]
    print('xyzpy: growing batch', batch_id)
    crop.grow(batch_id, **grow_kwargs)
"""

xyzpy.gen.cropping._BASE_CLUSTER_GROW_SINGLE = Multiline-String¶

Show Value

"""    batch_ids = {batch_ids}
    crop.grow(batch_ids, **grow_kwargs)
"""

xyzpy.gen.cropping._BASE_CLUSTER_SCRIPT_END = Multiline-String¶

Show Value

"""EOM
{launcher} -c "$SCRIPT"
echo 'XYZPY script finished'
"""

xyzpy.gen.cropping.gen_cluster_script(crop, scheduler, batch_ids=None, *, mode='array', num_procs=None, num_threads=None, num_nodes=None, num_workers=None, subprocess='auto', log=None, raise_errors=None, max_memory=None, gpus=None, affinities=None, mem=None, mem_per_cpu=None, gigabytes=None, time=None, hours=None, minutes=None, seconds=None, conda_env=False, launcher=None, setup='#', shell_setup='', mpi=False, temp_gigabytes=1, output_directory=None, debugging=False, **kwargs)[source]¶

Generate a cluster script to grow a Crop.

Parameters:

crop (Crop) – The crop to grow.
scheduler ({'sge', 'pbs', 'slurm'}) – Whether to use a SGE, PBS or slurm submission script template.
batch_ids (int or tuple[int]) – Which batch numbers to grow, defaults to all missing batches. In 'array' mode the missing batches are found when the script is made, and array task i grows the i-th of them. If no batch has a result yet, task i simply grows batch i. In 'single' mode the missing batches are found when the job starts, so the same job can be resubmitted to resume.
mode ({'array', 'single'}) – How to distribute the batches, either as an array job with a single batch per job, or as a single job processing batches in parallel.
time (float or str, optional) – How much time to request, either as a number of hours, or as a string like "H:M:S" or "D-H:M:S". Cannot be used with hours, minutes or seconds. If no time is given the default is 1 hour.
hours (int, optional) – How many hours to request.
minutes (int, optional) – How many minutes to request.
seconds (int, optional) – How many seconds to request.
gigabytes (int, optional) – How much memory to request. For SGE and PBS the default is 2. For slurm no memory is requested by default, so the cluster default applies.
mem (int or str, optional) – Alias for gigabytes. For slurm, a string like "500M" is passed on as is. For SGE and PBS it must be a number of gigabytes.
mem_per_cpu (int or str, optional) – How much memory to request per cpu, slurm only.
num_procs (int, optional) – How many processes to request (threaded cores or MPI). For SGE and PBS the default is 1. For slurm --cpus-per-task is left out by default, and the thread count is read from SLURM_CPUS_PER_TASK. For slurm, cpus_per_task can be given instead.
num_threads (int, optional) – How many threads to use per process. Will be computed automatically based on num_procs and num_workers if not specified.
num_workers (int, optional) – How many workers to use for parallel growing, default is sequential. If specified, then generally num_workers * num_threads == num_procs.
subprocess (bool or "auto", optional) – Whether to grow each batch in a fresh process, see Crop.grow_subprocess(). "auto", the default, does this if log, max_memory, gpus or affinities is set, (note log is set by default).
log (bool, optional) – Save the output of each batch to logs/batch-{batch_id}.log in the crop directory, adding to any log from an earlier attempt. The default is True, unless subprocess=False. setup and launcher then only apply to the parent process, not to the process growing each batch.
raise_errors (bool, optional) – Whether a failed batch raises an error. The default is True in 'array' mode, so the array task is marked as failed, and False in 'single' mode, so the other batches carry on.
max_memory (int or str, optional) – Memory limit for each batch process, such as "100G". See Crop.grow_subprocess().
gpus (int, str, or sequence of int, optional) – GPU IDs to share between batch processes. See Crop.grow_subprocess().
affinities (int, str, or sequence of int, optional) – CPU core IDs to share between batch processes. See Crop.grow_subprocess().
num_nodes (int, optional) – How many nodes to request. For SGE and PBS the default is 1. For slurm, nodes can be given instead.
conda_env (bool or str, optional) – Whether to activate a conda environment before running the script, default: False. This is usually not needed, since the script runs the current Python interpreter directly. If True, the environment will be the same as the one used to launch the script. If a string, the environment will be the one specified by the string.
launcher (str, optional) – How to launch the script, default: the current Python interpreter. But could for example be 'mpiexec python' for an MPI program.
setup (str, optional) – Python script to run before growing, for things that shouldn’t be put in the crop function itself, e.g. one-time imports with side-effects like: "import tensorflow as tf; tf.enable_eager_execution()”.
shell_setup (str, optional) – Commands to be run by the shell before the python script is executed.
mpi (bool, optional) – Request MPI processes not threaded processes
temp_gigabytes (int, optional) – How much temporary on-disk memory.
output_directory (str, optional) – Directory for the scheduler’s own output files, such as slurm-{job}_{task}.out. The default is the crop directory. For SGE this sets -wd. For slurm and PBS the files are written where the job is submitted from, which grow_cluster() sets to this directory.
debugging (bool, optional) – Set the python log level to debugging.
kwargs (dict, optional) – Extra keyword arguments are taken to be extra resources to request in the header of the submission script, e.g. gres="gpu:1" will add "#SBATCH --gres=gpu:1" to the header if using slurm. For slurm, underscores in names are replaced by hyphens, so mail_type="END" adds "#SBATCH --mail-type=END". If you supply literal True or None as the value, then the key will be treated as a flag. E.g. {'requeue': None} will add "#SBATCH --requeue" to the header.

Return type:

str

xyzpy.gen.cropping.grow_cluster(crop, scheduler, batch_ids=None, *, hours=None, minutes=None, seconds=None, gigabytes=None, num_nodes=None, num_procs=None, num_threads=None, num_workers=None, subprocess='auto', log=None, conda_env=False, launcher=None, setup='#', shell_setup='', mpi=False, temp_gigabytes=1, output_directory=None, debugging=False, **kwargs)[source]¶

Automagically submit SGE, PBS, or slurm jobs to grow all missing results.

Parameters:

crop (Crop) – The crop to grow.
scheduler ({'sge', 'pbs', 'slurm'}) – Whether to use a SGE, PBS or slurm submission script template.
batch_ids (int or tuple[int]) – Which batch numbers to grow, defaults to all missing batches.
hours (int, optional) – How many hours to request. If no time is given the default is 1 hour.
minutes (int, optional) – How many minutes to request.
seconds (int, optional) – How many seconds to request.
gigabytes (int, optional) – How much memory to request. For SGE and PBS the default is 2. For slurm no memory is requested by default, so the cluster default applies.
num_nodes (int, optional) – How many nodes to request. For SGE and PBS the default is 1.
num_procs (int, optional) – How many processes to request (threaded cores or MPI). For SGE and PBS the default is 1. For slurm --cpus-per-task is left out by default, and the thread count is read from SLURM_CPUS_PER_TASK.
num_threads (int, optional) – How many threads to use per process. Will be computed automatically based on num_procs and num_workers if not specified.
num_workers (int, optional) – How many workers to use for parallel growing, default is sequential. If specified, then generally num_workers * num_threads == num_procs.
subprocess (bool or "auto", optional) – Whether to grow each batch in a fresh process. "auto", the default, does this if log or another process option is set.
log (bool, optional) – Save the output of each batch to logs/batch-{batch_id}.log in the crop directory. The default is True, unless subprocess=False.
conda_env (bool or str, optional) – Whether to activate a conda environment before running the script, default: False. This is usually not needed, since the script runs the current Python interpreter directly. If True, the environment will be the same as the one used to launch the script. If a string, the environment will be the one specified by the string.
launcher (str, optional) – How to launch the script, default: the current Python interpreter. But could for example be 'mpiexec python' for a MPI program.
setup (str, optional) – Python script to run before growing, for things that shouldnt’t be put in the crop function itself, e.g. one-time imports with side-effects like: "import tensorflow as tf; tf.enable_eager_execution()”.
shell_setup (str, optional) – Commands to be run by the shell before the python script is executed. E.g. conda activate my_env.
mpi (bool, optional) – Request MPI processes not threaded processes.
temp_gigabytes (int, optional) – How much temporary on-disk memory.
output_directory (str, optional) – Directory for the scheduler’s own output files. The job is submitted from here, and it is created if needed. The default is the crop directory.
debugging (bool, optional) – Set the python log level to debugging.
kwargs – See gen_cluster_script for all other options, such as time, mem, mode, raise_errors, max_memory and extra header resources.

Returns:

job_id – The slurm job id, if submitted with slurm.

Return type:

str or None

xyzpy.gen.cropping.gen_qsub_script(crop, batch_ids=None, *, scheduler='sge', **kwargs)[source]¶

Generate a qsub script to grow a Crop. Deprecated in favor of gen_cluster_script and will be removed in the future.

Parameters:

crop (Crop) – The crop to grow.
batch_ids (int or tuple[int]) – Which batch numbers to grow, defaults to all missing batches.
scheduler ({'sge', 'pbs'}, optional) – Whether to use an SGE or PBS submission script template.
kwargs – See gen_cluster_script for all other parameters.

xyzpy.gen.cropping.qsub_grow(crop, batch_ids=None, *, scheduler='sge', **kwargs)[source]¶

Automagically submit SGE or PBS jobs to grow all missing results. Deprecated in favor of grow_cluster and will be removed in the future.

Parameters:

crop (Crop) – The crop to grow.
batch_ids (int or tuple[int]) – Which batch numbers to grow, defaults to all missing batches.
scheduler ({'sge', 'pbs'}, optional) – Whether to use a SGE or PBS submission script template.
kwargs – See grow_cluster for all other parameters.

xyzpy.gen.cropping.clean_slurm_outputs(crop, job, directory=None, cancel_if_finished=True)[source]¶

Check and tidy the output files of a slurm array job growing crop.

Each array task prints which batch it grows. A task is finished once that batch has a result, and its output file is then deleted. The output of each batch itself is in logs/batch-{batch_id}.log by default.

Parameters:

crop (Crop) – The crop being grown.
job (int or str) – The slurm job id.
directory (str, optional) – Where the output files are, by default the crop directory.
cancel_if_finished (bool, optional) – Cancel finished tasks that slurm still reports as running.

Returns:

The number of output files found.

Return type:

int

xyzpy.gen.cropping.manage_slurm_outputs(crop, job, directory=None, wait_time=60)[source]¶: Repeatedly check the output files of a slurm array job growing crop, until it is ready to reap. See clean_slurm_outputs().