xyzpy.gen.combo_runner¶

Functions for systematically evaluating a function over all combinations.

Attributes¶

combo_runner_to_df

Functions¶

`infer_shape`(x)	Take a nested sequence and find its shape as if it were an array.
`nan_like_result`(res)	Take a single result of a function evaluation and calculate the same
`_submit`(executor, fn, args, *kwds)	Default method for submitting to a executor.
`_get_result`(future)
`_run_linear_executor`(executor, fn, settings[, ...])
`_run_linear_sequential`(fn, settings[, verbosity, desc])
`_unflatten`(store, all_combo_values[, all_nan])
`combo_runner_core`(fn, combos, constants[, cases, ...])
`combo_runner`(fn[, combos, cases, constants, split, ...])	Take a function `fn` and compute it over all combinations of named
`multi_concat`(results, dims)	Concatenate a nested list of xarray objects along several dimensions.
`get_ndim_first`(x, ndim)	Return the first element from the ndim-nested list x.
`results_to_ds`(results, combos, var_names, var_dims, ...)	Convert the output of combo_runner into a `xarray.Dataset`.
`results_to_df`(results_linear, settings, attrs, ...)	Convert the output of combo_runner into a `pandas.DataFrame`.
`combo_runner_to_ds`(fn, combos, var_names, *[, ...])	Evaluate a function over all cases and combinations and output to a

Module Contents¶

xyzpy.gen.combo_runner.infer_shape(x)[source]¶

Take a nested sequence and find its shape as if it were an array.

Examples

>>> x = [[10, 20, 30], [40, 50, 60]]
>>> infer_shape(x)
(2, 3)

xyzpy.gen.combo_runner.nan_like_result(res)[source]¶

Take a single result of a function evaluation and calculate the same sequence of scalars or arrays but filled entirely with nan.

Examples

>>> res = (True, [[10, 20, 30], [40, 50, 60]], -42.0, 'hello')
>>> nan_like_result(res)
(array(nan), array([[nan, nan, nan],
                    [nan, nan, nan]]), array(nan), None)

xyzpy.gen.combo_runner._submit(executor, fn, *args, **kwds)[source]¶

Default method for submitting to a executor.

Parameters:

executor (pool-executor like) – A multiprocessing.pool or an pool executor with API matching either concurrent.futures, or an ipyparallel view.
fn (callable) – The function to submit.
args – Supplied to fn.
kwds – Supplied to fn.

Return type:

future

xyzpy.gen.combo_runner._get_result(future)[source]¶

xyzpy.gen.combo_runner._run_linear_executor(executor, fn, settings, verbosity=1, desc=None)[source]¶

xyzpy.gen.combo_runner._run_linear_sequential(fn, settings, verbosity=1, desc=None)[source]¶

xyzpy.gen.combo_runner._unflatten(store, all_combo_values, all_nan=None)[source]¶

xyzpy.gen.combo_runner.combo_runner_core(fn, combos, constants, cases=None, split=False, flat=False, shuffle=False, parallel=False, num_workers=None, executor=None, verbosity=1, desc=None, info=None)[source]¶

xyzpy.gen.combo_runner.combo_runner(fn, combos=None, *, cases=None, constants=None, split=False, flat=False, shuffle=False, parallel=False, executor=None, num_workers=None, verbosity=1, desc=None)[source]¶

Take a function fn and compute it over all combinations of named variables values, optionally showing progress and in parallel.

Parameters:

fn (callable) – Function to analyse.
combos (dict_like[str, iterable]) – All combinations of each argument to values mapping will be computed. Each argument range thus gets a dimension in the output array(s).
cases (sequence of mappings, optional) – Optional list of specific configurations. If both combos and cases are given, then the function is computed for all sub-combinations in combos for each case in cases, arguments can thus only appear in one or the other. Note that missing combinations of arguments will be represented by nan if creating a nested array.
constants (dict, optional) – Constant function arguments. Unlike combos and cases, these won’t produce dimensions in the output result when flat=False.
split (bool, optional) – Whether to split (unzip) the outputs of fn into multiple output arrays or not.
flat (bool, optional) – Whether to return a flat list of results or to return a nested tuple suitable to be supplied to numpy.array.
shuffle (bool or int, optional) – If given, compute the results in a random order (using random.seed and random.shuffle), which can be helpful for distributing resources when not all cases are computationally equal.
parallel (bool, optional) – Process combos in parallel, default number of workers picked.
executor (executor-like pool, optional) – Submit all combos to this pool executor. Must have submit or apply_async methods and API matching either concurrent.futures or an ipyparallel view. Pools from multiprocessing.pool are also supported.
num_workers (int, optional) – Explicitly choose how many workers to use, None for automatic.
verbosity ({0, 1, 2}, optional) –
How much information to display:
- 0: nothing,
- 1: just progress,
- 2: postfix the current settings to the progress bar.
desc (str, optional) – Description to show in the progress bar, if verbosity > 0.

Returns:

data – Nested tuple containing all combinations of running fn if flat == False else a flat list of results.

Return type:

nested tuple

Examples

>>> def fn(a, b, c, d):
...     return str(a) + str(b) + str(c) + str(d)

Run all possible combos:

>>> xyz.combo_runner(
...     fn,
...     combos={
...         'a': [1, 2],
...         'b': [3, 4],
...         'c': [5, 6],
...         'd': [7, 8],
...     },
... )
100%|##########| 16/16 [00:00<00:00, 84733.41it/s]

(((('1357', '1358'), ('1367', '1368')),
  (('1457', '1458'), ('1467', '1468'))),
 ((('2357', '2358'), ('2367', '2368')),
  (('2457', '2458'), ('2467', '2468'))))

Run only a selection of cases:

>>> xyz.combo_runner(
...     fn,
...     cases=[
...         {'a': 1, 'b': 3, 'c': 5, 'd': 7},
...         {'a': 2, 'b': 4, 'c': 6, 'd': 8},
...     ],
... )
100%|##########| 2/2 [00:00<00:00, 31418.01it/s]
(((('1357', nan), (nan, nan)),
  ((nan, nan), (nan, nan))),
 (((nan, nan), (nan, nan)),
  ((nan, nan), (nan, '2468'))))

Run only certain cases of some args, but all combinations of others:

>>> xyz.combo_runner(
...     fn,
...     cases=[
...         {'a': 1, 'b': 3},
...         {'a': 2, 'b': 4},
...     ],
...     combos={
...         'c': [3, 4],
...         'd': [4, 5],
...     },
... )
100%|##########| 8/8 [00:00<00:00, 92691.80it/s]
(((('1334', '1335'), ('1344', '1345')),
  ((nan, nan), (nan, nan))),
 (((nan, nan), (nan, nan)),
  (('2434', '2435'), ('2444', '2445'))))

xyzpy.gen.combo_runner.multi_concat(results, dims)[source]¶: Concatenate a nested list of xarray objects along several dimensions.

xyzpy.gen.combo_runner.get_ndim_first(x, ndim)[source]¶: Return the first element from the ndim-nested list x.

xyzpy.gen.combo_runner.results_to_ds(results, combos, var_names, var_dims, var_coords, constants=None, attrs=None)[source]¶: Convert the output of combo_runner into a xarray.Dataset.

xyzpy.gen.combo_runner.results_to_df(results_linear, settings, attrs, resources, var_names)[source]¶: Convert the output of combo_runner into a pandas.DataFrame.

xyzpy.gen.combo_runner.combo_runner_to_ds(fn, combos, var_names, *, var_dims=None, var_coords=None, cases=None, constants=None, resources=None, attrs=None, shuffle=False, parse=True, to_df=False, parallel=False, num_workers=None, executor=None, verbosity=1, desc=None)[source]¶

Evaluate a function over all cases and combinations and output to a xarray.Dataset.

Parameters:

fn (callable) – Function to evaluate.
combos (dict_like[str, iterable]) – Mapping of each individual function argument to sequence of values.
var_names (str, sequence of strings, or None) – Variable name(s) of the output(s) of fn, set to None if fn outputs data already labelled in a Dataset or DataArray.
var_dims (sequence of either strings or string sequences, optional) – ‘Internal’ names of dimensions for each variable, the values for each dimension should be contained as a mapping in either var_coords (not needed by fn) or constants (needed by fn).
var_coords (mapping, optional) – Mapping of extra coords the output variables may depend on.
cases (sequence of dicts, optional) – Individual cases to run for some or all function arguments.
constants (mapping, optional) – Arguments to fn which are not iterated over, these will be recorded either as attributes or coordinates if they are named in var_dims.
resources (mapping, optional) – Like constants but they will not be recorded.
attrs (mapping, optional) – Any extra attributes to store.
parallel (bool, optional) – Process combos in parallel, default number of workers picked.
executor (executor-like pool, optional) – Submit all combos to this pool executor. Must have submit or apply_async methods and API matching either concurrent.futures or an ipyparallel view. Pools from multiprocessing.pool are also supported.
num_workers (int, optional) – Explicitly choose how many workers to use, None for automatic.
verbosity ({0, 1, 2}, optional) –
How much information to display:
- 0: nothing,
- 1: just progress,
- 2: postfix the current settings to the progress bar.
desc (str, optional) – Description to show in the progress bar, if verbosity > 0.

Returns:

ds – Multidimensional labelled dataset contatining all the results if to_df=False (the default), else a pandas dataframe with results as labelled rows.

Return type:

xarray.Dataset or pandas.DataFrame

xyzpy.gen.combo_runner.combo_runner_to_df¶