xyzpy.gen.combo_runner#

Functions for systematically evaluating a function over all combinations.

Functions

`combo_runner`(fn[, combos, cases, constants, ...])	Take a function `fn` and compute it over all combinations of named variables values, optionally showing progress and in parallel.
`combo_runner_core`(fn, combos, constants[, ...])
`combo_runner_to_ds`(fn, combos, var_names, *)	Evaluate a function over all cases and combinations and output to a `xarray.Dataset`.
`get_ndim_first`(x, ndim)	Return the first element from the ndim-nested list x.
`infer_shape`(x)	Take a nested sequence and find its shape as if it were an array.
`multi_concat`(results, dims)	Concatenate a nested list of xarray objects along several dimensions.
`nan_like_result`(res)	Take a single result of a function evaluation and calculate the same sequence of scalars or arrays but filled entirely with `nan`.
`results_to_df`(results_linear, settings, ...)	Convert the output of combo_runner into a `pandas.DataFrame`.
`results_to_ds`(results, combos, var_names, ...)	Convert the output of combo_runner into a `xarray.Dataset`.

xyzpy.gen.combo_runner.combo_runner(fn, combos=None, *, cases=None, constants=None, split=False, flat=False, shuffle=False, parallel=False, executor=None, num_workers=None, verbosity=1)[source]#

Take a function fn and compute it over all combinations of named variables values, optionally showing progress and in parallel.

Parameters

fn (callable) – Function to analyse.
combos (dict_like[str, iterable]) – All combinations of each argument to values mapping will be computed. Each argument range thus gets a dimension in the output array(s).
cases (sequence of mappings, optional) – Optional list of specific configurations. If both combos and cases are given, then the function is computed for all sub-combinations in combos for each case in cases, arguments can thus only appear in one or the other. Note that missing combinations of arguments will be represented by nan if creating a nested array.
constants (dict, optional) – Constant function arguments. Unlike combos and cases, these won’t produce dimensions in the output result when flat=False.
split (bool, optional) – Whether to split (unzip) the outputs of fn into multiple output arrays or not.
flat (bool, optional) – Whether to return a flat list of results or to return a nested tuple suitable to be supplied to numpy.array.
shuffle (bool or int, optional) – If given, compute the results in a random order (using random.seed and random.shuffle), which can be helpful for distributing resources when not all cases are computationally equal.
parallel (bool, optional) – Process combos in parallel, default number of workers picked.
executor (executor-like pool, optional) – Submit all combos to this pool executor. Must have submit or apply_async methods and API matching either concurrent.futures or an ipyparallel view. Pools from multiprocessing.pool are also supported.
num_workers (int, optional) – Explicitly choose how many workers to use, None for automatic.
verbosity ({0, 1, 2}, optional) –
How much information to display:
- 0: nothing,
- 1: just progress,
- 2: all information.

Returns

data – Nested tuple containing all combinations of running fn if flat == False else a flat list of results.

Return type

nested tuple

Examples

>>> def fn(a, b, c, d):
...     return str(a) + str(b) + str(c) + str(d)

Run all possible combos:

>>> xyz.combo_runner(
...     fn,
...     combos={
...         'a': [1, 2],
...         'b': [3, 4],
...         'c': [5, 6],
...         'd': [7, 8],
...     },
... )
100%|##########| 16/16 [00:00<00:00, 84733.41it/s]

(((('1357', '1358'), ('1367', '1368')),
  (('1457', '1458'), ('1467', '1468'))),
 ((('2357', '2358'), ('2367', '2368')),
  (('2457', '2458'), ('2467', '2468'))))

Run only a selection of cases:

>>> xyz.combo_runner(
...     fn,
...     cases=[
...         {'a': 1, 'b': 3, 'c': 5, 'd': 7},
...         {'a': 2, 'b': 4, 'c': 6, 'd': 8},
...     ],
... )
100%|##########| 2/2 [00:00<00:00, 31418.01it/s]
(((('1357', nan), (nan, nan)),
  ((nan, nan), (nan, nan))),
 (((nan, nan), (nan, nan)),
  ((nan, nan), (nan, '2468'))))

Run only certain cases of some args, but all combinations of others:

>>> xyz.combo_runner(
...     fn,
...     cases=[
...         {'a': 1, 'b': 3},
...         {'a': 2, 'b': 4},
...     ],
...     combos={
...         'c': [3, 4],
...         'd': [4, 5],
...     },
... )
100%|##########| 8/8 [00:00<00:00, 92691.80it/s]
(((('1334', '1335'), ('1344', '1345')),
  ((nan, nan), (nan, nan))),
 (((nan, nan), (nan, nan)),
  (('2434', '2435'), ('2444', '2445'))))

xyzpy.gen.combo_runner.combo_runner_to_df(fn, combos, var_names, *, var_dims=None, var_coords=None, cases=None, constants=None, resources=None, attrs=None, shuffle=False, parse=True, to_df=True, parallel=False, num_workers=None, executor=None, verbosity=1)#

Evaluate a function over all cases and combinations and output to a xarray.Dataset.

Parameters

fn (callable) – Function to evaluate.
combos (dict_like[str, iterable]) – Mapping of each individual function argument to sequence of values.
var_names (str, sequence of strings, or None) – Variable name(s) of the output(s) of fn, set to None if fn outputs data already labelled in a Dataset or DataArray.
var_dims (sequence of either strings or string sequences, optional) – ‘Internal’ names of dimensions for each variable, the values for each dimension should be contained as a mapping in either var_coords (not needed by fn) or constants (needed by fn).
var_coords (mapping, optional) – Mapping of extra coords the output variables may depend on.
cases (sequence of dicts, optional) – Individual cases to run for some or all function arguments.
constants (mapping, optional) – Arguments to fn which are not iterated over, these will be recorded either as attributes or coordinates if they are named in var_dims.
resources (mapping, optional) – Like constants but they will not be recorded.
attrs (mapping, optional) – Any extra attributes to store.
parallel (bool, optional) – Process combos in parallel, default number of workers picked.
executor (executor-like pool, optional) – Submit all combos to this pool executor. Must have submit or apply_async methods and API matching either concurrent.futures or an ipyparallel view. Pools from multiprocessing.pool are also supported.
num_workers (int, optional) – Explicitly choose how many workers to use, None for automatic.
verbosity ({0, 1, 2}, optional) –
How much information to display:
- 0: nothing,
- 1: just progress,
- 2: all information.

Returns

ds – Multidimensional labelled dataset contatining all the results if to_df=False (the default), else a pandas dataframe with results as labelled rows.

Return type

xarray.Dataset or pandas.DataFrame

xyzpy.gen.combo_runner.combo_runner_to_ds(fn, combos, var_names, *, var_dims=None, var_coords=None, cases=None, constants=None, resources=None, attrs=None, shuffle=False, parse=True, to_df=False, parallel=False, num_workers=None, executor=None, verbosity=1)[source]#

Evaluate a function over all cases and combinations and output to a xarray.Dataset.

Parameters

fn (callable) – Function to evaluate.
combos (dict_like[str, iterable]) – Mapping of each individual function argument to sequence of values.
var_names (str, sequence of strings, or None) – Variable name(s) of the output(s) of fn, set to None if fn outputs data already labelled in a Dataset or DataArray.
var_dims (sequence of either strings or string sequences, optional) – ‘Internal’ names of dimensions for each variable, the values for each dimension should be contained as a mapping in either var_coords (not needed by fn) or constants (needed by fn).
var_coords (mapping, optional) – Mapping of extra coords the output variables may depend on.
cases (sequence of dicts, optional) – Individual cases to run for some or all function arguments.
constants (mapping, optional) – Arguments to fn which are not iterated over, these will be recorded either as attributes or coordinates if they are named in var_dims.
resources (mapping, optional) – Like constants but they will not be recorded.
attrs (mapping, optional) – Any extra attributes to store.
parallel (bool, optional) – Process combos in parallel, default number of workers picked.
executor (executor-like pool, optional) – Submit all combos to this pool executor. Must have submit or apply_async methods and API matching either concurrent.futures or an ipyparallel view. Pools from multiprocessing.pool are also supported.
num_workers (int, optional) – Explicitly choose how many workers to use, None for automatic.
verbosity ({0, 1, 2}, optional) –
How much information to display:
- 0: nothing,
- 1: just progress,
- 2: all information.

Returns

ds – Multidimensional labelled dataset contatining all the results if to_df=False (the default), else a pandas dataframe with results as labelled rows.

Return type

xarray.Dataset or pandas.DataFrame

xyzpy.gen.combo_runner.get_ndim_first(x, ndim)[source]#: Return the first element from the ndim-nested list x.

xyzpy.gen.combo_runner.infer_shape(x)[source]#

Take a nested sequence and find its shape as if it were an array.

Examples

>>> x = [[10, 20, 30], [40, 50, 60]]
>>> infer_shape(x)
(2, 3)

xyzpy.gen.combo_runner.multi_concat(results, dims)[source]#: Concatenate a nested list of xarray objects along several dimensions.

xyzpy.gen.combo_runner.nan_like_result(res)[source]#

Take a single result of a function evaluation and calculate the same sequence of scalars or arrays but filled entirely with nan.

Examples

>>> res = (True, [[10, 20, 30], [40, 50, 60]], -42.0, 'hello')
>>> nan_like_result(res)
(array(nan), array([[nan, nan, nan],
                    [nan, nan, nan]]), array(nan), None)

xyzpy.gen.combo_runner.results_to_df(results_linear, settings, attrs, resources, var_names)[source]#: Convert the output of combo_runner into a pandas.DataFrame.

xyzpy.gen.combo_runner.results_to_ds(results, combos, var_names, var_dims, var_coords, constants=None, attrs=None)[source]#: Convert the output of combo_runner into a xarray.Dataset.