xyzpy.gen.combo_runner ====================== .. py:module:: xyzpy.gen.combo_runner .. autoapi-nested-parse:: Functions for systematically evaluating a function over all combinations. Attributes ---------- .. autoapisummary:: xyzpy.gen.combo_runner.combo_runner_to_df Functions --------- .. autoapisummary:: xyzpy.gen.combo_runner.infer_shape xyzpy.gen.combo_runner.nan_like_result xyzpy.gen.combo_runner._submit xyzpy.gen.combo_runner._get_result xyzpy.gen.combo_runner._run_linear_executor xyzpy.gen.combo_runner._run_linear_sequential xyzpy.gen.combo_runner._unflatten xyzpy.gen.combo_runner.combo_runner_core xyzpy.gen.combo_runner.combo_runner xyzpy.gen.combo_runner.multi_concat xyzpy.gen.combo_runner.get_ndim_first xyzpy.gen.combo_runner.results_to_ds xyzpy.gen.combo_runner.results_to_df xyzpy.gen.combo_runner.combo_runner_to_ds Module Contents --------------- .. py:function:: infer_shape(x) Take a nested sequence and find its shape as if it were an array. .. rubric:: Examples >>> x = [[10, 20, 30], [40, 50, 60]] >>> infer_shape(x) (2, 3) .. py:function:: nan_like_result(res) Take a single result of a function evaluation and calculate the same sequence of scalars or arrays but filled entirely with ``nan``. .. rubric:: Examples >>> res = (True, [[10, 20, 30], [40, 50, 60]], -42.0, 'hello') >>> nan_like_result(res) (array(nan), array([[nan, nan, nan], [nan, nan, nan]]), array(nan), None) .. py:function:: _submit(executor, fn, *args, **kwds) Default method for submitting to a executor. :param executor: A ``multiprocessing.pool`` or an pool executor with API matching either ``concurrent.futures``, or an ``ipyparallel`` view. :type executor: pool-executor like :param fn: The function to submit. :type fn: callable :param args: Supplied to ``fn``. :param kwds: Supplied to ``fn``. :rtype: future .. py:function:: _get_result(future) .. py:function:: _run_linear_executor(executor, fn, settings, verbosity=1, desc=None) .. py:function:: _run_linear_sequential(fn, settings, verbosity=1, desc=None) .. py:function:: _unflatten(store, all_combo_values, all_nan=None) .. py:function:: combo_runner_core(fn, combos, constants, cases=None, split=False, flat=False, shuffle=False, parallel=False, num_workers=None, executor=None, verbosity=1, desc=None, info=None) .. py:function:: combo_runner(fn, combos=None, *, cases=None, constants=None, split=False, flat=False, shuffle=False, parallel=False, executor=None, num_workers=None, verbosity=1, desc=None) Take a function ``fn`` and compute it over all combinations of named variables values, optionally showing progress and in parallel. :param fn: Function to analyse. :type fn: callable :param combos: All combinations of each argument to values mapping will be computed. Each argument range thus gets a dimension in the output array(s). :type combos: dict_like[str, iterable] :param cases: Optional list of specific configurations. If both ``combos`` and ``cases`` are given, then the function is computed for all sub-combinations in ``combos`` for each case in ``cases``, arguments can thus only appear in one or the other. Note that missing combinations of arguments will be represented by ``nan`` if creating a nested array. :type cases: sequence of mappings, optional :param constants: Constant function arguments. Unlike ``combos`` and ``cases``, these won't produce dimensions in the output result when ``flat=False``. :type constants: dict, optional :param split: Whether to split (unzip) the outputs of ``fn`` into multiple output arrays or not. :type split: bool, optional :param flat: Whether to return a flat list of results or to return a nested tuple suitable to be supplied to ``numpy.array``. :type flat: bool, optional :param shuffle: If given, compute the results in a random order (using ``random.seed`` and ``random.shuffle``), which can be helpful for distributing resources when not all cases are computationally equal. :type shuffle: bool or int, optional :param parallel: Process combos in parallel, default number of workers picked. :type parallel: bool, optional :param executor: Submit all combos to this pool executor. Must have ``submit`` or ``apply_async`` methods and API matching either ``concurrent.futures`` or an ``ipyparallel`` view. Pools from ``multiprocessing.pool`` are also supported. :type executor: executor-like pool, optional :param num_workers: Explicitly choose how many workers to use, None for automatic. :type num_workers: int, optional :param verbosity: How much information to display: - 0: nothing, - 1: just progress, - 2: postfix the current settings to the progress bar. :type verbosity: {0, 1, 2}, optional :param desc: Description to show in the progress bar, if ``verbosity > 0``. :type desc: str, optional :returns: **data** -- Nested tuple containing all combinations of running ``fn`` if ``flat == False`` else a flat list of results. :rtype: nested tuple .. rubric:: Examples >>> def fn(a, b, c, d): ... return str(a) + str(b) + str(c) + str(d) Run all possible combos:: >>> xyz.combo_runner( ... fn, ... combos={ ... 'a': [1, 2], ... 'b': [3, 4], ... 'c': [5, 6], ... 'd': [7, 8], ... }, ... ) 100%|##########| 16/16 [00:00<00:00, 84733.41it/s] (((('1357', '1358'), ('1367', '1368')), (('1457', '1458'), ('1467', '1468'))), ((('2357', '2358'), ('2367', '2368')), (('2457', '2458'), ('2467', '2468')))) Run only a selection of cases:: >>> xyz.combo_runner( ... fn, ... cases=[ ... {'a': 1, 'b': 3, 'c': 5, 'd': 7}, ... {'a': 2, 'b': 4, 'c': 6, 'd': 8}, ... ], ... ) 100%|##########| 2/2 [00:00<00:00, 31418.01it/s] (((('1357', nan), (nan, nan)), ((nan, nan), (nan, nan))), (((nan, nan), (nan, nan)), ((nan, nan), (nan, '2468')))) Run only certain cases of some args, but all combinations of others:: >>> xyz.combo_runner( ... fn, ... cases=[ ... {'a': 1, 'b': 3}, ... {'a': 2, 'b': 4}, ... ], ... combos={ ... 'c': [3, 4], ... 'd': [4, 5], ... }, ... ) 100%|##########| 8/8 [00:00<00:00, 92691.80it/s] (((('1334', '1335'), ('1344', '1345')), ((nan, nan), (nan, nan))), (((nan, nan), (nan, nan)), (('2434', '2435'), ('2444', '2445')))) .. py:function:: multi_concat(results, dims) Concatenate a nested list of xarray objects along several dimensions. .. py:function:: get_ndim_first(x, ndim) Return the first element from the ndim-nested list x. .. py:function:: results_to_ds(results, combos, var_names, var_dims, var_coords, constants=None, attrs=None) Convert the output of combo_runner into a :class:`xarray.Dataset`. .. py:function:: results_to_df(results_linear, settings, attrs, resources, var_names) Convert the output of combo_runner into a :class:`pandas.DataFrame`. .. py:function:: combo_runner_to_ds(fn, combos, var_names, *, var_dims=None, var_coords=None, cases=None, constants=None, resources=None, attrs=None, shuffle=False, parse=True, to_df=False, parallel=False, num_workers=None, executor=None, verbosity=1, desc=None) Evaluate a function over all cases and combinations and output to a :class:`xarray.Dataset`. :param fn: Function to evaluate. :type fn: callable :param combos: Mapping of each individual function argument to sequence of values. :type combos: dict_like[str, iterable] :param var_names: Variable name(s) of the output(s) of `fn`, set to None if fn outputs data already labelled in a Dataset or DataArray. :type var_names: str, sequence of strings, or None :param var_dims: 'Internal' names of dimensions for each variable, the values for each dimension should be contained as a mapping in either `var_coords` (not needed by `fn`) or `constants` (needed by `fn`). :type var_dims: sequence of either strings or string sequences, optional :param var_coords: Mapping of extra coords the output variables may depend on. :type var_coords: mapping, optional :param cases: Individual cases to run for some or all function arguments. :type cases: sequence of dicts, optional :param constants: Arguments to `fn` which are not iterated over, these will be recorded either as attributes or coordinates if they are named in `var_dims`. :type constants: mapping, optional :param resources: Like `constants` but they will not be recorded. :type resources: mapping, optional :param attrs: Any extra attributes to store. :type attrs: mapping, optional :param parallel: Process combos in parallel, default number of workers picked. :type parallel: bool, optional :param executor: Submit all combos to this pool executor. Must have ``submit`` or ``apply_async`` methods and API matching either ``concurrent.futures`` or an ``ipyparallel`` view. Pools from ``multiprocessing.pool`` are also supported. :type executor: executor-like pool, optional :param num_workers: Explicitly choose how many workers to use, None for automatic. :type num_workers: int, optional :param verbosity: How much information to display: - 0: nothing, - 1: just progress, - 2: postfix the current settings to the progress bar. :type verbosity: {0, 1, 2}, optional :param desc: Description to show in the progress bar, if ``verbosity > 0``. :type desc: str, optional :returns: **ds** -- Multidimensional labelled dataset contatining all the results if ``to_df=False`` (the default), else a pandas dataframe with results as labelled rows. :rtype: xarray.Dataset or pandas.DataFrame .. py:data:: combo_runner_to_df