xyzpy
#
Subpackages#
Submodules#
Package Contents#
Classes#
Compare the performance of various |
|
Running covariance class. |
|
Running mean & standard deviation using Welford's |
|
A very simple context manager class for timing blocks. |
|
Encapsulates all the details describing a single 'crop', that is, |
|
Container class for collecting and aggregating data to disk. |
|
Container class with all the information needed to systematically |
|
Like a Harvester, but randomly samples combos and writes the table of |
|
Basic |
|
A |
|
Functions#
|
Benchmark the time it takes to run |
|
|
|
Given |
|
Compute the real size of a python object. Taken from |
|
Turn any iterable into a progress bar, with notebook option |
|
Split a nested iterable at a specified level, i.e. in numpy language |
|
Evaluate a function over all cases and combinations and output to a |
|
Take a function |
|
Takes a list of |
|
Simple case runner that outputs the raw tuple of results. |
|
Find all cases in a dataset with missing data. |
|
Does the dataset or dataarray |
|
Convert maybe |
|
|
|
Automatically process a batch of cases into results. Should be run in an |
|
Automatically load all the crops found in the current directory. |
|
|
|
Convenient decorator to automatically wrap a function as a |
|
Automatically turn an array into a xarray dataset. Transpose |
|
Cache this function to disk, using joblib. |
|
Print out information about the range and any missing values for an |
|
Load a dataframe from disk. |
|
Loads a xarray dataset. Basically |
|
Glob files based on base_name, merge them, save this new dataset if |
|
Save a dataframe to disk. |
|
Saves a xarray dataset. |
|
Save dataset |
|
Make sure all the dimensions on all the variables appear in the |
|
Drop values across all dimensions for which all values are NaN. |
|
Creates a color map for a single hue, with bright defaults. |
|
Creates a color map for a single hue. |
|
Convert lists of colors between formats |
|
Auto version of |
|
Auto version of |
|
Auto version of |
|
Auto version of |
|
From |
|
Dataset histogram. |
|
From |
|
From |
|
Visualize |
|
Visualize all entries of a tensor, with indices mapped into the plane |
|
|
|
|
|
Auto version of |
|
Auto version of |
|
Auto version of |
|
From |
|
From |
|
From |
Attributes#
- xyzpy.benchmark(fn, setup=None, n=None, min_t=0.1, repeats=3, get='min', starmap=False)[source]#
Benchmark the time it takes to run
fn
.- Parameters:
fn (callable) – The function to time.
setup (callable, optional) – If supplied the function that sets up the argument for
fn
.n (int, optional) – If supplied, the integer to supply to
setup
offn
.min_t (float, optional) – Aim to repeat function enough times to take up this many seconds.
repeats (int, optional) – Repeat the whole procedure (with setup) this many times in order to take the minimum run time.
get ({'min', 'mean'}, optional) – Return the minimum or mean time for each run.
starmap (bool, optional) – Unpack the arguments from
setup
, if given.
- Returns:
t – The minimum, averaged, time to run
fn
in seconds.- Return type:
Examples
Just a parameter-less function:
>>> import xyzpy as xyz >>> import numpy as np >>> xyz.benchmark(lambda: np.linalg.eig(np.random.randn(100, 100))) 0.004726233000837965
The same but with a setup and size parameter
n
specified:>>> setup = lambda n: np.random.randn(n, n) >>> fn = lambda X: np.linalg.eig(X) >>> xyz.benchmark(fn, setup, 100) 0.0042192734545096755
- class xyzpy.Benchmarker(kernels, setup=None, names=None, benchmark_opts=None, data_name=None)[source]#
Compare the performance of various
kernels
. Internally this makes use ofbenchmark()
,Harvester()
and xyzpys plotting functionality.- Parameters:
kernels (sequence of callable) – The functions to compare performance with.
setup (callable, optional) – If given, setup each benchmark run by suppling the size argument
n
to this function first, then feeding its output to each of the functions.names (sequence of str, optional) – Alternate names to give the function, else they will be inferred.
benchmark_opts (dict, optional) – Supplied to
benchmark()
.data_name (str, optional) – If given, the file name the internal harvester will use to store results persistently.
- harvester#
The harvester that runs and accumulates all the data.
- Type:
xyz.Harvester
- ds#
Shortcut to the harvester’s full dataset.
- Type:
- property ds#
- run(ns, kernels=None, **harvest_opts)[source]#
Run the benchmarks. Each run accumulates rather than overwriting the results.
- Parameters:
ns (sequence of int or int) – The sizes to run the benchmarks with.
kernels (sequence of str, optional) – If given, only run the kernels with these names.
harvest_opts – Supplied to
harvest_combos()
.
- xyzpy.estimate_from_repeats(fn, *fn_args, rtol=0.02, tol_scale=1.0, get='stats', verbosity=0, min_samples=5, max_samples=1000000, **fn_kwargs)[source]#
- Parameters:
fn (callable) – The function that estimates a single value.
fn_args – Supplied to
fn
.optional – Supplied to
fn
.rtol (float, optional) – Relative tolerance for error on mean.
tol_scale (float, optional) – The expected ‘scale’ of the estimate, this modifies the aboslute tolerance near zero to
rtol * tol_scale
, default: 1.0.get ({'stats', 'samples', 'mean'}, optional) – Just get the
RunningStatistics
object, or the actual samples too, or just the actual mean estimate.verbosity ({ 0, 1, 2}, optional) –
How much information to show:
0
: nothing1
: progress bar just with iteration rate,2
: progress bar with running stats displayed.
min_samples (int, optional) – Take at least this many samples before checking for convergence.
max_samples (int, optional) – Take at maximum this many samples.
fn_kwargs – Supplied to
fn
.optional – Supplied to
fn
.
- Returns:
rs (RunningStatistics) – Statistics about the random estimation.
samples (list[float]) – If
get=='samples'
, the actual samples.
Examples
Estimate the sum of
n
random numbers:>>> import numpy as np >>> import xyzpy as xyz >>> def fn(n): ... return np.random.rand(n).sum() ... >>> stats = xyz.estimate_from_repeats(fn, n=10, verbosity=3) 59: 5.13(12): : 58it [00:00, 3610.84it/s] RunningStatistics(mean=5.13(12), count=59)
- xyzpy.format_number_with_error(x, err)[source]#
Given
x
with errorerr
, format a string showing the relevant digits ofx
with two significant digits of the error bracketed, and overall exponent if necessary.Examples
>>> print_number_with_uncertainty(0.1542412, 0.0626653) '0.154(63)'
>>> print_number_with_uncertainty(-128124123097, 6424) '-1.281241231(64)e+11'
- xyzpy.progbar(it=None, nb=False, **kwargs)[source]#
Turn any iterable into a progress bar, with notebook option
- Parameters:
it (iterable) – Iterable to wrap with progress bar
nb (bool) – Whether to display the notebook progress bar
**kwargs (dict-like) – additional options to send to tqdm
- class xyzpy.RunningCovariance[source]#
Running covariance class.
- property covar#
The covariance.
- property sample_covar#
The covariance with “Bessel’s correction”.
- class xyzpy.RunningCovarianceMatrix(n=2)[source]#
- property count#
- property covar_matrix#
- property sample_covar_matrix#
- to_uncertainties(bias=True)[source]#
Convert the accumulated statistics to correlated uncertainties, from which new quantities can be calculated with error automatically propagated.
- Parameters:
bias (bool, optional) – If False, use the sample covariance with “Bessel’s correction”.
- Returns:
values – The sequence of correlated variables.
- Return type:
tuple of uncertainties.ufloat
Examples
Estimate quantities of two perfectly correlated sequences.
>>> rcm = xyz.RunningCovarianceMatrix() >>> rcm.update_from_it((1, 3, 2), (2, 6, 4)) >>> x, y = rcm.to_uncertainties(rcm)
Calculated quantities like sums have the error propagated:
>>> x + y 6.0+/-2.4494897427831783
But the covariance is also taken into account, meaning the ratio here can be estimated with zero error:
>>> x / y 0.5+/-0
- class xyzpy.RunningStatistics[source]#
Running mean & standard deviation using Welford’s algorithm. This is a very efficient way of keeping track of the error on the mean for example.
Examples
>>> rs = RunningStatistics() >>> rs.update(1.1) >>> rs.update(1.4) >>> rs.update(1.2) >>> rs.update_from_it([1.5, 1.3, 1.6]) >>> rs.mean 1.3499999046325684
>>> rs.std # standard deviation 0.17078252585383266
>>> rs.err # error on the mean 0.06972167422092768
- property var#
- property std#
- property err#
- property rel_err#
- class xyzpy.Timer[source]#
A very simple context manager class for timing blocks.
Examples
>>> from xyzpy import Timer >>> with Timer() as timer: ... print('Doing some work!') ... Doing some work! >>> timer.t 0.00010752677917480469
- xyzpy.unzip(its, zip_level=1)[source]#
Split a nested iterable at a specified level, i.e. in numpy language transpose the specified ‘axis’ to be the first.
- Parameters:
its (iterable (of iterables (of iterables ...))) – ‘n-dimensional’ iterable to split
zip_level (int) – level at which to split the iterable, default of 1 replicates
zip(*its)
behaviour.
Example
>>> x = [[(1, True), (2, False), (3, True)], [(7, True), (8, False), (9, True)]] >>> nums, bools = unzip(x, 2) >>> nums ((1, 2, 3), (7, 8, 9)) >>> bools ((True, False, True), (True, False, True))
- xyzpy.combo_runner_to_df#
- xyzpy.combo_runner_to_ds(fn, combos, var_names, *, var_dims=None, var_coords=None, cases=None, constants=None, resources=None, attrs=None, shuffle=False, parse=True, to_df=False, parallel=False, num_workers=None, executor=None, verbosity=1)[source]#
Evaluate a function over all cases and combinations and output to a
xarray.Dataset
.- Parameters:
fn (callable) – Function to evaluate.
combos (dict_like[str, iterable]) – Mapping of each individual function argument to sequence of values.
var_names (str, sequence of strings, or None) – Variable name(s) of the output(s) of fn, set to None if fn outputs data already labelled in a Dataset or DataArray.
var_dims (sequence of either strings or string sequences, optional) – ‘Internal’ names of dimensions for each variable, the values for each dimension should be contained as a mapping in either var_coords (not needed by fn) or constants (needed by fn).
var_coords (mapping, optional) – Mapping of extra coords the output variables may depend on.
cases (sequence of dicts, optional) – Individual cases to run for some or all function arguments.
constants (mapping, optional) – Arguments to fn which are not iterated over, these will be recorded either as attributes or coordinates if they are named in var_dims.
resources (mapping, optional) – Like constants but they will not be recorded.
attrs (mapping, optional) – Any extra attributes to store.
parallel (bool, optional) – Process combos in parallel, default number of workers picked.
executor (executor-like pool, optional) – Submit all combos to this pool executor. Must have
submit
orapply_async
methods and API matching eitherconcurrent.futures
or anipyparallel
view. Pools frommultiprocessing.pool
are also supported.num_workers (int, optional) – Explicitly choose how many workers to use, None for automatic.
verbosity ({0, 1, 2}, optional) –
How much information to display:
0: nothing,
1: just progress,
2: all information.
- Returns:
ds – Multidimensional labelled dataset contatining all the results if
to_df=False
(the default), else a pandas dataframe with results as labelled rows.- Return type:
- xyzpy.combo_runner(fn, combos=None, *, cases=None, constants=None, split=False, flat=False, shuffle=False, parallel=False, executor=None, num_workers=None, verbosity=1)[source]#
Take a function
fn
and compute it over all combinations of named variables values, optionally showing progress and in parallel.- Parameters:
fn (callable) – Function to analyse.
combos (dict_like[str, iterable]) – All combinations of each argument to values mapping will be computed. Each argument range thus gets a dimension in the output array(s).
cases (sequence of mappings, optional) – Optional list of specific configurations. If both
combos
andcases
are given, then the function is computed for all sub-combinations incombos
for each case incases
, arguments can thus only appear in one or the other. Note that missing combinations of arguments will be represented bynan
if creating a nested array.constants (dict, optional) – Constant function arguments. Unlike
combos
andcases
, these won’t produce dimensions in the output result whenflat=False
.split (bool, optional) – Whether to split (unzip) the outputs of
fn
into multiple output arrays or not.flat (bool, optional) – Whether to return a flat list of results or to return a nested tuple suitable to be supplied to
numpy.array
.shuffle (bool or int, optional) – If given, compute the results in a random order (using
random.seed
andrandom.shuffle
), which can be helpful for distributing resources when not all cases are computationally equal.parallel (bool, optional) – Process combos in parallel, default number of workers picked.
executor (executor-like pool, optional) – Submit all combos to this pool executor. Must have
submit
orapply_async
methods and API matching eitherconcurrent.futures
or anipyparallel
view. Pools frommultiprocessing.pool
are also supported.num_workers (int, optional) – Explicitly choose how many workers to use, None for automatic.
verbosity ({0, 1, 2}, optional) –
How much information to display:
0: nothing,
1: just progress,
2: all information.
- Returns:
data – Nested tuple containing all combinations of running
fn
ifflat == False
else a flat list of results.- Return type:
nested tuple
Examples
>>> def fn(a, b, c, d): ... return str(a) + str(b) + str(c) + str(d)
Run all possible combos:
>>> xyz.combo_runner( ... fn, ... combos={ ... 'a': [1, 2], ... 'b': [3, 4], ... 'c': [5, 6], ... 'd': [7, 8], ... }, ... ) 100%|##########| 16/16 [00:00<00:00, 84733.41it/s] (((('1357', '1358'), ('1367', '1368')), (('1457', '1458'), ('1467', '1468'))), ((('2357', '2358'), ('2367', '2368')), (('2457', '2458'), ('2467', '2468'))))
Run only a selection of cases:
>>> xyz.combo_runner( ... fn, ... cases=[ ... {'a': 1, 'b': 3, 'c': 5, 'd': 7}, ... {'a': 2, 'b': 4, 'c': 6, 'd': 8}, ... ], ... ) 100%|##########| 2/2 [00:00<00:00, 31418.01it/s] (((('1357', nan), (nan, nan)), ((nan, nan), (nan, nan))), (((nan, nan), (nan, nan)), ((nan, nan), (nan, '2468'))))
Run only certain cases of some args, but all combinations of others:
>>> xyz.combo_runner( ... fn, ... cases=[ ... {'a': 1, 'b': 3}, ... {'a': 2, 'b': 4}, ... ], ... combos={ ... 'c': [3, 4], ... 'd': [4, 5], ... }, ... ) 100%|##########| 8/8 [00:00<00:00, 92691.80it/s] (((('1334', '1335'), ('1344', '1345')), ((nan, nan), (nan, nan))), (((nan, nan), (nan, nan)), (('2434', '2435'), ('2444', '2445'))))
- xyzpy.case_runner_to_df#
- xyzpy.case_runner_to_ds(fn, fn_args, cases, var_names, var_dims=None, var_coords=None, combos=None, constants=None, resources=None, attrs=None, shuffle=False, to_df=False, parse=True, parallel=False, num_workers=None, executor=None, verbosity=1)[source]#
Takes a list of
cases
to runfn
over, possibly in parallel, and outputs axarray.Dataset
.- Parameters:
fn (callable) – Function to evaluate.
fn_args (str or iterable[str]) – Names and order of arguments to
fn
, can beNone
ifcases
are supplied as dicts.cases (iterable[tuple] or iterable[dict]) – List of configurations used to generate results.
var_names (str or iterable of str) – Variable name(s) of the output(s) of
fn
.var_dims (sequence of either strings or string sequences, optional) – ‘Internal’ names of dimensions for each variable, the values for each dimension should be contained as a mapping in either var_coords (not needed by fn) or constants (needed by fn).
var_coords (mapping, optional) – Mapping of extra coords the output variables may depend on.
combos (dict_like[str, iterable], optional) – If specified, run all combinations of some arguments in these mappings.
constants (mapping, optional) – Arguments to fn which are not iterated over, these will be recorded either as attributes or coordinates if they are named in var_dims.
resources (mapping, optional) – Like constants but they will not be recorded.
attrs (mapping, optional) – Any extra attributes to store.
shuffle (bool or int, optional) – If given, compute the results in a random order (using
random.seed
andrandom.shuffle
), which can be helpful for distributing resources when not all cases are computationally equal.parse (bool, optional) – Whether to perform parsing of the inputs arguments.
parallel (bool, optional) – Process combos in parallel, default number of workers picked.
executor (executor-like pool, optional) – Submit all combos to this pool executor. Must have
submit
orapply_async
methods and API matching eitherconcurrent.futures
or anipyparallel
view. Pools frommultiprocessing.pool
are also supported.num_workers (int, optional) – Explicitly choose how many workers to use, None for automatic.
verbosity ({0, 1, 2}, optional) –
How much information to display:
0: nothing,
1: just progress,
2: all information.
- Returns:
ds – Dataset with minimal covering coordinates and all cases evaluated.
- Return type:
- xyzpy.case_runner(fn, fn_args, cases, combos=None, constants=None, split=False, shuffle=False, parse=True, parallel=False, executor=None, num_workers=None, verbosity=1)[source]#
Simple case runner that outputs the raw tuple of results.
- Parameters:
fn (callable) – Function with which to evalute cases with
fn_args (tuple) – Names of case arguments that fn takes, can be
None
if each case is adict
.cases (iterable[tuple] or iterable[dict]) – List of specific configurations that
fn_args
should take. Iffn_args
isNone
, each case should be adict
.combos (dict_like[str, iterable], optional) – Optional specification of sub-combinations.
constants (dict, optional) – Constant function arguments.
split (bool, optional) – See
combo_runner()
.shuffle (bool or int, optional) – If given, compute the results in a random order (using
random.seed
andrandom.shuffle
), which can be helpful for distributing resources when not all cases are computationally equal.parallel (bool, optional) – Process combos in parallel, default number of workers picked.
executor (executor-like pool, optional) – Submit all combos to this pool executor. Must have
submit
orapply_async
methods and API matching eitherconcurrent.futures
or anipyparallel
view. Pools frommultiprocessing.pool
are also supported.num_workers (int, optional) – Explicitly choose how many workers to use, None for automatic.
verbosity ({0, 1, 2}, optional) –
How much information to display:
0: nothing,
1: just progress,
2: all information.
- Returns:
results
- Return type:
list of fn output for each case
- xyzpy.find_missing_cases(ds, ignore_dims=None, method='isnull', show_progbar=False)[source]#
Find all cases in a dataset with missing data.
- Parameters:
ds (xarray.Dataset) – Dataset in which to find missing data
ignore_dims (set (optional)) – internal variable dimensions (i.e. to ignore)
show_progbar (bool (optional)) – Show the current progress
- Returns:
Function arguments and missing cases.
- Return type:
missing_fn_args, missing_cases
- xyzpy.is_case_missing(ds, setting, method='isnull')[source]#
Does the dataset or dataarray
ds
not contain any non-null data for locationsetting
?Note that this only returns true if all data across all variables is completely missing at the location.
- Parameters:
ds (xarray.Dataset or xarray.DataArray) – Dataset or dataarray to look in.
setting (dict[str, hashable]) – Coordinate location to check.
- Returns:
missing
- Return type:
- xyzpy.parse_into_cases(combos=None, cases=None, ds=None, method='isnull')[source]#
Convert maybe
combos
and maybecases
to a single list ofcases
only, also optionally filtering based on whether any data at each location is already present in Dataset or DataArrayds
.- Parameters:
combos (dict_like[str, iterable], optional) – Parameter combinations.
cases (iterable[dict], optional) – Parameter configurations.
ds (xarray.Dataset or xarray.DataArray, optional) – Dataset or DataArray in which to check for existing data.
- Returns:
new_cases – The combined and possibly filtered list of cases.
- Return type:
iterable[dict]
- class xyzpy.Crop(*, fn=None, name=None, parent_dir=None, save_fn=None, batchsize=None, num_batches=None, shuffle=False, farmer=None, autoload=True)[source]#
Bases:
object
Encapsulates all the details describing a single ‘crop’, that is, its location, name, and batch size/number. Also allows tracking of crop’s progress, and experimentally, automatic submission of workers to grid engine to complete un-grown cases. Can also be instantiated directly from a
Runner
orHarvester
orCrop
instance.- Parameters:
fn (callable, optional) – Target function - Crop name will be inferred from this if not given explicitly. If given, Sower will also default to saving a version of fn to disk for cropping.grow to use.
name (str, optional) – Custom name for this set of runs - must be given if fn is not.
parent_dir (str, optional) – If given, alternative directory to put the “.xyz-{name}/” folder in with all the cases and results.
save_fn (bool, optional) – Whether to save the function to disk for cropping.grow to use. Will default to True if fn is given.
batchsize (int, optional) – How many cases to group into a single batch per worker. By default, batchsize=1. Cannot be specified if num_batches is.
num_batches (int, optional) – How many total batches to aim for, cannot be specified if batchsize is.
farmer ({xyzpy.Runner, xyzpy.Harvester, xyzpy.Sampler}, optional) – A Runner, Harvester or Sampler, instance, from which the fn can be inferred and which can also allow the Crop to reap itself straight to a dataset or dataframe.
autoload (bool, optional) – If True, check for the existence of a Crop written to disk with the same location, and if found, load it.
See also
- property runner#
- property all_nan_result#
Get a stand-in result for cases which are missing still.
- property num_sown_batches#
Total number of batches to be run/grown.
- property num_results#
- fn#
- choose_batch_settings(*, combos=None, cases=None)[source]#
Work out how to divide all cases into batches, i.e. ensure that
batchsize * num_batches >= num_cases
.
- load_function()[source]#
Load the saved function from disk, and try to re-insert it back into Harvester or Runner if present.
- prepare(combos=None, cases=None, fn_args=None)[source]#
Write information about this crop and the supplied combos to disk. Typically done at start of sow, not when Crop instantiated.
- sow_combos(combos, cases=None, constants=None, shuffle=False, verbosity=1, batchsize=None, num_batches=None)[source]#
Sow combos to disk to be later grown, potentially in batches. Note if you have already sown this Crop, as long as the number of batches hasn’t changed (e.g. you have just tweaked the function or a constant argument), you can safely resow and only the batches will be overwritten, i.e. the results will remain.
- Parameters:
combos (dict_like[str, iterable]) – The combinations to sow for all or some function arguments.
cases (iterable or mappings, optional) – Optionally provide an sequence of individual cases to sow for some or all function arguments.
constants (mapping, optional) – Provide additional constant function values to use when sowing.
shuffle (bool or int, optional) – If given, sow the combos in a random order (using
random.seed
andrandom.shuffle
), which can be helpful for distributing resources when not all cases are computationally equal.verbosity (int, optional) – How much information to show when sowing.
batchsize (int, optional) – If specified, set a new batchsize for the crop.
num_batches (int, optional) – If specified, set a new num_batches for the crop.
- sow_cases(fn_args, cases, combos=None, constants=None, verbosity=1, batchsize=None, num_batches=None)[source]#
Sow cases to disk to be later grown, potentially in batches.
- Parameters:
fn_args (iterable[str] or str) – The names and order of the function arguments, can be
None
if each case is supplied as adict
.cases (iterable or mappings, optional) – Sequence of individual cases to sow for all or some function arguments.
combos (dict_like[str, iterable]) – Combinations to sow for some or all function arguments.
constants (mapping, optional) – Provide additional constant function values to use when sowing.
verbosity (int, optional) – How much information to show when sowing.
batchsize (int, optional) – If specified, set a new batchsize for the crop.
num_batches (int, optional) – If specified, set a new num_batches for the crop.
- reap_combos(wait=False, clean_up=None, allow_incomplete=False)[source]#
Reap already sown and grown results from this crop.
- Parameters:
wait (bool, optional) – Whether to wait for results to appear. If false (default) all results need to be in place before the reap.
clean_up (bool, optional) – Whether to delete all the batch files once the results have been gathered. If left as
None
this will be automatically set tonot allow_incomplete
.allow_incomplete (bool, optional) – Allow only partially completed crop results to be reaped, incomplete results will all be filled-in as nan.
- Returns:
results – ‘N-dimensional’ tuple containing the results.
- Return type:
nested tuple
- reap_combos_to_ds(var_names=None, var_dims=None, var_coords=None, constants=None, attrs=None, parse=True, wait=False, clean_up=None, allow_incomplete=False, to_df=False)[source]#
Reap a function over sowed combinations and output to a Dataset.
- Parameters:
var_names (str, sequence of strings, or None) – Variable name(s) of the output(s) of fn, set to None if fn outputs data already labelled in a Dataset or DataArray.
var_dims (sequence of either strings or string sequences, optional) – ‘Internal’ names of dimensions for each variable, the values for each dimension should be contained as a mapping in either var_coords (not needed by fn) or constants (needed by fn).
var_coords (mapping, optional) – Mapping of extra coords the output variables may depend on.
constants (mapping, optional) – Arguments to fn which are not iterated over, these will be recorded either as attributes or coordinates if they are named in var_dims.
resources (mapping, optional) – Like constants but they will not be recorded.
attrs (mapping, optional) – Any extra attributes to store.
wait (bool, optional) – Whether to wait for results to appear. If false (default) all results need to be in place before the reap.
clean_up (bool, optional) – Whether to delete all the batch files once the results have been gathered. If left as
None
this will be automatically set tonot allow_incomplete
.allow_incomplete (bool, optional) – Allow only partially completed crop results to be reaped, incomplete results will all be filled-in as nan.
to_df (bool, optional) – Whether to reap to a
xarray.Dataset
or apandas.DataFrame
.
- Returns:
Multidimensional labelled dataset contatining all the results.
- Return type:
xarray.Dataset or pandas.Dataframe
- reap_runner(runner, wait=False, clean_up=None, allow_incomplete=False, to_df=False)[source]#
Reap a Crop over sowed combos and save to a dataset defined by a
Runner
.
- reap_harvest(harvester, wait=False, sync=True, overwrite=None, clean_up=None, allow_incomplete=False)[source]#
Reap a Crop over sowed combos and merge with the dataset defined by a
Harvester
.
- reap_samples(sampler, wait=False, sync=True, clean_up=None, allow_incomplete=False)[source]#
Reap a Crop over sowed combos and merge with the dataframe defined by a
Sampler
.
- reap(wait=False, sync=True, overwrite=None, clean_up=None, allow_incomplete=False)[source]#
Reap sown and grown combos from disk. Return a dataset if a runner or harvester is set, otherwise, the raw nested tuple.
- Parameters:
wait (bool, optional) – Whether to wait for results to appear. If false (default) all results need to be in place before the reap.
sync (bool, optional) – Immediately sync the new dataset with the on-disk full dataset or dataframe if a harvester or sampler is used.
overwrite (bool, optional) – How to compare data when syncing to on-disk dataset. If
None
, (default) merge as long as no conflicts.True
: overwrite with the new data.False
, discard any new conflicting data.clean_up (bool, optional) – Whether to delete all the batch files once the results have been gathered. If left as
None
this will be automatically set tonot allow_incomplete
.allow_incomplete (bool, optional) – Allow only partially completed crop results to be reaped, incomplete results will all be filled-in as nan.
- Return type:
nested tuple or xarray.Dataset
- xyzpy.grow(batch_number, crop=None, fn=None, num_workers=None, check_mpi=True, verbosity=2, debugging=False)[source]#
Automatically process a batch of cases into results. Should be run in an “.xyz-{fn_name}” folder.
- Parameters:
batch_number (int) – Which batch to ‘grow’ into a set of results.
crop (xyzpy.Crop) – Description of where and how to store the cases and results.
fn (callable, optional) – If specified, the function used to generate the results, otherwise the function will be loaded from disk.
num_workers (int, optional) – If specified, grow using a pool of this many workers. This uses
joblib.externals.loky
to spawn processes.check_mpi (bool, optional) – Whether to check if the process is rank 0 and only save results if so - allows mpi functions to be simply used. Defaults to true, this should only be turned off if e.g. a pool of workers is being used to run different
grow
instances.verbosity ({0, 1, 2}, optional) – How much information to show.
debugging (bool, optional) – Set logging level to DEBUG.
- xyzpy.load_crops(directory='.')[source]#
Automatically load all the crops found in the current directory.
- class xyzpy.Harvester(runner, data_name=None, chunks=None, engine='h5netcdf', full_ds=None)[source]#
Bases:
object
Container class for collecting and aggregating data to disk.
- Parameters:
runner (Runner) – Performs the runs and describes the results.
data_name (str, optional) – Base file path to save data to.
chunks (int or dict, optional) – If not None, passed to xarray so that the full dataset is loaded and merged into with on-disk dask arrays.
engine (str, optional) – Engine to use to save and load datasets.
full_ds (xarray.Dataset) – Initialize the Harvester with this dataset as the intitial full dataset.
Members –
------- –
full_ds – Dataset containing all data harvested so far, by default synced to disk.
last_ds (xarray.Dataset) – Dataset containing just the data from the last harvesting run.
- property fn#
- property last_ds#
Dataset containing the last runs’ data.
- property full_ds#
Dataset containing all saved runs.
- save_full_ds(new_full_ds=None, engine=None)[source]#
Save full_ds onto disk.
- Parameters:
new_full_ds (xarray.Dataset, optional) – Save this dataset as the new full dataset, else use the current full datset.
engine (str, optional) – Engine to use to save and load datasets.
- add_ds(new_ds, sync=True, overwrite=None, chunks=None, engine=None)[source]#
Merge a new dataset into the in-memory full dataset.
- Parameters:
new_ds (xr.Dataset or xr.DataArray) – Data to be merged into the full dataset.
sync (bool, optional) – If True (default), load and save the disk dataset before and after merging in the new data.
overwrite ({None, False, True}, optional) –
How to combine data from the new run into the current full_ds:
None
(default): attempt the merge and only raise if data conflicts.True
: overwrite conflicting current data with that from the new dataset.False
: drop any conflicting data from the new dataset.
chunks (int or dict, optional) – If not None, passed to xarray so that the full dataset is loaded and merged into with on-disk dask arrays.
engine (str, optional) – Engine to use to save and load datasets.
- expand_dims(name, value, engine=None)[source]#
Add a new coordinate dimension with
name
andvalue
. The change is immediately synced with the on-disk dataset. Useful if you want to expand the parameter space along a previously constant argument.
- drop_sel(labels=None, *, errors='raise', engine=None, **labels_kwargs)[source]#
Drop specific values of coordinates from this harvester and its dataset. See http://xarray.pydata.org/en/latest/generated/xarray.Dataset.drop_sel.html. The change is immediately synced with the on-disk dataset. Useful for tidying uneeded data points.
- harvest_combos(combos, *, sync=True, overwrite=None, chunks=None, engine=None, **runner_settings)[source]#
Run combos, automatically merging into an on-disk dataset.
- Parameters:
combos (dict_like[str, iterable]) – The combos to run. The only difference here is that you can supply an ellipse
...
, meaning the all values for that coordinate will be loaded from the current full dataset.sync (bool, optional) – If True (default), load and save the disk dataset before and after merging in the new data.
overwrite ({None, False, True}, optional) –
None
(default): attempt the merge and only raise if data conflicts.True
: overwrite conflicting current data with that from the new dataset.False
: drop any conflicting data from the new dataset.
chunks (bool, optional) – If not None, passed passed to xarray so that the full dataset is loaded and merged into with on-disk dask arrays.
engine (str, optional) – Engine to use to save and load datasets.
runner_settings – Supplied to
combo_runner()
.
- harvest_cases(cases, *, sync=True, overwrite=None, chunks=None, engine=None, **runner_settings)[source]#
Run cases, automatically merging into an on-disk dataset.
- Parameters:
sync (bool, optional) – If True (default), load and save the disk dataset before and after merging in the new data.
overwrite ({None, False, True}, optional) –
What to do regarding clashes with old data:
None
(default): attempt the merge and only raise if data conflicts.True
: overwrite conflicting current data with that from the new dataset.False
: drop any conflicting data from the new dataset.
chunks (bool, optional) – If not None, passed passed to xarray so that the full dataset is loaded and merged into with on-disk dask arrays.
engine (str, optional) – Engine to use to save and load datasets.
runner_settings – Supplied to
case_runner()
.
- xyzpy.label(var_names, fn_args=None, var_dims=None, var_coords=None, constants=None, resources=None, attrs=None, harvester=False, sampler=False, engine=None, **default_runner_settings)[source]#
Convenient decorator to automatically wrap a function as a
Runner
orHarvester
.- Parameters:
var_names (str, sequence of str, or None) – The ordered name(s) of the ouput variable(s) of fn. Set this explicitly to None if fn outputs already labelled data as a
Dataset
orDataArray
.fn_args (str, or sequence of str, optional) – The ordered name(s) of the input arguments(s) of fn. This is only needed if the cases or combos supplied are not dict-like.
var_dims (dict-like, optional) – Mapping of output variables to their named internal dimensions, can be the names of
constants
.var_coords (dict-like, optional) – Mapping of output variables named internal dimensions to the actual values they take.
constants (dict-like, optional) – Constants arguments to be supplied to fn. These can be used as ‘var_dims’, and will be saved as coords if so, otherwise as attributes.
resources (dict-like, optional) – Like constants but not saved to the the dataset, e.g. if very big.
attrs (dict-like, optional) – Any other miscelleous information to be saved with the dataset.
harvester (bool or str, optional) – If
True
, wrap the runner as aHarvester
, if a string, create the harvester with that as thedata_name
.default_runner_settings – These keyword arguments will be supplied as defaults to any runner.
Examples
Declare a function as a runner directly:
>>> import xyzpy as xyz >>> @xyz.label(var_names=['sum', 'diff']) ... def foo(x, y): ... return x + y, x - y ... >>> foo <xyzpy.Runner> fn: <function foo at 0x7f1fd8e5b1e0> fn_args: ('x', 'y') var_names: ('sum', 'diff') var_dims: {'sum': (), 'diff': ()} >>> foo(1, 2) # can still call it normally (3, -1)
- class xyzpy.Runner(fn, var_names, fn_args=None, var_dims=None, var_coords=None, constants=None, resources=None, attrs=None, **default_runner_settings)[source]#
Bases:
object
Container class with all the information needed to systematically run a function over many parameters and capture the output in a dataset.
- Parameters:
fn (callable) – Function that produces a single instance of a result.
var_names (str, sequence of str, or None) – The ordered name(s) of the ouput variable(s) of fn. Set this explicitly to None if fn outputs already labelled data as a
Dataset
orDataArray
.fn_args (str, or sequence of str, optional) – The ordered name(s) of the input arguments(s) of fn. This is only needed if the cases or combos supplied are not dict-like.
var_dims (dict-like, optional) – Mapping of output variables to their named internal dimensions, can be the names of
constants
.var_coords (dict-like, optional) – Mapping of output variables named internal dimensions to the actual values they take.
constants (dict-like, optional) – Constants arguments to be supplied to fn. These can be used as ‘var_dims’, and will be saved as coords if so, otherwise as attributes.
resources (dict-like, optional) – Like constants but not saved to the the dataset, e.g. if very big.
attrs (dict-like, optional) – Any other miscelleous information to be saved with the dataset.
default_runner_settings – These keyword arguments will be supplied as defaults to any runner.
- property last_ds#
- fn_args#
- var_names#
- var_dims#
- var_coords#
- constants#
- resources#
- run_combos(combos, constants=(), **runner_settings)[source]#
Run combos using the function map and save to dataset.
- Parameters:
combos (dict_like[str, iterable]) – The values of each function argument with which to evaluate all combinations.
constants (dict, optional) – Extra constant arguments for this run, repeated arguments will take precedence over stored constants but for this run only.
runner_settings – Keyword arguments supplied to
combo_runner()
.
- run_cases(cases, constants=(), fn_args=None, **runner_settings)[source]#
Run cases using the function and save to dataset.
- Parameters:
cases (sequence of mappings or tuples) – A sequence of cases.
constants (dict (optional)) – Extra constant arguments for this run, repeated arguments will take precedence over stored constants but for this run only.
runner_settings – Supplied to
case_runner()
.
- class xyzpy.Sampler(runner, data_name=None, default_combos=None, full_df=None, engine='pickle')[source]#
Like a Harvester, but randomly samples combos and writes the table of results to a
pandas.DataFrame
.- Parameters:
runner (xyzpy.Runner) – Runner describing a labelled function to run.
data_name (str, optional) – If given, the on-disk file to sync results with.
default_combos (dict_like[str, iterable], optional) – The default combos to sample from (which can be overridden).
full_df (pandas.DataFrame, optional) – If given, use this dataframe as the initial ‘full’ data.
engine ({'pickle', 'csv', 'json', 'hdf', ...}, optional) – How to save and load the on-disk dataframe. See
load_df()
andsave_df()
.
- full_df#
Dataframe describing all data harvested so far.
- Type:
- last_df#
Dataframe describing the data harvested on the previous run.
- Type:
pandas.Dataframe
- property fn#
- property full_df#
The dataframe describing all data harvested so far.
- property last_df#
The dataframe describing the last set of data harvested.
- save_full_df(new_full_df=None, engine=None)[source]#
Save full_df onto disk.
- Parameters:
new_full_df (pandas.DataFrame, optional) – Save this dataframe as the new full dataframe, else use the current
full_df
.engine (str, optional) – Which engine to save the dataframe with, if None use the default.
- add_df(new_df, sync=True, engine=None)[source]#
Merge a new dataset into the in-memory full dataset.
- Parameters:
new_df (pandas.DataFrame or dict) – Data to be appended to the full dataset.
sync (bool, optional) – If True (default), load and save the disk dataframe before and after merging in the new data.
engine (str, optional) – Which engine to save the dataframe with.
- sample_combos(n, combos=None, engine=None, **case_runner_settings)[source]#
Sample the target function many times, randomly choosing parameter combinations from
combos
(orSampleHarvester.default_combos
).- Parameters:
n (int) – How many samples to run.
combos (dict_like[str, iterable], optional) – A mapping of function arguments to potential choices. Any keys in here will override
default_combos
. You can also suppply a callable to manually return a random choice e.g. from a probability distribution.engine (str, optional) – Which method to use to sync with the on-disk dataframe.
case_runner_settings – Supplied to
case_runner()
and so ontocombo_runner()
. This includesparallel=True
etc.
- class xyzpy.RayExecutor(*args, default_remote_opts=None, **kwargs)[source]#
Basic
concurrent.futures
like interface usingray
.Example usage:
from xyzpy import RayExecutor # create a pool that by default requests a single gpu per task pool = RayExecutor( num_cpus=4, num_gpus=4, default_remote_opts={"num_gpus": 1}, )
- _maybe_inject_remote_opts(remote_opts=None)[source]#
Return the default remote options, possibly overriding some with those supplied by a
submit call
.
- submit(fn, *args, pure=False, remote_opts=None, **kwargs)[source]#
Remotely run
fn(*args, **kwargs)
, returning aRayFuture
.
- class xyzpy.RayGPUExecutor(*args, gpus_per_task=1, **kwargs)[source]#
Bases:
RayExecutor
A
RayExecutor
that by default requests a single gpu per task.
- xyzpy.auto_xyz_ds(x, y_z=None)[source]#
Automatically turn an array into a xarray dataset. Transpose
y_z
if necessary to automatically match dimension sizes.- Parameters:
x (array_like) – The x-coordinates.
y_z (array_like, optional) – The y-data, possibly varying with coordinate z.
- xyzpy.cache_to_disk(fn=None, *, cachedir=_DEFAULT_FN_CACHE_PATH, **kwargs)[source]#
Cache this function to disk, using joblib.
- xyzpy.check_runs(obj, dim='run', var=None, sel=())[source]#
Print out information about the range and any missing values for an integer dimension.
- xyzpy.load_ds(file_name, engine='h5netcdf', load_to_mem=None, create_new=False, chunks=None, **kwargs)[source]#
Loads a xarray dataset. Basically
xarray.open_dataset
with some different defaults and convenient behaviour.- Parameters:
file_name (str) – Name of file to open.
engine ({'h5netcdf', 'netcdf4', 'joblib', 'zarr'}, optional) – Engine used to load file.
load_to_mem (bool, optional) – Ince opened, load from disk into memory. Defaults to
True
ifchunks=None
.create_new (bool, optional) – If no file exists make a blank one.
chunks (int or dict) – Passed to
xarray.open_dataset
so that data is stored usingdask.array
.
- Returns:
ds – Loaded Dataset.
- Return type:
- xyzpy.merge_sync_conflict_datasets(base_name, engine='h5netcdf', combine_first=False)[source]#
Glob files based on base_name, merge them, save this new dataset if it contains new info, then clean up the conflicts.
- xyzpy.save_ds(ds, file_name, engine='h5netcdf', **kwargs)[source]#
Saves a xarray dataset.
- Parameters:
ds (xarray.Dataset) – The dataset to save.
file_name (str) – Name of the file to save to.
engine ({'h5netcdf', 'netcdf4', 'joblib', 'zarr'}, optional) – Engine used to save file with.
- Return type:
None
- xyzpy.save_merge_ds(ds, fname, overwrite=None, **kwargs)[source]#
Save dataset
ds
, but check for an existing dataset with that name first, and if it exists, merge the two before saving.- Parameters:
ds (xarray.Dataset) – The dataset to save.
fname (str) – The file name.
overwrite ({None, False, True}, optional) –
How to merge the dataset with the existing dataset.
None: the datasets will be merged in there are no conflicts
False: data will be taken from old dataset if conflicting
True: data will be taken from new dataset if conflicting
- xyzpy.sort_dims(ds)[source]#
Make sure all the dimensions on all the variables appear in the same order.
- xyzpy.cimple_bright(hue, sat1=0.8, sat2=0.9, val1=0.97, val2=0.3, hue_shift=0.0, name='cimple_bright')[source]#
Creates a color map for a single hue, with bright defaults.
- xyzpy.cimple(hue, sat1=0.4, sat2=1.0, val1=0.95, val2=0.35, hue_shift=0.0, name='cimple', auto_adjust_sat=0.2)[source]#
Creates a color map for a single hue.
- xyzpy.convert_colors(cols, outformat, informat='MATPLOTLIB')[source]#
Convert lists of colors between formats
- xyzpy.auto_heatmap(x, **heatmap_opts)[source]#
Auto version of
heatmap()
that accepts array arguments by converting them to aDataset
first.
- xyzpy.auto_histogram(x, **histogram_opts)[source]#
Auto version of
histogram()
that accepts array arguments by converting them to aDataset
first.
- xyzpy.auto_lineplot(x, y_z, **lineplot_opts)[source]#
Auto version of
lineplot()
that accepts array arguments by converting them to aDataset
first.
- xyzpy.auto_scatter(x, y_z, **scatter_opts)[source]#
Auto version of
scatter()
that accepts array arguments by converting them to aDataset
first.
- xyzpy.heatmap(ds, x, y, z, **kwargs)[source]#
From
ds
plot variablez
as a function ofx
andy
using a 2D heatmap.- Parameters:
ds (xarray.Dataset) – Dataset to plot from.
x (str) – Dimension to plot along the x-axis.
y (str) – Dimension to plot along the y-axis.
z (str, optional) – Variable to plot as colormap.
row (str, optional) – Dimension to vary over as a function of rows.
col (str, optional) – Dimension to vary over as a function of columns.
plot_opts – See
xyzpy.plot.core.PLOTTER_DEFAULTS
.
- class xyzpy.HeatMap(ds, x, y, z, **kwargs)[source]#
Bases:
PlotterMatplotlib
,xyzpy.plot.core.AbstractHeatMap
- xyzpy.histogram(ds, x, z=None, **plot_opts)[source]#
Dataset histogram.
- Parameters:
ds (xarray.Dataset) – The dataset to plot.
x (str, sequence of str) – The variable(s) to plot the probability density of. If sequence, plot a histogram of each instead of using a
z
coordinate.z (str, optional) – If given, range over this coordinate a plot a histogram for each.
row (str, optional) – Dimension to vary over as a function of rows.
col (str, optional) – Dimension to vary over as a function of columns.
plot_opts – See
xyzpy.plot.core.PLOTTER_DEFAULTS
.
- class xyzpy.Histogram(ds, x, z=None, **kwargs)[source]#
Bases:
PlotterMatplotlib
,xyzpy.plot.core.AbstractHistogram
- xyzpy.lineplot(ds, x, y, z=None, y_err=None, x_err=None, **plot_opts)[source]#
From
ds
plot lines ofy
as a function ofx
, optionally for varyingz
.- Parameters:
ds (xarray.Dataset) – Dataset to plot from.
x (str) – Dimension to plot along the x-axis.
y (str or tuple[str]) – Variable(s) to plot along the y-axis. If tuple, plot each of the variables - instead of
z
.z (str, optional) – Dimension to plot into the page.
y_err (str, optional) – Variable to plot as y-error.
x_err (str, optional) – Variable to plot as x-error.
row (str, optional) – Dimension to vary over as a function of rows.
col (str, optional) – Dimension to vary over as a function of columns.
plot_opts – See
xyzpy.plot.core.PLOTTER_DEFAULTS
.
- class xyzpy.LinePlot(ds, x, y, z=None, *, y_err=None, x_err=None, **kwargs)[source]#
Bases:
PlotterMatplotlib
,xyzpy.plot.core.AbstractLinePlot
- xyzpy.scatter(ds, x, y, z=None, y_err=None, x_err=None, **plot_opts)[source]#
From
ds
plot a scatter ofy
againstx
, optionally for varyingz
.- Parameters:
ds (xarray.Dataset) – Dataset to plot from.
x (str) – Quantity to plot along the x-axis.
y (str or tuple[str]) – Quantity(s) to plot along the y-axis. If tuple, plot each of the variables - instead of
z
.z (str, optional) – Dimension to plot into the page.
y_err (str, optional) – Variable to plot as y-error.
x_err (str, optional) – Variable to plot as x-error.
row (str, optional) – Dimension to vary over as a function of rows.
col (str, optional) – Dimension to vary over as a function of columns.
plot_opts – See
xyzpy.plot.core.PLOTTER_DEFAULTS
.
- class xyzpy.Scatter(ds, x, y, z=None, **kwargs)[source]#
Bases:
PlotterMatplotlib
,xyzpy.plot.core.AbstractScatter
- xyzpy.visualize_matrix(array, max_mag=None, magscale='linear', alpha_map=True, alpha_pow=1 / 2, legend=True, legend_loc='auto', legend_size=0.15, legend_bounds=None, legend_resolution=3, facecolor=None, rasterize=4096, rasterize_dpi=300, figsize=(5, 5), ax=None)[source]#
Visualize
array
as a 2D colormapped image.- Parameters:
array (array_like or Sequence[array_like]) – A 2D (or 1D) array or sequence of arrays to visualize.
max_mag (float, optional) – The maximum magnitude to use for the color mapping. If not provided, the maximum magnitude in the array will be used.
magscale ("linear" or float, optional) – How to scale the magnitude of the array values. If “linear”, then the magnitude is used directly. If a float, then the magnitude is raised to this power before being used, which can help to show variation among small values.
alpha_map (bool, optional) – Whether to map the tensor value magnitudes to pixel alpha.
alpha_pow (float, optional) – The power to raise the magnitude to when mapping to alpha.
legend (bool, optional) – Whether to show a legend (colorbar). If the array has complex dtype then the legend will be a colorwheel.
legend_loc (str or tuple[float], optional) – Where to place the legend. If “auto”, then the legend will be placed outside the plot rectangle, otherwise it should be a tuple of
(x, y)
coordinates in axes space.legend_size (float, optional) – The size of the legend, in relation to the size of the plot axes.
legend_bounds (tuple[float], optional) – The bounds of the legend, as
(x, y, width, height)
in axes space. If not provided, the bounds will be computed fromlegend_loc
andlegend_size
.legend_resolution (int, optional) – The number of different colors to show in the legend.
facecolor (str, optional) – The background color of the plot, by default transparent.
rasterize (int or float, optional) – Whether to rasterize the plot. If a number, then rasterize if the number of pixels in the plot is greater than this value.
rasterize_dpi (float, optional) – The dpi to use when rasterizing.
figsize (tuple[float], optional) – The size of the figure to create, if
ax
is not provided.ax (matplotlib.Axis, optional) – The axis to draw to. If not provided, a new figure will be created.
show_and_close (bool, optional) – If
True
(the default) then show and close the figure, otherwise return the figure and axis.
- Returns:
fig (matplotlib.Figure) – The figure containing the plot, or
None
ifax
was provided.ax (matplotlib.Axis) – The axis or axes containing the plot(s).
- xyzpy.visualize_tensor(array, spacing_factor=1.0, max_projections=None, projection_overlap_spacing=1.05, angles=None, scales=None, skew_angle_factor='auto', skew_scale_factor=0.05, max_mag=None, magscale='linear', size_map=True, size_pow=1 / 2, size_scale=1.0, alpha_map=True, alpha_pow=1 / 2, alpha=0.8, marker='o', linewidths=0, show_lattice=True, lattice_opts=None, compass=False, compass_loc='auto', compass_size=0.1, compass_bounds=None, compass_labels=None, compass_opts=None, legend=True, legend_loc='auto', legend_size=0.15, legend_bounds=None, legend_resolution=3, interleave_projections=False, reverse_projections=False, facecolor=None, rasterize=4096, rasterize_dpi=300, figsize=(5, 5), ax=None)[source]#
Visualize all entries of a tensor, with indices mapped into the plane and values mapped into a color wheel.
- Parameters:
array (numpy.ndarray) – The tensor to visualize.
spacing_factor (float, optional) – How to scale the dimensions relative to each other. If 1.0, then each dimension will have the same extent, and smaller dimensions will be sparser. If 0.0, the each dimension will have an extent propoertional to its size, with matching density.
max_projections (int, optional) – The maximum number of different projection directions / angles to use. If specified and less than the number of dimensions, then multiple dimensions will share the same angle but with different scales.
projection_overlap_spacing (float, optional) – When grouping multiple dimensions to the same angle, how much to increase the spacing at each scale so as to emphasize each.
angles (sequence[float], optional) – An explicit list of angles to use for each direction, in radians, with zero pointing straight down. If not provided, then the angles will be calculated automatically.
scales (sequence[float], optional) – An explicit list of scales to use for each direction. If not provided, then the scales will be calculated automatically.
skew_angle_factor (float, optional) – When there are more than two dimensions, a factor to scale the rotations by to avoid overlapping data points. If 0.0 then the angles will be evenly spaced.
skew_scale_factor (float, optional) – When there are more than two dimensions, a factor to scale the scales by to avoid overlapping data points, that shortens non-perpendicular directions.
max_mag (float, optional) – The maximum magnitude to use for the color mapping. If not provided, the maximum magnitude in the array will be used.
magscale ("linear" or float, optional) – How to scale the magnitude of the array values. If “linear”, then the magnitude is used directly. If a float, then the magnitude is raised to this power before being used, which can help to show variation among small values.
size_map (bool, optional) – Whether to map the tensor value magnitudes to marker size.
size_scale (float, optional) – An overall factor to scale the marker size by.
alpha_map (bool, optional) – Whether to map the tensor value magnitudes to marker alpha.
alpha_pow (float, optional) – The power to raise the magnitude to when mapping to alpha.
alpha (float, optional) – The overall alpha to use for all markers if
not alpha_map
.marker (str, optional) – The marker to use for the markers.
linewidths (float, optional) – The linewidth to use for the markers.
show_lattice (bool, optional) – Show a thin grey line connecting adjacent array coordinate points.
lattice_opts (dict, optional) – Options to pass to
maplotlib.Axis.scatter
for the lattice grid.compass (bool, optional) – Whether to show a compass indicating the orientation of each dimension.
compass_loc ((float, float), optional) – Where to place the compass.
compass_size (float, optional) – The size of the compass.
compass_bounds (tuple[float], optional) – Explicit bounds of the compass, as
(x, y, width, height)
.compass_labels (sequence[str], optional) – Explicit labels for the compass, in order of the dimensions.
compass_opts (dict, optional) – Extra options for the compass arrows.
legend (bool, optional) – Whether to show a legend (colorbar). If the array has complex dtype then the legend will be a colorwheel.
legend_loc (str or tuple[float], optional) – Where to place the legend. If “auto”, then the legend will be placed outside the plot rectangle, otherwise it should be a tuple of
(x, y)
coordinates in axes space.legend_size (float, optional) – The size of the legend, in relation to the size of the plot axes.
legend_bounds (tuple[float], optional) – Explicit bounds of the legend, as
(x, y, width, height)
in axes space.legend_resolution (int, optional) – The number of different colors to show in the legend.
interleave_projections (bool, optional) – If
True
and grouping dimensions, then they are assigned round robin fashion rather than blocks.False
matches the behavior of fusing.reverse_projections (bool, optional) – Whether to reverse the order of the projections.
facecolor (str, optional) – The background color of the plot, by default transparent.
rasterize (int or float, optional) – Whether to rasterize the plot. If a number, then rasterize if the size of the array is greater than this value.
rasterize_dpi (float, optional) – The dpi to use when rasterizing.
figsize (tuple, optional) – The size of the figure to create, if
ax
is not provided.ax (matplotlib.Axis, optional) – The axis to draw to. If not provided, a new figure will be created.
- Returns:
fig (matplotlib.Figure) – The figure containing the plot, or
None
ifax
was provided.ax (matplotlib.Axis) – The axis containing the plot.
- xyzpy.auto_iheatmap(x, **iheatmap_opts)[source]#
Auto version of
iheatmap()
that accepts array arguments by converting them to aDataset
first.
- xyzpy.auto_ilineplot(x, y_z, **lineplot_opts)[source]#
Auto version of
ilineplot()
that accepts array arguments by converting them to aDataset
first.
- xyzpy.auto_iscatter(x, y_z, **iscatter_opts)[source]#
Auto version of
iscatter()
that accepts array arguments by converting them to aDataset
first.
- xyzpy.iheatmap(ds, x, y, z, **kwargs)[source]#
From
ds
plot variablez
as a function ofx
andy
using a 2D heatmap. Interactive,- Parameters:
ds (xarray.Dataset) – Dataset to plot from.
x (str) – Dimension to plot along the x-axis.
y (str) – Dimension to plot along the y-axis.
z (str, optional) – Variable to plot as colormap.
row (str, optional) – Dimension to vary over as a function of rows.
col (str, optional) – Dimension to vary over as a function of columns.
plot_opts – See
xyzpy.plot.core.PLOTTER_DEFAULTS
.
- xyzpy.ilineplot(ds, x, y, z=None, y_err=None, x_err=None, **kwargs)[source]#
From
ds
plot lines ofy
as a function ofx
, optionally for varyingz
. Interactive,- Parameters:
ds (xarray.Dataset) – Dataset to plot from.
x (str) – Dimension to plot along the x-axis.
y (str or tuple[str]) – Variable(s) to plot along the y-axis. If tuple, plot each of the variables - instead of
z
.z (str, optional) – Dimension to plot into the page.
y_err (str, optional) – Variable to plot as y-error.
x_err (str, optional) – Variable to plot as x-error.
row (str, optional) – Dimension to vary over as a function of rows.
col (str, optional) – Dimension to vary over as a function of columns.
plot_opts – See
xyzpy.plot.core.PLOTTER_DEFAULTS
.
- xyzpy.iscatter(ds, x, y, z=None, y_err=None, x_err=None, **kwargs)[source]#
From
ds
plot a scatter ofy
againstx
, optionally for varyingz
. Interactive.- Parameters:
ds (xarray.Dataset) – Dataset to plot from.
x (str) – Quantity to plot along the x-axis.
y (str or tuple[str]) – Quantity(s) to plot along the y-axis. If tuple, plot each of the variables - instead of
z
.z (str, optional) – Dimension to plot into the page.
y_err (str, optional) – Variable to plot as y-error.
x_err (str, optional) – Variable to plot as x-error.
row (str, optional) – Dimension to vary over as a function of rows.
col (str, optional) – Dimension to vary over as a function of columns.
plot_opts – See
xyzpy.plot.core.PLOTTER_DEFAULTS
.