‘Farming’ with Harvester Example

This examples demonstrates ‘farming’ runs, that is, using a Harvester to perform and combine succesive, disjoint sets of runs.

First we define a simple trig function to explore and label with a Runner:

%config InlineBackend.figure_formats = ['svg']
import numpy as np

import xyzpy as xyz


def trig(f, x, amp, phi):

    trig_fn = getattr(np, f)
    err = 0.2 + 0.1 * np.random.randn()

    return amp * trig_fn(x - phi), err


r = xyz.Runner(trig, ["f(x)", "ef(x)"])

The harvester is composed of this Runner and logic for merging for saving each new set of runs.

h = xyz.Harvester(r, data_name="trig.h5")

We perform one set of runs first:

combos_1 = {
    "f": ["sin", "cos"],
    "x": np.linspace(-5, 5, 101),
    "amp": [1, 2, 3],
    "phi": [0.4, 0.8],
}

h.harvest_combos(combos_1)

100%|##########| 1212/1212 [00:00<00:00, 871954.79it/s]

Which we can plot:

fig, axs = h.full_ds.xyz.plot(
    x="x",
    y="f(x)",
    color="f",
    col="amp",
    row="phi",
    err="ef(x)",
)

Then we can define and run a completely disjoint set of runs:

combos_2 = {
    "f": ["sin", "cos"],
    "x": np.linspace(-5, 5, 101),
    "amp": [4, 5],
    "phi": [1.2, 1.6],
}

h.harvest_combos(combos_2)

100%|##########| 808/808 [00:00<00:00, 283883.20it/s]

If we plot the current full_ds we’ll see both sets of runs are present, but there are also blank spaces:

fig, axs = h.full_ds.xyz.plot(
    x="x",
    y="f(x)",
    color="f",
    col="amp",
    row="phi",
    err="ef(x)",
)

Finally let’s add tan into the mix, but only for the central combinations:

combos_3 = {
    "f": ["tan"],
    "x": np.linspace(-5, 5, 101),
    "amp": [2, 3, 4],
    "phi": [0.8, 1.2],
}

h.harvest_combos(combos_3)

100%|##########| 606/606 [00:00<00:00, 1130168.17it/s]

We’ll specify some plotting limits to avoid the infinities as well:

fig, axs = h.full_ds.xyz.plot(
    x="x",
    y="f(x)",
    color="f",
    col="amp",
    row="phi",
    err="ef(x)",
    ylim=(-4, 4),
)

All runs have also been accumulated into the on-disk:

!ls *.h5

trig.h5

xyz.load_ds("trig.h5")

<xarray.Dataset> Size: 98kB
Dimensions:  (f: 3, x: 101, amp: 5, phi: 4)
Coordinates:
  * f        (f) <U3 36B 'cos' 'sin' 'tan'
  * x        (x) float64 808B -5.0 -4.9 -4.8 -4.7 -4.6 ... 4.6 4.7 4.8 4.9 5.0
  * amp      (amp) int64 40B 1 2 3 4 5
  * phi      (phi) float64 32B 0.4 0.8 1.2 1.6
Data variables:
    f(x)     (f, x, amp, phi) float64 48kB 0.6347 0.8855 nan nan ... nan nan nan
    ef(x)    (f, x, amp, phi) float64 48kB 0.08426 0.1342 nan ... nan nan nan

xarray.Dataset

• Dimensions:
  • f: 3
  • x: 101
  • amp: 5
  • phi: 4
• Coordinates: (4)
  • f
    (f)
    <U3
    'cos' 'sin' 'tan'

    array(['cos', 'sin', 'tan'], dtype='<U3')

  • x
    (x)
    float64
    -5.0 -4.9 -4.8 -4.7 ... 4.8 4.9 5.0

    array([-5. , -4.9, -4.8, -4.7, -4.6, -4.5, -4.4, -4.3, -4.2, -4.1, -4. , -3.9,
           -3.8, -3.7, -3.6, -3.5, -3.4, -3.3, -3.2, -3.1, -3. , -2.9, -2.8, -2.7,
           -2.6, -2.5, -2.4, -2.3, -2.2, -2.1, -2. , -1.9, -1.8, -1.7, -1.6, -1.5,
           -1.4, -1.3, -1.2, -1.1, -1. , -0.9, -0.8, -0.7, -0.6, -0.5, -0.4, -0.3,
           -0.2, -0.1,  0. ,  0.1,  0.2,  0.3,  0.4,  0.5,  0.6,  0.7,  0.8,  0.9,
            1. ,  1.1,  1.2,  1.3,  1.4,  1.5,  1.6,  1.7,  1.8,  1.9,  2. ,  2.1,
            2.2,  2.3,  2.4,  2.5,  2.6,  2.7,  2.8,  2.9,  3. ,  3.1,  3.2,  3.3,
            3.4,  3.5,  3.6,  3.7,  3.8,  3.9,  4. ,  4.1,  4.2,  4.3,  4.4,  4.5,
            4.6,  4.7,  4.8,  4.9,  5. ])

  • amp
    (amp)
    int64
    1 2 3 4 5

    array([1, 2, 3, 4, 5])

  • phi
    (phi)
    float64
    0.4 0.8 1.2 1.6

    array([0.4, 0.8, 1.2, 1.6])

• Data variables: (2)
  • f(x)
    (f, x, amp, phi)
    float64
    0.6347 0.8855 nan ... nan nan nan

    array([[[[ 0.63469288,  0.88551952,         nan,         nan],
             [ 1.26938575,  1.77103903,         nan,         nan],
             [ 1.90407863,  2.65655855,         nan,         nan],
             [        nan,         nan,  3.98616839,  3.80093037],
             [        nan,         nan,  4.98271049,  4.75116296]],
    
            [[ 0.55437434,  0.83471278,         nan,         nan],
             [ 1.10874867,  1.66942557,         nan,         nan],
             [ 1.66312301,  2.50413835,         nan,         nan],
             [        nan,         nan,  3.93307375,  3.9063505 ],
             [        nan,         nan,  4.91634219,  4.88293813]],
    
            [[ 0.46851667,  0.77556588,         nan,         nan],
             [ 0.93703334,  1.55113176,         nan,         nan],
             [ 1.40555001,  2.32669764,         nan,         nan],
             [        nan,         nan,  3.84068115,  3.97273968],
             [        nan,         nan,  4.80085143,  4.96592459]],
    
            ...,
    
    ...
            ...,
    
            [[        nan,         nan,         nan,         nan],
             [        nan,  2.31564256,  0.98693346,         nan],
             [        nan,  3.47346385,  1.48040019,         nan],
             [        nan,  4.63128513,  1.97386692,         nan],
             [        nan,         nan,         nan,         nan]],
    
            [[        nan,         nan,         nan,         nan],
             [        nan,  2.84705297,  1.24946615,         nan],
             [        nan,  4.27057945,  1.87419923,         nan],
             [        nan,  5.69410593,  2.4989323 ,         nan],
             [        nan,         nan,         nan,         nan]],
    
            [[        nan,         nan,         nan,         nan],
             [        nan,  3.55555955,  1.54711218,         nan],
             [        nan,  5.33333932,  2.32066827,         nan],
             [        nan,  7.1111191 ,  3.09422436,         nan],
             [        nan,         nan,         nan,         nan]]]],
          shape=(3, 101, 5, 4))

  • ef(x)
    (f, x, amp, phi)
    float64
    0.08426 0.1342 nan ... nan nan nan

    array([[[[ 0.08425505,  0.13421237,         nan,         nan],
             [ 0.08009853,  0.12193447,         nan,         nan],
             [ 0.17960569,  0.09757148,         nan,         nan],
             [        nan,         nan,  0.25595408,  0.27181408],
             [        nan,         nan,  0.34085658,  0.18261051]],
    
            [[ 0.13830657,  0.29806074,         nan,         nan],
             [ 0.18063535,  0.33739076,         nan,         nan],
             [ 0.09776002,  0.27889984,         nan,         nan],
             [        nan,         nan,  0.17098707,  0.18386664],
             [        nan,         nan,  0.1273715 ,  0.18144898]],
    
            [[ 0.16595503,  0.3087557 ,         nan,         nan],
             [ 0.27272486,  0.12808078,         nan,         nan],
             [ 0.23814073,  0.08733166,         nan,         nan],
             [        nan,         nan,  0.29556813,  0.26351899],
             [        nan,         nan,  0.15816957,  0.13026945]],
    
            ...,
    
    ...
            ...,
    
            [[        nan,         nan,         nan,         nan],
             [        nan,  0.17363923,  0.41435647,         nan],
             [        nan,  0.30295065,  0.12129371,         nan],
             [        nan,  0.24506176,  0.21738481,         nan],
             [        nan,         nan,         nan,         nan]],
    
            [[        nan,         nan,         nan,         nan],
             [        nan,  0.26900323,  0.27789384,         nan],
             [        nan,  0.19739542,  0.13781966,         nan],
             [        nan,  0.09465954,  0.2555604 ,         nan],
             [        nan,         nan,         nan,         nan]],
    
            [[        nan,         nan,         nan,         nan],
             [        nan,  0.1772658 ,  0.19082714,         nan],
             [        nan,  0.12797979,  0.24934414,         nan],
             [        nan,  0.2369927 ,  0.1380617 ,         nan],
             [        nan,         nan,         nan,         nan]]]],
          shape=(3, 101, 5, 4))

For now just clean this up.

h.delete_ds()