‘Farming’ with Harvester Example

This examples demonstrates ‘farming’ runs, that is, using a Harvester to perform and combine succesive, disjoint sets of runs.

First we define a simple trig function to explore and label with a Runner:

%config InlineBackend.figure_formats = ['svg']
import xyzpy as xyz
import numpy as np


def trig(f, x, amp, phi):

    trig_fn = getattr(np, f)
    err = 0.2 + 0.1 * np.random.randn()

    return amp * trig_fn(x - phi), err


r = xyz.Runner(trig, ['f(x)', 'ef(x)'])

The harvester is composed of this Runner and logic for merging for saving each new set of runs.

h = xyz.Harvester(r, data_name='trig.h5')

We perform one set of runs first:

combos_1 = {
    'f': ['sin', 'cos'],
    'x': np.linspace(-5, 5, 101),
    'amp': [1, 2, 3],
    'phi': [0.4, 0.8],
}

h.harvest_combos(combos_1)

100%|##########| 1212/1212 [00:00<00:00, 548026.78it/s]

Which we can plot:

fig, axs = h.full_ds.xyz.plot(
    x='x',
    y='f(x)',
    color='f',
    col='amp',
    row='phi',
    err='ef(x)',
)

Then we can define and run a completely disjoint set of runs:

combos_2 = {
    'f': ['sin', 'cos'],
    'x': np.linspace(-5, 5, 101),
    'amp': [4, 5],
    'phi': [1.2, 1.6],
}

h.harvest_combos(combos_2)

100%|##########| 808/808 [00:00<00:00, 461464.82it/s]

If we plot the current full_ds we’ll see both sets of runs are present, but there are also blank spaces:

fig, axs = h.full_ds.xyz.plot(
    x='x',
    y='f(x)',
    color='f',
    col='amp',
    row='phi',
    err='ef(x)',
)

Finally let’s add tan into the mix, but only for the central combinations:

combos_3 = {
    'f': ['tan'],
    'x': np.linspace(-5, 5, 101),
    'amp': [2, 3, 4],
    'phi': [0.8, 1.2],
}

h.harvest_combos(combos_3)

100%|##########| 606/606 [00:00<00:00, 84516.47it/s]

We’ll specify some plotting limits to avoid the infinities as well:

fig, axs = h.full_ds.xyz.plot(
    x='x',
    y='f(x)',
    color='f',
    col='amp',
    row='phi',
    err='ef(x)',
    ylim=(-4, 4),
)

All runs have also been accumulated into the on-disk:

!ls *.h5

trig.h5

xyz.load_ds('trig.h5')

<xarray.Dataset>
Dimensions:  (f: 3, x: 101, amp: 5, phi: 4)
Coordinates:
  * f        (f) <U3 'cos' 'sin' 'tan'
  * x        (x) float64 -5.0 -4.9 -4.8 -4.7 -4.6 -4.5 ... 4.6 4.7 4.8 4.9 5.0
  * amp      (amp) int64 1 2 3 4 5
  * phi      (phi) float64 0.4 0.8 1.2 1.6
Data variables:
    f(x)     (f, x, amp, phi) float64 0.6347 0.8855 nan nan ... nan nan nan nan
    ef(x)    (f, x, amp, phi) float64 0.1428 0.1683 nan nan ... nan nan nan nan

xarray.Dataset

• Dimensions:
  • f: 3
  • x: 101
  • amp: 5
  • phi: 4
• Coordinates: (4)
  • f
    (f)
    <U3
    'cos' 'sin' 'tan'

    array(['cos', 'sin', 'tan'], dtype='<U3')

  • x
    (x)
    float64
    -5.0 -4.9 -4.8 -4.7 ... 4.8 4.9 5.0

    array([-5. , -4.9, -4.8, -4.7, -4.6, -4.5, -4.4, -4.3, -4.2, -4.1, -4. , -3.9,
           -3.8, -3.7, -3.6, -3.5, -3.4, -3.3, -3.2, -3.1, -3. , -2.9, -2.8, -2.7,
           -2.6, -2.5, -2.4, -2.3, -2.2, -2.1, -2. , -1.9, -1.8, -1.7, -1.6, -1.5,
           -1.4, -1.3, -1.2, -1.1, -1. , -0.9, -0.8, -0.7, -0.6, -0.5, -0.4, -0.3,
           -0.2, -0.1,  0. ,  0.1,  0.2,  0.3,  0.4,  0.5,  0.6,  0.7,  0.8,  0.9,
            1. ,  1.1,  1.2,  1.3,  1.4,  1.5,  1.6,  1.7,  1.8,  1.9,  2. ,  2.1,
            2.2,  2.3,  2.4,  2.5,  2.6,  2.7,  2.8,  2.9,  3. ,  3.1,  3.2,  3.3,
            3.4,  3.5,  3.6,  3.7,  3.8,  3.9,  4. ,  4.1,  4.2,  4.3,  4.4,  4.5,
            4.6,  4.7,  4.8,  4.9,  5. ])

  • amp
    (amp)
    int64
    1 2 3 4 5

    array([1, 2, 3, 4, 5])

  • phi
    (phi)
    float64
    0.4 0.8 1.2 1.6

    array([0.4, 0.8, 1.2, 1.6])

• Data variables: (2)
  • f(x)
    (f, x, amp, phi)
    float64
    0.6347 0.8855 nan ... nan nan nan

    array([[[[ 0.63469288,  0.88551952,         nan,         nan],
             [ 1.26938575,  1.77103903,         nan,         nan],
             [ 1.90407863,  2.65655855,         nan,         nan],
             [        nan,         nan,  3.98616839,  3.80093037],
             [        nan,         nan,  4.98271049,  4.75116296]],
    
            [[ 0.55437434,  0.83471278,         nan,         nan],
             [ 1.10874867,  1.66942557,         nan,         nan],
             [ 1.66312301,  2.50413835,         nan,         nan],
             [        nan,         nan,  3.93307375,  3.9063505 ],
             [        nan,         nan,  4.91634219,  4.88293813]],
    
            [[ 0.46851667,  0.77556588,         nan,         nan],
             [ 0.93703334,  1.55113176,         nan,         nan],
             [ 1.40555001,  2.32669764,         nan,         nan],
             [        nan,         nan,  3.84068115,  3.97273968],
             [        nan,         nan,  4.80085143,  4.96592459]],
    
            ...,
    
    ...
    
            ...,
    
            [[        nan,         nan,         nan,         nan],
             [        nan,  2.31564256,  0.98693346,         nan],
             [        nan,  3.47346385,  1.48040019,         nan],
             [        nan,  4.63128513,  1.97386692,         nan],
             [        nan,         nan,         nan,         nan]],
    
            [[        nan,         nan,         nan,         nan],
             [        nan,  2.84705297,  1.24946615,         nan],
             [        nan,  4.27057945,  1.87419923,         nan],
             [        nan,  5.69410593,  2.4989323 ,         nan],
             [        nan,         nan,         nan,         nan]],
    
            [[        nan,         nan,         nan,         nan],
             [        nan,  3.55555955,  1.54711218,         nan],
             [        nan,  5.33333932,  2.32066827,         nan],
             [        nan,  7.1111191 ,  3.09422436,         nan],
             [        nan,         nan,         nan,         nan]]]])

  • ef(x)
    (f, x, amp, phi)
    float64
    0.1428 0.1683 nan ... nan nan nan

    array([[[[ 0.14282348,  0.16833407,         nan,         nan],
             [ 0.29630199,  0.27659591,         nan,         nan],
             [ 0.23544534,  0.05450563,         nan,         nan],
             [        nan,         nan,  0.41023704,  0.11508055],
             [        nan,         nan,  0.24422762,  0.18136717]],
    
            [[ 0.11383026,  0.32967943,         nan,         nan],
             [ 0.3087783 ,  0.26756931,         nan,         nan],
             [ 0.21746075,  0.11802664,         nan,         nan],
             [        nan,         nan,  0.29838091,  0.15216249],
             [        nan,         nan,  0.18888552,  0.30664376]],
    
            [[ 0.16448738,  0.13023006,         nan,         nan],
             [ 0.00737546,  0.25038216,         nan,         nan],
             [ 0.30586435,  0.17383722,         nan,         nan],
             [        nan,         nan,  0.23027202,  0.01239187],
             [        nan,         nan,  0.21390776,  0.0981793 ]],
    
            ...,
    
    ...
    
            ...,
    
            [[        nan,         nan,         nan,         nan],
             [        nan,  0.25483371,  0.44080294,         nan],
             [        nan,  0.19518592,  0.30508751,         nan],
             [        nan,  0.21793418,  0.20504778,         nan],
             [        nan,         nan,         nan,         nan]],
    
            [[        nan,         nan,         nan,         nan],
             [        nan,  0.2123759 ,  0.16462831,         nan],
             [        nan,  0.16495827,  0.41636505,         nan],
             [        nan,  0.3635263 ,  0.12342688,         nan],
             [        nan,         nan,         nan,         nan]],
    
            [[        nan,         nan,         nan,         nan],
             [        nan,  0.11175819,  0.32226433,         nan],
             [        nan,  0.34458939,  0.10992671,         nan],
             [        nan,  0.24311122,  0.27206747,         nan],
             [        nan,         nan,         nan,         nan]]]])

• Indexes: (4)
  • f
    PandasIndex

    PandasIndex(Index(['cos', 'sin', 'tan'], dtype='object', name='f'))

  • x
    PandasIndex

    PandasIndex(Index([             -5.0,              -4.9,              -4.8,
                        -4.7,              -4.6,              -4.5,
                        -4.4,              -4.3,              -4.2,
                        -4.1,
           ...
                         4.1, 4.200000000000001, 4.300000000000001,
                         4.4,               4.5, 4.600000000000001,
           4.700000000000001, 4.800000000000001,               4.9,
                         5.0],
          dtype='float64', name='x', length=101))

  • amp
    PandasIndex

    PandasIndex(Index([1, 2, 3, 4, 5], dtype='int64', name='amp'))

  • phi
    PandasIndex

    PandasIndex(Index([0.4, 0.8, 1.2, 1.6], dtype='float64', name='phi'))

• Attributes: (0)

For now just clean this up.

h.delete_ds()