xyzpy.gen.ray_executor

Module Contents

Classes

RayFuture	Basic concurrent.futures like future wrapping a ray ObjectRef.
RayExecutor	Basic concurrent.futures like interface using ray.
RayGPUExecutor	A RayExecutor that by default requests a single gpu per task.

Functions

get_ray()
_unpack_futures_tuple(x)
_unpack_futures_list(x)
_unpack_futures_dict(x)
_unpack_futures_identity(x)
_unpack_futures(x)	Allows passing futures by reference - takes e.g. args and kwargs and
get_remote_fn(fn, **remote_opts)	Cached retrieval of remote function.
get_fn_as_remote_object(fn)
get_deploy(**remote_opts)	Alternative for 'non-function' callables - e.g. partial

Attributes

_unpack_dispatch

xyzpy.gen.ray_executor.get_ray()

class xyzpy.gen.ray_executor.RayFuture(obj)

Basic concurrent.futures like future wrapping a ray ObjectRef.

__slots__ = ('_obj', '_cancelled')

result(timeout=None)

done()

cancel()

xyzpy.gen.ray_executor._unpack_futures_tuple(x)

xyzpy.gen.ray_executor._unpack_futures_list(x)

xyzpy.gen.ray_executor._unpack_futures_dict(x)

xyzpy.gen.ray_executor._unpack_futures_identity(x)

xyzpy.gen.ray_executor._unpack_dispatch

xyzpy.gen.ray_executor._unpack_futures(x)

Allows passing futures by reference - takes e.g. args and kwargs and replaces all RayFuture objects with their underyling ObjectRef within all nested tuples, lists and dicts.

[Subclassing ObjectRef might avoid needing this.]

xyzpy.gen.ray_executor.get_remote_fn(fn, **remote_opts)

Cached retrieval of remote function.

xyzpy.gen.ray_executor.get_fn_as_remote_object(fn)

xyzpy.gen.ray_executor.get_deploy(**remote_opts)

Alternative for ‘non-function’ callables - e.g. partial functions - pass the callable object too.

class xyzpy.gen.ray_executor.RayExecutor(*args, default_remote_opts=None, **kwargs)

Basic concurrent.futures like interface using ray.

Example usage:

from xyzpy import RayExecutor

# create a pool that by default requests a single gpu per task
pool = RayExecutor(
    num_cpus=4,
    num_gpus=4,
    default_remote_opts={"num_gpus": 1},
)

_maybe_inject_remote_opts(remote_opts=None)

Return the default remote options, possibly overriding some with those supplied by a submit call.

submit(fn, *args, pure=False, remote_opts=None, **kwargs)

Remotely run fn(*args, **kwargs), returning a RayFuture.

map(func, *iterables, remote_opts=None)

Remote map func over arguments iterables.

scatter(data)

Push data into the distributed store, returning an ObjectRef that can be supplied to submit calls for example.

shutdown()

Shutdown the parent ray cluster, this RayExecutor instance itself does not need any cleanup.

class xyzpy.gen.ray_executor.RayGPUExecutor(*args, gpus_per_task=1, **kwargs)

Bases: RayExecutor

A RayExecutor that by default requests a single gpu per task.