xyzpy.gen.ray_executor

Attributes

_unpack_dispatch

Classes

RayFuture	Basic concurrent.futures like future wrapping a ray ObjectRef.
RayExecutor	Basic concurrent.futures like interface using ray.
RayGPUExecutor	A RayExecutor that by default requests a single gpu per task.

Functions

get_ray()	Import and return the ray module (cached).
_unpack_futures_tuple(x)
_unpack_futures_list(x)
_unpack_futures_dict(x)
_unpack_futures_identity(x)
_unpack_futures(x)	Allows passing futures by reference - takes e.g. args and kwargs and
get_remote_fn(fn, **remote_opts)	Cached retrieval of remote function.
get_fn_as_remote_object(fn)	Store fn in the Ray object store and return an ObjectRef.
get_deploy(**remote_opts)	Alternative for 'non-function' callables - e.g. partial

Module Contents

xyzpy.gen.ray_executor.get_ray()

Import and return the ray module (cached).

class xyzpy.gen.ray_executor.RayFuture(obj)

Basic concurrent.futures like future wrapping a ray ObjectRef.

__slots__ = ('_obj', '_cancelled')

_obj

_cancelled = False

result(timeout=None)

done()

cancel()

xyzpy.gen.ray_executor._unpack_futures_tuple(x)

xyzpy.gen.ray_executor._unpack_futures_list(x)

xyzpy.gen.ray_executor._unpack_futures_dict(x)

xyzpy.gen.ray_executor._unpack_futures_identity(x)

xyzpy.gen.ray_executor._unpack_dispatch

xyzpy.gen.ray_executor._unpack_futures(x)

Allows passing futures by reference - takes e.g. args and kwargs and replaces all RayFuture objects with their underyling ObjectRef within all nested tuples, lists and dicts.

[Subclassing ObjectRef might avoid needing this.]

xyzpy.gen.ray_executor.get_remote_fn(fn, **remote_opts)

Cached retrieval of remote function.

xyzpy.gen.ray_executor.get_fn_as_remote_object(fn)

Store fn in the Ray object store and return an ObjectRef.

xyzpy.gen.ray_executor.get_deploy(**remote_opts)

Alternative for ‘non-function’ callables - e.g. partial functions - pass the callable object too.

class xyzpy.gen.ray_executor.RayExecutor(*args, default_remote_opts=None, **kwargs)

Basic concurrent.futures like interface using ray.

Example usage:

from xyzpy import RayExecutor

# create a pool that by default requests a single gpu per task
pool = RayExecutor(
    num_cpus=4,
    num_gpus=4,
    default_remote_opts={"num_gpus": 1},
)

default_remote_opts

_maybe_inject_remote_opts(remote_opts=None)

Return the default remote options, possibly overriding some with those supplied by a submit call.

submit(fn, *args, pure=False, remote_opts=None, **kwargs)

Remotely run fn(*args, **kwargs), returning a RayFuture.

map(func, *iterables, remote_opts=None)

Remote map func over arguments iterables.

scatter(data)

Push data into the distributed store, returning an ObjectRef that can be supplied to submit calls for example.

shutdown()

Shutdown the parent ray cluster, this RayExecutor instance itself does not need any cleanup.

class xyzpy.gen.ray_executor.RayGPUExecutor(*args, gpus_per_task=1, **kwargs)

Bases: RayExecutor

A RayExecutor that by default requests a single gpu per task.