PlasmaCalcs.tools.multiprocessing.TaskArray

class PlasmaCalcs.tools.multiprocessing.TaskArray(tasks, *, shape=None, checks=True, **kw_super)

a container for multiple tasks; each Task is a function, args, & kwargs.

Calling self will perform all tasks, returning the result (and updating self.result as well).

tasks: array-like of Tasks, iterables, or callables.: the Tasks to be performed.

any non-Task input will be used to create a Task.

E.g. TaskArray([f1, (f2, args2), Task(f3, args3, kwargs3)])

–> [Task(f1), Task(f2, args2), Task(f3, args3, kwargs3)]
shape: None or tuple: shape that the array of tasks should have.

if None, infer tasks shape using self.task_nest_shape(tasks).

if provided, use the specified shape instead of trying to infer it.
checks: bool: whether to perform checks on the inputs, and possibly convert inputs if needed.

if False, assumes that tasks is already a numpy array of Task objects.

assign_task_idx: {assign_task_idx} printable_process_name: {printable_process_name} errors_ok: {errors_ok} result_missing: {result_missing}

__init__(tasks, *, shape=None, checks=True, **kw_super): should set self.data = something related to stuff.

Methods

Attributes

assign_task_idx(): assign task.i for tasks in self, based on their positions in self.

coarsen(ncoarse=5, *, idx=None): return a TaskPartition containing TaskGroups of size ncoarse.

Useful for coarsening a TaskContainer for more efficient multiprocessing;

grouping tasks together can reduce the overhead of multiprocessing,

while still allowing for parallel processing as the groups are run in parallel.

if idx is provided, only group the tasks with those indices.

classmethod empty(shape): return a TaskArray of shape shape, filled with UNSET_TASK.

enumerate(idx=None): iterate through i in idx, yielding (i, self[i]) pairs.

If idx is None, iterate through all objs in self (see self._enumerate_all).

property errors_ok: tuple of Exception types which are okay for tasks to raise.

setting self.errors_ok = False –> use empty tuple, i.e. no errors are okay.

setting self.errors_ok = errtype –> use errors_ok = (errtype,).

setting errors_ok will crash if it includes any parent class of KeyboardInterrupt,

e.g. errors_ok=BaseException will crash, but errors_ok=Exception will be fine.

See also: self.errors_ok_tuple

errors_ok_tuple(value=UNSET): returns tuple of okay errors. UNSET –> self.errors_ok.

False –> (). errtype –> (errtype,).

if result includes any parent class of KeyboardInterrupt, raises InputError.

e.g. errors_ok_tuple(BaseException) will crash, but errors_ok_tuple(Exception) will be fine.

init_result(): set self.result = container with similar shape as self, filled with RESULT_MISSING.

Then, return self.result.

The idea is that self.result[idx] will correspond to the result of self[idx].

new_empty(fill=UNSET): return a new array of the same shape as self, filled with the value fill.

property printable_process_name: return the name to be used for progress updates, if any.

If None, use the default: “[type(self)].__call__”.

size(idx=None): return the number of objects in the container, or in idx if provided.

static task_nest_shape(nested_list): returns the implied shape for numpy object array of tasks from nested_list.

This will be the most natural shape to use if each element of nested_list

is an iterable (e.g. tuple) or callable (e.g. function or Task),

and there is no desire to iter(f) for any callable f in the nested_list.