thepipe.core

The core of thepipe framework.

Module Contents

thepipe.core.STATS_LIMIT = 100000

thepipe.core.MODULE_CONFIGURATION = pipeline.toml

thepipe.core.RESERVED_ARGS

class thepipe.core.Blob(*args, **kwargs)

Bases: collections.OrderedDict

A simple (ordered) dict with a fancy name. This should hold the data.

class thepipe.core.Module(name=None, **parameters)

The module which can be attached to the pipeline

configure(self)

Configure module, like instance variables etc.

expose(self, obj, name)

Expose an object as a service to the Pipeline

print(self, *args, **kwargs)

property name(self)

The name of the module

property parameters(self)

property processed_parameters(self)

add(self, name, value)

Add the parameter with the desired value to the dict

get(self, name, default=None)

Return the value of the requested parameter or default if None.

require(self, name)

Return the value of the requested parameter or raise an error.

require_service(self, name, why='')

Add a service requirement with an optional reason

prepare(self)

Prepare! Executed between configure and the first process

process(self, blob)

Knead the blob and return it

finish(self)

Clean everything up.

pre_finish(self)

Do the last few things before calling finish()

open_file(self, filename, gzipped=False)

Open the file with filename

class thepipe.core.Pipeline(blob=None, timeit=False, configfile=None, stats_limit=100000)

The holy pipeline which holds everything together.

If initialised with timeit=True, all modules will be monitored, otherwise only the overall statistics and modules with timeit=True will be shown.

Parameters

timeit: bool, optional [default=False]

Display time profiling statistics for the pipeline?

configfile: str, optional [default=’pipeline.toml’]

Path to a configuration file (TOML format) which contains parameters for attached modules.

stats_limit: int, optional [default=100000]

The number of cycles to keep track of when using timeit=True

load_configuration(self, configfile)

attach(self, module_factory, name=None, **kwargs)

Attach a module to the pipeline system

drain(self, cycles=None)

Execute _drain while trapping KeyboardInterrupt

finish(self)

Call finish() on each attached module

class thepipe.core.ServiceManager

Takes care of pipeline services.

register(self, name, service)

Service registration

Args:

name: Name of the provided service service: Reference to the service

get_missing_services(self, services)

Check if all required services are provided

Args:

services: List the service names which are required

Returns:

List with missing services