thepipe.provenance

Provenance tracking inspired by the ctapipe approach.

Module Contents

thepipe.provenance.ENV_VARS_TO_LOG = ['PATH', 'LD_LIBRARY_PATH', 'DYLD_LIBRARY_PATH', 'USER', 'HOME', 'SHELL', 'VIRTUAL_ENV', 'CONDA_DEFAULT_ENV', 'CONDA_PREFIX', 'CONDA_EXE', 'CONDA_PROMOMPT_MODIFIER', 'CONDA_SHLVL']

thepipe.provenance.ENV_VARS_IN_CI_TO_LOG = ['APPVEYOR', 'CI', 'CIRCLECI', 'CONTINUOUS_INTEGRATION', 'GITHUB_ACTIONS', 'GITLAB_CI', 'TF_BUILD', 'TRAVIS']

thepipe.provenance.log

thepipe.provenance.python_packages()

All installed Python packages.

LRU cached, assuming no package installations during runtime.

class thepipe.provenance.Singleton

Bases: type

Singleton metaclass

instance

class thepipe.provenance.Provenance

The provenance manager.

property outfile(self)

start_activity(self, name)

Starts a new activity and returns its UUID for future reference

finish_activity(self, uuid, status='completed')

Finishes an activity with the given UUID

record_configuration(self, configuration)

Record configuration parameters (e.g. of the pipeline)

record_input(self, url, comment='')

record_output(self, url, comment='')

property current_activity(self)

activity(self, name)

property provenance(self)

property backlog(self)

as_json(self, **kwargs)

Dump provenance as JSON string. kwargs are passed to json.dumps

reset(self)

thepipe.provenance.isotime(timestamp)

ISO 8601 formatted date in UTC from unix timestamp

thepipe.provenance.now()

Returns the ISO 8601 formatted time in UTC

thepipe.provenance.system_state()

thepipe.provenance.system_provenance()

Provenance information of the system configuration

thepipe.provenance.duration(start, stop)

Return the duration in seconds between two ISO 8601 time strings in