alphatwirl.loop package

Submodules

alphatwirl.loop.Collector module

class alphatwirl.loop.Collector.Collector(resultsCombinationMethod, deliveryMethod=None)

Bases: object

This class collects results, i.e., this class combines results of readers and deliver them.

Methods for combination and delivery are specified at the instantiation.

Readers are typically instances of the same class initialized in the same way. Each reader reads a data set..

The method collect is called with a list of pairs of a data set and a reader after the event loop. It returns the combined results.

collect(dataset_readers_list)

class alphatwirl.loop.Collector.NullDeliveryMethod

Bases: object

deliver(results)

alphatwirl.loop.CollectorComposite module

class alphatwirl.loop.CollectorComposite.CollectorComposite(progressReporter=None)

Bases: object

A composite of collectors.

This class is a composite in the composite pattern.

Examples of collectors are instances of Collector, NullCollector, and this class.

add(collector)

add a collector

Parameters:collector – the collector to be added

collect(dataset_readers_list)

collect results

Returns:a list of results

alphatwirl.loop.CollectorDelegate module

class alphatwirl.loop.CollectorDelegate.CollectorDelegate(collector)

Bases: object

collect(dataset_readers_list)

alphatwirl.loop.DatasetIntoEventBuildersSplitter module

class alphatwirl.loop.DatasetIntoEventBuildersSplitter.DatasetIntoEventBuildersSplitter(EventBuilder, eventBuilderConfigMaker, maxEvents=-1, maxEventsPerRun=-1, maxFiles=-1, maxFilesPerRun=1)

Bases: object

alphatwirl.loop.EventLoop module

class alphatwirl.loop.EventLoop.EventLoop(build_events, reader)

Bases: object

An event loop

alphatwirl.loop.EventLoopProgressReportWriter module

class alphatwirl.loop.EventLoopProgressReportWriter.EventLoopProgressReportWriter

Bases: object

A progress report writer of an event loop

write(taskid, config, event)

alphatwirl.loop.EventLoopRunner module

class alphatwirl.loop.EventLoopRunner.EventLoopRunner(progressMonitor=None)

Bases: object

This class runs instances of EventLoop and keeps the results. It will return the results when end() is called.

begin()

run(eventLoop)

end()

alphatwirl.loop.EventsInDatasetReader module

class alphatwirl.loop.EventsInDatasetReader.EventsInDatasetReader(eventLoopRunner, reader, collector, split_into_build_events)

Bases: object

This class manages objects involved in reading events in data sets.

On receiving a data set, this class calls the function split_into_build_events(), which splits the data set into chunks, creates the function build_events() for each chunk, and returns a list of the functions. Then, for each build_events(), This class creates a copy of the reader, creates an event loop, and send it to the event loop runner.

At the end, this class receives results from the event loop runner and have the collector collect them.

begin()

read(dataset)

end()

alphatwirl.loop.MPEventLoopRunner module

class alphatwirl.loop.MPEventLoopRunner.MPEventLoopRunner(communicationChannel)

Bases: object

This class (concurrently) runs instances of EventLoop.

An instance of this class needs to be initialized with a communication channel with workers that actually run the EventLoop:

runner = MPEventLoopRunner(communicationChannel)

An example of a communication channel is an instance of CommunicationChannel.

The method begin() does nothing in the current version:

runner.begin()

In older versions, multiple processes are forked in this method.

Then, you can give an EventLoop with the method run():

runner.run(eventLoop1)

This class will send the EventLoop to a worker through the communication channel. The worker, then, runs the EventLoop.

You can call the method run() mutiple times:

runner.run(eventLoop2)
runner.run(eventLoop3)
runner.run(eventLoop4)

If workers are in the background, this method immediately returns. Worker are concurrently running the event loops in the background. If the worker is in the foreground, this method won’t return until the worker finishes running the event loop. Whether workers are in the background or foreground depends on the communication channel with which this class is initialized.

After giving all event loops that you need to run to this class, you need to call the method end():

results = runner.end()

If workers are in the background, this method will wait until workers finish running all event loops. If the worker is in the foreground, this method immediately returns. This method returns the results, the list of the values eventLoops return, sorted in the order given with run().

begin()

does nothing.

Older versions of this class had implementations.

run(eventLoop)

run the event loop in the background.

Parameters:eventLoop (EventLoop) – an event loop to run

end()

wait until all event loops end and returns the results.

alphatwirl.loop.NullCollector module

class alphatwirl.loop.NullCollector.NullCollector

Bases: object

collect(*_, **__)

alphatwirl.loop.ReaderComposite module

class alphatwirl.loop.ReaderComposite.ReaderComposite

Bases: object

A composite of event readers”

This class is a composite in the composite pattern.

Examples of event readers are instances of Summarizer and this class.

When event() is called, it calls event() of each reader in the order in which the readers are added. If a reader returns False, it won’t call the remaining readers.

add(reader)

begin(event)

event(event)

end()

alphatwirl.loop.splitfuncs module

alphatwirl.loop.splitfuncs.create_file_start_length_list(file_nevents_list, max_events_per_run=-1, max_events_total=-1, max_files_per_run=1)

Module contents