AlphaTwirl is a python library that loops over event data and summarizes them into multi-dimensional categorical data as data frames. Event data, input to AlphaTwirl, are data with one entry (or row) for one event: for example, data in ROOT TTrees with one entry per collision event of an LHC experiment at CERN. Event data are often large—too large to be loaded in memory—because they have as many entries as events. Multi-dimensional categorical data, the output of AlphaTwirl, have one row for one category. They are usually small—small enough to be loaded in memory—because they only have as many rows as categories. Users can, for example, import them as data frames into R and pandas, which usually load all data in memory, and can perform categorical data analyses with a rich set of data operations available in R and pandas.
Contents:
alphatwirl package¶
Subpackages¶
alphatwirl.binning package¶
Submodules¶
alphatwirl.binning.Binning module¶
alphatwirl.binning.Combine module¶
alphatwirl.binning.Echo module¶
alphatwirl.binning.ReturnTrue module¶
alphatwirl.binning.Round module¶
alphatwirl.binning.RoundLog module¶
Module contents¶
alphatwirl.cmsedm package¶
Submodules¶
alphatwirl.cmsedm.CMSEDMEventBuilder module¶
alphatwirl.cmsedm.CMSEDMEvents module¶
alphatwirl.cmsedm.EventBuilderConfig module¶
alphatwirl.cmsedm.EventBuilderConfigMaker module¶
Module contents¶
alphatwirl.collector package¶
Submodules¶
alphatwirl.collector.ToDataFrame module¶
alphatwirl.collector.ToDataFrameWithDatasetColumn module¶
alphatwirl.collector.ToTupleList module¶
alphatwirl.collector.ToTupleListWithDatasetColumn module¶
alphatwirl.collector.WriteListToFile module¶
alphatwirl.collector.WritePandasDataFrameToFile module¶
Module contents¶
alphatwirl.concurrently package¶
Submodules¶
alphatwirl.concurrently.CommunicationChannel module¶
-
class
alphatwirl.concurrently.CommunicationChannel.CommunicationChannel(dropbox)[source]¶ Bases:
objectA communication channel with workers in other processes.
(This docstring is outdated.)
You can send tasks to workers through this channel. The workers, running in other processes, execute the tasks in the background. You can receive the results of the tasks also through this channel.
An instance of this class can be created with two optional arguments:
nprocesses, the number of workers to be created, andprogressMonitor:progressBar = ProgressBar() progressMonitor = BProgressMonitor(progressBar) channel = CommunicationChannel(nprocesses=10, progressMonitor=progressMonitor)
Workers will be created when
begin()is called:channel.begin()
In
begin(), this class gives each worker aprogressReporter, which is created by theprogressMonitor.Now, you are ready to send a task. A task is a function or any object which is callable and picklable. A task can take any number of arguments. If a task takes a named argument
progressReporter, the worker will give theprogressReporterto the task. A value that a task returns is the result of the task and must be picklable. For example, an instance ofEventLoopcan be a task. You can send a task with the methodput:channel.put(task1, 10, 20, A=30)
Here, 10, 20, A=30 are the arguments to the task.
This class sends the task to a worker. The worker which receives the task will first try to call the task with the
progressReporterin addition to the arguments. If the task doesn’t take theprogressReporter, it calls only with the arguments.You can send multiple tasks:
channel.put(task2) channel.put(task3, 100, 200) channel.put(task4, A='abc') channel.put(task5)
They will be executed by workers.
You can receive the results of the tasks with the method
receive():results = channel.receive()
This method will wait until all tasks are finished. If a task gives a
progressReportto theprogressReporter, the report will be used, for example, to update progress bars on the screen.When all tasks end, results will be returned. The return value
resultsis a list of results of the tasks. They are sorted in the order in which the tasks are originally put.After receiving the results, you can put more tasks:
channel.put(task6) channel.put(task7)
And you can receive the results of them:
results = channel.receive()
If there are no more tasks to be done, you should call the method
end:channel.end()
This will end all background processes.
alphatwirl.concurrently.CommunicationChannel0 module¶
-
class
alphatwirl.concurrently.CommunicationChannel0.CommunicationChannel0(progressMonitor=None)[source]¶ Bases:
objectA communication channel for the single process mode
An alternative to
CommunicationChannel. However, unlikeCommunicationChannel, this class does not send tasks to workers. Instead, it directly executes tasks.This class has the same interface as the class
CommunicationChannel. When this class is used as a substitute forCommunicationChannel, the tasks will be sequentially executed in the foreground.
alphatwirl.concurrently.HTCondorJobSubmitter module¶
alphatwirl.concurrently.MultiprocessingDropbox module¶
alphatwirl.concurrently.SubprocessRunner module¶
alphatwirl.concurrently.TaskPackage module¶
alphatwirl.concurrently.TaskPackageDropbox module¶
alphatwirl.concurrently.Worker module¶
alphatwirl.concurrently.WorkingArea module¶
alphatwirl.concurrently.example_load module¶
alphatwirl.concurrently.example_load_result module¶
alphatwirl.concurrently.example_submit_htcondor module¶
alphatwirl.concurrently.run module¶
Module contents¶
alphatwirl.configure package¶
Submodules¶
alphatwirl.configure.TableConfigCompleter module¶
-
class
alphatwirl.configure.TableConfigCompleter.TableConfigCompleter(defaultSummaryClass=<class 'alphatwirl.summary.Count.Count'>, defaultWeight=WeightCalculatorOne(), defaultOutDir='.', createOutFileName=<alphatwirl.configure.TableFileNameComposer.TableFileNameComposer object>)[source]¶ Bases:
objectan example complete config:
tblcfg = { 'keyAttrNames': ('met_pt',), 'binnings': (MockBinning(),), 'keyIndices': None 'valAttrNames' : None, 'valIndices' : None, 'keyOutColumnNames': ('met_pt',), 'valOutColumnNames': ('n', 'nvar'), 'weight': MockWeight(), 'summaryClass': Count, 'sort': True, 'outFile': True, 'outFileName': 'tbl_n_component_met_pt.txt', 'outFilePath': '/tmp/tbl_n_component_met_pt.txt', }
alphatwirl.configure.TableFileNameComposer module¶
-
class
alphatwirl.configure.TableFileNameComposer.TableFileNameComposer(default_prefix='tbl_n', default_suffix='txt', default_var_separator='.', default_idx_separator='-')[source]¶ Bases:
object- Compose a name of a file to store the table from the column names
- and indices.
For example, if column names are ‘var1’, ‘var2’, and ‘var3’ and indices are 1, None and 2, the file name will be ‘tbl_n_component.var1-1.var2.var3-2.txt’
alphatwirl.configure.build_counter_collector_pair module¶
alphatwirl.configure.build_progressMonitor_communicationChannel module¶
Module contents¶
alphatwirl.delphes package¶
Submodules¶
alphatwirl.delphes.DelphesEventBuilder module¶
alphatwirl.delphes.DelphesEvents module¶
alphatwirl.delphes.EventBuilderConfig module¶
-
class
alphatwirl.delphes.EventBuilderConfig.EventBuilderConfig(inputPaths, treeName, maxEvents, start, dataset, name)¶ Bases:
tuple-
dataset¶ Alias for field number 4
-
inputPaths¶ Alias for field number 0
-
maxEvents¶ Alias for field number 2
-
name¶ Alias for field number 5
-
start¶ Alias for field number 3
-
treeName¶ Alias for field number 1
-
alphatwirl.delphes.EventBuilderConfigMaker module¶
alphatwirl.delphes.TClonesArrayWrap module¶
Module contents¶
alphatwirl.heppyresult package¶
Submodules¶
alphatwirl.heppyresult.Analyzer module¶
alphatwirl.heppyresult.Component module¶
alphatwirl.heppyresult.ComponentLoop module¶
alphatwirl.heppyresult.ComponentReaderComposite module¶
alphatwirl.heppyresult.EventBuilder module¶
alphatwirl.heppyresult.EventBuilderConfig module¶
alphatwirl.heppyresult.EventBuilderConfigMaker module¶
alphatwirl.heppyresult.HeppyResult module¶
-
class
alphatwirl.heppyresult.HeppyResult.HeppyResult(path, componentNames=None, excludeList=['Chunks', 'failed'], componentHasTheseFiles=['config.pck', 'config.txt'], isComponent=None)[source]¶ Bases:
objectA Heppy result
Parameters: - path (str) – the path to the Heppy result
- componentNames (list, optional) – the list of the names of the components to read. If not given, all components except the ones listed in
excludeListwill be read. - excludeList (list, optional) – a list of the names of the directory in the Heppy result directory which are to be excluded to be considered as components. if not given,
DEFAULT_EXCLUDE_LISTwill be used. - componentHasTheseFiles (list, optional) – the directories with there files are considered as components. if not given,
DEFAULT_COMPONENT_HAS_THESE_FILESwill be used. - isComponent (function, optional) – a function that determines if a directory is a Heppy component. if not give,
IsComponentwill be used.
alphatwirl.heppyresult.ReadComponentConfig module¶
alphatwirl.heppyresult.ReadCounter module¶
alphatwirl.heppyresult.ReadVersionInfo module¶
alphatwirl.heppyresult.TblBranch module¶
alphatwirl.heppyresult.TblBrilCalc module¶
alphatwirl.heppyresult.TblComponentConfig module¶
alphatwirl.heppyresult.TblCounter module¶
-
class
alphatwirl.heppyresult.TblCounter.TblCounter(analyzerName, fileName, outPath, levels=None, columnNames=None)[source]¶ Bases:
objectThis class reads counter files of HeppyResult.
Parameters: - analyzerName (str) – the name of the Heppy analyzer, e.g., skimAnalyzerCount
- fileName (str) – the name of the counter file, e.g., SkimReport.txt
- outPath (str) – a path to the output file
- levels (list) – a list of the levels to read. If not given, all levels will be read
- columnNames (list) – a list of the column names of the output file. If not given, the levels will be used
alphatwirl.heppyresult.TblCounterLong module¶
-
class
alphatwirl.heppyresult.TblCounterLong.TblCounterLong(analyzerName, fileName, outPath, levels=None, columnNames=('component', 'level', 'count'))[source]¶ Bases:
objectAn alternative class to TblCounter.
While TblCounter writes results in the wide format, this class writes results in the long format. Eventually, this class will replace TblCounter.
Parameters: - analyzerName (str) – the name of the Heppy analyzer, e.g., skimAnalyzerCount
- fileName (str) – the name of the counter file, e.g., SkimReport.txt
- outPath (str) – a path to the output file
- levels (list) – a list of the levels to read. If not given, all levels will be read
- columnNames (list) – a list of the column names of the output file. the default is (‘component’, ‘level’, ‘count’)
alphatwirl.heppyresult.TblTree module¶
Module contents¶
alphatwirl.loop package¶
Submodules¶
alphatwirl.loop.Collector module¶
-
class
alphatwirl.loop.Collector.Collector(resultsCombinationMethod, deliveryMethod=None)[source]¶ Bases:
objectThis class collects results, i.e., this class combines results of readers and deliver them.
Methods for combination and delivery are specified at the instantiation.
Readers are typically instances of the same class initialized in the same way. Each reader reads a data set..
The method
collectis called with a list of pairs of a data set and a reader after the event loop. It returns the combined results.
alphatwirl.loop.CollectorComposite module¶
-
class
alphatwirl.loop.CollectorComposite.CollectorComposite(progressReporter=None)[source]¶ Bases:
objectA composite of collectors.
This class is a composite in the composite pattern.
Examples of collectors are instances of
Collector,NullCollector, and this class.
alphatwirl.loop.CollectorDelegate module¶
alphatwirl.loop.DatasetIntoEventBuildersSplitter module¶
alphatwirl.loop.EventLoop module¶
alphatwirl.loop.EventLoopProgressReportWriter module¶
alphatwirl.loop.EventLoopRunner module¶
alphatwirl.loop.EventsInDatasetReader module¶
-
class
alphatwirl.loop.EventsInDatasetReader.EventsInDatasetReader(eventLoopRunner, reader, collector, split_into_build_events)[source]¶ Bases:
objectThis class manages objects involved in reading events in data sets.
On receiving a data set, this class calls the function split_into_build_events(), which splits the data set into chunks, creates the function build_events() for each chunk, and returns a list of the functions. Then, for each build_events(), This class creates a copy of the reader, creates an event loop, and send it to the event loop runner.
At the end, this class receives results from the event loop runner and have the collector collect them.
alphatwirl.loop.MPEventLoopRunner module¶
-
class
alphatwirl.loop.MPEventLoopRunner.MPEventLoopRunner(communicationChannel)[source]¶ Bases:
objectThis class (concurrently) runs instances of
EventLoop.An instance of this class needs to be initialized with a communication channel with workers that actually run the
EventLoop:runner = MPEventLoopRunner(communicationChannel)
An example of a communication channel is an instance of
CommunicationChannel.The method
begin()does nothing in the current version:runner.begin()
In older versions, multiple processes are forked in this method.
Then, you can give an
EventLoopwith the methodrun():runner.run(eventLoop1)
This class will send the
EventLoopto a worker through the communication channel. The worker, then, runs theEventLoop.You can call the method
run()mutiple times:runner.run(eventLoop2) runner.run(eventLoop3) runner.run(eventLoop4)
If workers are in the background, this method immediately returns. Worker are concurrently running the event loops in the background. If the worker is in the foreground, this method won’t return until the worker finishes running the event loop. Whether workers are in the background or foreground depends on the communication channel with which this class is initialized.
After giving all event loops that you need to run to this class, you need to call the method
end():results = runner.end()
If workers are in the background, this method will wait until workers finish running all event loops. If the worker is in the foreground, this method immediately returns. This method returns the results, the list of the values eventLoops return, sorted in the order given with
run().
alphatwirl.loop.NullCollector module¶
alphatwirl.loop.ReaderComposite module¶
-
class
alphatwirl.loop.ReaderComposite.ReaderComposite[source]¶ Bases:
objectA composite of event readers”
This class is a composite in the composite pattern.
Examples of event readers are instances of
Summarizerand this class.When
event()is called, it callsevent()of each reader in the order in which the readers are added. If a reader returnsFalse, it won’t call the remaining readers.
alphatwirl.loop.splitfuncs module¶
Module contents¶
alphatwirl.misc package¶
Submodules¶
alphatwirl.misc.listToAlignedText module¶
alphatwirl.misc.list_to_aligned_text module¶
Module contents¶
alphatwirl.progressbar package¶
Submodules¶
alphatwirl.progressbar.BProgressMonitor module¶
-
class
alphatwirl.progressbar.BProgressMonitor.BProgressMonitor(presentation)[source]¶ Bases:
objectA progress monitor of tasks.
This class monitors the progress of tasks and present it.
Tasks can be concurrently executed. The progress can be presented in an arbitrary way specified at the initialization.
This class monitors the progress and present it in the background by using
ProgressReportPickup.In order for a progress to be monitored by this class, an object that executes a task needs to send the
ProgressReportthrough theProgressReportercreated by this class.Examples
A presentation method needs to be specified at the initialization. For example, to use
ProgressBar:presentation = ProgressBar() monitor = BProgressMonitor(presentation)
After the initialization, start monitoring:
monitor.begin()
Then, create as many reporters (
ProgressReporter) as the number of the tasks whose progresses need to be monitored:reporter1 = monitor.createReporter() reporter2 = monitor.createReporter() reporter3 = monitor.createReporter()
These reporters can be given to objects which execute the tasks. These objects can be in other processes as long as the reporters are properly passed to them.
If a
ProgressReportis given to these reporters:reporter1.report(report)
the report will be received by
ProgressReportPickup, which will use the report to present the progress in the way specified.When all tasks are finished, end monitoring:
monitor.end()
This will ends
ProgressReportPickup, which is running in a different process.
alphatwirl.progressbar.NullProgressMonitor module¶
alphatwirl.progressbar.ProgressBar module¶
alphatwirl.progressbar.ProgressMonitor module¶
alphatwirl.progressbar.ProgressPrint module¶
alphatwirl.progressbar.ProgressReport module¶
-
class
alphatwirl.progressbar.ProgressReport.ProgressReport(name, done, total, taskid=None)[source]¶ Bases:
objectA progress report
Parameters: - name (str) – the name of the task. if
taskidisNone, used to identify the task - done (int) – the number of the iterations done so far
- total (int) – the total iterations to be done
- taskid (immutable, optional) – if given, used to identify the task. useful if multiple tasks have the same name
- name (str) – the name of the task. if
alphatwirl.progressbar.ProgressReportPickup module¶
alphatwirl.progressbar.ProgressReporter module¶
-
class
alphatwirl.progressbar.ProgressReporter.ProgressReporter(queue)[source]¶ Bases:
objectA progress reporter
This class sends a
ProgressReportto a progress monitor, e.g.,ProgressMonitororBProgressMonitor, which, for example, uses the reports to updateProgressBaron the screen.An instance of this class is initialized with a message queue:
reporter = ProgressReporter(queue)
A reporter, an instance of this class, is typically created and initialized by a progress monitor (
ProgressMonitororBProgressMonitor), which keeps the other end of the message queue. A reporter and a monitor might be running in different processes.A report, an instance of
ProgressReport, can be sent as:reporter.report(report)
This method can be frequently called multiple times. However, after sending one report, the reporter wait for a certain
interval(0.1 seconds by default) before sending another report. Reports received within this interval will be discarded. The exception for this is the last report. The last report, which indicates the completion of the task, will be always sent to the progress monitor regardless of whether it is given within the interval.-
report(report)[source]¶ send
reportto a progress monitorParameters: report (ProgressReport) – a progress report
-
Module contents¶
alphatwirl.roottree package¶
Submodules¶
alphatwirl.roottree.BEventBuilder module¶
alphatwirl.roottree.BEvents module¶
alphatwirl.roottree.Branch module¶
-
class
alphatwirl.roottree.Branch.Branch(name, array, countarray)[source]¶ Bases:
objectThis class encloses an array.array object, which is typically used to set the address of a branch of a ROOT TTree. This class is useful for fast access to contents of TTree.
While TTree can hold many types of data, including objects of user defined classes, this class can be used only for objects of simple types, such as Int_t, Double_t, and arrays of simple types.
Examples
Suppose jet_pt is an instance of this class and the corresponding object in TTree is an array.
>>> len(jet_pt) 3
>>> [v for v in jet_pt] [127.16558074951172, 68.16969299316406, 53.75463104248047]
When the corresponding object in TTree is not an array but a simple type, the length is 1 and the only element is the value.
>>> len(mht) 1
>>> mht[0] 73.7677
alphatwirl.roottree.BranchAddressManager module¶
-
class
alphatwirl.roottree.BranchAddressManager.BranchAddressManager[source]¶ Bases:
objectThe branch address manager for ROOT TTree
This class manages array.array objects used for branch addresses of ROOT TTree. The main purpose of this class is to prevent multiple arrays from being created for the same branch.
All instances of this class share the data.
-
arrayDict= {}¶
-
counterArrayDict= {}¶
-
getArrays(tree, branchName)[source]¶ return the array.array objects for the branch and its counter branch
This method returns a pair of the array.array objects. The first one is for the given tree and branch name. The second one is for its counter branch. The second one will be None when the branch does not have a counter. A pair of None will be returned when the tree does not have the branch.
-
alphatwirl.roottree.BranchAddressManagerForVector module¶
alphatwirl.roottree.BranchBuilder module¶
alphatwirl.roottree.EventBuilder module¶
alphatwirl.roottree.EventBuilderConfig module¶
alphatwirl.roottree.Events module¶
-
class
alphatwirl.roottree.Events.Events(tree, maxEvents=-1, start=0)[source]¶ Bases:
objectAn iterative object for events.
You can use this class to iterate over entries in a ROOT TTree.
You can instantiate this class with a TTree object and an optionally a maximum number of entries to loop over:
inputFile = ROOT.TFile.Open(inputPath) tree = inputFile.Get(treeName) events = Events(tree)
Then, the “for” loop for the tree entries can be:
for event in events:
Note: “event” and “events” are the same object. In each iteration, “event” (and “events”) is loaded with the next entry in the tree.
A content of the tree, e.g., a branch, can be accessed as an attribute of “event”:
event.jet_pt
In order to access to a particular entry, you can use an index. For example, to get 11th entry (the index for the first entry is 0):
event = events[10]
Note: Again “event” and “events” are the same object.
alphatwirl.roottree.inspect module¶
Module contents¶
alphatwirl.selection package¶
Subpackages¶
alphatwirl.selection.modules package¶
Submodules¶
alphatwirl.selection.modules.Count module¶
alphatwirl.selection.modules.LambdaStr module¶
-
class
alphatwirl.selection.modules.LambdaStr.LambdaStr(lambda_str, name=None)[source]¶ Bases:
objectselect events to which a lambda returns True.
A lambda should be given as a string to __init__ and will be evaluated in begin(). This is because a lambda is not picklable.
In the multiprocessing mode, __init__() is called in the main process. Then, the instance will be pickled and sent to subprocesses. begin() will be called in the subprocesses.
alphatwirl.selection.modules.basic module¶
-
class
alphatwirl.selection.modules.basic.All(name=None)[source]¶ Bases:
objectselect events that meet all conditions
alphatwirl.selection.modules.with_count module¶
-
class
alphatwirl.selection.modules.with_count.AllwCount(name=None)[source]¶ Bases:
objectselect events that meet all conditions
Module contents¶
Submodules¶
Module contents¶
alphatwirl.summary package¶
Submodules¶
alphatwirl.summary.BackrefMultipleArrayReader module¶
alphatwirl.summary.Count module¶
-
class
alphatwirl.summary.Count.Count(val=None, weight=1, contents=None)[source]¶ Bases:
objectParameters: - val – If None, initialize with 0. i.e., not counted. Otherwise, typically an empty tuple, counted with the weight. Ignored if contents are given.
- weight (float) – The weight
- contents – Specified contents unless None
alphatwirl.summary.KeyValueComposer module¶
-
class
alphatwirl.summary.KeyValueComposer.KeyValueComposer(keyAttrNames=None, binnings=None, keyIndices=None, valAttrNames=None, valIndices=None)[source]¶ Bases:
objectThis class composes keys and values for the event
(this docstring is under development.)
This class can be used with BEvents.
This class supports inclusive indices ‘*’
This class supports back references.