Add Profiling module: the way to estimate time of nodes and framework overhead
The profiling module located in dagflow/tools/profiling
and has the next structure:
-
profiler.py
: The baseProfiling
class for all profiling classes. Contains methods for working with tables of results and making profiling reports. -
timer_profiler.py
:TimerProfiler
- The base class for time-related profiling. Should not be used directly. -
node_profiler.py
:NodeProfiler
- Estimates the time of individual nodes. -
framework.py
:FrameworkProfiling
- Estimates the interaction time between nodes. In other words, it measures framework overhead time. -
count_calls_profiler.py
:CountCallsProfiler
- Estimates the number of calls for each node after fitting the model. -
memory_profiler.py
:MemoryProfiler
- Estimates the size of edges (in bytes) of the given nodes. -
delay_node.py
:DelayNode
- A node that sleeps for a certain time, used for profiler debugging and testing.
Usage example:
nodes = graph._nodes
node_profiler = NodeProfiler(nodes)
report = node_profiler.estimate_target_nodes().print_report()
report.to_csv('output/test_node_report.csv')
framework_profiler = FrameworkProfiler(nodes)
report = framework_profiler.estimate_framework_time().print_report()
report.to_json('output/test_framework_report.json')
calls_profiler = CountCallsProfiler(nodes)
calls_profiler.estimate_calls()
report = calls_profiler.make_report(aggregations=['single', 'std'])
memory_profiler = MemoryProfiler(nodes)
report = memory_profiler.estimate_target_nodes().print_report()
Edited by George Ponomarev