Skip to content

Add Profiling module: the way to estimate time of nodes and framework overhead

George Ponomarev requested to merge feature/profiling into master

The profiling module located in dagflow/tools/profiling and has the next structure:

  • profiler.py : The base Profiling class for all profiling classes. Contains methods for working with tables of results and making profiling reports.
  • timer_profiler.py : TimerProfiler - The base class for time-related profiling. Should not be used directly.
  • node_profiler.py : NodeProfiler - Estimates the time of individual nodes.
  • framework.py : FrameworkProfiling - Estimates the interaction time between nodes. In other words, it measures framework overhead time.
  • count_calls_profiler.py : CountCallsProfiler - Estimates the number of calls for each node after fitting the model.
  • memory_profiler.py : MemoryProfiler - Estimates the size of edges (in bytes) of the given nodes.
  • delay_node.py : DelayNode - A node that sleeps for a certain time, used for profiler debugging and testing.

Usage example:

nodes = graph._nodes

node_profiler = NodeProfiler(nodes)
report = node_profiler.estimate_target_nodes().print_report()
report.to_csv('output/test_node_report.csv')

framework_profiler = FrameworkProfiler(nodes)
report = framework_profiler.estimate_framework_time().print_report()
report.to_json('output/test_framework_report.json')

calls_profiler = CountCallsProfiler(nodes)
calls_profiler.estimate_calls()
report = calls_profiler.make_report(aggregations=['single', 'std'])

memory_profiler = MemoryProfiler(nodes)
report = memory_profiler.estimate_target_nodes().print_report()
Edited by George Ponomarev

Merge request reports

Loading