How do I profile a Python script to measure execution time?
Project Euler and other coding contests often have time constraints, and developers frequently compare solution performance. In Python, common approaches like adding timing code to main can be cumbersome. What are the best methods for profiling Python program execution time, including built-in tools and third-party libraries that provide detailed performance analysis?
Python offers several powerful tools for profiling script execution time, ranging from built-in modules like timeit and cProfile to advanced third-party libraries such as line_profiler and py-spy. The optimal method depends on your specific needs - whether you need quick timing measurements, detailed function profiling, or comprehensive performance analysis with visualization.
Contents
- Built-in Python Profiling Tools
- Third-Party Profiling Libraries
- Choosing the Right Profiling Method
- Practical Examples for Coding Contests
- Advanced Profiling Techniques
Built-in Python Profiling Tools
Python comes with several built-in modules that are perfect for timing and profiling without requiring additional installations.
timeit Module
The timeit module is ideal for measuring the execution time of small code snippets with high precision. It automatically handles timing overhead and provides statistical measurements.
import timeit
# Time a simple function
def example_function():
return sum(range(1000))
# Measure execution time
execution_time = timeit.timeit(example_function, number=1000)
print(f"Average execution time: {execution_time/1000:.6f} seconds")
# Time a code snippet directly
snippet_time = timeit.timeit('"-".join(str(n) for n in range(100))',
number=10000)
print(f"Snippet time: {snippet_time:.6f} seconds")
Key features:
timeit.timeit(stmt, setup, timer, number)- main timing functiontimeit.repeat(stmt, repeat, number)- runs timing multiple times for statistical analysistimeit.default_timer()- platform-appropriate timer function
cProfile Module
For more detailed profiling of entire scripts or functions, cProfile provides comprehensive performance statistics including function call counts, timing, and cumulative time.
import cProfile
def fibonacci(n):
if n <= 1:
return n
return fibonacci(n-1) + fibonacci(n-2)
def main():
# Profile the fibonacci function
cProfile.run('fibonacci(20)')
if __name__ == "__main__":
main()
This output shows:
- Number of function calls
- Total time spent in each function
- Time per call
- Cumulative time
sys Module for Simple Timing
For basic script timing, the sys module provides simple timing functions:
import sys
import time
start_time = time.time()
# Your code here
result = sum(range(1000000))
end_time = time.time()
print(f"Script executed in {end_time - start_time:.4f} seconds")
Third-Party Profiling Libraries
When built-in tools aren’t sufficient, several third-party libraries offer advanced profiling capabilities.
line_profiler
For detailed line-by-line timing, line_profiler is essential for identifying exactly where your code spends its time.
Installation:
pip install line_profiler
Usage:
# Decorate the function you want to profile
from line_profiler import LineProfiler
def example_function():
total = 0
for i in range(1000):
total += i * i
return total
# Create and run the profiler
lp = LineProfiler()
lp_wrapper = lp(example_function)
lp_wrapper()
lp.print_stats()
py-spy
py-spy is a sampling profiler that can profile running Python processes without modifying your code or restarting your application.
Installation:
pip install py-spy
Usage:
# Profile a running Python process
py-spy top --pid <process_id>
# Profile a script
py-spy top -m your_script.py
# Profile and save to file
py-spy record -o profile.svg -- your_script.py
memory_profiler
For memory usage profiling, especially important in coding contests where memory limits often exist:
Installation:
pip install memory_profiler
Usage:
from memory_profiler import profile
@profile
def memory_intensive_function():
data = []
for i in range(10000):
data.append([i] * 1000)
return data
if __name__ == "__main__":
memory_intensive_function()
pyinstrument
A sampling profiler with a beautiful HTML output:
Installation:
pip install pyinstrument
Usage:
import pyinstrument
def your_function():
# Your code here
pass
# Profile and get HTML report
profiler = pyinstrument.Profiler()
profiler.start()
your_function()
profiler.stop()
print(profiler.output_text(unicode=True, color=True))
viztracer
For comprehensive tracing and visualization:
Installation:
pip install viztracer
Usage:
python -m viztracer your_script.py
Choosing the Right Profiling Method
| Scenario | Recommended Tool | Why |
|---|---|---|
| Quick timing of small snippets | timeit |
High precision, low overhead |
| Function-level profiling | cProfile |
Built-in, comprehensive stats |
| Line-by-line timing | line_profiler |
Detailed per-line analysis |
| Live application profiling | py-spy |
No code modification needed |
| Memory usage analysis | memory_profiler |
Tracks memory allocation |
| Beautiful visualizations | pyinstrument |
HTML reports with flamegraphs |
| Comprehensive tracing | viztracer |
Detailed execution tracing |
Practical Examples for Coding Contests
Project Euler Problem Timing
For Project Euler problems, you often need to optimize algorithms:
import time
import cProfile
from functools import lru_cache
@lru_cache(maxsize=None)
def fibonacci_cached(n):
if n <= 1:
return n
return fibonacci_cached(n-1) + fibonacci_cached(n-2)
def fibonacci_naive(n):
if n <= 1:
return n
return fibonacci_naive(n-1) + fibonacci_naive(n-2)
# Compare performance
def compare_performance():
n = 35
# Naive approach
start = time.time()
result1 = fibonacci_naive(n)
naive_time = time.time() - start
# Cached approach
start = time.time()
result2 = fibonacci_cached(n)
cached_time = time.time() - start
print(f"Naive: {naive_time:.4f}s, Cached: {cached_time:.4f}s")
print(f"Speedup: {naive_time/cached_time:.1f}x")
# Detailed profiling
cProfile.run('compare_performance()', sort='cumulative')
Contest Solution Optimization
When optimizing solutions for coding contests:
import time
def timing_decorator(func):
def wrapper(*args, **kwargs):
start = time.perf_counter()
result = func(*args, **kwargs)
end = time.perf_counter()
print(f"{func.__name__} executed in {end-start:.6f} seconds")
return result
return wrapper
@timing_decorator
def contest_solution(n):
# Your optimized solution here
if n < 2:
return n
a, b = 0, 1
for _ in range(2, n + 1):
a, b = b, a + b
return b
# Usage
if __name__ == "__main__":
result = contest_solution(1000)
print(f"Result: {result}")
Advanced Profiling Techniques
Profiling Specific Code Blocks
For targeted profiling within larger applications:
import contextlib
import time
@contextlib.contextmanager
def timer(description):
start = time.time()
yield
end = time.time()
print(f"{description}: {end-start:.4f} seconds")
# Usage
with timer("Data processing"):
# Your code here
data = [i**2 for i in range(100000)]
with timer("Algorithm execution"):
# Your algorithm here
result = sum(data)
Memory and CPU Profiling Together
For comprehensive performance analysis:
import tracemalloc
import time
import cProfile
def comprehensive_profile(func):
def wrapper(*args, **kwargs):
# Start memory tracking
tracemalloc.start()
# Start CPU profiling
profiler = cProfile.Profile()
profiler.enable()
start_time = time.time()
result = func(*args, **kwargs)
end_time = time.time()
# Stop profiling
profiler.disable()
profiler.print_stats(sort='cumulative')
# Get memory stats
current, peak = tracemalloc.get_traced_memory()
tracemalloc.stop()
print(f"Execution time: {end_time - start_time:.4f} seconds")
print(f"Current memory usage: {current / 1024 / 1024:.2f} MB")
print(f"Peak memory usage: {peak / 1024 / 1024:.2f} MB")
return result
return wrapper
@comprehensive_profile
def memory_and_cpu_intensive_function():
# Your function here
data = []
for i in range(10000):
data.append([i] * 1000)
return len(data)
Profiling Asynchronous Code
For modern async applications:
import asyncio
import time
async def profile_async_function(func, *args, **kwargs):
start = time.time()
result = await func(*args, **kwargs)
end = time.time()
print(f"{func.__name__} executed in {end-start:.4f} seconds")
return result
async def async_example():
await asyncio.sleep(1)
return "Done"
# Usage
async def main():
result = await profile_async_function(async_example)
print(result)
if __name__ == "__main__":
asyncio.run(main())
Conclusion
Python offers a comprehensive ecosystem for profiling script execution time, from simple timing measurements to detailed performance analysis. For coding contests like Project Euler, start with timeit for quick measurements and cProfile for function-level analysis. When you need line-by-line timing, line_profiler is invaluable. For live applications, py-spy provides non-invasive profiling, while memory_profiler helps identify memory bottlenecks.
The key to effective profiling is choosing the right tool for your specific needs and using it systematically to identify performance bottlenecks. Remember that profiling is an iterative process - profile, optimize, profile again. This approach will help you develop faster, more efficient solutions for coding contests and real-world applications alike.
To get started, try timing your existing solutions with timeit and cProfile, then graduate to more detailed profiling as needed. The insights gained will not only help you solve problems faster but also improve your understanding of Python’s performance characteristics.