Lecture 10 — Higher-order functions and decorators¶
A Python function is an object. So a function can take functions as arguments and return new functions. A wrapper is a function that forwards each call to a stored function. Wrappers count, time and cache calls. The source code of the wrapped functions stays unchanged.
Notes: Lecture 10 ·
code: attractor at lecture-10
The notebook runs at the root of the attractor clone, at the tag lecture-10.
The kernel is Python (attractor 3.13), which setup_env.sh installs:
cd ~/GitHub/attractor
git fetch --tags --force
git switch --detach lecture-10
source setup_env.sh
import functools, inspect, math, os, subprocess, sys, tempfile, time
from pathlib import Path
import numpy as np
TAG = "lecture-10"
if not Path("tools.py").exists(): # outside a clone (Colab, a download): fetch the project
root = Path(tempfile.mkdtemp()) / "attractor"
subprocess.run(["git", "-c", "advice.detachedHead=false", "clone", "-q", "--depth", "1", "--branch", TAG,
"https://github.com/borisbolliet/attractor.git", str(root)], check=True)
os.chdir(root)
sys.path.insert(0, os.getcwd())
subprocess.run(["git", "describe", "--tags"], capture_output=True, text=True).stdout.strip()
'lecture-10'
1 Functions that take and return functions¶
Higher-order functions¶
A higher-order function is a function that takes a callable as an argument, returns a callable, or both.
Taking a callable is enough, and returning a callable is enough.
make_adder(n) of the previous lecture returns a new function, add.
A factory is a higher-order function that builds and returns a new function.
For example, make_adder is a factory.
sorted is a higher-order function.
The argument key of sorted is a function.
sorted calls the key function once per element.
sorted then orders the elements by the results.
sorted(["pear", "fig", "apple"]) # alphabetical
['apple', 'fig', 'pear']
sorted(["pear", "fig", "apple"], key=len) # by len of each word: 3, 4, 5
['fig', 'pear', 'apple']
A callable is an object that code can call like a function.
functools.partial takes a function and returns a new callable with some arguments fixed.
partial is the factory.
from_binary is the object that partial returns.
from_binary is a partial object.
A partial object is callable, but a partial object is not a function.
from_binary = functools.partial(int, base=2) # takes int, returns a new callable
type(from_binary), callable(from_binary)
(functools.partial, True)
from_binary("101") # int("101", base=2)
5
In the project, make_lotka_volterra of models.py takes four rates and returns a new function, rhs.
So make_lotka_volterra is a factory.
integrate of integrators.py takes two callables, step and rhs.
integrate returns arrays.
So integrate is higher-order, but integrate is not a factory.
from integrators import integrate, rk4_step
from models import CHEMOSTAT, make_lotka_volterra
rhs = make_lotka_volterra(**CHEMOSTAT) # the factory returns a function
rhs, rhs(0.0, np.array([1.5, 8.0]))
(<function models.make_lotka_volterra.<locals>.rhs(t, y)>, array([0.24, 3.08]))
print(inspect.getsource(integrate)) # step is called, rhs is passed on to it
def integrate(step, rhs, y0, t0, t1, h):
"""Integrate from t0 to t1 with a fixed step; returns (t, y) with y of shape (n+1, len(y0))."""
n = round((t1 - t0) / h)
y = np.array(y0, dtype=float)
ts, ys = [t0], [y]
for i in range(n):
y = step(rhs, t0 + i * h, y, h)
ts.append(t0 + (i + 1) * h)
ys.append(y)
return np.array(ts), np.array(ys)
t, y = integrate(rk4_step, rhs, np.array([1.5, 8.0]), 0.0, 40.0, 0.05) # two functions in, arrays out
y.shape
(801, 2)
Wrappers¶
A wrapper is a function that callers call in place of a stored function. The wrapper accepts any arguments. The wrapper forwards all these arguments to the stored function. The wrapper returns the result of the stored function. Before or after the forwarded call, the wrapper does work of its own. A wrapper is a closure. The stored function is a free variable of the wrapper. Python keeps the stored function in a cell.
def counted(f):
def wrapper(*args, **kwargs):
wrapper.calls += 1
return f(*args, **kwargs)
wrapper.calls = 0
return wrapper
hyp = counted(math.hypot)
hyp(3, 4), hyp(5, 12), hyp.calls
(5.0, 13.0, 2)
counted is higher-order.
counted takes f and returns wrapper.
wrapper uses the name f.
f is the parameter of counted, and f is not a parameter of wrapper.
A free variable is a name used in the body of a function but bound in the enclosing function.
So f is a free variable of wrapper.
When counted returns, the local names of counted would disappear with the call.
Python therefore keeps f in a cell attached to wrapper.
A closure is a function together with the cells of its free variables.
wrapper is a free variable of wrapper too, because the body of wrapper reads wrapper.calls.
hyp.__code__.co_freevars, hyp.__closure__[0].cell_contents
(('f', 'wrapper'), <function math.hypot>)
Each call of counted creates a new wrapper with its own cells.
All the wrappers share one code object:
hyp2 = counted(math.hypot)
hyp2 is hyp, hyp2.__code__ is hyp.__code__, hyp2.calls
(False, True, 0)
A wrapper must call its parameter.
In experiments/wrong_name.py, the body of the wrapper calls the global f instead of the parameter g.
As a result, the wrapped square calls double.
The wrapper has no cell for the wrapped function:
%run -m experiments.wrong_name
6 6
('wrapper',)
Tools that measure scientific code, such as counters and timers, must not change what the code computes. A wrapper that calls the wrong function can make two different models give the same result.
Forwarding of arguments¶
*args collects the positional arguments of a call into a tuple.
**kwargs collects the keyword arguments of a call into a dictionary.
The call f(*args, **kwargs) unpacks the tuple and the dictionary again.
The signature of a function is the list of parameters that the function accepts.
As a result, one wrapper works for functions of any signature:
def add(x, y):
return x + y
args, kwargs = (3, 4), {"x": 3, "y": 4}
add(*args), add(**kwargs) # add(3, 4) and add(x=3, y=4)
(7, 7)
add(args) # one argument: the tuple
--------------------------------------------------------------------------- TypeError Traceback (most recent call last) Cell In[14], line 1 ----> 1 add(args) # one argument: the tuple TypeError: add() missing 1 required positional argument: 'y'
A wrapper with named parameters accepts the arguments of one signature only.
Python raises TypeError for any other signature:
def counted_badly(f):
def wrapper(x):
wrapper.calls += 1
return f(x)
wrapper.calls = 0
return wrapper
counted_badly(abs)(-2)
2
counted_badly(math.hypot)(3, 4)
--------------------------------------------------------------------------- TypeError Traceback (most recent call last) Cell In[16], line 1 ----> 1 counted_badly(math.hypot)(3, 4) TypeError: counted_badly.<locals>.wrapper() takes 1 positional argument but 2 were given
Metadata of a wrapped function¶
A wrapper is a new function object.
The name of the wrapper is wrapper.
The wrapper does not have the docstring of the wrapped function.
inspect.signature reports (*args, **kwargs) for the wrapper.
The line @functools.wraps(f) above the inner def copies the name, qualified name, docstring and annotations of f.
functools.wraps also sets __wrapped__ to f.
inspect.signature follows __wrapped__ to report the original parameters.
def area(w, h):
"""Area of a w by h rectangle."""
return w * h
def bare(f):
def wrapper(*args, **kwargs):
return f(*args, **kwargs)
return wrapper
def neat(f):
@functools.wraps(f)
def wrapper(*args, **kwargs):
return f(*args, **kwargs)
return wrapper
b, n = bare(area), neat(area)
b.__name__, b.__doc__, inspect.signature(b)
('wrapper', None, <Signature (*args, **kwargs)>)
n.__name__, n.__doc__, inspect.signature(n), n.__wrapped__ is area
('area', 'Area of a w by h rectangle.', <Signature (w, h)>, True)
functools.wraps copies the metadata once, when Python applies the decorator.
functools.wraps does not change the code object of the wrapper.
n.__qualname__ is 'area'.
The code object of the wrapper keeps its own qualified name, n.__code__.co_qualname.
n.__qualname__, n.__code__.co_qualname
('area', 'neat.<locals>.wrapper')
2 Decorators¶
The decorator syntax¶
A decorator is a callable applied to a function object.
Python calls the decorator right after the def statement has created the function.
Python binds the return value of the decorator to the name of the function.
So @deco above def f is f = deco(f).
The decorator below prints a line when the def statement runs, before any call.
def loud(f):
print("decorating", f.__name__)
return f
@loud
def area(w, h):
return w * h
decorating area
area = loud(area); area(2, 3)
decorating area
6
The cell below applies the counter of tools.py to the chemostat right-hand side.
The cell uses the long spelling rhs = counted(rhs).
Each RK4 step calls the right-hand side four times.
So 800 RK4 steps make 4 × 800 = 3200 calls.
from tools import counted
rhs = counted(make_lotka_volterra(**CHEMOSTAT))
t, y = integrate(rk4_step, rhs, np.array([1.5, 8.0]), 0.0, 40.0, 0.05)
y.shape, rhs.calls
((801, 2), 3200)
rhs.reset(); rhs(0.0, np.array([1.0, 10.0])), rhs.calls
(array([0., 0.]), 1)
Decorator factories¶
A decorator factory is a function that returns a decorator.
In @factory(arg), Python first evaluates the call factory(arg).
The call returns a decorator.
Python then applies the returned decorator to the function.
functools.lru_cache is a decorator factory.
For example, functools.lru_cache(maxsize=32) returns a decorator that keeps at most 32 results.
The bare form @functools.lru_cache keeps the default of 128 entries.
@functools.lru_cache(maxsize=32)
def square(n):
return n * n
@functools.lru_cache
def cube(n):
return n ** 3
square.cache_info().maxsize, cube.cache_info().maxsize
(32, 128)
A factory written by hand nests three functions. The factory returns the decorator. The decorator returns the wrapper. The argument of the factory is a free variable of the wrapper:
def announce(message): # factory
def decorator(f): # decorator
def wrapper(*args, **kwargs): # wrapper
print(message)
return f(*args, **kwargs)
return wrapper
return decorator
@announce("Bonjour!")
def double(x):
return 2 * x
double(3), double.__code__.co_freevars
Bonjour!
(6, ('f', 'message'))
tools.py defines a dictionary REGISTRY and a decorator factory register(name).
The decorator returned by register stores the decorated function in REGISTRY under name.
The decorator returns the decorated function unchanged:
import tools
print(inspect.getsource(tools.register))
def register(name):
"""Decorator factory: store the decorated factory under `name` in REGISTRY and return it unchanged."""
def deco(factory):
REGISTRY[name] = factory
return factory
return deco
Import time and call time¶
The top-level code of a module is made of the statements outside any function or class.
Python runs the top-level code once, at the first import of the module.
Python then keeps the module object in sys.modules.
A def statement and its decorator are top-level code.
So the body of a decorator runs at import time, once per decorated function.
The body of a wrapper runs at call time, once per call.
experiments/import_time.py prints a line at import time and a line at call time.
The second import prints nothing:
!{sys.executable} -c "import experiments.import_time as m; m.area(2, 3); m.area(4, 5); import experiments.import_time"
decorator: registering area wrapper: calling area wrapper: calling area
Decorators add counting, timing or caching to many functions without editing the functions. The scientific code then stays separate from the tools that measure it.
A registry is a dictionary from names to functions.
A program uses a registry to call a function chosen by a name read at run time.
A decorator adds each function to the registry when the function is defined.
The import of models.py fills REGISTRY.
make_model then builds a right-hand side by name:
from tools import REGISTRY, make_model
import models
sorted(REGISTRY), REGISTRY is models.REGISTRY
(['lotka_volterra', 'rosenzweig_macarthur'], True)
make_model("lotka_volterra", **models.CHEMOSTAT).__qualname__
'make_lotka_volterra.<locals>.rhs'
Stacked decorators¶
Several decorators written above one def are stacked decorators.
Python applies stacked decorators from the bottom up.
So @A over @B over def g is g = A(B(g)).
experiments/stack_order.py prints which decorator Python applies first:
%run -m experiments.stack_order
B applied to g A applied to g 3 0.0 3 True True
3 Counting and timing¶
Counting calls¶
A count of calls depends on the algorithm and its input only. The recursive calls of a function look up the function by its module-level name. The decoration has bound the module-level name to the wrapper. As a result, the counter also counts the recursive calls:
@counted
def fib(n):
return n if n < 2 else fib(n - 1) + fib(n - 2)
fib(10), fib.calls
(55, 177)
fib.reset(); fib(20), fib.calls
(6765, 21891)
experiments/cost.py counts the right-hand-side calls of Euler's method and RK4 for five step sizes.
The script prints each count next to the error at t = 40 days:
%run -m experiments.cost
method h steps calls error at t=40 euler 0.1 400 400 2.9e+00 euler 0.05 800 800 2.7e+00 euler 0.01 4000 4000 5.5e-01 euler 0.005 8000 8000 2.7e-01 euler 0.001 40000 40000 5.3e-02 rk4 0.4 100 400 9.5e-04 rk4 0.2 200 800 1.3e-04 rk4 0.1 400 1600 1.1e-05 rk4 0.05 800 3200 7.5e-07 rk4 0.01 4000 16000 1.3e-09 RK4: 3200 calls for error < 1e-05 (h = 0.05) Euler: ~2e+08 calls (extrapolated from the slope of the table)
Timers, overhead and profiles¶
time.perf_counter() returns the value of a clock for measuring intervals.
The perf_counter clock never goes backwards.
The zero point of the perf_counter clock is undefined.
So only a difference of two readings has a meaning.
The timing wrapper of timed in tools.py reads the clock before and after the forwarded call.
The overhead of a wrapper is the work that the wrapper does itself, plus the cost of the extra call.
The overhead of timed is about 150 ns per call.
start = time.perf_counter()
total = sum(range(10**6))
elapsed = time.perf_counter() - start
total, elapsed
(499999500000, 0.007529916008934379)
A profiler is a tool that records each function call of a run with its time.
cProfile reports three values for each function:
ncalls, the number of calls.tottime, the time in the function's own lines.cumtime, the time in the function, including the functions that the function calls.
experiments/timing.py profiles one run of integrate.
The right-hand side is decorated with counted:
%run -m experiments.timing
integrate True
1 call, 0.005 s
12128 function calls (12127 primitive calls) in 0.007 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
800 0.003 0.000 0.006 0.000 integrators.py:10(rk4_step)
3200 0.002 0.000 0.002 0.000 models.py:14(rhs)
3203 0.001 0.000 0.001 0.000 {built-in method numpy.array}
3200 0.001 0.000 0.003 0.000 tools.py:8(wrapper)
1 0.000 0.000 0.007 0.007 <string>:1(<module>)
1 0.000 0.000 0.001 0.001 integrators.py:20(integrate)
1601 0.000 0.000 0.000 0.000 {method 'append' of 'list' objects}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
2 0.000 0.000 0.000 0.000 attrsettr.py:66(_get_attr_opt)
2/1 0.000 0.000 0.007 0.007 {built-in method builtins.exec}
1 0.000 0.000 0.000 0.000 {built-in method _heapq.heappush}
2 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:1390(_handle_fromlist)
2 0.000 0.000 0.000 0.000 attrsettr.py:43(__getattr__)
1 0.000 0.000 0.000 0.000 zmqstream.py:574(_handle_events)
7 0.000 0.000 0.000 0.000 enum.py:695(__call__)
1 0.000 0.000 0.000 0.000 asyncio.py:208(call_at)
1 0.000 0.000 0.000 0.000 socket.py:785(recv_multipart)
3 0.000 0.000 0.000 0.000 enum.py:1607(__and__)
12 0.000 0.000 0.000 0.000 enum.py:1589(_get_value)
1 0.000 0.000 0.000 0.000 base_events.py:805(call_at)
24 0.000 0.000 0.000 0.000 {built-in method builtins.isinstance}
1 0.000 0.000 0.000 0.000 events.py:36(__init__)
3 0.000 0.000 0.000 0.000 base_events.py:772(time)
1 0.000 0.000 0.000 0.000 events.py:113(__init__)
1 0.000 0.000 0.000 0.000 base_events.py:781(call_later)
1 0.000 0.000 0.000 0.000 zmqstream.py:615(_handle_recv)
1 0.000 0.000 0.000 0.000 selector_events.py:740(_process_events)
1 0.000 0.000 0.000 0.000 queue.py:112(empty)
1 0.000 0.000 0.000 0.000 zmqstream.py:654(_rebuild_io_state)
1 0.000 0.000 0.000 0.000 asyncio.py:198(_handle_events)
7 0.000 0.000 0.000 0.000 enum.py:1156(__new__)
1 0.000 0.000 0.000 0.000 ioloop.py:602(call_later)
1 0.000 0.000 0.000 0.000 iostream.py:677(_schedule_in_thread)
1 0.000 0.000 0.000 0.000 base_events.py:1955(_add_callback)
1 0.000 0.000 0.000 0.000 events.py:87(_run)
1 0.000 0.000 0.000 0.000 iostream.py:233(_handle_event)
1 0.000 0.000 0.000 0.000 zmqstream.py:547(_run_callback)
2 0.000 0.000 0.000 0.000 ioloop.py:547(time)
1 0.000 0.000 0.000 0.000 {method 'run' of '_contextvars.Context' objects}
2 0.000 0.000 0.000 0.000 {built-in method builtins.getattr}
1 0.000 0.000 0.000 0.000 abc.py:117(__instancecheck__)
1 0.000 0.000 0.000 0.000 enum.py:1596(__or__)
1 0.000 0.000 0.000 0.000 {built-in method _abc._abc_instancecheck}
1 0.000 0.000 0.000 0.000 zmqstream.py:533(sending)
1 0.000 0.000 0.000 0.000 zmqstream.py:677(_update_handler)
2 0.000 0.000 0.000 0.000 {built-in method builtins.max}
4 0.000 0.000 0.000 0.000 {built-in method builtins.len}
2 0.000 0.000 0.000 0.000 {built-in method builtins.hasattr}
3 0.000 0.000 0.000 0.000 {built-in method time.monotonic}
2 0.000 0.000 0.000 0.000 {method 'popleft' of 'collections.deque' objects}
1 0.000 0.000 0.000 0.000 {built-in method _contextvars.copy_context}
2 0.000 0.000 0.000 0.000 {built-in method time.time}
1 0.000 0.000 0.000 0.000 {method 'get' of 'dict' objects}
1 0.000 0.000 0.000 0.000 {built-in method builtins.round}
1 0.000 0.000 0.000 0.000 {method 'append' of 'collections.deque' objects}
2 0.000 0.000 0.000 0.000 zmqstream.py:529(receiving)
1 0.000 0.000 0.000 0.000 events.py:129(__lt__)
1 0.000 0.000 0.000 0.000 {method '__exit__' of '_thread.lock' objects}
1 0.000 0.000 0.000 0.000 base_events.py:554(_check_closed)
1 0.000 0.000 0.000 0.000 queue.py:266(_qsize)
2 0.000 0.000 0.000 0.000 {method 'upper' of 'str' objects}
1 0.000 0.000 0.000 0.000 base_events.py:2068(get_debug)
A profile shows which functions take the time of a run. The bookkeeping of the profiler slows each call that the profiler records. A benchmark measures how long one operation takes. A benchmark repeats the operation alone and times the repetitions.
Computing time is limited, on a laptop as on a cluster. A measurement before an optimisation directs the effort to the part of the code that takes the time.
4 Caching¶
Memoisation¶
Memoisation stores the result of a call under the arguments of the call.
A memoised function returns the stored result when a caller passes the same arguments again.
Memoisation is correct when the result depends on the arguments alone.
A cache is the store of results that a memoised function reads before computing.
The cache of functools.lru_cache is a dictionary keyed by the arguments.
So the arguments must be hashable.
A hashable object is an object whose hash value never changes.
A hit is a call that the wrapper answers from the cache.
A miss is a call whose result the wrapper computes and stores.
@functools.lru_cache(maxsize=None)
def fib(n):
return n if n < 2 else fib(n - 1) + fib(n - 2)
fib(80), fib.cache_info()
(23416728348467685, CacheInfo(hits=78, misses=81, maxsize=None, currsize=81))
A cache pays off when a call is slow:
@functools.lru_cache(maxsize=32)
def slow_square(n):
time.sleep(0.5)
return n * n
for _ in range(2):
start = time.perf_counter()
slow_square(30)
print(f"{time.perf_counter() - start:.3f} s")
slow_square.cache_info()
0.505 s 0.000 s
CacheInfo(hits=1, misses=1, maxsize=32, currsize=1)
A dictionary finds a key by its hash and then compares keys with ==.
The dictionary does not compare identities.
Two tuples built at run time are two objects with equal hashes.
As a result, each tuple finds the entry stored under the other tuple.
A list has no hash, because the contents of a list can change.
Python raises TypeError when a list is passed to a cached function.
a, b = tuple([1, 2]), tuple([1, 2])
a == b, hash(a) == hash(b), a is b
(True, True, False)
fib([80])
--------------------------------------------------------------------------- TypeError Traceback (most recent call last) Cell In[39], line 1 ----> 1 fib([80]) TypeError: unhashable type: 'list'
A cache can save hours in an analysis that repeats the same computations. A cached result is correct only while everything it depends on is unchanged.
Cached objects and aliasing¶
A cache returns the stored object itself. The cache makes no copy. A caller that writes into a cached result changes what every later caller receives. Python raises no error.
@functools.lru_cache(maxsize=32)
def make_list(n):
return [0] * n
a = make_list(3)
b = make_list(3)
a[0] = 99
a, b, a is b, make_list(3) # the body says [0] * n, the cache returns the changed list
([99, 0, 0], [99, 0, 0], True, [99, 0, 0])
experiments/fib_cache.py caches a loader of the chemostat runs by path.
Two calls of the loader return one array.
A write through the first result changes the second result:
%run -m experiments.fib_cache
23416728348467685 CacheInfo(hits=78, misses=81, maxsize=None, currsize=81) fib_plain(28): 0.03 s TypeError: unhashable type: 'numpy.ndarray' True CacheInfo(hits=1, misses=1, maxsize=128, currsize=1) -1.0 1.92
The exercises and their solutions are at the end of the notes.
The scripts of the exercises are in exercises/lecture10/.