Lecture 10 — Higher-order functions and decorators¶
Notes: Lecture 10 ·
code: attractor at lecture-10
# mutation
# changes object without changing identity
a=[1,2,3]
print(id(a))
a.append(4)
print(id(a))
4420225216 4420225216
# rebinding
# changes which object a name refers to
a=(1,2)
print(id(a))
a+=(3,)
print(id(a))
4420227840 4420318976
a
(1, 2, 3)
# aliasing is when multiple names refer to the same object
a=[1,2,3]
b=a
a is b
True
dir() returns callable and non-callable attributes of an object or in the current scope
functions like print, dir, sorted getattr are available through Python's builtins module, which Python makes accessible automatically.
a = [1,2] # a is a name bound to an instance of a class object list (a list object)
print(type(a))
dir(a)
<class 'list'>
['__add__', '__class__', '__class_getitem__', '__contains__', '__delattr__', '__delitem__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__getstate__', '__gt__', '__hash__', '__iadd__', '__imul__', '__init__', '__init_subclass__', '__iter__', '__le__', '__len__', '__lt__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__reversed__', '__rmul__', '__setattr__', '__setitem__', '__sizeof__', '__str__', '__subclasshook__', 'append', 'clear', 'copy', 'count', 'extend', 'index', 'insert', 'pop', 'remove', 'reverse', 'sort']
getattr(a,"__len__")
<method-wrapper '__len__' of list object at 0x1075a6940>
callable(getattr(a,"__len__"))
True
getattr(a,"__len__")()
2
dir()
['In', 'Out', '_', '_3', '_4', '_5', '_6', '_7', '__', '___', '__builtin__', '__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__session__', '__spec__', '_dh', '_i', '_i1', '_i2', '_i3', '_i4', '_i5', '_i6', '_i7', '_i8', '_ih', '_ii', '_iii', '_oh', 'a', 'b', 'exit', 'get_ipython', 'open', 'quit']
# locals()
#, globals()
# globals()
The notebook runs at the root of the attractor clone, at the tag lecture-10, with the kernel
Python (attractor 3.12) that setup_env.sh installs:
cd ~/GitHub/attractor
git fetch --tags --force
git switch --detach lecture-10
source setup_env.sh
import functools, inspect, math, os, subprocess, sys, tempfile, time # all part of stdl
from pathlib import Path
import numpy as np
TAG = "lecture-10"
if not Path("tools.py").exists(): # outside a clone (Colab, a download): fetch the project
root = Path(tempfile.mkdtemp()) / "attractor"
subprocess.run(["git", "-c", "advice.detachedHead=false", "clone", "-q", "--depth", "1", "--branch", TAG,
"https://github.com/borisbolliet/attractor.git", str(root)], check=True)
os.chdir(root)
sys.path.insert(0, os.getcwd())
subprocess.run(["git", "describe", "--tags"], capture_output=True, text=True).stdout.strip()
'lecture-10'
functools module in Python's standard library provides higher-order functions and tools to modify or extend callable objects without rewriting their core logic
1 Functions that take and return functions¶
Higher-order functions¶
A higher-order function takes a callable as an argument, or returns a callable, or both (i.e., both has a function as argument and returns a function).
In the previous lecture we saw the make_adder function factory: this is an example of a higher-order function.
Here is another very simple case of a higher-order function that takes a function as an argument (the function len):
sorted(["pear", "fig", "apple"], key=len)
['fig', 'pear', 'apple']
The next example is a higher-order function that has both a function as an argument (the function int) and returns a function.
from_binary = functools.partial(int, base=2)
type(from_binary), callable(from_binary)
(functools.partial, True)
Here, to be precise, from_binary is not a factory, but functools.partial is.
from_binary("101")
5
In the project, integrate of integrators.py takes two functions:
- the step of a method
- and the right-hand side
Additionally, make_lotka_volterra of models.py is a factory, a function that builds and
returns another function.
Next... some other example to illustrate how to inspect python code... first, just some imports:
from integrators import integrate, rk4_step # --> looks in integrators.py and imports the functions integrate and rk4_step
from models import CHEMOSTAT, make_lotka_volterra # --> looks in models.py and imports the disctionary CHEMOSTAT and the function make_lotka_volterra
We can use the inspect module (part of the Python standard library) to do some introspection when we are not sure:
inspect.signature(integrate) # Get a signature object for the passed callable --> here the passed callable is `integrate`
<Signature (step, rhs, y0, t0, t1, h)>
The signature lists the parameter names but doesnt give the types.
We can use getsource to get more details:
print(inspect.getsource(integrate))
def integrate(step, rhs, y0, t0, t1, h):
"""Integrate from t0 to t1 with a fixed step; returns (t, y) with y of shape (n+1, len(y0))."""
n = round((t1 - t0) / h)
y = np.array(y0, dtype=float)
ts, ys = [t0], [y]
for i in range(n):
y = step(rhs, t0 + i * h, y, h)
ts.append(t0 + (i + 1) * h)
ys.append(y)
return np.array(ts), np.array(ys)
step and rhs are callables, and in this case, functions.
Another useful inspect method is the getdoc method to print the docstrings.
inspect.getdoc(integrate)
'Integrate from t0 to t1 with a fixed step; returns (t, y) with y of shape (n+1, len(y0)).'
Here we illustrate how to introspect using the dunder __qualname__ (special attribute, means qualified name. Shows name of a callable and where it was defined (including possible enclosing scope))
CHEMOSTAT
{'a': 0.8, 'b': 0.08, 'c': 0.77, 'd': 0.77}
make_lotka_volterra.__qualname__
'make_lotka_volterra'
make_lotka_volterra.__name__
'make_lotka_volterra'
make_lotka_volterra(**CHEMOSTAT).__qualname__ , make_lotka_volterra(**CHEMOSTAT).__name__
('make_lotka_volterra.<locals>.rhs', 'rhs')
What does this tell us?
It tells us that rhs is a function, defined inside make_lotka_volterra (it is a local of that function, created by each call)
The __qualname__ showing <local> is evidence that the factory returns a function it built inside itself.
To see the difference try on a function that returns a number
def area(w, h):
return w * h # returns a number
area.__qualname__ # 'area'
'area'
area(2, 3).__qualname__
--------------------------------------------------------------------------- AttributeError Traceback (most recent call last) Cell In[70], line 1 ----> 1 area(2, 3).__qualname__ AttributeError: 'int' object has no attribute '__qualname__'
area.__qualname__
'area'
hasattr(area,"__qualname__")
True
area(1,2).__qualname__
--------------------------------------------------------------------------- AttributeError Traceback (most recent call last) Cell In[72], line 1 ----> 1 area(1,2).__qualname__ AttributeError: 'int' object has no attribute '__qualname__'
print(inspect.getsource(make_lotka_volterra(**CHEMOSTAT)))
def rhs(t, y):
x, z = y
return np.array([a * x - b * x * z, c * x * z - d * z])
More of the same:
def counter():
def tick():
pass
return tick
t = counter()
t.__name__ # 'tick'
'tick'
t.__qualname__ # 'counter.<locals>.tick'
'counter.<locals>.tick'
Wrappers¶
A wrapper accepts any arguments, forwards them to a function stored (when the wrapper was made), and returns that function's result, with work of its own before or after the call.
It is a closure: the stored function is a free variable of the wrapper, it is kept in a cell.
Slowly:
counted is higher-order. It takes a function, f, and returns a new function, wrapper.
It is also a factory, since it builds the function wrapper.
Here wrapper uses the name f, but the name f is not a parameter or a local variable of wrapper, it is the parameter of the enclosing function, counted. A name used in a function but bound in the function around it is a free variable.
More generally: A free variable is a variable referenced within a function that is neither local to that function nor global, but is resolved from an enclosing function scope.
def counted(f): # a decorator
def wrapper(*args, **kwargs): # the wrapper
wrapper.calls += 1
return f(*args, **kwargs)
wrapper.calls = 0
return wrapper
hyp = counted(math.hypot)
hyp(3, 4), hyp(5, 12), hyp.calls
(5.0, 13.0, 2)
math.hypot.calls
--------------------------------------------------------------------------- AttributeError Traceback (most recent call last) Cell In[33], line 1 ----> 1 math.hypot.calls AttributeError: 'builtin_function_or_method' object has no attribute 'calls'
type(math.hypot)
builtin_function_or_method
hyp(3, 4), hyp(5, 12), hyp.calls
(5.0, 13.0, 4)
hyp is counted(math.hypot)
False
type(hyp)
function
counted(math.hypot).__name__
'wrapper'
In this example we have created a wrapper that counts calls to function.
math.hypotis the original functionhypis the wrapped function
print(hyp.__qualname__)
counted.<locals>.wrapper
type(hyp.__qualname__)
str
hyp.__name__
'wrapper'
hyp.__qualname__
'counted.<locals>.wrapper'
def double(x):
return 2 * x
f = double
f is double # True
f.__name__ # 'double'
double.__name__ # 'double'
f.__name__ is double.__name__ # True
True
def square(x):
return x**2
f = square
print(f.__name__) # square
print(square is f) # True
square True
hyp.__code__
<code object wrapper at 0x114717530, file "/var/folders/hc/d14d998d3rv6_7vdcjskjrxc0000gp/T/ipykernel_88868/3306617900.py", line 2>
hyp2 = counted(math.hypot)
hyp2.__code__
<code object wrapper at 0x114717530, file "/var/folders/hc/d14d998d3rv6_7vdcjskjrxc0000gp/T/ipykernel_88868/3306617900.py", line 2>
make_lotka_volterra.__code__
<code object make_lotka_volterra at 0x107260490, file "/Users/boris/GitHub/attractor/models.py", line 11>
Two function objects share the same code object
type(hyp.__code__)
code
hyp.__code__.co_freevars ##
('f', 'wrapper')
Note the subtelty that here the name wrapper is a free variable of the inner function wrapper: it is bound (by the def statement in the enclosing scope of counted, not in the local scope of wrapper.
hyp.__closure__
(<cell at 0x107796380: builtin_function_or_method object at 0x104449c10>, <cell at 0x1147f97b0: function object at 0x1147efce0>)
hyp.__closure__[0]
<cell at 0x107796380: builtin_function_or_method object at 0x104449c10>
type(hyp.__closure__[0])
cell
dir(hyp.__closure__[0])
['__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'cell_contents']
hyp.__closure__[0].cell_contents.__name__
'hypot'
hyp.__closure__[0].cell_contents.__code__
--------------------------------------------------------------------------- AttributeError Traceback (most recent call last) Cell In[49], line 1 ----> 1 hyp.__closure__[0].cell_contents.__code__ AttributeError: 'builtin_function_or_method' object has no attribute '__code__'
hyp.__closure__[0].cell_contents
<function math.hypot>
hyp.__closure__[1].cell_contents.__name__
'wrapper'
hyp.__closure__[1].cell_contents.__code__
<code object wrapper at 0x11484fa30, file "/var/folders/hc/d14d998d3rv6_7vdcjskjrxc0000gp/T/ipykernel_103/3162283636.py", line 2>
why are these closure cells important?
counted(math.hypot)
<function __main__.counted.<locals>.wrapper(*args, **kwargs)>
Closure cell 0 holds a reference to math.hypot function object: Allows hyp to call the original function
Closure cell 1 holds a reference to wrapper function object, allos hyp to increment its own calls attribute.
In experiments/wrong_name.py the body calls the global f
instead of the parameter g.
%run -m experiments.wrong_name
6 6
('wrapper',)
f is referenced inside wrapper, but it is bound in the global scope, not an enclosing function scope. Therefore, it is a global variable, not a closure free-variable.
def double(x):
return 2 * x
def square(x):
return x * x
f = square
def counted(g):
def wrapper(*args, **kwargs):
wrapper.calls += 1
return f(*args, **kwargs)
wrapper.calls = 0
return wrapper
d, s = counted(double), counted(square)
print(d(3), s(3))
print(s.__code__.co_freevars)
9 9
('wrapper',)
Forwarding of arguments¶
In our example, *args collects the positional arguments of a call into a tuple and **kwargs the keyword arguments
into a dictionary.
f(*args, **kwargs) unpacks them, so this wrapper serves functions of any signature, the list of parameters a function accepts.
def add(x,y):
return x+y
args = (3,4)
add(args)
--------------------------------------------------------------------------- TypeError Traceback (most recent call last) Cell In[110], line 5 1 def add(x,y): 2 return x+y 3 4 args = (3,4) ----> 5 add(args) TypeError: add() missing 1 required positional argument: 'y'
add(*args)
7
def add(price1 = True, price2 = False):
return price1+price2
inspect.signature(add)
<Signature (price1=True, price2=False)>
add()
1
prices={'price1':90,'price2':10.1}
add(**prices)
100.1
def add(price1 : int, price2 : int):
return price1+price2
prices={'price1':90,'price2':10.1}
add(**prices)
100.1
A wrapper with named parameters fits one signature only:
def counted_x(f):
def wrapper(x):
wrapper.calls += 1
return f(x)
wrapper.calls = 0
return wrapper
c = counted_x(abs)
c(-2), c.calls
(2, 1)
counted_x(math.hypot)(3, 4)
--------------------------------------------------------------------------- TypeError Traceback (most recent call last) Cell In[202], line 1 ----> 1 counted_x(math.hypot)(3, 4) TypeError: counted_x.<locals>.wrapper() takes 1 positional argument but 2 were given
Metadata of a wrapped function¶
A wrapper is a new function object named wrapper, without the docstring of the function it wraps,
and inspect.signature reports (*args, **kwargs).
Usinge @functools.wraps(f) above the inner def
copies the name, qualified name, docstring and annotations of f, and sets __wrapped__,
which inspect.signature follows.
def area(w, h):
"""Area of a w by h rectangle."""
return w * h
def bare(f):
def wrapper(*args, **kwargs):
print("hi\n")
return f(*args, **kwargs)
return wrapper
def neat(f):
@functools.wraps(f)
def wrapper(*args, **kwargs):
print("bonjour\n")
return f(*args, **kwargs)
return wrapper
b, n = bare(area), neat(area)
b.__name__, b.__doc__, inspect.signature(b)
('wrapper', None, <Signature (*args, **kwargs)>)
b(9,10)
hi
90
n(9,10)
bonjour
90
area.__doc__
'Area of a w by h rectangle.'
b.__doc__
n.__doc__
'Area of a w by h rectangle.'
functools.wraps creates a __wrapped__ attribute
n.__name__, n.__doc__, inspect.signature(n), n.__wrapped__ is area
('area', 'Area of a w by h rectangle.', <Signature (w, h)>, True)
n.__wrapped__(9,10)
90
n.__qualname__
'area'
area.__code__.co_qualname
'area'
n.__code__.co_qualname
'neat.<locals>.wrapper'
2 Decorators¶
The decorator syntax¶
A decorator is a callable applied to a function right after the def statement has created it.
Its return value is bound to the function's name: @deco above def f is f = deco(f).
The line below appears when the def runs, before any call.
def loud(f):
print("decorating", f.__name__)
return f
@loud
def area(w, h):
return w * h
@counted
def area2(w, h):
return w * h
decorating area
area = loud(area); area(2, 3)
decorating area
6
area2(2, 3),area2.calls
(6, 1)
callable(area)
True
The counter of tools.py on the chemostat right-hand side, in the long spelling
rhs = counted(rhs): 800 RK4 steps make 4 × 800 = 3200 calls.
from tools import counted
rhs = counted(make_lotka_volterra(**CHEMOSTAT))
rhs.calls
3200
t, y = integrate(rk4_step, rhs, np.array([1.5, 8.0]), 0.0, 40.0, 0.05)
y.shape, rhs.calls
((801, 2), 3200)
rhs.reset()
rhs(0.0, np.array([1.0, 10.0])), rhs.calls
(array([0., 0.]), 1)
Decorator factories¶
A decorator factory is a function whose return value is a decorator. It adds another level to decorators.
In @factory(arg) the call first returns the decorator, which is then applied.
def announce(message): # Decorator factory
def decorator(f): # Decorator
def wrapper(*args, **kwargs): # Wrapper
print(message)
return f(*args, **kwargs)
return wrapper
return decorator
@announce("Bonjour!")
def double(x):
return 2 * x
double(3)
Bonjour!
6
btw in this example message is also a free variable
double.__closure__
(<cell at 0x116e4b460: function object at 0x116ef25c0>, <cell at 0x116e4beb0: str object at 0x116fd0370>)
In the next example, lru_cache stands for Least Recently Used cache, It is a decorator that remembers the results of previous function calls so that repeated calls with the same arguments do not need to repeat the computation.
functools.lru_cache(maxsize=32) is one,
and the bare @functools.lru_cache keeps the default of 128 entries.
@functools.lru_cache(maxsize=32)
def square(n):
time.sleep(0.5)
return n * n
@functools.lru_cache
def cube(n):
return n ** 3
square.cache_info().maxsize, cube.cache_info().maxsize
(32, 128)
square(1)
1
square(1)
1
square(2)
4
start = time.perf_counter()
square(30)
print(time.perf_counter() - start)
9.050010703504086e-05
square.cache_info()
CacheInfo(hits=2, misses=3, maxsize=32, currsize=3)
Note that a decorator or decorator factory does not need to wrap or change the behavior of a function.
tools.py holds a dictionary REGISTRY and a factory register(name), whose decorator stores
the decorated function under name and returns it unchanged:
import tools
print(inspect.getsource(tools.register))
def register(name):
"""Decorator factory: store the decorated factory under `name` in REGISTRY and return it unchanged."""
def deco(factory):
REGISTRY[name] = factory
return factory
return deco
Import time and call time¶
The top-level code of a module runs once, at its first import, and the module is then kept in
sys.modules.
A def and its decorator are top-level code, so a decorator's body runs at
import time, once per decorated function, and its wrapper's body at call time, once per call.
experiments/import_time.py prints in both; the second import prints nothing:
!{sys.executable} -c "import experiments.import_time as m; m.area(2, 3);m.area(2, 3);m.area(2, 3); import experiments.import_time"
decorator: registering area wrapper: calling area wrapper: calling area wrapper: calling area
A registry is a dictionary from names to functions,
for example:
REGISTRY = {
"lorenz": make_lorenz,
"pendulum": make_pendulum,
}
it is filled by a decorator as each function is
defined. Importing models.py fills REGISTRY.
from tools import REGISTRY, make_model
import models
sorted(REGISTRY), REGISTRY is models.REGISTRY
(['lotka_volterra', 'rosenzweig_macarthur'], True)
For instance, this would then allow you to build a right-hand side by name:
make_model("lotka_volterra", **models.CHEMOSTAT)
<function models.make_lotka_volterra.<locals>.rhs(t, y)>
make_model("lotka_volterra", **models.CHEMOSTAT).__qualname__
'make_lotka_volterra.<locals>.rhs'
make_model("lotka_volterra", **models.CHEMOSTAT).__name__
'rhs'
Stacked decorators¶
Stacked decorators apply from the bottom up: @A over @B over def g is g = A(B(g)).
experiments/stack_order.py prints which decorator is applied first:
%run -m experiments.stack_order
B applied to g A applied to g 3 0.0 3 True True
Note that functools.wraps copies attributes from the wrapped function to the wrapper when the decorator is applied. It does not keep them synchronised.
3 Counting and timing¶
Counting calls¶
A count of calls depends on the algorithm and its input only. Under the counter, a recursive function counts its inner calls too, since the recursion goes through the name that the decoration rebound:
@counted
def fib(n):
return n if n < 2 else fib(n - 1) + fib(n - 2)
fib(10), fib.calls
(55, 177)
fib.reset(); fib(20), fib.calls
(6765, 21891)
experiments/cost.py counts the right-hand-side calls of Euler's method and RK4 for five step
sizes, against the error at t = 40 days:
%run -m experiments.cost
method h steps calls error at t=40 euler 0.1 400 400 2.9e+00 euler 0.05 800 800 2.7e+00 euler 0.01 4000 4000 5.5e-01 euler 0.005 8000 8000 2.7e-01 euler 0.001 40000 40000 5.3e-02 rk4 0.4 100 400 9.5e-04 rk4 0.2 200 800 1.3e-04 rk4 0.1 400 1600 1.1e-05 rk4 0.05 800 3200 7.5e-07 rk4 0.01 4000 16000 1.3e-09 RK4: 3200 calls for error < 1e-05 (h = 0.05) Euler: ~2e+08 calls (extrapolated from the slope of the table)
Timers, overhead and profiles¶
time.perf_counter() reads the clock for intervals. It never goes back and its zero means nothing,
so only a difference of two readings counts.
A timing wrapper, timed of tools.py, takes the
two readings around the forwarded call. Its own work, the overhead, is about 100 ns per call.
start = time.perf_counter()
total = sum(range(10**6))
elapsed = time.perf_counter() - start
total, elapsed
(499999500000, 0.02331004198640585)
A profiler records every call of a run with its time.
cProfile reports for each function
ncalls, tottime, the time in its own lines, and cumtime, the time including the functions it calls.
experiments/timing.py profiles one run of integrate with the right-hand side under counted:
%run -m experiments.timing
integrate True
1 call, 0.01945204 s
12130 function calls (12129 primitive calls) in 0.010 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
800 0.005 0.000 0.009 0.000 integrators.py:10(rk4_step)
3200 0.002 0.000 0.003 0.000 models.py:14(rhs)
3203 0.001 0.000 0.001 0.000 {built-in method numpy.array}
3200 0.001 0.000 0.004 0.000 tools.py:8(wrapper)
1 0.000 0.000 0.010 0.010 <string>:1(<module>)
1 0.000 0.000 0.004 0.004 integrators.py:20(integrate)
1601 0.000 0.000 0.000 0.000 {method 'append' of 'list' objects}
2/1 0.000 0.000 0.010 0.010 {built-in method builtins.exec}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
2 0.000 0.000 0.000 0.000 attrsettr.py:66(_get_attr_opt)
2 0.000 0.000 0.000 0.000 attrsettr.py:43(__getattr__)
1 0.000 0.000 0.000 0.000 zmqstream.py:574(_handle_events)
1 0.000 0.000 0.000 0.000 socket.py:785(recv_multipart)
7 0.000 0.000 0.000 0.000 enum.py:720(__call__)
3 0.000 0.000 0.000 0.000 enum.py:1561(__and__)
12 0.000 0.000 0.000 0.000 enum.py:1543(_get_value)
1 0.000 0.000 0.000 0.000 asyncio.py:216(call_at)
1 0.000 0.000 0.000 0.000 events.py:36(__init__)
1 0.000 0.000 0.000 0.000 base_events.py:771(call_at)
24 0.000 0.000 0.000 0.000 {built-in method builtins.isinstance}
1 0.000 0.000 0.000 0.000 zmqstream.py:615(_handle_recv)
1 0.000 0.000 0.000 0.000 asyncio.py:206(_handle_events)
1 0.000 0.000 0.000 0.000 base_events.py:1907(_add_callback)
1 0.000 0.000 0.000 0.000 iostream.py:677(_schedule_in_thread)
1 0.000 0.000 0.000 0.000 events.py:111(__init__)
1 0.000 0.000 0.000 0.000 zmqstream.py:654(_rebuild_io_state)
2 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:1390(_handle_fromlist)
1 0.000 0.000 0.000 0.000 zmqstream.py:547(_run_callback)
1 0.000 0.000 0.000 0.000 base_events.py:747(call_later)
3 0.000 0.000 0.000 0.000 base_events.py:738(time)
1 0.000 0.000 0.000 0.000 queue.py:97(empty)
1 0.000 0.000 0.000 0.000 selector_events.py:750(_process_events)
7 0.000 0.000 0.000 0.000 enum.py:1128(__new__)
1 0.000 0.000 0.000 0.000 events.py:86(_run)
1 0.000 0.000 0.000 0.000 ioloop.py:604(call_later)
1 0.000 0.000 0.000 0.000 iostream.py:233(_handle_event)
2 0.000 0.000 0.000 0.000 {built-in method builtins.getattr}
3 0.000 0.000 0.000 0.000 {built-in method builtins.max}
2 0.000 0.000 0.000 0.000 ioloop.py:549(time)
1 0.000 0.000 0.000 0.000 {built-in method _abc._abc_instancecheck}
1 0.000 0.000 0.000 0.000 {method 'run' of '_contextvars.Context' objects}
1 0.000 0.000 0.000 0.000 abc.py:117(__instancecheck__)
1 0.000 0.000 0.000 0.000 {built-in method _heapq.heappush}
1 0.000 0.000 0.000 0.000 selectors.py:275(_key_from_fd)
1 0.000 0.000 0.000 0.000 {built-in method builtins.round}
1 0.000 0.000 0.000 0.000 enum.py:1550(__or__)
1 0.000 0.000 0.000 0.000 zmqstream.py:677(_update_handler)
1 0.000 0.000 0.000 0.000 zmqstream.py:533(sending)
2 0.000 0.000 0.000 0.000 {built-in method builtins.hasattr}
2 0.000 0.000 0.000 0.000 {method 'upper' of 'str' objects}
3 0.000 0.000 0.000 0.000 {built-in method time.monotonic}
2 0.000 0.000 0.000 0.000 {built-in method time.time}
1 0.000 0.000 0.000 0.000 events.py:127(__lt__)
4 0.000 0.000 0.000 0.000 {built-in method builtins.len}
1 0.000 0.000 0.000 0.000 {built-in method _contextvars.copy_context}
1 0.000 0.000 0.000 0.000 {built-in method builtins.min}
2 0.000 0.000 0.000 0.000 {method 'popleft' of 'collections.deque' objects}
1 0.000 0.000 0.000 0.000 queue.py:209(_qsize)
2 0.000 0.000 0.000 0.000 zmqstream.py:529(receiving)
1 0.000 0.000 0.000 0.000 {method 'append' of 'collections.deque' objects}
1 0.000 0.000 0.000 0.000 {method '__exit__' of '_thread.lock' objects}
1 0.000 0.000 0.000 0.000 base_events.py:543(_check_closed)
1 0.000 0.000 0.000 0.000 base_events.py:2017(get_debug)
cProfile is primarily a profiler, not a precise microbenchmarking tool.
- Profiling: Which functions consume the most execution time?
- Benchmarking: How long does a particular operation take under controlled conditions?
4 Caching¶
Memoisation¶
Memoisation stores the result of a call under its arguments and returns it when the same arguments come again. It is correct when the result depends on the arguments alone.
functools.lru_cache keeps a cache, a dictionary keyed by the arguments, so the arguments must be hashable, with a hash that
never changes.
A call answered from the cache is a hit, one computed and stored a miss.
functools.lru_cache is a Python standard-library implementation of memoisation.
@functools.lru_cache(maxsize=None)
def fib(n):
return n if n < 2 else fib(n - 1) + fib(n - 2)
fib(80), fib.cache_info()
(23416728348467685, CacheInfo(hits=78, misses=81, maxsize=None, currsize=81))
hash(3), hash("hello"),hash("helLo")
(3, 5346668813046773645, -4140080607657821313)
# distinction between hash and is
a = (1, 2)
b = (1, 2)
print(a == b) # True
print(hash(a) == hash(b)) # True
print(a is b)
True True False
fib([80])
--------------------------------------------------------------------------- TypeError Traceback (most recent call last) Cell In[273], line 1 ----> 1 fib([80]) TypeError: unhashable type: 'list'
Lists are mutable. Their contents can change, so they are not hashable.
Cached objects and aliasing¶
functools.lru_cache stores references to Python objects, not copies of those objects.
If a cached result is mutable (for example, a list or a NumPy array), modifying it will affect what subsequent calls receive.
Example with a cached list:
import functools
@functools.lru_cache(maxsize=32)
def make_list(n):
return [0] * n
a = make_list(3)
print(a)
[0, 0, 0]
The first call creates a list object and stores it in the cache.
Next:
b = make_list(3)
print(a is b)
# True
True
What happens if we mutate the cached result?
a[0] = 99
print(a)
# [99, 0, 0]
print(b)
# [99, 0, 0]
[99, 0, 0] [99, 0, 0]
Because a and b are aliases, the mutation is visible through both names.
c = make_list(3)
print(c)
# [99, 0, 0]
print(c is a)
# True
[99, 0, 0] True
Eventhough the function says return [0]*n
See another example here:
@functools.lru_cache
def digits(n):
return [int(c) for c in str(n)]
a = digits(2026)
a.append(0)
b = digits(2026)
b, b is a, digits.cache_info().hits
([2, 0, 2, 6, 0], True, 1)
experiments/fib_cache.py caches a loader of the chemostat runs by path (the expensive operation is often not necessarily computation: it can be reading and parsing data from disk).
Two calls return one array, and a write through the first result changes the second:
%run -m experiments.fib_cache
23416728348467685 CacheInfo(hits=78, misses=81, maxsize=None, currsize=81) fib_plain(28): 0.02 s TypeError: unhashable type: 'numpy.ndarray' True CacheInfo(hits=1, misses=1, maxsize=128, currsize=1) -1.0 1.92