Lecture 12 — From scripts to an installable package¶
A package can be installed into an environment. Python then imports the installed package from any folder.
Notes: Lecture 12 ·
code: attractor at lecture-12-start
The notebook runs at the root of the attractor clone, at the tag lecture-12-start, with the kernel
Python (attractor 3.13). setup_env.sh now also installs the project as an editable install:
cd ~/GitHub/attractor
git fetch --tags --force
git switch --detach lecture-12-start
source setup_env.sh
The cells that build wheels work in a temporary clone. The wheel cells never change the clone of the notebook.
import importlib, importlib.util, json, math, os, shutil, subprocess, sys, tempfile, zipfile
from pathlib import Path
TAG = "lecture-12-start"
REPO = "https://github.com/borisbolliet/attractor.git"
if not Path("src/attractor").exists(): # outside a clone (Colab, a download): fetch and install the project
root = Path(tempfile.mkdtemp()) / "attractor"
subprocess.run(["git", "-c", "advice.detachedHead=false", "clone", "-q", "--branch", TAG, REPO, str(root)],
check=True)
os.chdir(root)
if importlib.util.find_spec("attractor") is None: # e.g. Colab: install the project into this kernel
subprocess.run([sys.executable, "-m", "pip", "install", "-q", "-e", ".[fit]"], check=True)
sys.path.insert(0, str(root / "src"))
BIN = Path(sys.executable).parent # the bin/ folder of the kernel's environment
def sh(cmd, cwd=None, tail=None):
"""Run a shell command, print its output (the last `tail` lines) and its exit status if not 0."""
env = {**os.environ, "PIP_DISABLE_PIP_VERSION_CHECK": "1", "PYTHON_COLORS": "0"}
env["PATH"] = f"{BIN}{os.pathsep}{env['PATH']}"
r = subprocess.run(cmd, shell=True, cwd=cwd, capture_output=True, text=True, env=env)
lines = (r.stdout + r.stderr).rstrip().splitlines()
print("\n".join(lines[-tail:] if tail else lines))
if r.returncode:
print(f"(exit status {r.returncode})")
subprocess.run(["git", "describe", "--tags"], capture_output=True, text=True).stdout.strip()
'lecture-12-start'
1 The import system¶
The import search path¶
The import search path is a list of folders, sys.path. The statement import name uses sys.path
in three steps.
- Python searches the folders of
sys.pathin order. - Python imports the first module or package called
namethat Python finds. - If no folder holds a module or package called
name, Python raisesModuleNotFoundError.
For python script.py, the first entry is the folder of the script. For python -c, python -m
and the prompt, the first entry is the current folder. Then come the folders of PYTHONPATH. Then
comes the standard library. Last comes site-packages.
for entry in sys.path:
print(repr(entry))
'/opt/homebrew/Cellar/python@3.13/3.13.11/Frameworks/Python.framework/Versions/3.13/lib/python313.zip' '/opt/homebrew/Cellar/python@3.13/3.13.11/Frameworks/Python.framework/Versions/3.13/lib/python3.13' '/opt/homebrew/Cellar/python@3.13/3.13.11/Frameworks/Python.framework/Versions/3.13/lib/python3.13/lib-dynload' '' '/private/var/folders/hc/d14d998d3rv6_7vdcjskjrxc0000gp/T/tmp.fPX0zXIu3w/attractor/.venv/lib/python3.13/site-packages' '/private/var/folders/hc/d14d998d3rv6_7vdcjskjrxc0000gp/T/tmp.fPX0zXIu3w/attractor/src'
sh(f'{sys.executable} -c "import sys; print(repr(sys.path[0]))"')
''
For python -c, the first entry is ''. The empty string stands for the current folder.
Python finds a file of the current folder before the standard library. In the next cell, a new
folder holds a file random.py:
folder = Path(tempfile.mkdtemp())
(folder / "random.py").write_text('print("random.py of this folder")')
sh(f'{sys.executable} -c "import random"', cwd=folder)
sh(f'{sys.executable} -c "import random"')
random.py of this folder
In the new folder, the file random.py shadows the module random of the standard library. A
program that uses the standard module random then fails. Python raises an error for a missing
attribute:
sh(f'{sys.executable} -c "import random; random.randint(1, 6)"', cwd=folder, tail=1)
AttributeError: module 'random' has no attribute 'randint' (consider renaming '/private/var/folders/hc/d14d998d3rv6_7vdcjskjrxc0000gp/T/tmppknfqoa5/random.py' since it has the same name as the standard library module named 'random' and prevents importing that standard library module) (exit status 1)
importlib.util.find_spec("no_such_module") is None
True
For python script.py, the folder of the script is first on sys.path. So the script finds only the
modules of its own folder:
(folder / "scripts").mkdir()
(folder / "scripts" / "where.py").write_text("import sys; print(sys.path[0])")
sh(f"{sys.executable} scripts/where.py", cwd=folder)
/private/var/folders/hc/d14d998d3rv6_7vdcjskjrxc0000gp/T/tmppknfqoa5/scripts
In data analysis, a notebook may run in a directory containing a local copy of a module, such as fit.py. If that directory appears before site-packages in sys.path, Python imports the local copy instead of the installed version. The analysis then uses the local code, which may differ from the intended version, without raising an error.
Packages and relative imports¶
A package is a module that contains other modules, its submodules. A regular package is a
folder that holds a file __init__.py. Importing the package runs the file __init__.py. The package
stores its folder in __path__. Python searches for the submodules of the package in the folder of __path__.
An absolute import names a module by its full name, as in from pkg.mod import f. A relative
import names a module from the package of the importing module, as in from .mod import f.
hasattr(json, "__path__"), hasattr(math, "__path__")
(True, False)
import json.decoder
json.decoder.__name__, json.decoder.__package__
('json.decoder', 'json')
importlib.util.resolve_name(".decoder", "json"), importlib.util.resolve_name("..mod", "pkg.sub")
('json.decoder', 'pkg.mod')
The modules of the project lie in src/attractor/. The modules
import each other by relative imports, such as from .trajectory import Trajectory in
integrators.py:
print(Path("src/attractor/integrators.py").read_text().splitlines()[4])
from .trajectory import Trajectory
A module of a package run as a file belongs to no package. Python raises ImportError at the first
relative import of the module:
sh(f"{sys.executable} src/attractor/cli.py", tail=1)
ImportError: attempted relative import with no known parent package (exit status 1)
The src layout¶
In the flat layout, the package folder lies in the root of the project. In the src layout, the
package folder lies in src/. Commands run from the root of the project. The root is then the first
entry of sys.path. With the src layout, the root holds no package folder. As a result, import attractor imports the installed package. In this notebook, the editable install makes Python find
the package in src/:
import attractor
attractor.__file__
'/private/var/folders/hc/d14d998d3rv6_7vdcjskjrxc0000gp/T/tmp.fPX0zXIu3w/attractor/src/attractor/__init__.py'
sh(f'{sys.executable} -c "import attractor; print(attractor.__version__)"', cwd="/")
1.0.0
2 Project metadata and installation¶
Distributions and the project file¶
A distribution is what pip installs. A distribution holds import packages and their metadata. The metadata give the name, the version, the interpreters accepted and the distributions required. The name of a distribution and the name imported may differ.
import importlib.metadata as md
md.packages_distributions()["dotenv"]
['python-dotenv']
meta = md.metadata("python-dotenv")
meta["Version"], meta["Requires-Python"], md.requires("python-dotenv")
('1.2.4', '>=3.10', ['click>=5.0; extra == "cli"'])
A project declares its metadata in pyproject.toml. [build-system] names
the build backend. [project] holds the metadata. [tool.name] holds the settings of one tool.
In TOML, [project.scripts] is the table scripts inside the table project. The module tomllib
reads the file into nested dictionaries:
import tomllib
doc = tomllib.loads(Path("pyproject.toml").read_text())
list(doc)
['build-system', 'project', 'tool']
doc["build-system"]
{'requires': ['setuptools>=77'], 'build-backend': 'setuptools.build_meta'}
doc["project"]["dependencies"], doc["project"]["optional-dependencies"], doc["project"]["scripts"]
(['numpy>=1.26'],
{'fit': ['scipy>=1.11'], 'plot': ['matplotlib>=3.8']},
{'attractor-simulate': 'attractor.cli:simulate',
'attractor-periods': 'attractor.cli:periods',
'attractor-fit': 'attractor.cli:fit'})
doc["tool"]
{'setuptools': {'packages': {'find': {'where': ['src']}}}}
Version specifiers and optional dependencies¶
Two versions are compared component by component, as integers. A missing component counts as 0. A
version specifier is a list of clauses separated by commas. A version satisfies the specifier
when the version satisfies all the clauses. ~=V accepts V and the later versions that agree with V
in every component but the last.
from packaging.specifiers import SpecifierSet
from packaging.version import Version
Version("1.10.1") > Version("1.9.3"), Version("1.15") == Version("1.15.0")
(True, True)
releases = ["1.9.3", "1.10.1", "1.11.4", "1.15.3", "1.18.1"]
specs = ["~=1.11.2", "~=1.11", "==1.15", "==1.15.*", "!=1.11.*", ">=1.10,<1.15", "<=1.15"]
for spec in specs:
print(f"{spec:<13}", list(SpecifierSet(spec).filter(releases)))
~=1.11.2 ['1.11.4'] ~=1.11 ['1.11.4', '1.15.3', '1.18.1'] ==1.15 [] ==1.15.* ['1.15.3'] !=1.11.* ['1.9.3', '1.10.1', '1.15.3', '1.18.1'] >=1.10,<1.15 ['1.10.1', '1.11.4'] <=1.15 ['1.9.3', '1.10.1', '1.11.4']
An optional dependency is a requirement of one feature only. The table
[project.optional-dependencies] groups optional dependencies under names, called extras. In the
metadata, an optional requirement names its extra:
md.requires("attractor")
['numpy>=1.26', 'scipy>=1.11; extra == "fit"', 'matplotlib>=3.8; extra == "plot"']
Only attractor-fit uses SciPy. Only the option --plot uses Matplotlib. So
cli.py imports each of the two libraries inside the
function that uses the library. As a result, import attractor loads neither SciPy nor Matplotlib:
"scipy" in sys.modules, "matplotlib" in sys.modules
(False, False)
Editable installs and lock files¶
An editable install, pip install -e ., works in three steps.
- pip writes the metadata of the project into
site-packages. - pip writes into
site-packagesa file ending in.pth, which names the foldersrc/of the project. - At each start-up of Python, the module
siteappends the foldersrc/tosys.path.
import sysconfig
site_packages = Path(sysconfig.get_paths()["purelib"])
for pth in sorted(site_packages.glob("__editable__*attractor*.pth")):
print(pth.name, "->", pth.read_text().strip())
__editable__.attractor-1.0.0.pth -> /private/var/folders/hc/d14d998d3rv6_7vdcjskjrxc0000gp/T/tmp.fPX0zXIu3w/attractor/src
md.version("attractor"), attractor.__version__
('1.0.0', '1.0.0')
setup_env.sh installs the lock first. setup_env.sh then installs the project without
dependencies. The command pip check then reports any unsatisfied requirement. Without --no-deps,
pip would replace a locked version that does not satisfy a requirement:
sh(f"{sys.executable} -m pip check")
No broken requirements found.
sh(f'{sys.executable} -m pip install --dry-run "numpy<2.5"', tail=1)
Would install numpy-2.4.6
A published result depends on the code, the data and the software environment that ran them. Reproducing the result later requires the same environment. A lock file records this environment.
3 Console commands and the public interface¶
Console commands¶
An entry point is a named reference to an object of a package. An entry point is written
module:attribute. Each line of [project.scripts] defines a console command.
- At install time, pip writes a launcher into the folder
bin/of the environment. - When the command runs, the launcher imports the module and calls the function without arguments.
- The launcher passes the return value to
sys.exit.Nonegives the exit status 0.
(ep,) = md.entry_points(group="console_scripts", name="dotenv")
ep.value, ep.module, ep.attr
('dotenv.__main__:cli', 'dotenv.__main__', 'cli')
sorted(ep.name for ep in md.entry_points(group="console_scripts") if ep.module == "attractor.cli")
['attractor-fit', 'attractor-periods', 'attractor-simulate']
print((BIN / "attractor-simulate").read_text())
#!/private/var/folders/hc/d14d998d3rv6_7vdcjskjrxc0000gp/T/tmp.fPX0zXIu3w/attractor/.venv/bin/python
import sys
from attractor.cli import simulate
if __name__ == '__main__':
if sys.argv[0].endswith('.exe'):
sys.argv[0] = sys.argv[0][:-4]
sys.exit(simulate())
The first line of the launcher names the interpreter of the environment by its absolute path. The package is installed in that environment. So the command runs from any folder. The next cell runs the command from the root of the file system:
sh("attractor-simulate", cwd="/")
801 points, step 0.05, algae period 8.16 days
Each function reads its options from sys.argv, or from a list. As a result, each function can
also be called from Python:
from attractor.cli import simulate
simulate(["--h", "0.1"])
401 points, step 0.1, algae period 8.18 days
The public interface¶
The public interface of a package is the set of names that other code may import from the
package and rely on. A package lists its public names in __all__. __init__.py imports the public
names from the submodules. Names that start with an underscore are private by convention.
json.__all__, "codecs" in vars(json)
(['dump', 'dumps', 'load', 'loads', 'JSONDecoder', 'JSONDecodeError', 'JSONEncoder'], True)
names = {}
exec("from json import *", names)
sorted(name for name in names if name != "__builtins__")
['JSONDecodeError', 'JSONDecoder', 'JSONEncoder', 'dump', 'dumps', 'load', 'loads']
The interface of attractor, in __init__.py:
attractor.__all__
['DynamicalSystem', 'LotkaVolterra', 'RosenzweigMacArthur', 'Goodwin', 'CHEMOSTAT', 'Euler', 'RK4', 'integrate', 'Trajectory', 'peaks', 'cycle_period', 'load_run', 'lynx_hare']
attractor.RK4 is attractor.integrators.RK4, "rk4_step" in attractor.__all__
(True, False)
Semantic versions¶
A semantic version MAJOR.MINOR.PATCH states what a release does to the public interface. A release
that breaks code written for the previous one increments MAJOR. A release that adds to the interface
and breaks nothing increments MINOR. A release that only fixes faults increments PATCH. NumPy 2.0.0
removed the alias np.NaN:
import numpy as np
np.__version__
'2.5.4'
np.NaN
--------------------------------------------------------------------------- AttributeError Traceback (most recent call last) Cell In[39], line 1 ----> 1 np.NaN File /private/var/folders/hc/d14d998d3rv6_7vdcjskjrxc0000gp/T/tmp.fPX0zXIu3w/attractor/.venv/lib/python3.13/site-packages/numpy/__init__.py:763, in __getattr__(attr) 760 raise AttributeError(__former_attrs__[attr], name=None) 762 if attr in __expired_attributes__: --> 763 raise AttributeError( 764 f"`np.{attr}` was removed in the NumPy 2.0 release. " 765 f"{__expired_attributes__[attr]}", 766 name=None 767 ) 769 raise AttributeError(f"module {__name__!r} has no attribute {attr!r}") AttributeError: `np.NaN` was removed in the NumPy 2.0 release. Use `np.nan` instead.
"1.26.4" in SpecifierSet("~=1.26"), "2.0.0" in SpecifierSet("~=1.26")
(True, False)
4 Wheels¶
The contents of a wheel¶
A wheel is a zip archive that pip unpacks into site-packages. A wheel holds three kinds of files.
- The import packages.
- The declared package data.
- A folder
name-version.dist-info, with the filesMETADATA,WHEEL,RECORDandentry_points.txt.
A wheel does not hold tests, documentation or scripts outside the package.
dist = md.distribution("python-dotenv")
print(dist.read_text("WHEEL").strip())
print(dist.read_text("entry_points.txt").strip())
Wheel-Version: 1.0 Generator: setuptools (84.0.0) Root-Is-Purelib: true Tag: py3-none-any [console_scripts] dotenv = dotenv.__main__:cli
The next cell builds the wheel of the project in a temporary clone at lecture-12-start. The
version is 1.0.0. pip first installs setuptools into a temporary environment. As a result, the cell
needs the network.
work = Path(tempfile.mkdtemp())
sh(f"git clone -q {REPO} attractor && cd attractor && git -c advice.detachedHead=false switch -q --detach {TAG}",
cwd=work)
clone = work / "attractor"
sh(f"{sys.executable} -m pip wheel . --no-deps -w dist -q", cwd=clone)
wheel100 = clone / "dist/attractor-1.0.0-py3-none-any.whl"
print("\n".join(zipfile.ZipFile(wheel100).namelist()))
attractor/__init__.py attractor/cli.py attractor/cycles.py attractor/datasets.py attractor/integrators.py attractor/models.py attractor/tools.py attractor/trajectory.py attractor-1.0.0.dist-info/licenses/LICENSE attractor-1.0.0.dist-info/METADATA attractor-1.0.0.dist-info/WHEEL attractor-1.0.0.dist-info/entry_points.txt attractor-1.0.0.dist-info/top_level.txt attractor-1.0.0.dist-info/RECORD
print("\n".join(zipfile.ZipFile(wheel100).read("attractor-1.0.0.dist-info/METADATA").decode().splitlines()[:12]))
Metadata-Version: 2.4 Name: attractor Version: 1.0.0 Summary: Predator-prey models, fixed-step integrators and cycle periods License-Expression: MIT Requires-Python: >=3.12 Description-Content-Type: text/markdown License-File: LICENSE Requires-Dist: numpy>=1.26 Provides-Extra: fit Requires-Dist: scipy>=1.11; extra == "fit" Provides-Extra: plot
Package data¶
Package data are the files of a package that are not modules. setuptools puts a data file into the
wheel only if the data file is declared in [tool.setuptools.package-data]. The code of a package
reads a data file through importlib.resources.files(package). The object returned by
files(package) represents the folder of the installed package:
from importlib.resources import files
rc = files("matplotlib").joinpath("mpl-data/matplotlibrc")
rc.is_file(), rc.read_text().splitlines()[0]
(True, '#### MATPLOTLIBRC FORMAT')
files("matplotlib").joinpath("mpl-data/no_such_file").is_file()
False
lynx_hare of datasets.py reads a table of the
package through files("attractor"). With the editable install, Python finds the file in src/:
year, lynx, hare = attractor.lynx_hare()
len(year), int(year[0]), float(hare.max())
(21, 1900, 77.4)
The wheel of 1.0.0 lacks the table. The next cell installs the wheel and its extra fit into a new
environment. In the new environment, attractor-fit fails on the missing file. pip downloads NumPy
and SciPy unless the cache of pip holds NumPy and SciPy.
fresh = work / "fresh"
sh(f"{sys.executable} -m venv {fresh}")
sh(f'{fresh}/bin/python -m pip install -q "{wheel100}[fit]"')
sh(f"{fresh}/bin/attractor-fit", tail=1)
FileNotFoundError: [Errno 2] No such file or directory: '/private/var/folders/hc/d14d998d3rv6_7vdcjskjrxc0000gp/T/tmpyw4kfo_w/fresh/lib/python3.13/site-packages/attractor/data/lynx_hare.csv' (exit status 1)
At lecture-12, two lines declare the table. The version becomes 1.0.1
(pyproject.toml):
sh("git -c advice.detachedHead=false switch -q --detach lecture-12 && git diff lecture-12-start lecture-12 -- pyproject.toml",
cwd=clone)
diff --git a/pyproject.toml b/pyproject.toml index 0e456c5..061d1f9 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "attractor" -version = "1.0.0" +version = "1.0.1" description = "Predator-prey models, fixed-step integrators and cycle periods" readme = "README.md" license = "MIT" @@ -22,3 +22,6 @@ attractor-fit = "attractor.cli:fit" [tool.setuptools.packages.find] where = ["src"] + +[tool.setuptools.package-data] +attractor = ["data/*.csv"]
sh(f"{sys.executable} -m pip wheel . --no-deps -w dist -q", cwd=clone)
wheel101 = clone / "dist/attractor-1.0.1-py3-none-any.whl"
[name for name in zipfile.ZipFile(wheel101).namelist() if name.endswith(".csv")]
['attractor/data/lynx_hare.csv']
sh(f'{fresh}/bin/python -m pip install -q "{wheel101}[fit]"')
sh(f"{fresh}/bin/attractor-fit")
LotkaVolterra(a=0.481, b=0.0248, c=0.0275, d=0.926) H0 = 34.91, L0 = 3.86, rms residual 3.76 thousand pelts
The editable install of this notebook still reports 1.0.0. pip wrote the metadata of the editable
install at install time, at lecture-12-start.
Compatibility tags¶
The last three fields of a wheel's name are its compatibility tags. The three tags are the Python tag, the ABI tag and the platform tag. The interpreter accepts an ordered list of tags. pip installs a wheel only if one of its tags is in the list. Among several compatible wheels, pip prefers the wheel whose tag comes first.
print(md.distribution("numpy").read_text("WHEEL"))
Wheel-Version: 1.0 Generator: meson Root-Is-Purelib: false Tag: cp313-cp313-macosx_14_0_arm64 Generator: delocate 0.13.0
from packaging.tags import sys_tags
accepted = [str(t) for t in sys_tags()]
len(accepted), accepted[:3]
(683, ['cp313-cp313-macosx_15_0_arm64', 'cp313-cp313-macosx_15_0_universal2', 'cp313-cp313-macosx_14_0_arm64'])
The tag of a NumPy wheel compiled for CPython 3.12 is not in the list. The tag of the pure Python wheel of the project is in the list:
"cp312-cp312-macosx_14_0_arm64" in accepted, "py3-none-any" in accepted
(False, True)
Scientific software is useful to others only if it installs on their machines. The tags of a wheel decide which machines can install the package.
The field requires-python decides which interpreters may install the project. For example,
Python 3.10.18 does not satisfy the specifier of requires-python:
spec = SpecifierSet(doc["project"]["requires-python"])
spec, "3.13.11" in spec, "3.10.18" in spec
(<SpecifierSet('>=3.12')>, True, False)
The exercises and their solutions are at the end of the notes. The scripts of the exercises are in
exercises/lecture12/.