Lecture 12 — From scripts to an installable package¶

A Python package is a collection of Python modules and possibly subpackages, organised under a common name. It allows related code to be structured and imported using that name.

A package is typically a directory containing an init.py file.

A package can be installed into a Python environment, making it importable from any working directory using that environment.

Python then imports the installed package from any folder.

Notes: Lecture 12 · code: attractor at lecture-12-start

The notebook runs at the root of the attractor clone, at the tag lecture-12-start, with the kernel Python (attractor 3.13). setup_env.sh now also installs the project as an editable install:

cd ~/GitHub/attractor
git fetch --tags --force
git switch --detach lecture-12-start
source setup_env.sh

The cells that build wheels work in a temporary clone. The wheel cells never change the clone of the notebook.

In [157]:

import importlib, importlib.util, json, math, os, shutil, subprocess, sys, tempfile, zipfile
from pathlib import Path

TAG = "lecture-12-start"
REPO = "https://github.com/borisbolliet/attractor.git"
if not Path("src/attractor").exists():   # outside a clone (Colab, a download): fetch and install the project
    root = Path(tempfile.mkdtemp()) / "attractor"
    subprocess.run(["git", "-c", "advice.detachedHead=false", "clone", "-q", "--branch", TAG, REPO, str(root)],
                   check=True)
    os.chdir(root)
    subprocess.run([sys.executable, "-m", "pip", "install", "-q", "-e", ".[fit]"], check=True)
    sys.path.insert(0, str(root / "src"))
BIN = Path(sys.executable).parent        # the bin/ folder of the kernel's environment

def sh(cmd, cwd=None, tail=None):
    """Run a shell command, print its output (the last `tail` lines) and its exit status if not 0."""
    env = {**os.environ, "PIP_DISABLE_PIP_VERSION_CHECK": "1", "PYTHON_COLORS": "0"}
    env["PATH"] = f"{BIN}{os.pathsep}{env['PATH']}"
    r = subprocess.run(cmd, shell=True, cwd=cwd, capture_output=True, text=True, env=env)
    lines = (r.stdout + r.stderr).rstrip().splitlines()
    print("\n".join(lines[-tail:] if tail else lines))
    if r.returncode:
        print(f"(exit status {r.returncode})")

subprocess.run(["git", "describe", "--tags"], capture_output=True, text=True).stdout.strip()

import importlib, importlib.util, json, math, os, shutil, subprocess, sys, tempfile, zipfile from pathlib import Path TAG = "lecture-12-start" REPO = "https://github.com/borisbolliet/attractor.git" if not Path("src/attractor").exists(): # outside a clone (Colab, a download): fetch and install the project root = Path(tempfile.mkdtemp()) / "attractor" subprocess.run(["git", "-c", "advice.detachedHead=false", "clone", "-q", "--branch", TAG, REPO, str(root)], check=True) os.chdir(root) subprocess.run([sys.executable, "-m", "pip", "install", "-q", "-e", ".[fit]"], check=True) sys.path.insert(0, str(root / "src")) BIN = Path(sys.executable).parent # the bin/ folder of the kernel's environment def sh(cmd, cwd=None, tail=None): """Run a shell command, print its output (the last `tail` lines) and its exit status if not 0.""" env = {**os.environ, "PIP_DISABLE_PIP_VERSION_CHECK": "1", "PYTHON_COLORS": "0"} env["PATH"] = f"{BIN}{os.pathsep}{env['PATH']}" r = subprocess.run(cmd, shell=True, cwd=cwd, capture_output=True, text=True, env=env) lines = (r.stdout + r.stderr).rstrip().splitlines() print("\n".join(lines[-tail:] if tail else lines)) if r.returncode: print(f"(exit status {r.returncode})") subprocess.run(["git", "describe", "--tags"], capture_output=True, text=True).stdout.strip()

Out[157]:

'lecture-12-start'

1 The import system¶

The import search path¶

The import search path is a list of folders, sys.path. The statement import name uses sys.path in three steps.

1. Python searches the folders of sys.path in order.
2. Python imports the first module or package called name that Python finds.
3. If no folder holds a module or package called name, Python raises ModuleNotFoundError.

For python path/to/script.py, the first entry is the folder of the script.

For python -c, python -m and the prompt, the first entry is the current folder. Then come the folders of PYTHONPATH.

Then comes the standard library. Last comes site-packages.

In [72]:

sys.executable

sys.executable

Out[72]:

'/Users/boris/GitHub/attractor/.venv/bin/python'

In [73]:

for entry in sys.path:
    print(repr(entry))

for entry in sys.path: print(repr(entry))

'/opt/homebrew/Cellar/python@3.13/3.13.11/Frameworks/Python.framework/Versions/3.13/lib/python313.zip'
'/opt/homebrew/Cellar/python@3.13/3.13.11/Frameworks/Python.framework/Versions/3.13/lib/python3.13'
'/opt/homebrew/Cellar/python@3.13/3.13.11/Frameworks/Python.framework/Versions/3.13/lib/python3.13/lib-dynload'
''
'/Users/boris/GitHub/attractor/.venv/lib/python3.13/site-packages'
'/Users/boris/GitHub/attractor/src'

Entry in sys.path	Purpose
.../lib/python313.zip	Optional ZIP archive containing Python standard-library modules (often absent).
.../lib/python3.13	Python's standard library (os, pathlib, json, etc.).
.../lib/python3.13/lib-dynload	Compiled standard-library extension modules (.so files).
''	Current working directory.
.../.venv/lib/python3.13/site-packages	Third-party packages installed in the virtual environment.
.../attractor/src	Project source directory, allowing direct imports of local packages (often via an editable installation).

In [74]:

sh(f'{sys.executable} -c "import sys; print(repr(sys.path[0]))"')

sh(f'{sys.executable} -c "import sys; print(repr(sys.path[0]))"')

''

For python -c, the first entry is ''. The empty string stands for the current folder.

Python finds a file of the current folder before the standard library. In the next cell, a new folder holds a file random.py:

In [76]:

folder = Path(tempfile.mkdtemp())
(folder / "random.py").write_text('print("random.py of this folder")')
sh(f'{sys.executable} -c "import random"', cwd=folder)

folder = Path(tempfile.mkdtemp()) (folder / "random.py").write_text('print("random.py of this folder")') sh(f'{sys.executable} -c "import random"', cwd=folder)

random.py of this folder

In [77]:

sh(f'{sys.executable} -c "import random"')

sh(f'{sys.executable} -c "import random"')

In the new folder, the file random.py shadows the module random of the standard library.

A program that uses the standard module random then fails.

Python raises an error for a missing attribute:

In [78]:

sh(f'{sys.executable} -c "import random; random.randint(1, 6)"', cwd=folder, tail=1)

sh(f'{sys.executable} -c "import random; random.randint(1, 6)"', cwd=folder, tail=1)

AttributeError: module 'random' has no attribute 'randint' (consider renaming '/private/var/folders/hc/d14d998d3rv6_7vdcjskjrxc0000gp/T/tmphnhmtp0e/random.py' since it has the same name as the standard library module named 'random' and prevents importing that standard library module)
(exit status 1)

In [79]:

importlib.util.find_spec("no_such_module") is None

importlib.util.find_spec("no_such_module") is None

Out[79]:

True

For python script.py, the folder of the script is first on sys.path. So the script finds only the modules of its own folder:

In [18]:

# dir(folder)

# dir(folder)

In [80]:

(folder / "scripts").mkdir()
(folder / "scripts" / "where.py").write_text("import sys; print(sys.path[0])")
sh(f"{sys.executable} scripts/where.py", cwd=folder)

(folder / "scripts").mkdir() (folder / "scripts" / "where.py").write_text("import sys; print(sys.path[0])") sh(f"{sys.executable} scripts/where.py", cwd=folder)

/private/var/folders/hc/d14d998d3rv6_7vdcjskjrxc0000gp/T/tmphnhmtp0e/scripts

In data analysis, a notebook may run in a directory containing a local copy of a module, such as simulate.py. If that directory appears before site-packages in sys.path, Python imports the local copy instead of the installed version. The analysis then uses the local code, which may differ from the intended version, without raising an error.

In [84]:

import simulate

import simulate

In [85]:

H

H

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[85], line 1
----> 1 H

NameError: name 'H' is not defined

In [86]:

from simulate import H

from simulate import H

In [87]:

H

H

Out[87]:

0.05

Packages and relative imports¶

A package is a special kind of module that can contain other modules, called its submodules, and other packages, called its subpackages.

A regular package is a directory containing an __init__.py file.

Importing the package executes its __init__.py file. The package stores its search locations in __path__, typically a list containing its directory.

Python searches for the package's submodules in the locations listed in __path__.

An absolute import names a module by its full name, as in from pkg.mod import f.

A relative import specifies a module relative to the package of the importing module. One leading dot (.) refers to the current package, two dots (..) to its parent package, and three dots (...) to its grandparent package.

For example, from .mod import f imports from the current package, while from ..utils import f imports from a sibling module of the parent package.

In [20]:

hasattr(json, "__path__"), hasattr(math, "__path__")

hasattr(json, "__path__"), hasattr(math, "__path__")

Out[20]:

(True, False)

In [88]:

json.__path__

json.__path__

Out[88]:

['/opt/homebrew/Cellar/python@3.13/3.13.11/Frameworks/Python.framework/Versions/3.13/lib/python3.13/json']

In [21]:

import json.decoder
json.decoder.__name__, json.decoder.__package__

import json.decoder json.decoder.__name__, json.decoder.__package__

Out[21]:

('json.decoder', 'json')

In [22]:

importlib.util.resolve_name(".decoder", "json"), importlib.util.resolve_name("..mod", "pkg.sub")

importlib.util.resolve_name(".decoder", "json"), importlib.util.resolve_name("..mod", "pkg.sub")

Out[22]:

('json.decoder', 'pkg.mod')

The modules of the project lie in src/attractor/. The modules import each other by relative imports, such as from .trajectory import Trajectory in integrators.py:

In [23]:

print(Path("src/attractor/integrators.py").read_text().splitlines()[4])

print(Path("src/attractor/integrators.py").read_text().splitlines()[4])

from .trajectory import Trajectory

A module of a package run as a file belongs to no package. Python raises ImportError at the first relative import of the module:

In [89]:

sh(f"{sys.executable} src/attractor/cli.py", tail=3)

sh(f"{sys.executable} src/attractor/cli.py", tail=3)

  File "/Users/boris/GitHub/attractor/src/attractor/cli.py", line 10, in <module>
    from .cycles import cycle_period
ImportError: attempted relative import with no known parent package
(exit status 1)

Example

Consider this package structure:

project/
└── pkg/
    ├── __init__.py
    ├── utils.py
    └── analysis/
        ├── __init__.py
        ├── fitting.py
        └── models/
            ├── __init__.py
            ├── other.py
            └── linear.py

Inside linear.py (module pkg.analysis.models.linear):

from . import other       # from pkg.analysis.models
from .. import fitting    # from pkg.analysis
from ... import utils     # from pkg

The src layout¶

Flat layout vs src layout¶

In the flat layout, the package directory lies directly in the project root:

attractor/
├── pyproject.toml
├── attractor/
│   ├── __init__.py
│   ├── models.py
│   └── utils.py
└── tests/
    └── test_models.py

In the src layout, the package directory lies inside src/:

attractor/
├── pyproject.toml
├── src/
│   └── attractor/
│       ├── __init__.py
│       ├── models.py
│       └── utils.py
└── tests/
    └── test_models.py

Suppose commands are run from the project root. This directory is normally the first entry of sys.path.

• Flat layout: Python can import attractor directly from the project root, even if the package has not been installed.
• Src layout: The project root does not directly contain an attractor package. Python must find it through another search path, typically provided by an installation.

As a result, with the src layout, import attractor normally requires the package to be installed (or src/ to be added explicitly to sys.path).

In this notebook, the editable installation (pip install -e .) makes Python find the package in src/. Changes to the source code are therefore reflected without reinstalling the package.

In [92]:

import attractor
attractor.__file__

import attractor attractor.__file__

Out[92]:

'/Users/boris/GitHub/attractor/src/attractor/__init__.py'

In [93]:

sh(f'{sys.executable} -c "import attractor; print(attractor.__version__)"', cwd="/")

sh(f'{sys.executable} -c "import attractor; print(attractor.__version__)"', cwd="/")

1.0.0

2 Project metadata and installation¶

Distributions and the project file¶

A distribution package (or distribution) is what pip installs.

A distribution contains Python modules and/or import packages, together with their metadata.

The metadata specify the distribution's name, version, supported Python versions, and required dependencies.

The name of a distribution and the name used to import its modules or packages may differ.

Example:

pip install scikit-learn

import sklearn

In [95]:

import importlib.metadata as md
md.packages_distributions()["dotenv"]

import importlib.metadata as md md.packages_distributions()["dotenv"]

Out[95]:

['python-dotenv']

In [96]:

meta = md.metadata("python-dotenv")
meta["Version"], meta["Requires-Python"], md.requires("python-dotenv")

meta = md.metadata("python-dotenv") meta["Version"], meta["Requires-Python"], md.requires("python-dotenv")

Out[96]:

('1.2.4', '>=3.10', ['click>=5.0; extra == "cli"'])

A project declares its metadata in pyproject.toml.

[build-system] names the build backend.

[project] holds the metadata.

[tool.name] holds the settings of one tool.

In TOML, [project.scripts] is the table scripts inside the table project.

The module tomllib reads the file into nested dictionaries:

In [101]:

import tomllib
doc = tomllib.loads(Path("pyproject.toml").read_text())
list(doc)

import tomllib doc = tomllib.loads(Path("pyproject.toml").read_text()) list(doc)

Out[101]:

['build-system', 'project', 'tool']

In [102]:

doc["build-system"]

doc["build-system"]

Out[102]:

{'requires': ['setuptools>=77'], 'build-backend': 'setuptools.build_meta'}

In [103]:

doc["project"]["dependencies"], doc["project"]["optional-dependencies"], doc["project"]["scripts"]

doc["project"]["dependencies"], doc["project"]["optional-dependencies"], doc["project"]["scripts"]

Out[103]:

(['numpy>=1.26'],
 {'fit': ['scipy>=1.11'], 'plot': ['matplotlib>=3.8']},
 {'attractor-simulate': 'attractor.cli:simulate',
  'attractor-periods': 'attractor.cli:periods',
  'attractor-fit': 'attractor.cli:fit'})

In [100]:

doc["tool"]

doc["tool"]

Out[100]:

{'setuptools': {'packages': {'find': {'where': ['src']}}}}

Version specifiers and optional dependencies¶

The numerical components of a version are compared as integers, from left to right. Missing components count as zero.

Versions may also contain suffixes, such as rc1 for a release candidate.

A version specifier consists of clauses separated by commas. A version satisfies the specifier when it satisfies all the clauses.

The compatible release operator ~=V accepts versions greater than or equal to V that agree with V in all numerical components except the last.

In [104]:

from packaging.specifiers import SpecifierSet
from packaging.version import Version

Version("1.10.1") > Version("1.9.3"), Version("1.15") == Version("1.15.0")

from packaging.specifiers import SpecifierSet from packaging.version import Version Version("1.10.1") > Version("1.9.3"), Version("1.15") == Version("1.15.0")

Out[104]:

(True, True)

In [105]:

releases = ["1.9.3", "1.10.1", "1.11.4", "1.15.3", "1.18.1"]
specs = ["~=1.11.2", "~=1.11", "==1.15", "==1.15.*", "!=1.11.*", ">=1.10,<1.15", "<=1.15"]
for spec in specs:
    print(f"{spec:<13}", list(SpecifierSet(spec).filter(releases)))

releases = ["1.9.3", "1.10.1", "1.11.4", "1.15.3", "1.18.1"] specs = ["~=1.11.2", "~=1.11", "==1.15", "==1.15.*", "!=1.11.*", ">=1.10,<1.15", "<=1.15"] for spec in specs: print(f"{spec:<13}", list(SpecifierSet(spec).filter(releases)))

~=1.11.2      ['1.11.4']
~=1.11        ['1.11.4', '1.15.3', '1.18.1']
==1.15        []
==1.15.*      ['1.15.3']
!=1.11.*      ['1.9.3', '1.10.1', '1.15.3', '1.18.1']
>=1.10,<1.15  ['1.10.1', '1.11.4']
<=1.15        ['1.9.3', '1.10.1', '1.11.4']

An optional dependency is a requirement of one feature only. The table [project.optional-dependencies] groups optional dependencies under names, called extras. In the metadata, an optional requirement names its extra:

In [106]:

md.requires("attractor")

md.requires("attractor")

Out[106]:

['numpy>=1.26',
 'scipy>=1.11; extra == "fit"',
 'matplotlib>=3.8; extra == "plot"']

Only attractor-fit uses SciPy.

Only the option --plot uses Matplotlib.

So cli.py imports each of the two libraries inside the function that uses the library.

As a result, import attractor loads neither SciPy nor Matplotlib:

In [107]:

"scipy" in sys.modules, "matplotlib" in sys.modules

"scipy" in sys.modules, "matplotlib" in sys.modules

Out[107]:

(False, True)

For more on import and recent additions in Python 3.15 see: https://docs.python.org/3.15/reference/simple_stmts.html#lazy-imports

Also this is the opportunity to learn about the concept of PEP (Python Enhancement Proposals): https://peps.python.org/

Editable installs and lock files¶

An editable install, pip install -e ., works in three steps.

1. pip writes the metadata of the project into site-packages.
2. pip writes into site-packages a file ending in .pth, which names the folder src/ of the project.
3. At each start-up of Python, the module site appends the folder src/ to sys.path.

In [108]:

import sysconfig
site_packages = Path(sysconfig.get_paths()["purelib"])
for pth in sorted(site_packages.glob("__editable__*attractor*.pth")):
    print(pth.name, "->", pth.read_text().strip())

import sysconfig site_packages = Path(sysconfig.get_paths()["purelib"]) for pth in sorted(site_packages.glob("__editable__*attractor*.pth")): print(pth.name, "->", pth.read_text().strip())

__editable__.attractor-1.0.0.pth -> /Users/boris/GitHub/attractor/src

In [109]:

md.version("attractor"), attractor.__version__

md.version("attractor"), attractor.__version__

Out[109]:

('1.0.0', '1.0.0')

setup_env.sh first installs the dependencies at their locked versions.

It then installs the project with --no-deps, preventing pip from modifying those dependencies.

The command pip check reports any missing or incompatible dependencies.

Without --no-deps, pip could replace a locked dependency version if it does not satisfy the project's requirements:

In [112]:

sh(f"{sys.executable} -m pip check")

sh(f"{sys.executable} -m pip check")

No broken requirements found.

In [113]:

sh(f'{sys.executable} -m pip install --dry-run "numpy<2.5"', tail=1)

sh(f'{sys.executable} -m pip install --dry-run "numpy<2.5"', tail=1)

Would install numpy-2.4.6

Remember that a a published result depends on the software environment as well as on the code and the data.

3 Console commands and the public interface¶

Console commands¶

An entry point is a reference to a Python object, written module:attribute, such as attractor.cli:main.

Each entry in [project.scripts] associates a terminal command with a Python function. When the distribution is installed, an executable command is created that imports and calls this function.

For example, attractor-fit = "attractor.cli:main" creates the command attractor-fit, which calls main() from the module attractor.cli.

1. At install time, pip writes a launcher into the folder bin/ of the environment.
2. When the command runs, the launcher imports the module and calls the function without arguments.
3. The launcher passes the return value to sys.exit. None gives the exit status 0.

In [115]:

(ep,) = md.entry_points(group="console_scripts", name="dotenv")
ep.value, ep.module, ep.attr

(ep,) = md.entry_points(group="console_scripts", name="dotenv") ep.value, ep.module, ep.attr

Out[115]:

('dotenv.__main__:cli', 'dotenv.__main__', 'cli')

In [116]:

sorted(ep.name for ep in md.entry_points(group="console_scripts") if ep.module == "attractor.cli")

sorted(ep.name for ep in md.entry_points(group="console_scripts") if ep.module == "attractor.cli")

Out[116]:

['attractor-fit', 'attractor-periods', 'attractor-simulate']

In [117]:

print((BIN / "attractor-simulate").read_text())

print((BIN / "attractor-simulate").read_text())

#!/Users/boris/GitHub/attractor/.venv/bin/python3.13
import sys
from attractor.cli import simulate
if __name__ == '__main__':
    if sys.argv[0].endswith('.exe'):
        sys.argv[0] = sys.argv[0][:-4]
    sys.exit(simulate())

The first line of the launcher names the interpreter of the environment by its absolute path. The package is installed in that environment. So the command runs from any folder. The next cell runs the command from the root of the file system:

In [121]:

sh("attractor-simulate", cwd="/Users/boris/Desktop")

sh("attractor-simulate", cwd="/Users/boris/Desktop")

801 points, step 0.05, algae period 8.16 days

Each function reads its options from sys.argv, or from a list. As a result, each function can also be called from Python:

In [122]:

from attractor.cli import simulate
simulate(["--h", "0.1"])

from attractor.cli import simulate simulate(["--h", "0.1"])

401 points, step 0.1, algae period 8.18 days

The public interface¶

The public interface of a package is the set of names that other code may import from the package and rely on.

A package lists its public names in __all__. __init__.py imports the public names from the submodules. It defines the public interface of a package.

Names that start with an underscore are private by convention.

In [136]:

json.__all__, "codecs" in vars(json) # diff between public interface and namespace
# the namespace contains all the names available in a module, 
# whereas the public interface contains the names that users are intended to rely on.

json.__all__, "codecs" in vars(json) # diff between public interface and namespace # the namespace contains all the names available in a module, # whereas the public interface contains the names that users are intended to rely on.

Out[136]:

(['dump',
  'dumps',
  'load',
  'loads',
  'JSONDecoder',
  'JSONDecodeError',
  'JSONEncoder'],
 True)

In [139]:

names = {}
exec("from json import *", names)
sorted(name for name in names if name != "__builtins__")

names = {} exec("from json import *", names) sorted(name for name in names if name != "__builtins__")

Out[139]:

['JSONDecodeError',
 'JSONDecoder',
 'JSONEncoder',
 'dump',
 'dumps',
 'load',
 'loads']

In [145]:

from json import *

from json import *

In [146]:

JSONDecoder

JSONDecoder

Out[146]:

json.decoder.JSONDecoder

In [147]:

codecs

codecs

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[147], line 1
----> 1 codecs

NameError: name 'codecs' is not defined

In [148]:

json.codecs

json.codecs

Out[148]:

<module 'codecs' from '/opt/homebrew/Cellar/python@3.13/3.13.11/Frameworks/Python.framework/Versions/3.13/lib/python3.13/codecs.py'>

The interface of attractor, in __init__.py:

In [149]:

attractor.__all__

attractor.__all__

Out[149]:

['DynamicalSystem',
 'LotkaVolterra',
 'RosenzweigMacArthur',
 'Goodwin',
 'CHEMOSTAT',
 'Euler',
 'RK4',
 'integrate',
 'Trajectory',
 'peaks',
 'cycle_period',
 'load_run',
 'lynx_hare']

In [150]:

attractor.RK4 is attractor.integrators.RK4, "rk4_step" in attractor.__all__

attractor.RK4 is attractor.integrators.RK4, "rk4_step" in attractor.__all__

Out[150]:

(True, False)

Semantic versions¶

A semantic version MAJOR.MINOR.PATCH states what a release does to the public interface.

• A release that breaks code written for the previous one increments MAJOR.
• A release that adds to the interface and breaks nothing increments MINOR.
• A release that only fixes faults increments PATCH.

NumPy 2.0.0 removed the alias np.NaN:

In [151]:

import numpy as np
np.__version__

import numpy as np np.__version__

Out[151]:

'2.5.4'

In [152]:

raises-exception

np.NaN

np.NaN

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[152], line 1
----> 1 np.NaN

File ~/GitHub/attractor/.venv/lib/python3.13/site-packages/numpy/__init__.py:763, in __getattr__(attr)
    760     raise AttributeError(__former_attrs__[attr], name=None)
    762 if attr in __expired_attributes__:
--> 763     raise AttributeError(
    764         f"`np.{attr}` was removed in the NumPy 2.0 release. "
    765         f"{__expired_attributes__[attr]}",
    766         name=None
    767     )
    769 raise AttributeError(f"module {__name__!r} has no attribute {attr!r}")

AttributeError: `np.NaN` was removed in the NumPy 2.0 release. Use `np.nan` instead.

In [154]:

"1.26.4" in SpecifierSet("~=1.26"), "2.0.0" in SpecifierSet("~=1.26")
# SpecifierSet represents a set of acceptable package versions.
# The operator ~= is called the compatible release operator.

"1.26.4" in SpecifierSet("~=1.26"), "2.0.0" in SpecifierSet("~=1.26") # SpecifierSet represents a set of acceptable package versions. # The operator ~= is called the compatible release operator.

Out[154]:

(True, False)

4 Wheels¶

The contents of a wheel¶

A wheel is a zip archive that pip unpacks into site-packages. A wheel holds three kinds of files.

1. The import packages.
2. The declared package data.
3. A folder name-version.dist-info, with the files METADATA, WHEEL, RECORD and entry_points.txt.

A wheel does not hold tests, documentation or scripts outside the package.

In [156]:

dist = md.distribution("python-dotenv")
print(dist.read_text("WHEEL").strip())
print("-"*12)
print(dist.read_text("entry_points.txt").strip())

dist = md.distribution("python-dotenv") print(dist.read_text("WHEEL").strip()) print("-"*12) print(dist.read_text("entry_points.txt").strip())

Wheel-Version: 1.0
Generator: setuptools (84.0.0)
Root-Is-Purelib: true
Tag: py3-none-any
------------
[console_scripts]
dotenv = dotenv.__main__:cli

The next cell builds the wheel of the project in a temporary clone at lecture-12-start. The version is 1.0.0. pip first installs setuptools into a temporary environment. (As a result, the cell needs the network.)

In [158]:

work = Path(tempfile.mkdtemp())
sh(f"git clone -q {REPO} attractor && cd attractor && git -c advice.detachedHead=false switch -q --detach {TAG}",
   cwd=work)
clone = work / "attractor"
sh(f"{sys.executable} -m pip wheel . --no-deps -w dist -q", cwd=clone)
wheel100 = clone / "dist/attractor-1.0.0-py3-none-any.whl"
print("\n".join(zipfile.ZipFile(wheel100).namelist()))

work = Path(tempfile.mkdtemp()) sh(f"git clone -q {REPO} attractor && cd attractor && git -c advice.detachedHead=false switch -q --detach {TAG}", cwd=work) clone = work / "attractor" sh(f"{sys.executable} -m pip wheel . --no-deps -w dist -q", cwd=clone) wheel100 = clone / "dist/attractor-1.0.0-py3-none-any.whl" print("\n".join(zipfile.ZipFile(wheel100).namelist()))

attractor/__init__.py
attractor/cli.py
attractor/cycles.py
attractor/datasets.py
attractor/integrators.py
attractor/models.py
attractor/tools.py
attractor/trajectory.py
attractor-1.0.0.dist-info/licenses/LICENSE
attractor-1.0.0.dist-info/METADATA
attractor-1.0.0.dist-info/WHEEL
attractor-1.0.0.dist-info/entry_points.txt
attractor-1.0.0.dist-info/top_level.txt
attractor-1.0.0.dist-info/RECORD

In [159]:

print("\n".join(zipfile.ZipFile(wheel100).read("attractor-1.0.0.dist-info/METADATA").decode().splitlines()[:12]))

print("\n".join(zipfile.ZipFile(wheel100).read("attractor-1.0.0.dist-info/METADATA").decode().splitlines()[:12]))

Metadata-Version: 2.4
Name: attractor
Version: 1.0.0
Summary: Predator-prey models, fixed-step integrators and cycle periods
License-Expression: MIT
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.26
Provides-Extra: fit
Requires-Dist: scipy>=1.11; extra == "fit"
Provides-Extra: plot

Package data¶

Package data are the files of a package that are not modules.

setuptools puts a data file into the wheel only if the data file is declared in [tool.setuptools.package-data].

The code of a package reads a data file through importlib.resources.files(package).

The object returned by files(package) represents the folder of the installed package:

In [160]:

from importlib.resources import files
rc = files("matplotlib").joinpath("mpl-data/matplotlibrc")
rc.is_file(), rc.read_text().splitlines()[0:10]

from importlib.resources import files rc = files("matplotlib").joinpath("mpl-data/matplotlibrc") rc.is_file(), rc.read_text().splitlines()[0:10]

Out[160]:

(True,
 ['#### MATPLOTLIBRC FORMAT',
  '',
  '## DO NOT EDIT THIS FILE, MAKE A COPY FIRST',
  '##',
  '## This is a sample Matplotlib configuration file - you can find a copy',
  '## of it on your system in site-packages/matplotlib/mpl-data/matplotlibrc',
  '## (relative to your Python installation location).',
  '## DO NOT EDIT IT!',
  '##',
  '## If you wish to change your default style, copy this file to one of the'])

In [162]:

rc

rc

Out[162]:

PosixPath('/Users/boris/GitHub/attractor/.venv/lib/python3.13/site-packages/matplotlib/mpl-data/matplotlibrc')

In [163]:

files("matplotlib").joinpath("mpl-data/no_such_file").is_file()

files("matplotlib").joinpath("mpl-data/no_such_file").is_file()

Out[163]:

False

lynx_hare of datasets.py reads a table of the package through files("attractor"). With the editable install, Python finds the file in src/:

In [164]:

year, lynx, hare = attractor.lynx_hare()
len(year), int(year[0]), float(hare.max())

year, lynx, hare = attractor.lynx_hare() len(year), int(year[0]), float(hare.max())

Out[164]:

(21, 1900, 77.4)

The wheel of 1.0.0 lacks the table. The next cell installs the wheel and its extra fit into a new environment. In the new environment, attractor-fit fails on the missing file. pip downloads NumPy and SciPy unless the cache of pip holds NumPy and SciPy.

In [59]:

fresh = work / "fresh"
sh(f"{sys.executable} -m venv {fresh}")
sh(f'{fresh}/bin/python -m pip install -q "{wheel100}[fit]"')
sh(f"{fresh}/bin/attractor-fit", tail=1)

fresh = work / "fresh" sh(f"{sys.executable} -m venv {fresh}") sh(f'{fresh}/bin/python -m pip install -q "{wheel100}[fit]"') sh(f"{fresh}/bin/attractor-fit", tail=1)

FileNotFoundError: [Errno 2] No such file or directory: '/private/var/folders/hc/d14d998d3rv6_7vdcjskjrxc0000gp/T/tmplx9k49h_/fresh/lib/python3.13/site-packages/attractor/data/lynx_hare.csv'
(exit status 1)

At lecture-12, two lines declare the table. The version becomes 1.0.1 (pyproject.toml):

In [60]:

sh("git -c advice.detachedHead=false switch -q --detach lecture-12 && git diff lecture-12-start lecture-12 -- pyproject.toml",
   cwd=clone)

sh("git -c advice.detachedHead=false switch -q --detach lecture-12 && git diff lecture-12-start lecture-12 -- pyproject.toml", cwd=clone)

diff --git a/pyproject.toml b/pyproject.toml
index 0e456c5..061d1f9 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "attractor"
-version = "1.0.0"
+version = "1.0.1"
 description = "Predator-prey models, fixed-step integrators and cycle periods"
 readme = "README.md"
 license = "MIT"
@@ -22,3 +22,6 @@ attractor-fit = "attractor.cli:fit"
 
 [tool.setuptools.packages.find]
 where = ["src"]
+
+[tool.setuptools.package-data]
+attractor = ["data/*.csv"]

In [61]:

sh(f"{sys.executable} -m pip wheel . --no-deps -w dist -q", cwd=clone)
wheel101 = clone / "dist/attractor-1.0.1-py3-none-any.whl"
[name for name in zipfile.ZipFile(wheel101).namelist() if name.endswith(".csv")]

sh(f"{sys.executable} -m pip wheel . --no-deps -w dist -q", cwd=clone) wheel101 = clone / "dist/attractor-1.0.1-py3-none-any.whl" [name for name in zipfile.ZipFile(wheel101).namelist() if name.endswith(".csv")]

Out[61]:

['attractor/data/lynx_hare.csv']

In [62]:

sh(f'{fresh}/bin/python -m pip install -q "{wheel101}[fit]"')
sh(f"{fresh}/bin/attractor-fit")

sh(f'{fresh}/bin/python -m pip install -q "{wheel101}[fit]"') sh(f"{fresh}/bin/attractor-fit")

LotkaVolterra(a=0.481, b=0.0248, c=0.0275, d=0.926)
H0 = 34.91, L0 = 3.86, rms residual 3.76 thousand pelts

The editable install of this notebook still reports 1.0.0. pip wrote the metadata of the editable install at install time, at lecture-12-start.

Compatibility tags¶

The last three fields of a wheel's filename are its compatibility tags:

• Python tag: the Python implementation and version (e.g. cp313 for CPython 3.13).
• ABI tag: the binary interface required by compiled extensions (e.g. cp313 or none).
• Platform tag: the operating system and architecture (e.g. macosx_14_0_arm64 or any).

The Python environment has an ordered list of supported tag combinations, from most to least preferred.

pip considers a wheel compatible if at least one of its tag combinations appears in this list.

When choosing between compatible wheels of the same distribution version, pip generally prefers the wheel matching the earliest tag in the list.

py3-none-any is a wheel compatibility tag indicating that a wheel is compatible with Python 3, requires no specific Python binary interface (ABI), and is platform-independent.

This is typically a pure-Python wheel, containing Python code without platform-specific compiled extensions.

In [63]:

print(md.distribution("numpy").read_text("WHEEL"))

print(md.distribution("numpy").read_text("WHEEL"))

Wheel-Version: 1.0
Generator: meson
Root-Is-Purelib: false
Tag: cp313-cp313-macosx_14_0_arm64
Generator: delocate 0.13.0

In [168]:

from packaging.tags import sys_tags
accepted = [str(t) for t in sys_tags()]
len(accepted), accepted[:10]

from packaging.tags import sys_tags accepted = [str(t) for t in sys_tags()] len(accepted), accepted[:10]

Out[168]:

(683,
 ['cp313-cp313-macosx_15_0_arm64',
  'cp313-cp313-macosx_15_0_universal2',
  'cp313-cp313-macosx_14_0_arm64',
  'cp313-cp313-macosx_14_0_universal2',
  'cp313-cp313-macosx_13_0_arm64',
  'cp313-cp313-macosx_13_0_universal2',
  'cp313-cp313-macosx_12_0_arm64',
  'cp313-cp313-macosx_12_0_universal2',
  'cp313-cp313-macosx_11_0_arm64',
  'cp313-cp313-macosx_11_0_universal2'])

The tag of a NumPy wheel compiled for CPython 3.12 is not in the list. The tag of the pure Python wheel of the project is in the list:

In [65]:

"cp312-cp312-macosx_14_0_arm64" in accepted, "py3-none-any" in accepted

"cp312-cp312-macosx_14_0_arm64" in accepted, "py3-none-any" in accepted

Out[65]:

(False, True)

A simulation code with compiled modules is often shared as a wheel built on the laptop of its author. A wheel tagged cp313-cp313-macosx_14_0_arm64 installs only under CPython 3.13 on macOS 14 or later with arm64. On the Linux cluster of a colleague, pip refuses the wheel as not supported on this platform.

The field requires-python decides which interpreters may install the project. For example, Python 3.10.18 does not satisfy the specifier of requires-python:

In [66]:

spec = SpecifierSet(doc["project"]["requires-python"])
spec, "3.13.11" in spec, "3.10.18" in spec

spec = SpecifierSet(doc["project"]["requires-python"]) spec, "3.13.11" in spec, "3.10.18" in spec

Out[66]:

(<SpecifierSet('>=3.12')>, True, False)

The exercises and their solutions are at the end of the notes. The scripts of the exercises are in exercises/lecture12/.