Packaging Compiled Projects¶
There are a variety of ways to package compiled projects. In the past, the only way to do it was to use setuptools/distutils, which required using lots of fragile internals - distutils was intended primarily to compile CPython, and setuptools tried to stay away from changing the compile portions except as required. Now, however, we have several very nice options for compiled packages!
The most exciting developments have been new native build backends:
scikit-build-core: Builds C/C++/Fortran using CMake.
meson-python: Builds C/C++/Fortran using Meson.
maturin: Builds Rust using Cargo. Written entirely in Rust!
[enscons][]: Builds C/C++ using SCONs. (Aging now, but this was the first native backend!)
There are also classic setuptools plugins:
scikit-build: Builds C/C++/Fortran using CMake.
setuptools-rust: Builds Rust using Cargo.
pyproject.toml: build-system¶
PY001 Packages must have a pyproject.toml file PP001 that
selects the backend:
[build-system]
requires = ["pybind11", "scikit-build-core>=0.12"]
build-backend = "scikit_build_core.build"[build-system]
requires = ["meson-python>=0.18", "pybind11"]
build-backend = "mesonpy"[build-system]
requires = ["maturin>=1.9,<2"]
build-backend = "maturin"pyproject.toml: project table¶
The metadata is specified in a standards-based format:
[project]
name = "package"
version = "0.1.0"
authors = [
{ name = "My Name", email = "me@email.com" },
]
description = "A great package."
readme = "README.md"
license = "BSD-3-Clause"
license-files = ["LICENSE"]
requires-python = ">=3.10"
classifiers = [
"Development Status :: 1 - Planning",
"Intended Audience :: Science/Research",
"Intended Audience :: Developers",
"Operating System :: OS Independent",
"Programming Language :: Python",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3 :: Only",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
"Programming Language :: Python :: 3.14",
"Topic :: Scientific/Engineering",
"Typing :: Typed",
]
dependencies = []
[project.urls]
Homepage = "https://github.com/org/package"
"Bug Tracker" = "https://github.com/org/package/issues"
Discussions = "https://github.com/org/package/discussions"
Changelog = "https://github.com/org/package/releases"In this example, "package" is the name of the thing you are working on. You
can read more about each field, and all allowed fields, in
packagingname and version fields are strictly required. Note that “Homepage” is
special, and replaces the old url setting.
If you use the above configuration, you need README.md and LICENSE files,
since they are explicitly specified.
License¶
The license can be done one of two ways.
The modern way is to use the license field and an SPDX identifier
expression. You can specify a list of files globs in license-files. You
need hatchling>=1.26, flit-core>=1.11 (1.12 for complex license statements),
pdm-backend>=2.4, setuptools>=77, meson-python>=0.18, maturin>=1.9.2,
poetry-core>=2.2, or scikit-build-core>=0.12 to support this. You can also
specify license-files as a list with globs for license files. If you don’t,
most backends will discover common license file names by default.
The classic convention uses one or more Trove Classifiers to specify the
license. There also was a license.file field, required by meson-python, but
other tools often did the wrong thing (such as load the entire file into the
metadata’s free-form one line text field that was intended to describe
deviations from the classifier license(s)).
classifiers = [
"License :: OSI Approved :: BSD License",
]You should not include the License :: classifiers if you use the license
field PP007.
Extras¶
Sometimes you want to ship a package with optional dependencies. For example,
you might have extra requirements that are only needed for running a CLI, or for
plotting. Users must opt-in to get these dependencies by adding them to the
package or wheel name when installing, like package[cli,mpl].
Here is an example of a simple extras:
[project.optional-dependencies]
cli = [
"click",
]
mpl = [
"matplotlib >=2.0",
]Self dependencies can be used by using the name of the package, such as
all = ["package[cli,mpl]"], (requires Pip 21.2+).
Command line¶
If you want to ship an “app” that a user can run from the command line, you need
to add a script entry point. The form is:
[project.scripts]
cliapp = "package.__main__:main"The format is command line app name as the key, and the value is the path to the
function, followed by a colon, then the function to call. If you use
__main__.py as the file, then python -m followed by the module will also
work to call the app (__name__ will be "__main__" in that case).
Development dependencies¶
The proper way to specify dependencies exclusively used for development tasks
(such as pytest, ruff, packages for generating documentation, etc.) is to
use dependency-groups. Dependency-groups are recommended over requirement files
because they are formally standardized (i.e. they will be more portable going
forward) and they are more composable. In contrast with extras,
dependency-groups are not available when installing your package via PyPI, but
they are available for local installation (and can be installed separately from
your package); the dev group is even installed, by default, when using uv’s
high level commands like uv run and uv sync. PP006 Here is an
example:
[dependency-groups]
test = [
"pytest >=9",
"pytest-cov >=7",
]
dev = [
{ include-group = "test" },
]
docs = [
"sphinx>=7.0",
"myst_parser>=0.13",
"sphinx_copybutton",
"sphinx_autodoc_typehints",
"furo>=2023.08.17",
]You can include one dependency group in another. Most tools allow you to install
groups using --group, like pip (25.1+), uv pip, and the high level uv
interface. It’s possible to install a package’s dependency group without
installing the package itself, but usually you’ll want to instruct your tool to
install both (the high level uv interface does this automatically). Nox, Tox,
and cibuildwheel all support groups, too. The dependency-groups package also
provides tools to get the dependencies.
Tool section in pyproject.toml¶
These tools all read the project table. They also have extra configuration
options in tool.* settings.
[tool.scikit-build]
minimum-version = "build-system.requires"
build-dir = "build/{wheel_tag}"These options are not required, but can improve your experience.
No tool.meson-python configuration required for this example.
[tool.maturin]
module-name = "package._core"
python-source = "src"
sdist-generator = "git" # default is cargoMaturin assumes you follow Rust’s package structure, so we need a little bit of configuration here to follow the convention of the other tools here.
Backend specific files¶
Example CMakeLists.txt file (using pybind11, so include pybind11 in
build-system.requires too):
cmake_minimum_required(VERSION 3.15...3.26)
project(${SKBUILD_PROJECT_NAME} LANGUAGES CXX)
set(PYBIND11_FINDPYTHON ON)
find_package(pybind11 CONFIG REQUIRED)
pybind11_add_module(_core MODULE src/main.cpp)
install(TARGETS _core DESTINATION ${SKBUILD_PROJECT_NAME})Scikit-build-core will use your .gitignore to help it avoid adding ignored
files to your distributions; it also has a default ignore for common cache
files, so you can get started without one, but it’s recommended.
Example meson.build file (using pybind11, so include pybind11 in
build-system.requires too):
project(
'package',
'cpp',
version: '0.1.0',
license: 'BSD',
meson_version: '>= 1.1.0',
default_options: [
'cpp_std=c++11',
],
)
py = import('python').find_installation(pure: false)
pybind11_dep = dependency('pybind11')
py.extension_module('_core',
'src/main.cpp',
subdir: 'package',
install: true,
dependencies : [pybind11_dep],
)
install_subdir('src/package', install_dir: py.get_install_dir() / 'package', strip_directory: true)Meson requires that your source be tracked by version control. In a real project, you will likely be doing this, but when trying out a build backend you might not think to set up a git repo to build it.
Example Cargo.toml file:
[package]
name = "package"
version = "0.1.0"
edition = "2021"
[lib]
name = "_core"
# "cdylib" is necessary to produce a shared library for Python to import from.
crate-type = ["cdylib"]
[dependencies]
rand = "0.9.2"
[dependencies.pyo3]
version = "0.27.2"
# "extension-module" tells pyo3 we want to build an extension module (skips linking against libpython.so)
# "abi3-py310" tells pyo3 (and maturin) to build using the stable ABI with minimum Python version 3.10
features = ["extension-module", "abi3-py310"]Example compiled file¶
This example will make a _core extension inside your package; this pattern
allows you to easily provide both Python files and compiled extensions, and
keeping the details of your compiled extension private. You can select whatever
name you wish, though, or even make your compiled extension a top level module.
Example src/main.cpp file:
#include <pybind11/pybind11.h>
int add(int i, int j) { return i + j; }
namespace py = pybind11;
PYBIND11_MODULE(_core, m) {
m.doc() = R"pbdoc(
Pybind11 example plugin
-----------------------
.. currentmodule:: python_example
.. autosummary::
:toctree: _generate
add
subtract
)pbdoc";
m.def("add", &add, R"pbdoc(
Add two numbers
Some other explanation about the add function.
)pbdoc");
m.def("subtract", [](int i, int j) { return i - j; }, R"pbdoc(
Subtract two numbers
Some other explanation about the subtract function.
)pbdoc");
}Example src/main.cpp file:
#include <pybind11/pybind11.h>
int add(int i, int j) { return i + j; }
namespace py = pybind11;
PYBIND11_MODULE(_core, m) {
m.doc() = R"pbdoc(
Pybind11 example plugin
-----------------------
.. currentmodule:: python_example
.. autosummary::
:toctree: _generate
add
subtract
)pbdoc";
m.def("add", &add, R"pbdoc(
Add two numbers
Some other explanation about the add function.
)pbdoc");
m.def("subtract", [](int i, int j) { return i - j; }, R"pbdoc(
Subtract two numbers
Some other explanation about the subtract function.
)pbdoc");
}Example src/lib.rs file:
use pyo3::prelude::*;
/// A Python module implemented in Rust. The name of this function must match
/// the `lib.name` setting in the `Cargo.toml`, else Python will not be able to
/// import the module.
#[pymodule]
mod _core {
use super::*;
#[pyfunction]
fn add(x: i64, y: i64) -> i64 {
x + y
}
#[pyfunction]
fn subtract(x: i64, y: i64) -> i64 {
x - y
}
#[pymodule_init]
fn pymodule_init(m: &Bound<'_, PyModule>) -> PyResult<()> {
m.add("__version__", env!("CARGO_PKG_VERSION"))?;
Ok(())
}
}Package structure¶
The recommendation (followed above) is to have source code in /src, and the
Python package files in /src/<package>. The compiled files also can go in
/src.
Versioning¶
Check the documentation for the tools above to see what forms of dynamic versioning the tool supports.
Including/excluding files in the SDist¶
Each tool uses a different mechanism to include or remove files from the SDist, though the defaults are reasonable.
Distributing¶
Unlike pure Python, you’ll need to build redistributable wheels for each platform and supported Python version if you want to avoid compilation on the user’s system using cibuildwheel. See the CI page on wheels for a suggested workflow.
Special considerations¶
NumPy¶
Modern versions of NumPy (1.25+) allow you to target older versions when building, which is highly recommended, and this became required in NumPy 2.0. Now you add:
#define NPY_TARGET_VERSION NPY_1_22_API_VERSION(Where that number is whatever version you support as a minimum) then make sure
you build with NumPy 1.25+ (or 2.0+). Before 1.25, it was necessary to actually
pin the oldest NumPy you supported (the oldest-supported-numpy package is the
easiest method). If you support Python < 3.9, you’ll have to use the old method
for those versions.
If using pybind11, you don’t need NumPy at build-time in the first place.