=======================================
Build System Support
=======================================
What is it?
-------------
Python packaging has come `a long way `_.
The traditional ``setuptools`` way of packaging Python modules
uses a ``setup()`` function within the ``setup.py`` script. Commands such as
``python setup.py bdist`` or ``python setup.py bdist_wheel`` generate a
distribution bundle and ``python setup.py install`` installs the distribution.
This interface makes it difficult to choose other packaging tools without an
overhaul. Because ``setup.py`` scripts allowed for arbitrary execution, it
proved difficult to provide a reliable user experience across environments
and history.
`PEP 517 `_ therefore came to
rescue and specified a new standard to
package and distribute Python modules. Under PEP 517:
a ``pyproject.toml`` file is used to specify what program to use
for generating distribution.
Then, two functions provided by the program, ``build_wheel(directory: str)``
and ``build_sdist(directory: str)`` create the distribution bundle at the
specified ``directory``. The program is free to use its own configuration
script or extend the ``.toml`` file.
Lastly, ``pip install *.whl`` or ``pip install *.tar.gz`` does the actual
installation. If ``*.whl`` is available, ``pip`` will go ahead and copy
the files into ``site-packages`` directory. If not, ``pip`` will look at
``pyproject.toml`` and decide what program to use to 'build from source'
(the default is ``setuptools``)
With this standard, switching between packaging tools becomes a lot easier. ``build_meta``
implements ``setuptools``' build system support.
How to use it?
--------------
Starting with a package that you want to distribute. You will need your source
scripts, a ``pyproject.toml`` file and a ``setup.cfg`` file::
~/meowpkg/
pyproject.toml
setup.cfg
meowpkg/__init__.py
The pyproject.toml file is required to specify the build system (i.e. what is
being used to package your scripts and install from source). To use it with
setuptools, the content would be::
[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"
The ``setuptools`` package implements the ``build_sdist``
command and the ``wheel`` package implements the ``build_wheel``
command; the latter is a dependency of the former
exposed via :pep:`517` hooks.
Use ``setuptools``' :ref:`declarative config ` to
specify the package information::
[metadata]
name = meowpkg
version = 0.0.1
description = a package that meows
[options]
packages = find:
.. _building:
Now generate the distribution. To build the package, use
`PyPA build `_::
$ pip install -q build
$ python -m build
And now it's done! The ``.whl`` file and ``.tar.gz`` can then be distributed
and installed::
dist/
meowpkg-0.0.1.whl
meowpkg-0.0.1.tar.gz
$ pip install dist/meowpkg-0.0.1.whl
or::
$ pip install dist/meowpkg-0.0.1.tar.gz
Dynamic build dependencies and other ``build_meta`` tweaks
----------------------------------------------------------
With the changes introduced by :pep:`517` and :pep:`518`, the
``setup_requires`` configuration field was made deprecated in ``setup.cfg`` and
``setup.py``, in favour of directly listing build dependencies in the
``requires`` field of the ``build-system`` table of ``pyproject.toml``.
This approach has a series of advantages and gives package managers and
installers the ability to inspect in advance the build requirements and
perform a series of optimisations.
However some package authors might still need to dynamically inspect the final
users machine before deciding these requirements. One way of doing that, as
specified by :pep:`517`, is to "tweak" ``setuptools.build_meta`` by using a
:pep:`in-tree backend <517#in-tree-build-backends>`.
.. tip:: Before implementing a *in-tree* backend, have a look on
:pep:`PEP 508 <508#environment-markers>`. Most of the times, dependencies
with **environment markers** are enough to differentiate operating systems
and platforms.
If you add the following configuration to your ``pyprojec.toml``:
.. code-block:: toml
[build-system]
requires = ["setuptools", "wheel"]
build-backend = "backend"
backend-path = ["_custom_build"]
then you should be able to implement a thin wrapper around ``build_meta`` in
the ``_custom_build/backend.py`` file, as shown in the following example:
.. code-block:: python
from setuptools import build_meta as _orig
prepare_metadata_for_build_wheel = _orig.prepare_metadata_for_build_wheel
build_wheel = _orig.build_wheel
build_sdist = _orig.build_sdist
def get_requires_for_build_wheel(self, config_settings=None):
return _orig.get_requires_for_build_wheel(config_settings) + [...]
def get_requires_for_build_sdist(self, config_settings=None):
return _orig.get_requires_for_build_sdist(config_settings) + [...]
Note that you can override any of the functions specified in :pep:`PEP 517
<517#build-backend-interface>`, not only the ones responsible for gathering
requirements.
.. important:: Make sure your backend script is included in the :doc:`source
distribution `, otherwise the build will fail.
This can be done by using a SCM_/VCS_ plugin (like :pypi:`setuptools-scm`
and :pypi:`setuptools-svn`), or by correctly setting up :ref:`MANIFEST.in
`.
If this is the first time you are using a customised backend, please have a
look on the generated ``.tar.gz`` and ``.whl``.
On POSIX systems that can be done with ``tar -tf dist/*.tar.gz``
and ``unzip -l dist/*.whl``.
On Windows systems you can rename the ``.whl`` to ``.zip`` to be able to
inspect it on the file explorer, and use the same ``tar`` command in a
command prompt (alternativelly there are GUI programs like `7-zip`_ that
handle ``.tar.gz``).
In general the backend script should be present in the ``.tar.gz`` (so the
project can be build from the source) but not in the ``.whl`` (otherwise the
backend script would end up being distributed alongside your package).
See ":doc:`/userguide/package_discovery`" for more details about package
files.
.. _SCM: https://en.wikipedia.org/wiki/Software_configuration_management
.. _VCS: https://en.wikipedia.org/wiki/Version_control
.. _7-zip: https://www.7-zip.org