Poetry#

We are using Poetry for the dependency management in both sds-data-manager and imap_processing. All dependency management should go through Poetry, following our standard Poetry Environment.

Poetry overview#

Poetry is a tool which provides the ability to manage dependencies in Python. Although there are lots of tools which do this, including the built-in pip, Poetry has several specific advantages:

  1. Poetry allows you to “lock” dependency versions in place.

  2. Poetry will automatically determine if dependencies are compatible.

  3. Poetry can provide dependency groups, which allow you to specify which subset of dependencies you would like to install.

These advantages ensure that we can all install the same versions of tools, that these versions are compatible with each other, and that we can manage dependencies in a fine tuned way. When you run poetry install to install your dependencies, you can be confident that they are the dependency versions that other people have been testing with.

Poetry also provides tools for automatically bumping dependency versions only where it makes sense. You can specify when a dependency should get a version change using Poetry’s extensive dependency specification formatting.

Finally, Poetry manages a virtual environment tied to the project directory, which allows developers to install dependency versions in an isolated environment. This means that, for example, if you have a different project using the same package with a different version, you can have each project with its own version kept separate from conflicts. See Installing and Activating the Virtual Environment for how to activate it.

We have a Poetry style guide for specific recommendations about using Poetry in these projects.

Using Poetry#

Poetry can be installed with the official instructions.

The dependencies for the project are all specified in the pyproject.toml file in the base level of the repository.

The Poetry project has a great basic usage guide which introduces the basic capabilities. This covers the basic commands which are used in installing and managing Poetry.

You can use Poetry in addition to other Python tools. Unlike with Conda, installing Poetry should not interfere with other projects which may use other tools. You can also use different methods for virtual environments if you prefer, which is covered under the Poetry shell section. Poetry uses pip to install under the hood.

Adding Dependencies#

Poetry has a command to add a new dependency, with an optional version specification:

# Add a new dependency
poetry add pendulum@^2.0.5

# Add a new dependency with default latest version
poetry add pendulum

These dependencies are then added automatically to the pyproject.toml file. The main project dependencies go under the [project] table as a dependencies list, using PEP 508 version specifiers (e.g. numpy>=1.24,<2). Dependencies in [tool.poetry.group.*] sections continue to use Poetry’s caret notation. The main project dependencies are always installed.

You can also update the pyproject.toml file directly, using the existing formatting or the Poetry documentation on it as a guide.

After you update any dependencies, you will need to update the lock file::

poetry lock

This will create a new version of the poetry.lock file, which should be committed to the repository. Our pre-commit tools also do this step automatically if needed.

Optional extras#

This project uses PEP 621 optional dependencies (extras) to separate dependencies into logical groups. If you are installing the project as an end user, you do not need the development tools. The testing environment does not need the documentation generation dependencies. Before you add a dependency to the main dependencies, ask yourself if it would make more sense in one of the existing extras (dev, test, doc, tools, map_visualization).

To add a dependency to an existing extra, update the [project.optional-dependencies] section in pyproject.toml directly using PEP 508 version specifiers. These extras can be installed selectively when running poetry install or pip install.

Installing and Activating the Virtual Environment#

To install the Poetry project, you can use the install command:

# We use dynamic versioning, which requires a plugin to be installed first
poetry self add "poetry-dynamic-versioning[plugin]"

# Install main dependencies only
poetry install

# Install all extras (dev, test, doc, tools, map_visualization)
poetry install --all-extras

# Install with specific extras
poetry install --extras "test doc"

By default, this command will install dependencies out of the poetry.lock file.

Note

poetry shell was removed in Poetry 2. To activate the Poetry-managed virtual environment, use:

source $(poetry env info --path)/bin/activate

# To exit the virtual environment
deactivate

On Windows (PowerShell):

& (poetry env info --path)\Scripts\activate.ps1

You can also run a single command inside the environment without activating it using poetry run:

poetry run pytest

Using a plain venv instead of Poetry’s virtual environment#

If you prefer to manage your virtual environment with the standard Python venv module rather than Poetry, you can do so by creating the environment first and then running poetry install (or pip install) inside it. Poetry will detect the active virtual environment and install into it instead of creating its own:

# Create and activate a venv
python3 -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate.ps1

# Install via Poetry (uses the active venv)
poetry self add "poetry-dynamic-versioning[plugin]"
poetry install --all-extras

# Or install directly with pip using the pyproject.toml extras
pip install -e ".[dev,test,tools]"

The venv approach avoids the need for poetry shell entirely: your shell is already in the virtual environment after running source .venv/bin/activate.

Poetry will, by default, not create a new virtual environment if it detects that it is running in a virtual environment already. So, for example, you can use a Conda environment by activating the environment first, and then running poetry install.

There are also settings surrounding the virtual environment that you can change to suit your workflow.