Dependencies and Continuous Integration

Dependencies

⏩ Dependencies are other pieces of software your project needs to work.

🤔 They save time by reusing existing solutions, but can make your project harder to maintain if they change or are no longer available.

Application

A program that is run by a user

  • command line tool
  • script
  • web application

Pin versions to ensure reproducibility, e.g. numpy==1.11.0

Library

A program that is used by another program

  • numpy
  • scikit-learn

Make the requirements as loose as possible, e.g. numpy>=1.11.0

Dependency resolution

  • Python projects often depend on many packages, which may depend on others (“dependency tree”).
  • Different packages may require conflicting versions of the same dependency.
  • Historically, Python tools did not always resolve these conflicts well, leading to “dependency hell.” 😈
  • Ensuring everyone gets the same working set of packages has been difficult, especially for larger projects. 🤪

Modern tools like uv aim to solve these problems with faster and more reliable 😎

uv

  • 🚀 A single tool to replace pip, pip-tools, pipx, poetry, pyenv, twine, virtualenv, and more.
  • ⚡️ 10-100x faster than pip.
  • 🗂️ Provides comprehensive project management, with a universal lockfile.
  • ❇️ Runs scripts, with support for inline dependency metadata.
  • 🐍 Installs and manages Python versions.
  • 🛠️ Runs and installs tools published as Python packages.
  • 🔩 Includes a pip-compatible interface for a performance boost with a familiar CLI.
  • 💾 Disk-space efficient, with a global cache for dependency deduplication.
  • ⏬ Installable without Rust or Python via curl or pip.
  • 🖥️ Supports macOS, Linux, and Windows.

Dependency management

uv is the recommended tool for managing a Python project including dependencies.

Example of pinning versions:

pyproject.toml
dependencies = [
  "numpy==1.11.0",
  "scipy==0.17.0",
  "matplotlib==1.5.1",
]

Or using a range of versions:

pyproject.toml
dependencies = [
  "numpy>=1.11.0",
  "scipy>=0.17.0",
  "matplotlib>=1.5.1,<=2.0.0"
]

Install dependencies:

$ uv sync

Development dependencies

I.e. dependencies needed for testing, building documentation, linting, etc. not needed to run the package.

pyproject.toml
[dependency-groups]
dev = [
    "pytest>=8.4.1",
]

Dependency management using uv

  • Add a dependency:
$ uv add matplotlib
  • Remove a dependency:
$ uv remove seaborn
  • Add development dependency:
$ uv add --dev pytest

Creating an installable package

Create a new library project:

$ uv init --lib

Start a Python session:

$ uv run python
>>> import mini
>>> mini.foo()
42

Run tests:

$ uv run pytest
...

tests/test_foo.py .                       [100%]

=============== 1 passed in 0.01s ===============

Virtual environments

  • Creates a clean environment for each project
  • Allows different versions of a package to coexist on your machine
  • Can be used to create a reproducible environment for a project
  • Virtual environments are managed by uv

Continuous Integration (CI)

  • Running tests on every commit in a well defined environment ensures that the code is working as expected.

  • It solves the “it works on my machine” problem.

  • Executing code on a remote server is a good way to ensure that the code is working as expected.

  • Example of CI services:

    • GitHub Actions
    • Azure Pipelines
    • Travis CI

GitHub Actions

  • Workflow are stored in the .github/workflows folder.
  • Workflow is described in a YAML file.
  • YAML is whitespace sensitive (like Python).
  • YAML can contain lists, dictionaries and strings, and can be nested.
$ tree mikeio/.github/
mikeio/.github/
└── workflows
    ├── docs.yml
    ├── full_test.yml
    ├── notebooks_test.yml
    ├── perf_test.yml
    └── python-publish.yml

Workflow example

name: Quick test

on: # when to run the workflow
  push:
    branches: [ main]
  pull_request:
    branches: [ main ]

jobs: # which jobs to run
  build: # descriptive name 🙄
    runs-on: ubuntu-latest # on what operating system

    steps:
    - uses: actions/checkout@v3
    - name: Set up uv
      uses: astral-sh/setup-uv@v6
      with:
        python-version: "3.13"
    - name: Install dependencies
      run: |
        uv sync

    - name: Test with pytest
      run: |
        uv run pytest

🙂🚀

☹️

Benefits of CI

  • Run tests on every commit
  • Test on different operating systems
  • Test on different Python versions
  • Create API documentation
  • Publish package to PyPI or similar package repository

Triggers

  • push and pull_request are the most common triggers
  • schedule can be used to run the workflow on a schedule
  • workflow_dispatch can be used to trigger the workflow manually
on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]
  schedule:
    - cron: '0 0 * * 0'
  workflow_dispatch:

Jobs

  • Operating system
  • Python version
  • …
...
jobs:
  build:
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [ubuntu-latest, windows-latest]
        python-version: ["3.10","3.13"]
...

GitHub Releases

  • GitHub releases are a way to publish software releases.

  • You can upload files, write release notes and tag the release.

  • As a minimum, the release will contain the source code at the time of the release.

  • Creating a release can trigger other workflows, e.g. publishing a package to PyPI.

https://github.com/pydata/xarray/releases/tag/v2022.12.0

Summary

  • Application vs library
  • Use a separate virtual environment for each project
  • Use GitHub Actions to run tests on every commit
  • Use GitHub Releases to publish software releases