Modern Python Packaging: From Zero to Green CI/CD Pipeline in One Afternoon

Photo by Christina @ wocintechchat.com M on Unsplash

So we all know this problem — we start with some new Python project and need functionality from another code project. We copy-paste the .py files and then try to adapt the imports and install the missing dependencies by trial and error. And we are thinking — shouldn’t this be easier?

We could of course use a Git Submodule and add the Python code like that. So that would mean one of the folders in our Python project is another Git repository. But this snake also bites — we are still missing the dependencies of the code and — to be honest — working with Git Submodules wasn’t the most fun I ever had. Continuously making sure that every developer also pulled the right version configured in the Git Submodule seemed tricky.

But do not despair — there are much more elegant ways to share and re-use code!

You can package your Python code and publish it to a public or private Python registry. A Python registry is basically a storage hub for your Python packages. Some options to host your own Python registry are Nexus, Artifactory or a self-managed GitLab.

I personally came to this problem when I was creating data pipelines with the workflow-orchestration tool Apache Airflow and was always using the same code to connect to my databases. So I decided to create a Python package with my database connectors, create a CI/CD pipeline that tests and packages this code and then publishes it directly to the Python registry in GitLab. You can even add tools for dependency checks (e.g. pip audit) or check for security holes in your own code using bandit. We can even generate beautiful docs!

Setting up the project structure and the CI/CD pipeline for that for the first time in a new infrastructure is always a hassle — so let’s do it together and you can re-use this base code for the future!

Set up the project structure with uv

I recently switched to using **uv** as my package manager. It is fast and you can install Python with it directly. I recently converted to uv after using poetry for years and I have never looked back! Check out my uv cheatsheet.

Install uv for your platform: https://docs.astral.sh/uv/getting-started/installation/

Verify that the installation worked using this command in the terminal (it should print the uv version that is installed on your system):

uv --info

Now let’s install Python 3.13 with uv (yes, it can do that!! Installing Python platform-agnostic — isn’t it amazing??):

uv python install 3.13

Uv gives us an option to scaffold the code structure of our package via the command line:

uv init --name <NAME OF YOUR PACKAGE> --package

Side note on naming the package (naming things being an important part of our life as programmers…): Do use the naming convention for Python packages defined by PEP 8. Mainly this includes:

• Use lowercase names.
• Hyphens are common for distribution/package names on PyPI, while Python import names typically use underscores, e.g. most_amazing_package
• Use simple and short names
• Do not use abbreviations
• Make sure that the project or package names do not conflict with the existing Python standard library modules, e.g. pathlib

Now let’s create the virtual environment:

uv venv

Let’s look at what defines our virtual environment when using uv: If you have been using poetry before, you already know the “pyproject.toml” file — it contains information about your project and your dependencies. Additionally, we get the “uv.lock” file which states detailed information about the dependencies and from which source they were installed.

Good practice is also to add a “.gitignore” file that is specialised on Python to stop Git from committing some files that should not be added to Git (cache files, secrets…). Here is an example from GitHub: https://github.com/github/gitignore/blob/main/Python.gitignore

Add a “LICENSE” file with the license that fits to your Python package. The file contains text that explains your license.

Additionally, add a “README.md” so you can add information about setting up the local development environment to it. Get ideas from a readme generator like this one: https://readme.so/

Now you are ready to add your code! Uv already created the package structure for us. Your code goes in the created source code folder that is named after the package, in my example this looks like this (“column_statistics.py” is where my package code lies):

folder content

To add a Python dependency that is necessary for your package code, use “uv add”:

uv add pandas

You can check on the dependencies using:

uv tree

This will give you a nice overview of your dependencies in the terminal.

output of uv tree

Write a test

Of course we want to test the code that we just wrote before we distribute it! There are two popular Python packages for creating tests — unittest and pytest. unittest comes with the Python standard libraries, so it is already installed. It uses a class-based approach. pytest works with basic functions and has some other nice features like fixtures, so I prefer to use pytest. We need to install it as a dependency, but of course we don’t want to add it to the package dependencies, only to the development dependencies. With uv you have the option to install packages for certain groups — so for example you can now add pytest to the dev group and it will only be installed when someone is building the Python environment to further develop the package, but not when the package itself is installed in another project. So let’s to that via the terminal:

uv add pytest --group dev

You will see that uv edited the “pyproject.toml” and the “uv.lock” file. In the “pyproject.toml” you will find a new group in the dependencies that specifies pytest as a dev dependency. Here is an extracted section of the “pyproject.toml” file that shows the package dependencies vs. the dev dependencies in a separate group:

dependencies = [
    "pandas>=2.2",
]

[dependency-groups]
dev = [
    "pytest>=9.0.3",
    "pytest-xdist>=3",
]

Now add a folder called “tests”. Add a test called. Pytest will automatically parse the tests folder for .py files that start with* “test_”**.

screenshot of files

You can set up running the tests with the test explorer in your IDE. Here is a tutorial on that for VSCode: https://code.visualstudio.com/docs/debugtest/testing

Set up pre-commit

Next up in our tool set, we have pre-commit. This is a nifty little Python package that helps us to only commit clean code to our Git repository.

Install the pre-commit Python package as a dev dependency:

uv add pre-commit --group dev

With pre-commit there are many different options of what you can set up — you can run tools like black or run self-defined terminal commands. Create a new file in your root directory called “.pre-commit-config.yaml”.

Here is an example for the file content:

repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v5.0.0
    hooks:
      - id: trailing-whitespace
      - id: end-of-file-fixer
      - id: check-yaml
      - id: check-toml
      - id: check-merge-conflict
      - id: check-added-large-files
        args: ["--maxkb=10240"]
      - id: debug-statements

  - repo: https://github.com/pycqa/isort
    rev: 5.13.2
    hooks:
      - id: isort
        args: ["--profile", "black", "--line-length", "80"]

  - repo: https://github.com/psf/black-pre-commit-mirror
    rev: 24.1.1
    hooks:
      - id: black
        language_version: python3.13
        args: ["--line-length=80"]

  - repo: https://github.com/pycqa/flake8
    rev: 7.0.0
    hooks:
      - id: flake8
        args: ["--max-line-length=80"]

With the first section (pre-commit-hook) you can run some basic file checks, e.g. if all your yaml files contain valid yaml syntax. Isort sorts your imports. Black auto-formats your code so it is pretty. Flake8 is a popular linter. You can also add mypy — a static type checker — or we could add a custom command that runs our tests that we created with pytest.

So now we set up the configuration of pre-commit via the yaml file — now we need to install the actual pre-commit hooks so that they are set up via out local Git installation:

uv run pre-commit install

Now try to stage and commit your files! You will see the output of the pre-commit run.

terminal output of pre-commit

Black will automatically format your files — you just have to re-add them with Git and commit again. Whoop — beautiful, readable code!

Set up mkdocs

Of course, as every good programmer wants, we would like to set up a documentation. With the package mkdocs you can generate one automatically — provided that the functions in the code are documented using docstrings (a specific way of documenting your functions, modules etc.). I prefer to use the Google docstring format as this is the style that I see most. Mkdocs will create a static HTML page that you can host somewhere — you can even use GitLab Pages or GitHub Pages. I like adding the publishing of this page to my CI/CD pipeline. Whenever I create a new release version of my Python package, the documentation is automatically generated and deployed as a static HTML page. Developers using my package can then easily find out how my package works.

To get mkdocs to help us out with the auto-generation of the docs, we need to install it (plus some extra helper packages) as a dev package:

uv add mkdocs mkdocs-material mkdocstrings[python] --group dev

Next we need to set up yet another yaml. In the root directory of your project, create a file called “mkdocs.yaml”.

Here is my example (you need to adapt a couple of lines for your personal project). I like using the material theme and, as already mentioned, the Google style docstrings.

site_name: data-analytics
site_description: Column-wise statistics and helpers for pandas DataFrames.
edit_uri: https://github.com/kolbl/data-analytics/edit/main/docs/

docs_dir: docs

theme:
  name: material
  features:
    - content.code.copy

plugins:
  - search
  - mkdocstrings:
      handlers:
        python:
          paths: [src]
          options:
            docstring_style: google
            show_source: true
            show_root_heading: true
            members_order: source

nav:
  - Home: index.md
  - API reference: api.md

markdown_extensions:
  - pymdownx.highlight:
      anchor_linenums: true
  - pymdownx.superfences

strict: true

In the nav section you see two markdown files referenced — we need to create those: In your root directory, create a folder “docs” and subsequently create “api.md” and “index.md” in that folder.

file content of the docs folder

In the api.md, we specify what of the API of our Python package should be parsed by mkdocs for docstrings. In my case there is only a single file, so that’s what is included in my “api.md”:

# API reference

::: data_analytics.column_statistics

In the “index.md” you can add whatever you would like — it is basically the starting point of your static documentation page. Here is my example:

# data-analytics

Small utilities for exploratory work with **pandas** (descriptive statistics).

## Install

```bash
uv add "data-analytics==0.1.0.dev5" --extra-index-url https://test.pypi.org/simple/  --index-strategy unsafe-best-match

Quick example

import pandas as pd
from data_analytics.column_statistics import calculate_min_of_column

df = pd.DataFrame({"a": [1.0, 2.0, 3.0], "b": [4.0, 5.0, 6.0]})
min_a = calculate_min_of_column(df, "a")  # 1.0

Documentation

• API reference — generated from docstrings.

Source

github.com/kolbl/data-analytics

You can check locally if the documentation can be built:

uv run mkdocs build --strict

You will see a new folder called **“site” **pop up in your project. Now serve it locally with:

uv run mkdocs serve --dev-addr 0.0.0.0:8000

Now open [http://localhost:8000/](http://localhost:8001/) locally in a browser. You should see something similar to this:

![screenshot of automated docs using mkdocs](https://miro.medium.com/v2/resize:fit:1349/1*DxtMroVXXWMzsTqf6Gx9zQ.png)

*screenshot of automated docs using mkdocs*

Isn’t it fantastic?? In the nav, you can switch from the index page (maps to **“index.md”**, here called “Home”) to the API reference (**“api.md”**).

## Write the CI/CD pipeline

I love automating scripts and seeing many green check boxes, so of course we need to add a CI/CD pipeline to our package. Here is an example using GitHub Actions (most of the commands can be re-used for other platforms):

Pre-commit, tests, and security (pip-audit, bandit) run in parallel. On pushes

to main, after all succeed, builds a PEP 440 dev release (<base>.dev<run_number>)

and publishes with uv. MkDocs is built and deployed to GitHub Pages only after

publish succeeds.

Add a PyPI API token as repository secret PYPI_API_TOKEN (Account settings → API tokens).

Pages: Settings → Pages → Source: GitHub Actions.

name: CI

on: push: branches: [main] pull_request: branches: [main] workflow_dispatch:

concurrency: group: ${{ github.workflow }}-${{ github.ref }} cancel-in-progress: true

jobs: pre-commit: runs-on: ubuntu-latest steps:

• uses: actions/checkout@v4
• uses: astral-sh/setup-uv@v5 with: python-version: "3.13" enable-cache: true
• name: Install dependencies run: uv sync --frozen --group dev
• name: Run pre-commit on all files run: uv run pre-commit run --all-files

test: runs-on: ubuntu-latest steps:

• uses: actions/checkout@v4
• uses: astral-sh/setup-uv@v5 with: python-version: "3.13" enable-cache: true
• name: Install dependencies run: uv sync --frozen --group dev
• name: Run tests in parallel run: uv run pytest -n auto

security: runs-on: ubuntu-latest steps:

• uses: actions/checkout@v4
• uses: astral-sh/setup-uv@v5 with: python-version: "3.13" enable-cache: true
• name: Install dependencies run: uv sync --frozen --group dev
• name: pip audit (installed environment) run: uv run pip-audit --skip-editable
• name: bandit (src/data_analytics) run: uv run bandit -r src/data_analytics -f txt

publish: needs: [pre-commit, test, security] if: github.event_name == 'push' && github.ref == 'refs/heads/main' runs-on: ubuntu-latest permissions: contents: read steps:

• uses: actions/checkout@v4
• uses: astral-sh/setup-uv@v5 with: python-version: "3.13" enable-cache: true
• name: Install dependencies run: uv sync --frozen --group dev
• name: Set dev version (PEP 440) run: | base="$(uv version --short)" uv version "${base}.dev${{ github.run_number }}" --frozen
• name: Build distributions run: uv build
• name: Publish to PyPI env: UV_PUBLISH_TOKEN: ${{ secrets.PYPI_API_TOKEN }} UV_PUBLISH_URL: https://test.pypi.org/legacy/ run: uv publish

docs-build: needs: [publish] runs-on: ubuntu-latest permissions: contents: read pages: write id-token: write steps:

• uses: actions/checkout@v4
• uses: astral-sh/setup-uv@v5 with: python-version: "3.13" enable-cache: true
• name: Install dependencies run: uv sync --frozen --group dev
• name: Build MkDocs site run: uv run mkdocs build --strict
• uses: actions/upload-pages-artifact@v3 with: path: site

docs-deploy: needs: [docs-build] runs-on: ubuntu-latest concurrency: group: github-pages cancel-in-progress: false permissions: pages: write id-token: write environment: name: github-pages url: ${{ steps.deployment.outputs.page_url }} steps:

• id: deployment uses: actions/deploy-pages@v4

This is the pipeline visualised:

CI/CD pipeline using GitHub Actions

The first block runs our pre-commit checks, the tests and also some additional security checks with pip-audit (checks the dependencies of the Python package) and bandit (checks for security issues in your code).

But wait — why are we running the pre-commit checks again in the pipeline when we set them up locally? Well — not everyone is as meticulous as us. Trust no one. Not even yourself. It happens that people forget to install pre-commit locally, so the Git pre-commit hooks won’t be triggered once they try to locally commit their new code.

The next step builds and publishes the Python package using uv. Let’s look at this more closely:

  publish:
    needs: [pre-commit, test, security]
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    permissions:
      contents: read
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v5
        with:
          python-version: "3.13"
          enable-cache: true
      - name: Install dependencies
        run: uv sync --frozen --group dev
      - name: Set dev version (PEP 440)
        run: |
          base="$(uv version --short)"
          uv version "${base}.dev${{ github.run_number }}" --frozen
      - name: Build distributions
        run: uv build
      - name: Publish to PyPI
        env:
          UV_PUBLISH_TOKEN: ${{ secrets.PYPI_API_TOKEN }}
          UV_PUBLISH_URL: https://test.pypi.org/legacy/
        run: uv publish

The first couple of lines are just setting up the pipeline — the publish step needs to wait for the pre-commit, test and security jobs to finish.

What you can see further down is how we are installing the dependencies (we need some packages to help us build and publish the package). Then we get the version — either specify a dev version using some unique identifier like the run number of the pipeline run in GitHub or we use the version that is specified in the “pyproject.toml” via:

uv version --short

This will output the version, such as: 0.1.0. This value comes from our “pyproject.toml”, but you can add some other rules in your CI/CD pipeline. I recommend using Git tags here instead of reading the version from the file.

This is the final pyproject.toml for my example project:

[project]
name = "data-analytics"
version = "0.1.0"
description = "A Python Package for Data Analysis"
readme = "README.md"
license = "MIT"
license-files = ["LICEN[CS]E*"]
authors = [
    { name = "kolbl" }
]
requires-python = ">=3.13"
classifiers = [
    "Programming Language :: Python :: 3",
    "Operating System :: OS Independent",
]
dependencies = [
    "pandas>=2.2",
    "pip==26.1",
]

[dependency-groups]
dev = [
    "black>=26.3.1",
    "flake8>=7.3.0",
    "isort>=8.0.1",
    "bandit[toml]>=1.8",
    "mkdocs-material>=9.5",
    "mkdocstrings[python]>=0.26",
    "pip-audit>=2.7",
    "pre-commit>=4.5.1",
    "pytest>=9.0.3",
    "pytest-xdist>=3",
    "mkdocs>=1.6.1",
]
app = []

[build-system]
requires = ["uv_build>=0.11.1,<0.12.0"]
build-backend = "uv_build"

[tool.pytest.ini_options]
testpaths = ["tests"]

[tool.isort]
profile = "black"
line_length = 80

Here you see the dependency groups with our standard and dev groups.

When building and pupblishing our Python package, the data in this “pyproject.toml” becomes especially important since it gives all the information about our dependencies (Python packages, Python version). Also our readme and license is referenced.

The basic commands for building and publishing our Python package using uv are:

uv build
uv publish

In this learning example, I was using TestPyPI which is a public Python registry where people can try out publishing a package before publishing them to the PyPI, the public Python registry that you have probably become in contact with before. Basically it is a public hub where programmers get Python packages from. A Python package is basically not more than something like a compressed archive file with a specific structure and a list of dependencies. It is being distributed using a Python registry like the public PyPI or a private one that is only available inside of a company. To set up a private PyPI registry, you have several options, such as using Nexus or a self-managed GitLab.

It is common that the upload to a registry (the publishing step) needs authentication (token or user + password). This is managed using variables in your CI/CD platform and then referencing these variables in the CI/CD yaml, such as:

- name: Publish to PyPI
  env:
    UV_PUBLISH_TOKEN: ${{ secrets.PYPI_API_TOKEN }}
    UV_PUBLISH_URL: https://test.pypi.org/legacy/
  run: uv publish

After publishing the package to a Python registry, the last steps in your CI/CD pipeline build the documentation and then serve the docs via a GitHub page. Fancy, right?!

And there you have it — a green pipeline in the effort of an afternoon!

How to use the package

To install the package in another project, we can simply use “uv add” again. But by default uv will check for Python packages on the public PyPI, so we need to specify a different source by using the argument “extra-index-url” :

uv add my-package --extra-index-url
<PYTHON REGISTRY URL, e.g. https://test.pypi.org/simple/> --index-strategy
unsafe-best-match

Then you can easily use the package by importing it in your next Python script:

from my_package import my_function
result = my_function()
print(result)

It might be that your Python registry enforces authentication — in that case you need to supply that when downloading the package onto your machine.

Extra Challenge: dev container

If you are not familiar with Docker/Podman, then I would suggest for you to skip this step and come back after you have researched about containers.

So let’s say you work together with other programmers on your Python package. It can be tricky to set everything up on your own machine and you might run into some “doesn’t work on my machine” issues. But there is also a great solution for that — dev containers! You can set up a Dockerfile that defines an image that is then spun up by your IDE as a container using Docker or Podman. Then your team members and you develop inside this container. This is a super practical way to create a reusable environment that runs everywhere the same way and is easily reproducible.

To start this up, create a “.devcontainer” folder with a “Dockerfile” and a “devcontainer.json” file inside.

files needed for dev containers

Here is my “Dockerfile”:

FROM mcr.microsoft.com/devcontainers/python:1-3.13-bookworm

USER vscode
RUN curl -LsSf https://astral.sh/uv/0.11.1/install.sh | sh -s

ENV PATH="/home/vscode/.local/bin:${PATH}"

And here is the content of “devcontainer.json” (this needs to be adapted to your specific project):

{
  "name": "data-analytics",
  "build": {
    "context": "..",
    "dockerfile": "Dockerfile"
  },
  "customizations": {
    "vscode": {
      "extensions": [
        "ms-python.python",
        "ms-python.debugpy",
        "ms-python.vscode-pylance",
        "ms-python.black-formatter"
      ],
      "settings": {
        "python.defaultInterpreterPath": "${workspaceFolder}/.venv/bin/python",
        "python.testing.pytestEnabled": true,
        "python.testing.unittestEnabled": false,
        "python.testing.pytestArgs": ["tests"],
        "python.testing.pytestPath": "${workspaceFolder}/.venv/bin/pytest",
        "python.terminal.activateEnvironment": true,
        "editor.formatOnSave": true,
        "[python]": {
          "editor.defaultFormatter": "ms-python.black-formatter"
        }
      }
    }
  },
  "postCreateCommand": "uv sync --group dev && uv run pre-commit install",
  "remoteUser": "vscode"
}

The extra cherry on top would of course be to use GitHub Codespaces where you can use a container supplied by GitHub instead of getting the container running on your personal machine! Check out the information on that here: https://github.com/features/codespaces

So now you finally reached the end of this tutorial. Congratulations!

Whoa! “You’re a Python package programmer, Harry!”

The tools we have used here can be recycled in any other of your projects. Here is the link to my full code for this tutorial on GitHub (there is also a presentation of my workshop at the PyCon Austria included in the repo which has more information — and memes!): https://github.com/kolbl/data-analytics