Design Patterns in the Python Standard Library

We take a look at the usage of design patterns in a few core Python modules in the standard library.

Design Patterns

Any experienced software engineer who has spent some time in reading the literature of documented “design patterns” would be familiar with the term.

The term was popularized by the well known Gang of four design patterns book — which documents the schema, structure, usage and examples of some 23 object-oriented design patterns.

The “Gang of four” (G4) design patterns

Python — Design Patterns

Many of the “G4” design patterns find their usage in well known Python frameworks and open source code. Moreover, there is a clear usage of some of these patterns in the core Python standard library.

Being a dynamic and high level programming language, Python provides a number of constructs which makes many of the original design patterns kind of obsolete in the way Python solves problems.

In this article — we will mostly cover the creational patterns along with a couple of related structural patterns as a discussion of the entire gamut of design patterns covering the Python standard library would be too big for a single article.

The Singleton

The singleton pattern is a creational design pattern that restricts the instantiation of a class to a singular instance.

Every module in Python provides its own unique namespace which is unique, so a Singleton can be replaced with a well known name or constant or value defined at the top-level in a Python module.

For example here is a “Singleton” configuration value.

# module config
VALUE = 1

# import module config
>>> import config
>>> config.VALUE
1

Every module in Python can be thought of as a singleton since how many times you import it or import it in another name, it is the same module.

>>> import sys
>>> import sys as sysx
>>> sys is sysx
True

In Python, constants and boolean values like None and True/False have similar property. As well as the special Ellipsis type …

>>> n1 = None
>>> n2 = None
>>> n1 is n2
True
>>> b1, b2 = True, True
>>> b1 is b2
True
>>> e1 = ...
>>> e2 = ...
>>> type(e1)
<class 'ellipsis'>
>>> e1 is e2
True

Similarly some of the well known modules, implement Singletons in subtle ways. For example, the logging module.

>>> import logging
>>> log1 = logging.getLogger('app')
>>> log2 = logging.getLogger('app')
>>> log1 is log2
True

Given the same name, it returns the same logging.Logger object. This is an example of a registry based singleton or often called a *Multiton. The logging.getLogger function is a Factory Method which has the additional property of working with a registry of singletons. So this can also be thought of as implementing a Registry* pattern.

You can find this Registry + Singleton + Factory Method pattern repeating naturally in multiple Python modules. For example, the codecs module.

>>> import codecs
>>> c1=codecs.lookup('base64')
>>> c2=codecs.lookup('base64')
>>> c1 is c2
True

Similarly the re module when compiling regular expressions usually returns the same object due to internal caching — this can also be thought of as implementing a similar pattern as above.

>>> import re
>>> email_regex = r"^[\w\.-]+@[\w\.-]+\.\w+$"
>>> e1 = re.compile(email_regex)
>>> e2 = re.compile(email_regex)
>>> e1 is e2
True

You can test this out with larger regular expression strings as well.

It is possible to implement an actual Singleton — in terms of a class with a single instance associated to a single memory location — in Python in multiple ways, but the language makes it unnecessary to implement it in the traditional way. The Singleton problem as expressed in the original G4 book — becomes kind of moot in Python.

In fact it is more common in Python standard library (and external frameworks) to find a combination of Patterns implemented to solve problems than a single one.

The Flyweight

The Flyweight Design Pattern is a structural design pattern used to reduce memory usage by sharing common object data among multiple objects.

Python runtime naturally implements Flyweight as it caches many small integers and most small strings. This is evident from a simple check.

>>> x = 100
>>> y = 100
>>> x is y
True
>>> s1 = "Python"
>>> s2 = "Python"
>>> s1 is s2
True

One can write a small function to see how far this caching of numbers go.

>>> x,y = 0, 0
>>> while True:
...     if x is not y:
...             print(f"not cached {x}")
...             break
...     x += 1
...     y += 1
... 
not cached 257 

NOTE: The actual value where the caching breaks will depend on the Python version and probably your operating system as well.

The sys.intern function is the classic example of a Flyweight factory function in Python stdlib. It creates an internal version of a passed string using a dictionary so that strings with same values always return the same object — thus making lookups faster.

>>> import sys
>>> s1 = "This is a reasonably long string, likely not cached"
>>> s2 = "This is a reasonably long string, likely not cached"
>>> s1 is s2
False

A reasonably long string was not cached as expected, but on interning it,

>>> s1 = sys.intern("This is a reasonably long string, likely not cached")
>>> s2 = sys.intern("This is a reasonably long string, likely not cached")
>>> s1 is s2
True

The earlier example of re.compile can also be thought of as a Flyweight factory function since it uses internal mappings to cache the compiled patterns.

In fact almost any module level function in Python which implements caching with names using registries — usually implemented using dictionaries — can be thought to implement a Registry + Multiton + a minimal Flyweight pattern.

Another name for a cache which stores frequently computed values using well known keys/names is memoization which is a common pattern in Python.

The Facade

The Facade Design Pattern is a structural design pattern that provides a simplified, unified interface to a complex system or library.

A facade provides a single, clean API, hiding internal complexity for clients.

The os module in Python is pretty close to a classic facade. It hides OS and system level complexity across multiple operating systems and provides simple, easy to use functions such as,

>>> import os
>>> os.listdir(".")
['sys_examples.py']
>>> os.stat("sys_examples.py")
os.stat_result(st_mode=33188, st_ino=1184241, st_dev=66312, st_nlink=1, st_uid=1000, st_gid=1000, st_size
=304, st_atime=1779697295, st_mtime=1779697475, st_ctime=1779697475)
>>> os.chmod('sys_examples.py', 0o644)

Internally it uses delegation to make calls to the correct OS/system specific backend module for the function implementations like posix for nix systems and nt *for Windows for example.

Another facade is implemented by the pathlib module which facades over os module with the additional ability to make system calls to provide a layered facade stack.

>>> from pathlib import Path
>>> list(Path('.').glob('*'))[0]
PosixPath('sys_examples.py')
>>> p = _
>>> p.stat()
os.stat_result(st_mode=33188, st_ino=1184241, st_dev=66312, st_nlink=1, st_uid=1000, st_gid=1000, st_size
=304, st_atime=1779711983, st_mtime=1779711983, st_ctime=1779711985)
>>> p.chmod(0o644)

The shutil module can also be thought of as implementing a Facade — this time mimicking shell utilities like copy, move, rename or removal of files and folders.

The Factory Method (Function) / AbstractFactory

The Factory method pattern is a design pattern that uses factory methods to deal with the problem of creating objects without having to specify their exact classes.

A related pattern is the Abstract Factory which provides an interface for creating families of related or dependent objects without specifying their concrete classes.

Python stdlib modules implements a variation of this — which can be called as “Function as factory” — this is so common that we don’t even feel these as factory functions.

We have already seen examples of this above.

The pathlib.Path can be thought of as a function as factory.

>>> from pathlib import Path
>>> p = Path("file.txt")

Depending on the OS, this returns either a PosixPath or a WindowsPath or some other “OSPath”. The function can be thought of as a constructor returning different subclass instances.

We already encountered re.compile and logging.getLogger and codecs.lookup so not revisiting them.

Classmethod Factories

Many types in Python including the dict type most famously, implements methods which returns new instances of the type.

For example constructing a new dictionary from keys of another dictionary with default values (None) using dict.fromkeys .

>>> d={x: x+1 for x in range(5)}
>>> dict.fromkeys(d)
{0: None, 1: None, 2: None, 3: None, 4: None}

A similar method exists for collections.OrderedDict.

>>> from collections import OrderedDict
>>> OrderedDict.fromkeys(d)
OrderedDict([(0, None), (1, None), (2, None), (3, None), (4, None)])

Here are similar methods for some other well-known types:

>>> int.from_bytes(b"\x00\x10", "big")
16
>>> int.from_bytes(b"\x00\x10", "little")
4096
>>> bytes.fromhex("ffaa00")
b'\xff\xaa\x00'

Class Factories

Class factories are a powerful construct in Python starting with the most basic type which provides powerful meta-programming constructs.

However this pattern can be thought of as closer to AbstractFactory or a universal Meta Factory since it can be used to create any class — which is way more flexible than the related family of products (classes) as defined for the AbstractFactory pattern.

So the type goes one level deeper — since it is a ClassFactory — creating classes — whose output can then be used to create instances of the classes.

>>> Point = type("Point", (), {'x': 0.0, 'y': 0.0})
>>> Point()
<__main__.Point object at 0x7f0710314650>

Another example for class factories in a similar vein is the namedtuple of the collections module — which allows to define custom classes in a similar but in a very different way.

>>> from collections import NamedTuple
>>> Point=namedtuple("Point", field_names=('x', 'y'), defaults=(0.0, 0.0))
>>> Point()
Point(x=0.0, y=0.0)

NOTE: The typing.NamedTuple provides a more flexible way for creating namedtuples than the original collections.namedtuple.

The Builder

The builder pattern is a creational pattern that separates the construction of a complex object from its representation — so as to allow similar interfaces to build slightly different objects.

The email module in Python is the closest to a true builder pattern example. In this module, an email message is built step by step.

from email.message import EmailMessage

msg = EmailMessage()
msg["From"] = "hello@example.com"
msg["To"] = "person@example.com"
msg["Subject"] = "Hello"
msg.set_content("Hi there")

One constructs an email message step by step — first creating the object and then building parts of it. The construction is separated from the representation (fields) and the order of building the fields (parts) doesn’t matter.

The argparse module can be thought of as a declarative builder. The original argument parser itself is constructed first and then parts (arguments) are attached to it step by step later.

import argparse

parser = argparse.ArgumentParser(description="Parse a person")
parser.add_argument("--name", required=True)
parser.add_argument("--age", type=int)
parser.add_argument("--gender")
args = parser.parse_args()

There are also builders that are inspired by Java in Python standard library — for example the io.StringIO class, which allows to build long strings piece by piece and to finally fetch the built object using a method.

from io import StringIO

buf = StringIO()
buf.write("Welcome to the world.")
buf.write("This article talks about Python")
buf.write("It also talks about design patterns")
buf.write("...")
...

result = buf.getvalue()

The Prototype

The Prototype pattern is a creational design pattern that allows for creating new objects by copying an existing instance, known as a prototype.

Python’s way of prototyping or copying objects is to provide the support for these operations in a special copy module and to provide special dunder methods in classes namely copy and deepcopy to override these operations.

Also most mutable Python data structures support a copy method.

>>> l = list(range(5))
>>> l.copy()
[0, 1, 2, 3, 4]
>>> d = {x:x+1 for x in range(5)}
>>> d.copy()
{0: 1, 1: 2, 2: 3, 3: 4, 4: 5}
>>> s=set(l)
>>> s.copy()
{0, 1, 2, 3, 4}

The dataclasses module provides a powerful way to create cloned objects with slightly different values using its replace API.

from dataclasses import dataclass, replace

@dataclass
class Person:
    name: str
    age: int
    job: str

>>> p1 = Person("Anand", 50, "Software Architect")
>>> p2 = replace(p1, name="Pranav", age=20, job="Engineering Student")
>>> print(p1)
Person(name='Anand', age=50, job='Software Architect')
>>> print(p2)
Person(name='Pranav', age=20, job='Engineering Student')

As you can see, this is more powerful and can be absorbed into dynamic configurations using dictionaries.

For example, one can create a basic Person instance from the dataclass and then clone it using replace by reading the configuration from say a CSV file or Pandas dataframe and creating copies like illustrated above.

Summary

In this article, we discussed all creational G4 design patterns and a few related structural design patterns with actual practical examples of their usage in modules from the Python standard library.