Mastering Ansible Inventories — Part 2: Dynamic Inventories

A practical guide to dynamic inventories in Ansible: using both dynamic inventory scripts and plugins!

Not a Medium member? Read here!

Introduction

Dynamic inventories in Ansible can transform the way you manage and orchestrate hosts by automatically sourcing them from external environments such as cloud providers, databases, container clusters, and other dynamically changing systems.

As I mentioned in the previous part of this series:

Static inventories are defined manually and typically used for small, rarely changing infrastructures, as any updates must be done manually. In contrast, dynamic inventories automatically discover and define Ansible hosts, adapting as your infrastructure evolves, thus offering greater scalability and automating a decent amount of work.

Dynamic inventories automatically adapt to changes in your infrastructure to ensure the targets are always current. More of it, they can handle large-scale environments with thousands of hosts!

There are two primary approaches to leverage dynamic inventories in Ansible:

• Inventory scripts
• Inventory plugins

In this article, we will set up a lab working environment using Docker containers to see these concepts in action. I will provide a step-by-step guide to demonstrate how to utilize dynamic inventories effectively and show how they work. We’ll start by writing an inventory script in Python, and then shift to one of the built-in inventory plugins for comparison.

If you’re eager to learn, follow the lab and repeat the steps on your own. The best way to learn a skill is to practice it in a way that reflects how you intend to use it in the future.

For this lab, I’ve created a GitHub repository containing all the code you’ll need:

• https://github.com/0xtr1gger/Ansible_dynamic_inventories

Let’s dive in!

Dynamic inventory scripts

A dynamic inventory typically consists of a script or program that outputs a JSON-formatted list of target hosts in a way that Ansible understands. Such scripts can be written in any programming language, though Python is one of the most common choices.

Essentially, Ansible inventory scripts (and plugins) do the following:

1. Fetching data: Query data from the source of interest, e.g., Docker containers or cloud instances, to gather information about available hosts.
2. Formatting data: Structure the retrieved data in JSON for Ansible to understand.
3. Returning structured data: Provide organized data about discovered hosts, including host names, IP addresses, statuses, and other relevant details.

To understand how these scripts work, we will do the following:

1. Set up the lab

• We will set up a lab environment comprising several Docker containers acting as managed nodes, and the host machine taking the role of an Ansible controller.

2. Write a Python script for dynamic inventory

• We will craft our own Python script that will discover Docker containers running on the local machine and output information about them in JSON format.

3. Test the inventory

• We will use the ansible-inventory command to see how the dynamic inventory script works, and then execute an ad-hoc ping command across the discovered containers.

Setting up the lab

Before we start, create a project directory where all files will reside:

mkdir Ansible_dynamic_inventories && cd Ansible_dynamic_inventories

Next, we’ll launch several Docker containers for our environment. The first step is to build a custom Docket image based on Ubuntu to streamline the setup process for these containers as Ansible-managed nodes.

Here’s the Dockerfile:

# base image
FROM ubuntu:20.04 

# copying setup script
COPY ./setup.sh /usr/bin/setup.sh

# assigning execute permissions to the script and  running it
RUN chmod +x /usr/bin/setup.sh && /usr/bin/setup.sh

EXPOSE 22

# start SSH service
CMD ["/usr/sbin/sshd", "-D"]

The last command launches the SSH server in the background for the container to accept SSH connections. Above, the COPY command copies the setup.sh script from the host machine into the image, and then the RUNdirective assigns execution permissions to that script and run it.

The setup.sh script contains instructions necessary to configure the node. Here is how it looks like:

apt-get update
apt-get install -y openssh-server # install ssh
service ssh start                 # start SSH daemon
apt-get install -y python3        # install python3
apt-get install -y sudo           # install sudo

# create a runtime directory for sshd
mkdir -p /var/run/sshd

# set up a dedicated user for Ansible automation
useradd -m ansible -s /bin/bash
echo "ansible:ansible" | chpasswd
echo "ansible ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers

# create an SSH directory in /home/ansible
mkdir /home/ansible/.ssh

To build the image, run:

docker build -t ansible_node_image .

After that, launch the first container using the docker run command:

docker run -d --name ansible_node_one ansible_node_image /bin/bash

Ensure the container is up and running with docker ps:

docker ps

We don’t use any orchestration tools, such as Docker Compose, to launch containers for the lab. This is because we might need to frequently delete and recreate containers on the fly to test the dynamic inventory.

Configuring SSH keys

Before moving on, we need to set up SSH keys, which will be necessary to connect from Ansible controller later.

First, on the host machine, generate the keys by executing the following command:

# on Ansible Controller, the host
ssh-keygen -t rsa -b 4096

By default, the keys will be stored in your home directory under ~/.ssh. To allow the Ansible controller to connect to the node through SSH, we need to append the public key (.pub) to the /home/ansible/.ssh/authorized_keys file inside the Docker container.

The quickest way to do that is by using docker exec:

docker exec -it ansible_node_one bash -c "echo $(cat ~/.ssh/id_r  
sa.pub) >> /home/ansible/.ssh/authorized_keys"

If you like, you can verify that the SSH connection is working by simply initiating an SSH session to the Docker container:

ssh ansible@172.17.0.2

To get the IP address of a running Docker container, use the following command:

docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' ansible_node_one

Writing a dynamic inventory script

The next step is to write a dynamic inventory script, in Python.

The script should be able to connect to the local Docker daemon, scan for all currently running containers, and then output the results as a JSON string.

To understand how exactly the JSON-formatted inventory should look like, run the ansible-inventory command against any of your existing static inventories. This will give you an example of the expected format. To see more practical examples with this command, check out the first part of this series.

Here’s the complete script, discovery.py:

# discovery.py

import json   # formatting output
import docker # interacting with Docker API
import socket

def get_docker_containers():
    client = docker.from_env()
    containers = client.containers.list()
    inventory = {'_meta': {'hostvars': {}}}

    for container in containers:
        try:
            # get container's IP address
            container_ip = container.attrs['NetworkSettings']['Networks']['bridge']['IPAddress']
            inventory['all'] = inventory.get('all', {'hosts': []})
            inventory['all']['hosts'].append(container.name)
            inventory['_meta']['hostvars'][container.name] = {
    'ansible_host': container_ip,
    'container_id': container.id,
    'status': container.status,
    'ansible_user': 'ansible',
    'ansible_python_interpreter': '/usr/bin/python3.8'
            }
        except KeyError as e:
            print(f"Skipping container {container.name} due to error: {e}")

    return inventory

def main():
    # get the inventory file
    inventory = get_docker_containers()

    # convert the dictionary to JSON and output
    print(json.dumps(inventory, indent=4))

if __name__ == "__main__":
    main()

Explanations:

The workhorse of the script is the get_socket_containers function, which does the following:

1. Instantiates a client to communicate with the local Docker daemon using [docker.from_env()](https://docker-py.readthedocs.io/en/stable/client.html#docker.client.from_env).
2. Retrieves a list of all currently running containers (client.containers.list()).
3. Initializes a dictionary, inventory, to store information about each container.
4. Iterates through the running containers and captures relevant details such as the name, IP address, and status.
5. Iterates through the running containers and captures relevant details about each of them, including its name, IP address, and state, into the dictionary. The function also specifies two essential per-host inventory variables that should be already familiar to you: ansible_user and ansible_python_interpreter.
6. Returns the dictionary.

Note: Specifying the ansible_user variable is crucial; without it, Ansible will attempt to connect using the username of the controller.

The main() function calls get_docker_containers, takes its output, converts it to a JSON string, and prints it out.

Let’s run this script to see what it does:

python3 discovery.py

All necessary details in JSON format!

Using ansible-inventory and running ad-hoc commands with the dynamic inventory script

It’s time to utilize the dynamic inventory script with Ansible. This is straightforward; you merely need to provide the path to the script using the -i option in Ansible commands.

Before this, we need to do two things:

1. Enable the script inventory plugin in Ansible configuration:

In your Ansible configuration file, /etc/ansible/ansible.cfg, include the following:

[inventory]
enable_plugins = host_list, script, auto, yaml, ini, toml

This will allow Ansible to recognize the inventory generated by the script rather than treating it as an INI file, the default behavior.

1. Assign proper execute permissions to the script

Ensure that the Python script has the proper execute permissions:

chmod +x discovery.py

Now we can make the use of the script. First, let's check how it works with the ansible-inventory command, as we discussed in the previous part:

ansible-inventory -i ./discovery.py --list

In the output, you should see the container we just launched:

Next, let’s try to add some new containers and see how the script reacts. Remember to copy the SSH keys!

docker run -d --name ansible_node_two ansible_node_image && docker exec -it ansible_node_two bash -c "echo $(cat ~/.ssh/id_rsa.pub) >> /home/ansible/.ssh/authorized_keys"

docker run -d --name ansible_node_three ansible_node_image && docker exec -it ansible_node_three bash -c "echo $(cat ~/.ssh/id_rsa.pub) >> /home/ansible/.ssh/authorized_keys"

And then run the script again:

ansible-inventory -i ./discovery.py --list

All newly added containers are instantly recognized!

What happens if we kill one of the containers?

docker kill ansible_node_one

ansible-inventory -i ./discovery.py --list

Perfect! The inventory updates accordingly.

Finally, let’s try to ping our hosts with an ad-hoc command. Like before, provide the path to the script using the -i option. This applies to both ansible and ansible-playbook commands:

ansible -i ./discovery.py all -m ping

We took a look at dynamic inventory scripts and how they work. But do you really want to write a Python script each time? Barely. We should find a more efficient way out.

Dynamic inventory plugins

Dynamic inventory plugins and configurations

Inventory plugins are specialized scripts what enable Ansible to query external sources, such as cloud providers, to dynamically retrieve hosts and their associated variables.

There is a wide range of both built-in and community plugins available that can cater to various needs. Essentially, plugins are pre-built scripts similar to the one we created, but with significantly more extensive functionality.

Before crafting your own script, check for a ready-to-use plugin — you will probably find what you’re looking for. You should be able to find one that meets your needs. Here are some examples of dynamic inventory plugins:

⤬amazon.aws.aws_ec2 - AWS EC2:

• Retrieves EC2 instances directly from your AWS account, with options to filter by region, tags, and instance types.

⤬azure.azcollection.azure_rm - Azure RM (Resource Manager):

• Gathers resources from your Azure cloud, allowing you to manage VMs, databases, and other Azure resources.

⤬google.cloud.gcp_compute - GCP Google Cloud Platform Compute:

• Retrieves instances from GCP projects for resource management.

⤬openstack.cloud.openstack - OpenStack:

• Interfaces with OpenStack environments to gather host information.

⤬community.general.nmap - Nmap:

• Utilizes nmap to automatically discover hosts.

⤬community.docker.docker_containers - Docker:

• Lists Docker containers across a defined set of hosts.

Each plugin contains settings and parameters tailored to the data source it interacts with (e.g., AWS, Azure, Docker). Using and customizing plugins only requires you to create a YAML file outlining the necessary configurations.

To browse all available plugins, use the following command:

ansible-doc -t inventory --list

Partial output:

If a specific plugin is not present on your system, you can install the relevant collection from Ansible Galaxy using:

ansible-galaxy collection install community.docker

Taking advantage of the Docker inventory plugin

Instead of using our own Python script, we can leverage the community.docker.docker_containers dynamic inventory plugin. To make use of it, we will create a YAML configuration file for the inventory plugin.

The simplest setup looks as follows:

# dynamic_inventory.docker.yaml
plugin: community.docker.docker_containers  
docker_host: unix:///var/run/docker.sock

• The plugin directive specifies the complete name of the inventory plugin. Below that, you can define plugin-specific options, such as docker_host.
• The docker_host option specifies where to find the Docker daemon socket to establish a connection. The value set above is the default and refers to the local daemon, but you can change it to a URL pointing to a remote Docker socket instead, such as tcp://192.168.5.11:2376.

As a reference, here’s a snippet from the plugin documentation:

Note: Pay attention to the name of the configuration file; it must end with docker.yaml. Otherwise, the inventory simply won't be parsed and you will receive the following warning: `[WARNING]: Failed to parse /path/to/dynamic_inventory.yaml with auto plugin: inventory source '/path/to/dynamic_inventory.yaml' could not be verified by inventory plugin 'community.docker.docker_containers'`.*

For further details on the options for a specific plugin, use the ansible-doc command. For example, to get documentation of the community.docker.docker_containers plugin, run:

ansible-doc -t inventory community.docker.docker_containers

The above screenshot is taken from the output of this very command.

Applying dynamic inventory plugins

To apply a dynamic inventory plugin, simply provide a path to its YAML configuration within the -i option — similar to how we used our Python script earlier.

First, let’s test it using the ansible-inventory command:

ansible-inventory -i ./dynamic_inventory.docker.yaml --list

Works as expected! Notice that the community plugin gathers more extensive information about each node, far more than our homemade Python script. This highlights the main advantage of plugins: they save time and provide enhanced functionality beyond what one might think to implement.For shorter output, use the — graph option instead of — list:

For shorter output, use the --graph option instead of --list:

Finally, let’s run the beloved ping module on the hosts discovered by this inventory plugin:

It works!

Conclusion

In this guide, we’ve explored both dynamic inventory scripts and plugins in Ansible and learned how to utilize them. When properly set up, they can greatly simplify automation workflows by allowing inventories to dynamically adjust to infrastructure changes, which eliminates a fair amount of manual work.

In the conclusion of the previous part, I asked you a question:

What happens if we define a host in both a custom group and the ungrouped group?

The answer:

The host will be part of both groups. That means it can be targeted using either group name in your playbooks. Say, if you run a playbook targeting the custom group, it will include the host as expected. Conversely, if you target the ungrouped group, the host will also be included. In other works, this host can be referenced from two groups, a custom one and the ungrouped host. However, this is generally not recommended.

Remember to experiment and practice. Happy automating!