Staging Munki Updates with AutoPkg

An introduction to our new MunkiAutoStaging processor

How to automatically promote software updates in your Munki repository from testing to production with the AutoPkg processor **MunkiAutoStaging**.

AutoPkg is an indispensable tool for us Mac Admins, especially if you want to manage the Apple devices of numerous companies as a consultant. At wycomco, we have been using AutoPkg for several years to maintain our clients’ Munki repositories. Since 2020, we have also been providing our own recipe repository, in which we offer recipes that our customers need but that are not yet found in other recipe repos.

Background: What is AutoPkg?

The project’s maintainers answer this question at Github as follows:

AutoPkg is an automation framework for macOS software packaging and distribution, oriented towards the tasks one would normally perform manually to prepare third-party software for mass deployment to managed clients.

AutoPkg can be used to define workflows that are necessary for maintaining a software repository. Typically, these workflows include the following steps:

• Checking whether new app versions are available
• Downloading new packages
• Retrieving relevant meta-information
• Repackaging the software — if necessary
• Importing the packages into the software repository.

Simple AutoPkg run to import Firefox to a Munki repository

Thanks to the excellent Mac Admin community, there are workflows for many software titles, and the number is growing. These workflows, called “recipes”, can be run manually and the newly imported packages can be tested extensively and then deployed to the managed devices.

However, in my opinion, a good admin is a lazy admin, he should not be willing to repeat the same manual procedures over and over again like on an assembly line. At least if those procedures can be automated. That is also the reason why AutoPkg was developed in the first place. And wouldn’t it be a shame if executing AutoPkg itself was not automated as well?

Many admins certainly already do this: with tools like AutoPkgr or launchd, AutoPkg workflows can be easily automated, so that the admin has little or no effort in maintaining his repository. However, this also comes with risks.

Dealing with Software Updates

It happens from time to time that new versions of common software titles are buggy, although this is becoming rarer and rarer. Of course, this is not nice, but it can rarely be prevented. After all, software developers are only humans, and humans make mistakes.

And even if the software is supposedly error-free, it may no longer be compatible with other systems and applications in place. In many companies, the complexity of IT systems has increased to such an extent that individual dependencies can hardly be kept track of. Updating a browser could lead to employees no longer being able to access their ERP application because it may rely on functions that have long been deprecated and are therefore carelessly switched off by the browser developers.

We as admins have to fulfil our duty to minimise risks. In regards to software updates, there are ultimately two ways to do this:

1. Manually testing of updates
2. Let others test the updates

Manual Testing of Updates

In the past, this was very common: we install the latest app updates on our test devices and check all workflows — ideally with a comprehensive, constantly growing checklist. This still works today if at least the following requirements are met:

• We know all the workflows of our users.
• We have enough time to test everything.

The larger a company is, the more diverse and complex the workflows are. If I, as an administrator, am responsible for many different companies or departments, it quickly becomes impossible to know all the workflows sufficiently well. Sure, I can launch an application and see if it starts and if a new document can be created and saved. But whether the rendering process for a 3D model, which is processed via different nodes of a render farm, has problems with externally integrated textures, is something I as an IT admin actually don’t need to know. By contrast, users who have to deal with such tasks on a daily basis usually recognise these errors very quickly.

And on the second point: which admin answers ‘’yes’’ to the question whether he has enough time? 🤯

Let others test the Updates

Therefore, it is a logical step to delegate the testing of updates to some of the power users of a company or department. If each of those people can focus on his or her own, well-known processes, there is no need for a detailed testing checklist.

Of course, the respective employees should be informed in advance that they will be testing new software updates. This is the only way to raise their awareness of recognising problems and reporting them at an early stage.

Introducing: MunkiAutoStaging Processor

If everyone involved is in the know, we can also automate this test procedure. For this purpose, we at wycomco have developed our own AutoPkg Processor, which ensures that critical updates are kept in an explicit staging catalog for a certain period of time: MunkiAutoStaging.

How it works

This processor can be used in an AutoPkg recipe to automatically move a Munki package from a staging catalog to a production catalog after a certain period of time. To do this, you need to add the wycomco-recipes repository to the AutoPkg search path and reference the Processor in your Recipe using its name “ com.github.wycomco.SharedProcessors/MunkiAutoStaging”:

[embed]Add wycomco’s recipe repo

[embed]How to reference MunkiAutoStaging in an AutoPkg recipe

When the MunkiAutoStaging processor is run, it checks for all packages with the referenced name to determine when they were imported into the repository. For this, it looks at the creation_date of the _metadata dictionary, which is automatically set by munkiimport. If the package is still in the staging catalog, but the creation_date is older than the MUNKI_STAGING_DAYS, the package is automatically moved to the production catalog. For documentation purposes, the current timestamp is stored as promotion_date, as is the local user name under promoted_by:

[embed]Sample _metadata dictionary from a pkginfo file after automatically promoting a package to production

Configuration

The following input variables can be used to tailor the behaviour of the processor to the individual requirements for each app:

• MUNKI_STAGING_CATALOG

This is the name of the staging catalog. If it is not set up, AutoPkg will use the usual “testing” catalogue in the Munki universe. Since the processor always refers to a single Munki package, you can also name a different catalogue for each individual package, and thus divide the tests according to individual apps or app groups, for example via the catalogue “TestingFirefox” or “TestingAdobeApps”.

• MUNKI_PRODUCTION_CATALOG

This specifies the name of the catalog in which the “stable”, i.e. already tested, packages are kept. If nothing is defined here, AutoPkg uses the common “production” catalog.

• MUNKI_STAGING_DAYS

With this variable you can define how long a package should remain in the staging catalog after it has been imported into the munki repository. For some, less critical applications, you can specify low values like “1”, for more complex applications higher values like “7” or “14” are suitable.

Depending on how often AutoPkg is run, some fine tuning may be worthwhile. For example, if you run AutoPkg on a daily basis, you should perhaps enter a value of “0.9” for MUNKI_STAGING_DAYS instead of “1”. Depending on the runtime of the job, it could otherwise happen that the check takes place a few minutes before the actual grace period ends and the package is not moved to the production Catalog, even though it has already been in the repository for almost(!) a day.

Usage as part of an Override

Processors are usually used as part of a recipe. Now it would not be very practical if all recipe maintainers would add our new processor to their Munki recipes — after all, not every admin necessarily wants to implement such a staging system. Since AutoPkg 1.0, however, it is expected that one creates a so-called override for each recipe to be used. Overrides are basically just recipes and can therefore also include other processors. Since our MunkiAutoStaging Processor is usually executed after the Munki recipe, it is to be used at the very end of a workflow anyway.

So — if we want to implement the staging system as described — we can simply add another Processor step in our overrides.

Please make sure to set the catalog inside your (probably customised) pkgsinfo input variable to reflect the value of the MUNKI_STAGING_CATALOG variable. You, of course, may use the input variable itself:

[embed]Example Override — without ParentRecipeTrustInfo

Finally, after we have defined the input variables we want, the Processor takes over with each subsequent AutoPkg run:

Importing Microsoft Excel into the staging catalog “testing”

Automatically promoting Excel to production catalog after pre-defined amount of days

Caveats

Currently, there are two minor hurdles to using the MunkiAutoStaging Processor, but they are negligible, at least for our practice:

MakeCatalogs.munki

Usually, the last step in an AutoPkg run is to execute the recipe MakeCatalogs.munki. This checks whether a package was imported into the repository in the current run and then rebuilds the catalogs. As of today, the processor checks whether the output variable “pkginfo_repo_path” has been set by a processor, which is the case, for example, with the MunkiImport Processor. Our processor, however, sets the variable “munki_repo_changed” instead. This causes the catalogs not to be recreated unless a new package has also been imported into the repository.

We have already created a pull request that can solve this problem.

For us, this limitation is negligible because we also create the catalogs at a different time.

Parent Trust Info

If our processor is included in an override, AutoPkg does not check the validity of the processor. The necessary parent trust information is only generated for the parent recipes of the override and stored in the override. So if our processor is compromised — in our publicly available Github repository or locally on the system running AutoPkg — this otherwise very useful check can be completely bypassed.

Of course, since we trust our work and can ensure that no one makes changes in our repository, this is negligible for us.

Feedback

We love to hear feedback from you about this article and, of course, if you use the MunkiAutoStaging processor yourself, we’d love to hear about your experiences.