Architecting for cloud in a regulated industry

By Nicholas Chu, Domain Architect, Commodities and Global Markets Technology

Introduction

As a domain architect in the Commodities and Global Markets (CGM) technology team at Macquarie Group, a key focus of my role is to enact Macquarie’s cloud-first strategy when deploying new workloads, or migrating those that are existing while offering timely architectural decisions. The cloud allows us to design our technology solutions for the long-term and enables our businesses to scale with the demands of the market and regulation.

Architects at Macquarie are empowered to collaborate across teams, own our architectural designs end-to-end, and manage our systems in a unified representation that encapsulates both business and technical architecture.

In this post, I will detail a major migration critical for our business, challenges associated with the migration, how we overcame these and what we have in store next.

The problem statement

Within Macquarie’s Market Operations Division (MOD), we need a scalable application with the capability to route multiple value, high volumes of global payments, both incoming and outgoing in multiple currencies. It needs to have high availability and resiliency to meet our business, client and financial obligations. Earlier this year, we set out to migrate an existing application to AWS cloud to meet these needs.

When embarking on this project, we faced three distinct challenges that required architectural commitments and engineering implementation across different teams. These were:

External Challenges

· The application team must represent the current state 24x6 availability of the routing of payments for our internal and external customers

· The On-Premises Data Centre is scheduled for a hard date demolition

Application Challenges

· We needed to make the application cloud-native, ensuring it meets new payment standards. The deprecation of the application was imminent due to version and operating system specification and the adoption of new international payment standards.

Cloud Challenges

· Macquarie’s Cloud Control Plane observes AWS infrastructure as immutable as per AWS’ Well-Architected Framework. As such, this required transformation and modernisation of the vendor application to operate and function within those principles.

· Application of zero trust security principles is required upon any workload being deployed to the cloud through Macquarie’s Cloud Control Plane.

How we executed

Over the course of 12 months, we worked through architectural designs and decisions to address the above challenges. Our team of application engineers, cloud platform engineers, database admins, security analysts, solution architects and the Commodities and Global Markets business coalesced around the Agile methodology — a project management approach that involves breaking the project into phases and focuses on continuous collaboration and improvement. We identified the following major phases or ‘epics’ that covered automation, transformation, deployment and testing.

Within these epics, our talented team of engineers were empowered and accountable for driving a non-cloud native vendor application into an automated environment, removing the need for any manual operator intervention.

What we believed was at first a ‘lift and shift’ became an exercise of re-engineering and modernising. We used Infrastructure as Code and pipelines with the intent of immutable deployments on tearing-down and re-deploying the typical three-tiered application that consisted of web hosts, application hosts and databases. This pattern required the development of core scaffolding for the platform, which consisted of:

1. Leveraging repositories for code, configuration and binaries through Git;
2. Leveraging a code build pipeline for CI/CD through Bamboo and our Macquarie Cloud Control Plane;
3. Leveraging automation to deploy the application and environment through calls to YAML files in our Git repos;
4. Leveraging persistent AWS native file systems to store application state and configuration via EFS and FSx for Windows;
5. Applying application and infrastructure health checks for resilience through programmatic policies;
6. Applying security zero trust principles at the core of our code base through certificate issuing and rotation, authentication and authorisation policies, secrets and key material rotation and IAM policies.

Outcomes and what’s next

Through the engineering dedication of our teams and trust from our CGM technology leaders over the course of 12 months, we were able to successfully cutover our payments and routing platform into AWS in mid-April of this year. Since then, this critical workload has been operational on the AWS cloud and has successfully gone through our scheduled stack roll process. There has been no rollback or major issues since the migration to AWS, which is a testament to the collaborative effort of our teams and the result of considerable building and testing in a robust environment.

We are now in the process of reviewing the environment from an optimisation perspective with our FinOps team to enhance our compute, messaging volumes and database capacity. We are also exploring a migration of our EC2 instances to managed containers and targeting the use of AWS’ Transfer Family service as a replacement of our File Transfer work via our next generation cloud control plane (as described by my colleague Pranita Praveen here). This is a secure and AWS-managed service that enables us to move files into our application that further reinforces our architectural pillars.

This has laid the path for subsequent migrations, with each enabling our businesses to scale and operate with the speed and agility that the cloud provides.