Heralding a digital era for our annual staff review platform

By Rakesh Goel, Director of Engineering, Enterprise Technology

Overview

At Macquarie, our greatest asset is our people, and we are committed to ensuring the systems and processes we use each day enable a seamless employee experience.

A key focus of my role is on digitalisation to transform our platforms and processes to be agile and future-ready.

Recently, we transformed our global annual staff review platform from a legacy system reliant on some manual processes to a seamless digital experience for our thousands of people managers worldwide. This platform needed to handle high-use periods and multiple users simultaneously.

Our new rewards platform provides access to real-time data, enables optimal collaboration, and enhances the efficiency of our annual review process.

Team structure a recipe for success

Our project team across HR and Technology brought together diverse expertise to develop an intuitive, user-centric, and scalable digital platform. Our new rewards platform provides access to real-time data, enables optimal collaboration, and enhances the efficiency of our annual review process.

Our North Star was to deliver a seamless digital, real-time data-driven reward experience for our people, that is not only performant but also a cost effective, future-proof tech system. We were guided by five key principles:

Human centric — Focus on simplicity as a fundamental aspect in our design process.

Data-driven — Leverage data to create automated and actionable insights within our technology landscape.

Integrated — Support a modular enterprise with extensive integration options including with Finance, both inbound and outbound.

Automated — Enable straight-through processing and automation across our processes, minimise manual touchpoints, and leverage automation for speed to value.

Secure — Ensuring the security of our data and adhering to regulatory compliance standards is always paramount.

We worked across multidisciplinary groups, combining expertise from HR, HR Technology, Enterprise Technology, and Digital. We maintained effective communication through a change management approach, catering to the unique needs of our diverse business groups globally.

From a design perspective, we adopted a simplified and unified User Interface and User Experience approach, creating continuity across our Macquarie platforms.

Application architecture and agile culture: how we solved the problem

The core priorities for our team were:

· Technology selection — For APIs, frontend, database and whole infra.

· Application Deployment (GitOps Automation Components) — To deploy the platform with bootstrap that prepares code for services.

· Platform Deployment — To build the platform first using Macquarie’s inhouse secure Infrastructure as Code (IaC) tool.

· Managing infrastructure budget — Building real-time cloud platform with minimum running cost.

Technology selection

We designed a cloud agnostic solution with the initial build for AWS cloud. Our approach for three-tier architecture consisted of:

• Application tier — We chose NodeJS using typescript for our containerised microservices over building them in Java. This was the latest technology and provided better support for docker based development.
• Presentation tier — We built our frontend with ReactJS using a micro-frontend approach to make parallel development possible in our large engineering team.
• Data tier — We also opted to leverage our core database power over our lucrative design pattern where every microservice can have its own database. We chose PostgreSQL as our core database. This helped to abstract heavy concurrent calculations and data manipulations in the database and made our APIs lightweight. Later on in the cycle, it helped independently improve the performance on struggling complex and bulky components catering to heavy calculations.

Infrastructure enablers

Leveraging Kubernetes to empower scalability We used Kubernetes (EKS on AWS) for our end-to-end solution to address our functional and non-functional requirements.

Amazon Elastic Kubernetes Service (EKS) is a managed Kubernetes service that simplifies the deployment, management, and scaling of containerised applications using Kubernetes on AWS. By using this technology, we were able to unlock the benefits of selective scalability, cost effectiveness, auto-healing and security.

High-level interpretation of our Rewards platform

To build a scalable EKS implementation, it was essential for us to consider best practices around infrastructure setup, security, monitoring, and scaling mechanisms.

The EKS GitOps scope of work attempts to solve these problems which our developers faced when working with microservices using EKS on AWS.

1. Cluster setup We began by creating an EKS cluster using Macquarie’s proprietary IaC tool, ensuring it guarantees high availability and fault tolerance.

2. Node management Utilising Amazon EKS managed node groups for simplified node lifecycle management, including Zero Outage Upgrades, scaling, and health monitoring, allowed for easy integration with AWS Auto Scaling. For non-prod environments and selective prod areas we also used cost-effective Spot Instances with managed node groups, as well as a diversified Spot Fleet to reduce the risk of interruptions.

3. Scaling strategies To ensure scalability across our large organisation, we implemented Horizontal Pod Autoscaling (HPA) to automatically adjust the number of pod replicas based on CPU and memory usage. Some selected services needed high demand hence the separate scaling configurations.

We deployed the Kubernetes Cluster Autoscaler to automatically adjust the number of nodes in our EKS cluster, based on the resource requirements of our pods. The Cluster Autoscaler integrates seamlessly with EKS and works with both On-Demand and Spot Instances.

Using AWS Auto Scaling groups with dynamic scaling policies ensured that our nodes scaled based on actual demand. We then integrated these groups with CloudWatch alarms to trigger scaling events.

4. Security approach using best practices Underpinning our approach was our strong risk mindset. This was exemplified through the below considerations:

— IAM Roles and Policies: We used IAM roles for service accounts (IRSA) to grant least-privilege permissions to our Kubernetes pods, tailoring permissions to specific needs.

— Network Policies: We implemented Kubernetes network policies to control the flow of traffic between pods and external services. This helped to enforce security boundaries within the cluster.

5. Monitoring and logging To enable comprehensive monitoring of our EKS clusters, we used CloudWatch Container Insights, which provided metrics, logs, and insights into our container workloads and the underlying infrastructure.

— Centralised Logging: We ported all the logs — using Fluentd service to collect and forward to centralised logging tool — Splunk. — Prometheus and Grafana: Then we used Grafana for in-depth monitoring and custom dashboards. We also used Prometheus adapter to expose custom metrics for Horizontal Pod Autoscaling.

6. CI/CD Integration: To ensure our application was continuously deployed with minimal manual intervention, we integrated our EKS cluster with Macquarie’s internal tool that is based on CloudFormation for Infra deployment.

We used GitOps delivery tool ArgoCD for application deployments. These tools synchronise the desired state defined in a Git repository with the actual state of your Kubernetes clusters.

7. Cost optimisation: We continuously monitored and right-sized our workloads to avoid over-provisioning resources. We used AWS Cost Explorer to identify cost-saving opportunities.

We also maximised the use of Spot Instances for all non-prod environments and non-critical and stateless workloads to significantly reduce costs.

8. Disaster recovery and high availability: To ensure high availability and fault tolerance, we deployed our EKS nodes across multiple Availability Zones.

We used RDS snapshot for DB backups ensuring that backups are stored securely and can be restored quickly in case of failure.

A scalable EKS implementation on AWS requires careful planning and adherence to best practices across networking, security, scaling, and monitoring. By leveraging AWS’s managed services and Kubernetes features, we were able to build an EKS environment that met our applications’ performance, availability, and cost requirements while maintaining the flexibility and scalability inherent to Kubernetes.

Outcomes and what’s next: A new, digital era for our Reward system

Through leveraging the benefits of a strong risk mindset, multidisciplinary teams and with scalability and cost optimisation at the forefront, we were able to successfully implement our Reward Experience (RX) platform for thousands of people managers around the world.

Using the latest cloud technologies, we have built a future-proof, sustainable platform that provides an intuitive customer experience based on real-time data for quick and effective decision making. This resilient platform can scale up to cater for our global users during peak periods, mitigated our operational risk by replacing legacy spreadsheet-based tools and automating manual processes and controls.

Managers now receive key data and insights embedded in the platform, creating a seamless digital, data-driven reward experience for our people and ultimately advancing our digitalisation ambitions one solution at a time.

If you’re interested in finding out more about working in Technology at Macquarie, take a look at our careers page: Careers | Macquarie Group