Data Engineering and ML Platform — Part 1 — Platform Architecture

Platform Architecture, Codebase, and Interesting Features

Image Credits: Pradnyesh Mali

Introduction

Welcome to our comprehensive blog series that revolves around how we developed a data engineering and ML platform named Xpower-Boost for Xcellen PTE Ltd. We go through the why and how of the platform with in-depth technical details, and the challenges we faced while developing it. This series is divided into 3 parts, one related to the platform architecture (which you are reading), one related to the codebase, and lastly related to some of the interesting features we had to develop and how we tackled them in a reusable manner.

We intend to discuss some of our technical decisions and the positive and negative impact they had on the development of the project. There is one more area that generally does not come to light and is underrated, and that is the technical decisions that impact the speed of development and the efficiency of a development team, the QA team, and the product owner to work together cohesively.

Problem Statement

The platform is for business analysts with a varying level of data science knowledge where operations on tabular data ranging from the shape of (12 rows x 3 columns) to (70L rows x 60 columns) can be performed and analysis can be done including machine learning to take business decisions for commercial excellence in the pharmaceutical domain. The platform provides this in a non-complex way for the masses while also allowing for inherently complex operations. First, a solid foundation was built before adding domain-specific features,

• The product was developed as a generalized data engineering and ML platform that would enable the user to “enrich” the data.
• The user enriches the data as per requirements to transform it into a state that in itself might be the final output or might be further used for analysis.
• If required, users can train machine learning models such as classification, regression, and clustering.
• Users can understand the models utilizing both simple and easy-to-understand results as well as complex results which provide deeper insights some of which are powered by explainable AI.
• Further users should be able to predict the target classes, and target value on new data using the trained models.
• Later on, domain-specific requirements were addressed by developing features that used around 80% of the existing features.

Unlike general data engineering where tasks are often run in batches, on a set schedule, or triggered by an event with results available later, this platform allows users to interactively perform tasks one at a time and get immediate results.

The main challenge was that the generalized features required kept on evolving and when they were developed since the domain-specific requirements were not clearly defined as providing flexible functionality to the user was the goal, it was critical to develop the features in an extensible and reusable manner.

Also, the platform was not meant to consist of a fixed set of user flows with a defined start and end. Any point can be an exit point and there can be multiple entry points. Being a platform there was no fixed data schema that the uploaded CSV / Excel files would adhere to.

Table Of Contents

Feel free to jump around.

• Frontend Architecture
• Backend Architecture
• ML Worker Architecture
• OnDemand Server: A Novel Concept
• Data Storage — S3 and Postgres
• EFS
• CI / CD
• VPC

Platform Architecture

Let’s discuss about the platform architecture in which we talk about the various parts of the architecture

Codebase Components

Let’s start with the primary components that execute the codebase. They consist of frontend, backend, on-demand backend, and ML worker.

Frontend

In developing our platform, we chose Next.js with Static Site Generation (SSG) mode, primarily focusing on handling dynamic user interactions. Here’s an overview of how we shaped our frontend architecture and hosting choices:

1. Next.js with Static Site Generation (SSG) along with typescript: => Maintainable and Scalable: The choice of Next.js in SSG mode was driven by our platform’s need to handle dynamic interactions in a maintainable way with separation of concerns. Using typescript was an important decision for codebase readability. => Reusable HTML Pages: This approach generates HTML pages that can be reused across requests. => Client-Side Rendering Post-Load: Once these pages load in the user’s browser, client-side rendering takes over by rendering the data based on API responses.
2. Initial Hosting and Deployment Strategy: => AWS S3 for Static Site Hosting: Given that server-side rendering was not necessary for our platform, we chose AWS S3 for its efficiency in static site hosting. => AWS CloudFront as CDN: To ensure fast content delivery, AWS CloudFront was implemented as our Content Delivery Network (CDN). => Cost-Effectiveness and Resource Optimization: These choices were made to build a cost-effective platform and to avoid the use of unnecessary resources such as AWS EC2 machines.
3. Development and Deployment Integration: => Initial Serverless SDK Plugin Integration: In our GitHub actions, we initially integrated a serverless SDK plugin to manage the build and deployment process. =>Direct Route Access with Serverless SDK Plugin: This integration was also crucial for enabling direct route access due to incompatibilities in our codebase build configuration with AWS S3’s static site hosting features. => Transition to AWS Amplify: After facing reliability issues with the serverless plugin, we switched to AWS Amplify. This platform simplifies the build and deployment process and continues to use AWS Cloudfront for CDN, ensuring no compromise on performance.

*Jump to the table of contents*

Backend

In our journey to develop a platform specializing in operations on tabular data, we went ahead with Python for the development flexibility it offers along with the ocean of libraries supported by it for data science. To build our API, we chose Django REST framework. This decision was based on a logical assessment of our needs:

1. Efficiency in Development: Django REST framework offers a powerful and flexible toolkit for building Web APIs which has all batteries included such as authentication, middlewares, CRON jobs, Django ORM, and the built-in Django Admin panel for debugging.
2. Scalability and Maintainability: This framework supports our need for a scalable solution that can grow with our platform while ensuring ease of maintenance.

For deployment, we adopted a containerization strategy. Here’s why:

1. Consistent and Reliable Environment: By dockerizing our code, we ensured that our application runs consistently across different development environments while also retaining the ability to host our solution at a different place than AWS.
2. Simplified Deployment and Scaling: The use of Docker containers, deployed via AWS Elastic Container Service (ECS), simplified the deployment process. AWS ECS offers a scalable, high-performance container orchestration service that supports Docker containers and allows us to easily run and scale containerized applications on AWS. AWS ECS was used in EC2 mode and not via Fargate as the latter was costly. Also, AWS EKS was simply too complicated and not needed as well for our requirements.

*Jump to the table of contents*

ML Worker

To optimize the utilization of resources for our system, especially given the anticipated usage patterns and the frequency of machine learning (ML) model training by users, we opted against maintaining a standby fleet of ML workers. This decision was informed by a thorough evaluation of cost-effectiveness and efficiency.

Our chosen strategy involves dynamically launching an Amazon EC2 instance each time there is a need to train an ML model. This approach is underpinned by several key decisions and configurations:

1. EC2 with User Data Script: Each launched EC2 instance is equipped with a user data script. This script enables the instance to automatically associate itself with an AWS Elastic Container Service (ECS) Cluster upon launch.
2. ECS Cluster with Daemon Service: We configured our ECS Cluster with a service of the ‘Daemon’ type. To understand this choice, it’s essential to know the two types of services in AWS ECS: ‘Replica’ and ‘Daemon’. => Replica Service: This maintains a fixed number of tasks within the cluster. => Daemon Service: In contrast, the Daemon service runs a single task on each machine associated with the cluster.
3. Deployment of ML Worker as a Dockerized Task: When an EC2 instance joins the cluster, it triggers the execution of our ML code, which is designed as a dockerized Python codebase. This setup allows the codebase to function as a standalone worker within the ECS task.
4. Self-Termination of ML Worker: Upon completion of its task, the ML worker is programmed to self-terminate. It achieves this by first fetching its instance ID from AWS’s private network address (http://169.254.169.254/latest/meta-data/instance-id). It then uses the AWS SDK for Python (Boto3) to terminate itself by passing this instance ID in the terminate ec2 method of the ec2 client.

This architecture ensures that we only use resources when necessary, leading to a more cost-efficient system. It also offers the flexibility to spin up resources of different configurations if need be depending upon the data size and the time in which the user needs the results.

Image by Author

In addressing the communication needs between our backend and the machine learning (ML) worker, we prioritized asynchronous interaction. This decision was largely influenced by the ML worker’s initialization time, approximately 2 minutes and 30 seconds, during which it is not feasible to keep the user waiting. To facilitate this asynchronous communication, we implemented a queue-based approach, using Amazon Simple Queue Service (SQS) and AWS Lambda functions. Here’s how we structured this system:

1. Use of AWS SQS FIFO Queue: Our backend prepares a message in JSON format and publishes it to an AWS SQS FIFO (First-In-First-Out) queue. This choice ensures that messages are processed in the exact order they are sent.
2. Polling for Messages: The ML worker once it’s ready polls for a single message which is a list of tasks along with the entire job parameters.
3. Data Storage in AWS S3: The actual data that the ML worker processes is stored in AWS S3, a detail we will discuss later.
4. Intermediate Results Communication: For updating progress and sending intermediate results back to the backend, the ML worker publishes another JSON message into a separate AWS SQS queue. This approach, as opposed to direct API calls, ensures reliability as explained below.
5. Integration with AWS Lambda: The queue for intermediate results is configured with an AWS Lambda function. This function is triggered when a new message is available in the queue and is responsible for making the API call to the backend.
6. Handling Backend Availability Issues: To mitigate the risk of losing results due to backend downtime, the Lambda function only deletes the message from the queue after receiving a success status code from the backend. If the backend is unavailable, the message remains in the queue, and the Lambda function is triggered again after a set duration.
7. Configurable Visibility Timeout: The duration after which the Lambda function retries can be adjusted through the visibility timeout setting of the SQS queue. This flexibility allows us to retry communicating with the backend whenever there is a downtime.

By implementing this architecture, we ensure that the communication between our backend and ML worker is not only asynchronous but also robust and failsafe.

*Jump to the table of contents*

OnDemand Server: A Novel Concept

To manage the challenges posed by compute and memory-intensive operations and the potential risk of backend downtime due to multi-user concurrency, a decision was made to provide users with the option to launch a dedicated server. This server would exclusively host the backend code for the user, enhancing both performance and reliability. We implemented this solution using AWS EC2 and an on-demand cluster but faced a significant challenge in routing frontend requests to these dynamically launched servers. Here’s how we addressed this challenge:

1. Use of AWS EC2 for On-Demand Servers: Upon user request, an AWS EC2 instance is launched. This instance is designed to automatically associate itself with an on-demand ECS cluster, via the user data script that is executed during its initialization.
2. Dynamic Public IP Addressing: Each newly launched EC2 machine receives a random public IP address. This dynamic allocation of IP addresses presented a hurdle for implementing SSL certificates directly on these EC2 instances. Additionally, to avoid the costs associated with maintaining multiple Elastic IPs, we needed an alternative solution.
3. The Need for an Intermediary: The frontend could not directly communicate with the on-demand server as the server cannot be associated with an SSL certificate dynamically. So we needed an intermediary server between frontend and the on-demand server.
4. Adopting Nginx as a Dumb Forwarder: We opted to use Nginx, a powerful reverse proxy, uniquely. We refer to Nginx as a “dumb forwarder” in our setup, and here’s why: => Dynamic Forwarding: Nginx dynamically forwards these requests to the appropriate on-demand EC2 server. This is achieved without the need for complex configuration changes whenever a new server is launched or terminated. The way this works is by utilizing proxy_pass with the on-demand IP address as explained in detail below. => SSL Handling: By centralizing SSL handling at the Nginx level, we bypass the issue of attaching SSL certificates to each EC2 instance with variable IPs. => Cost-Effectiveness: This approach eliminates the need for multiple Elastic IPs, thus reducing costs.

Image by Author

As shown in the above diagram, first the frontend calls an API endpoint on the normal API which launches an ec2 instance. The private IP address of this instance is sent back to the frontend which is stored in local storage. Then whenever the frontend wants to make an API call to the on-demand server, it makes an API call to the proxy and passes the IP address of the on-demand server in request headers. Specifically, it passes the IP address as a value of the header with the key “forward”. Nginx has been configured such that it forwards the request to the IP address mentioned in the value of the forward request header. We only needed to upload the SSL certificate to the load balancer associated with the reverse proxy.

File: nginx.conf

location / {
            proxy_hide_header Access-Control-Allow-Origin;
            proxy_hide_header Access-Control-Allow-Headers;
            proxy_hide_header Access-Control-Allow-Methods;
            add_header Access-Control-Allow-Origin * always;
            add_header Access-Control-Allow-Headers * always;
            add_header Access-Control-Allow-Methods * always;
            if ($request_method = 'OPTIONS'){
            return 204;
            }
            proxy_pass_request_headers on;
            proxy_pass '\$http_forward';
        }

*Jump to the table of contents*

Others

Let’s talk about the rest of the architecture

Data Storage — S3 and Postgres

The decision to store datasets in the Parquet file format on AWS S3 was made after careful consideration of several factors, with a focus on optimizing data storage and retrieval efficiency. Parquet, known for its columnar storage approach, offers significant advantages over traditional row-based file formats like CSV. Here’s an overview of the key reasons behind our choice:

1. Significant Compression Benefits: Parquet provides approximately 90% compression compared to CSV. This means a 1 GB CSV file could potentially be reduced to around 100 MB when stored as a Parquet file. This compression led us to relinquish the benefit of => Reduced Data I/O Costs: With smaller file sizes, both disk storage and network transfer costs are significantly lowered.
2. Data Type Preservation: Parquet inherently stores data type information. This feature eliminates the need for data type casting after reading datasets as dataframes, streamlining the data processing workflow.
3. Efficiency in Column-Specific Operations: For certain features, it was identified that reading only specific columns of the dataset would suffice, rather than the entire set. Parquet’s columnar format is exceptionally advantageous in such scenarios, allowing for more efficient data access compared to row-based formats like CSV.

The metadata is stored separately as mentioned below,

1. Storing Metadata in a Database: To avoid repetitive reading of the entire dataset, essential metadata like the number of nulls per column and the range of numerical columns are stored in a database. This approach speeds up feature development and usage.
2. Sampling for Quick Data Overview: A sample of 2000 randomly selected rows is stored in the database. This sample provides users with a quick glimpse of the data without the need to load the entire dataset.
3. Schema Information: The schema of the dataset, including column names and data types, is also stored in the database for easy reference and manipulation.

For the database selection, Postgres was chosen due to its compatibility and advanced features:

1. Compatibility with Django ORM: Utilizing Django ORM with Postgres is straightforward, making it a suitable choice for our Django-based setup.
2. Advanced JSON Handling: Unlike MySQL, Postgres not only supports JSON fields but also provides the capability to perform queries on JSON data. This feature is crucial for storing and querying the metadata in JSON field columns, offering the flexibility our application requires.

*Jump to the table of contents*

EFS

The adoption of AWS Elastic File System (EFS) in our architecture served three primary purposes: enhancing dataset read performance and providing a shared filesystem for multiple EC2 instances and lambda functions. This choice was shaped by both performance considerations and architectural requirements.

1. Speeding Up Dataset Reads: => Caching Mechanism: Upon reading a dataset from S3, we cache it in EFS. This significantly speeds up future read operations, as EFS offers faster read speeds compared to S3 due to its design as a file system service. => Performance Benefit: This caching strategy greatly reduces data access time for frequently accessed datasets, improving overall system performance.
2. Shared File System for EC2 Instances: => Initial Architecture with Multiple EC2s: At the time EFS was adopted, our architecture did not include on-demand servers, and user requests were handled by load balancing among multiple EC2 instances. => Need for Shared Storage: A shared file system was essential to ensure that all EC2 instances had consistent and concurrent access to the same data. EFS, being a fully managed service, provided a simple and efficient solution to this need.
3. Exploration and Rejection of AWS EBS Multi-Attach: => Compatibility Issues: While AWS EBS Multi-Attach was considered, it was ultimately not chosen due to compatibility issues with our existing setup.
4. EFS for Lambda Function Dependencies: => Challenges with AWS Lambda Layers: The typical method of including dependencies in AWS Lambda functions is through Lambda Layers. However, we encountered a limitation with Lambda Layers’ size, capped at 250 MB. When bundling large libraries such as Pandas, Fastparquet, and Plotly, the size exceeded the Lambda Layers limit. => Solution with EFS: To circumvent this, we created a virtual environment in EFS to store these dependencies. We then set the PYTHONPATH in Lambda to point to this virtual environment in EFS. This approach allowed us to bypass the size constraints of Lambda Layers.

*Jump to the table of contents*

CI / CD

Here is the breakdown of our CI / CD workflow,

1. Frontend Development with AWS Amplify: => Seamless Integration and Testing: AWS Amplify is used for frontend development. It’s particularly beneficial for its feature of deploying pull requests to a preview link, allowing for testing before merging. This ensures that any frontend changes can be evaluated in a near-production environment, enhancing the quality assurance process. => Feature Branch Deployment: We also use AWS Amplify to deploy feature branches when needed to allow development testing while development is still in progress.
2. Backend and ML with Circle CI: => GDPR Compliance and Cost-Effectiveness: Circle CI is chosen for backend and ML CI/CD processes due to its GDPR compliance and generous free tier, which makes it an ideal choice for cost-effective and secure development. => Docker Image Management: The Docker images are built in Circle CI and then pushed to AWS Elastic Container Registry. This process ensures a streamlined and secure way of managing Docker images.
3. GitHub Actions for Unit Testing: For running unit tests, particularly when a pull request is raised, GitHub Actions is utilized. This integration allows for an automated and efficient testing workflow, ensuring that code changes are thoroughly tested before they are merged.
4. Incorporating AI in Development Flow: Recently, AI-powered PR code reviewers have been incorporated to make the development flow more efficient. This advanced tool helps in identifying potential issues, improving code quality, and streamlining the review process.
5. Feature Branch Deployment: There are instances in development where a feature branch of the backend needs to be available over the cloud persistently. A slight modification in the Circle CI’s config YAML file enables the deployment of any chosen feature branch to the development environment on AWS. This flexibility is crucial for testing and integrating new features without disrupting the main production environment.

*Jump to the table of contents*

AWS VPC

Virtual private cloud is considered as a first line of defence which kind of provides a trusted contact point feel for the components within it. To know more about it, you can reference this blog. https://medium.com/zeza-tech/secure-your-application-using-a-virtual-network-4b03396e9358

Conclusion

Concluding our discussion on Platform Architecture, we hope you’ve found insightful takeaways as to how we laid the foundation of this data engineering and ML platform. The journey continues in the next part of this series, where we delve into the Codebase (click here to read it), providing a closer look at its design and implementation. If you’re interested in directly jumping to the innovative features that the platform was built to provide, the final part discussing Interesting Features awaits you (click here to read it)

You can reach out to me on Linkedin susmit-vengurlekar or shoot me an email at susmit.py@gmail.com. You can see what I am up to on Github susmit.py

Interested in data engineering and ML? You’re in the right place! Keep an eye out for open roles at Zeza Tech.