Data Analytics Project: Australian Immigration Trend Analysis (Part 1: Data Collection)

TL;DR: This is a comprehensive project in Data Analytics, from start to finish, where I will explain each step and integrate technical skills wherever possible. This is the first part where I will explain about data collection with different data sources, methods, and tools used for the same.

Project objective: The objective of this project is to understand the trend of Australian migration, identify fields that were popularly invited for PR, etc.

Step 1: Data Collection

The First step is to identify possible data sources. Since most of our data is released by the Australian Government, we can find it in the following source types:

• Excel/Spreadsheet/PDFs
• APIs

The following are the data sources identified with the help of ChatGPT:

Prompt used in ChatGPT:

I’m working on a data project to analyze Australian migration trends over the years. I aim to identify all possible data sources that I can utilize to support my analysis. These can include public datasets, government reports, APIs, databases, research papers, cloud-hosted files, or even third-party platforms.

Please provide a list of data source types (structured, semi-structured, and unstructured) and specific examples relevant to Australian migration, such as visa types, population by region, country of origin, education/employment of migrants, or policy impact.

For each source, briefly mention:

Where can I access it (website or API)

Format (CSV, Excel, API, JSON, PDF, etc.)

Whether it’s free or requires access

Any tools I might need to extract it

After careful evaluation of suggestions generated, as all of them might not be helpful,the following are the data links finalised:

***https://www.abs.gov.au/statistics/people/population/overseas-migration***

***https://www.homeaffairs.gov.au/research-and-stats/files/report-migration-program-2023-24.pdf***

***https://www.homeaffairs.gov.au/research-and-stats/files/migration-trends-compendium-2023-24.pdf***

***https://www.homeaffairs.gov.au/research-and-statistics/statistics/visa-statistics/live/migration-program***

***https://api.dynamic.reports.employment.gov.au/***

So we have data sources in the form of Excel files, PDFs, and APIs. The Excel file and PDFs can be easily downloaded. So, let’s put our focus on the API.

First, let’s understand what an API is and how it is used in data extraction!

Consider that an API is like a language or a contract we use to request data from a website or service. Websites don’t understand human language, so we must communicate in a structured way they can understand, and that’s exactly what an API does.

But just because we ask for data doesn’t mean the website will give it to us! Some data is public (like weather updates), but other data might be private or sensitive (like personal account info). In such cases, the website will ask:

“Who are you, and do you have permission to access this?”

To prove this, we often need authentication, like a key, token, or login, which acts like a password to unlock the data. Hence, depending on data sensitivity, the authentic types are:

• No Auth: no login is needed, publicly available, anyone can easily access the data. e.g. weather data
• API Key: like a password, private data, included as a unique key every time you make a request. e.g. gym membership
• OAuth: getting permission to access private data, e.g., permitting access to Google Drive
• Bearer Token,/JWT: this is like using a one-time password to access data, e.g,. accessing Netflix movies where the user doesn’t have to enter the password for every movie once logged in (session management after login)

Here’s a quick summary:

Just like human beings communicate in different languages with each other, APIs are also of different types:

• REST API: easiest form, we just send a URL, and it will reply with data easily
• SOAP API: a bit strict, has formal rules, used in banking and government sectors
• GraphQL: very flexible, can be precise, e.g., GitHub API
• Webhooks: instead of us having to ask for data every time, the server sends us data automatically when something happens, e.g., notification of payment