Exploring Data Resources for Machine Learning: From Popular Hubs to Niche Repositories

In this article, I share the data sources that I have used and intend to use. My students experimentally generated some data, whereas others were obtained from data repositories. I hope the list of data repositories is useful to those aspiring to be data scientists.

The Importance of Data in Data Science

Data is said to be the water or medium that makes the wheel of data science or Machine Learning turn. It is said that data is everywhere. The term ubiquitous data refers to the pervasive occurrence of data in every aspect of our lives. Digital technologies in smartphones, smartwatches, and home devices constantly consume, generate, and transmit data.

But does the data fulfill our needs? The need might be to classify some herbs according to location as herbs from certain locations are better quality. Certain combinations of genes may predispose individuals to specific diseases, such as obesity. Are the data available to carry out such analysis?

Generating Data vs. Using Repositories

In the realm of data science, sourcing relevant and high-quality data is a foundational step. One way is to generate the data through experimentation as my student did. The other way is to use repositories of data for testing out different algorithms.

Kaggle: The Home of Data Science

One of the most popular repositories is Kaggle which is known as the home of data science. It is a leading platform for data science competitions and has more than 400,000 datasets. The data sets are divided according to categories such as computer science, education, and classification as well as can be searched using keywords and filters. Another advantage of Kaggle is that it has about 3,600 datasets that are related to computer vision or image analysis. I feel that this is an exciting area as computers are still not able to identify subtle nuances in photos as what humans are capable of.

There are thousands of machine learning models including LLMs and their codes in Kaggle. I believe other than courses at Udemy, I have learned a lot about data science from Kaggle.

UC Irvine Machine Learning Repository

Another popular and useful repository is the UC Irvine Machine Learning Repository. It has currently 670 datasets. The likable thing about the datasets is that one can see from the outset, the type of task such as classification or regression, the number of samples, and the number of features or variables in each dataset. I feel that the use of algorithms such as genetic algorithm for selecting variables is better done with large datasets with thousands of samples and at least a hundred variables. The task of identifying such datasets is easier in this repository.

GEO Dataset and ArrayExpress: Genomics Data Resources

Another dataset that I am familiar with is the GEO Dataset or Gene Expression Omnibus repository. The dataset can be searched using various keywords and species of animal involved in the study. The limitation of these data is that the number of samples is small in comparison to features which are expression levels of genes. The use of Google Scholar to search for research articles when combined with the use of this repository should help in obtaining suitable data. Another similar site is ArrayExpress — Functional Genomics Data. The GEO dataset and the Array Express are niche repositories that might interest researchers in the field of genomics.

The repositories that I have mentioned till now are the ones that I have some experience with their structured data. Most of the tasks involved classification and the results were satisfactory.

Challenges in Image Classification

The task that I have found difficult and with poor accuracy on the test set is image classification. I have used different datasets such as flowers dataset (consisting of daisies, dandelions, roses, sunflowers, and tulips), mushrooms, etc.

ImageNet: A Comprehensive Image Dataset

The datasets above are easier to work with as the number of categories is small but the more comprehensive dataset is the ImagNet which initially had more than 16 million images. The most used set of images in this repository is ILSVRC [https://www.kaggle.com/c/imagenet-object-localization-challenge/data]. This dataset is currently available on Kaggle and has more than 1,200,000 training images as well as 50,000 validation images. The number of test images is 100,000 and this dataset spans 1000 object classes.

I believe a better attempt at image analysis might involve using some of the classes of samples in ILSVRC that are easier to distinguish to check the parameters for convolution neural networks. Alternatively, I might try the technique of transfer learning to obtain better accuracy on the test set.

Exploring Graph Network Data Resources

My recent interest in expanding my knowledge in data science got me interested in graph networks. One of the first steps other than getting resources in terms of books and courses is to search for data repositories.

The Network Repository and Stanford Collections

The Network Repository is the largest network data and the first interactive repository with data from more than 30 domains. The site is intended to facilitate researchers to download and study the vast collection of network data.

Another data repository that I might be interested in is the Standford Biomedical Network Dataset collection. The examples of networks that I find related to my field are the chemical-gene interaction network (ChChSe-Decagon), drug-target interaction network (ChG-TargetDecagon), and disease-gene interaction network (DG-AssocMiner).

The network data related to other domains than biomedical are available in Stanford Large Network Dataset Collection. Examples of them are citation networks, cryptocurrency networks, and social networks.

The Evolution of Data Repositories

The repositories mentioned here may not be the most extensive, and they are exclusive of certain domains such as time series forecasting. The important thing about data science is that finding the right data source is a necessary skill. Furthermore, the repositories have evolved, too. They are constantly updated and at times merged.

Conclusion: Finding the Right Data

The right data source is crucial for effective data science. While repositories offer quality-assured, preprocessed datasets that save time, experimental data provides tailored, domain-specific insights. Balancing these options depends on the project’s needs and available expertise. Ultimately, whether using structured repositories or generating unique data, the focus should remain on obtaining high-quality, relevant datasets to drive meaningful analysis.

Image created by DALL·E by OpenAI-the author has the provenance and copyright