Data Visualization using Python — Part I

Grammar of Graphics using Plotnine Library

Data is being generated everywhere and every moment. The first step in any data science project is to have a feeling about the patterns in the data. But having a look into raw numbers rarely achives this goal. Just to have some idea, let us have a look into the US murders data for each state:

What do you learn from staring at this table? How quickly can you determine which states have the largest populations? Which states have the smallest? How large is a typical state? Is there a relationship between population size and total murders? How do murder rates vary across regions of the country? For most human brains, it is quite difficult to extract this information just by looking at the numbers. In contrast, the answer to all the questions above are readily available from examining this plot:

Figure 1: Visualization of US murders data (Created by the author)

A good visualization reminds us of the saying “a picture is worth a thousand words”. Data visualization provides a powerful way to communicate a data-driven finding. In some cases, the visualization is so convincing that no follow-up analysis is required. In fact, a large number of business decisions are based solely on data visualizations only.

The growing availability of informative datasets and software tools has led to increased reliance on data visualizations across many industries, academia, and government. A salient example is news organizations, which are increasingly embracing data journalism and including effective infographics as part of their reporting.

A particularly effective example is a Wall Street Journal article showing data related to the impact of vaccines on battling infectious diseases. One of the graphs shows measles cases by US state through the years with a vertical line demonstrating when the vaccine was introduced.

Figure 2: Measles cases in US states over the years (Created by the Author)

A New York Times chart provides a compelling example by summarizing scores from the NYC Regents Exams. According to the accompanying article, these scores are collected for various purposes, one of which is to determine a student’s eligibility for high school graduation. In New York City, a score of 65 is required to pass. The pattern of these test scores suggests something unusual.

Figure 3: NYC Regents Exams Scores (Created by the Author)

It isv interesting to note that the minimum passing grade is attained by most of the students, with very few scores just below the cutoff score of 65. This unexpected result is consistent with students close to passing having their scores bumped up.

This is an example of how data visualization can lead to discoveries which would otherwise be missed if we simply subjected the data to a battery of data analysis tools or procedures. Data visualization is the strongest tool of what we call exploratory data analysis (EDA). John W. Tukey, considered the father of EDA, once said,

“The greatest value of a picture is when it forces us to notice what we never expected to see.”

Many widely used data analysis tools were initiated by discoveries made via EDA. EDA is perhaps the most important part of data analysis, yet it is one that is often overlooked.

Data visualization has now a heavy influence in philanthropic and educational organizations. Hans Rosling, in his talks New Insights on Poverty and The Best Stats You’ve Ever Seen, forces us to notice the unexpected with a series of plots related to world health and economics. In his videos, he uses animated graphs to show us how the world is changing and how old narratives are no longer true.

Figure 4: Gapminder data on life expectancy and fertility rate (Created by the Author)

It is also important to note that mistakes, biases, systematic errors and other unexpected problems often lead to data that should be handled with care. Failure to discover these problems can give rise to flawed analyses and false discoveries. As an example, consider that measurement devices sometimes fail and that most data analysis procedures are not designed to detect these. Yet these data analysis procedures will still give you an answer. The fact that it can be difficult or impossible to notice an error just from the reported results makes data visualization particularly important.

In the following tutorials, we will learn the basics of data visualization and exploratory data analysis by using three motivating examples. We will use the plotnine package in Python to code, which is based on Grammar of graphics and very similar to ggplot2 library of R. To learn the very basics, we will start with a somewhat artificial example: heights reported by students. Then we will focus on two cases studies: 1) world health and economics and 2) infectious disease trends in the United States. Note that we do not cover interactive graphics.

The codes for all of the above plots are available at: https://github.com/in2boats/data_visualizations/blob/main/Data%20Visualization.ipynb