Topic Modeling Lyrics of Popular Songs

By Rachit Kumbalaparambil

Introduction

This tutorial uses a topic modeling technique called Latent Dirichlet Allocation (LDA), to identify themes in popular song lyrics over time.

LDA is a form of unsupervised learning which uses a probabilistic model of language to generate “topics” from a collection of documents, in our case, song lyrics.

Each document is modeled as a Bag of Words (BoW), meaning we just have the words that were used along with their frequencies, without information on word order. LDA sees these bags of words as a composition of all topics, having a weighting for each topic.

Each topic is a list of words and their associated probabilities of occurrence, and LDA determines these topics by looking at which words often appear together.

Topic modeling is in theory very useful in this case as we don’t have labeled data (genre not specified), and what we really want to do is identify themes that the lyrics are talking about. It would be difficult to analyze themes like loneliness, heartbreak, or love from just genre-labeled data. Musical genres are by no means hard boundaries either, as most artists may not fit into any given category.

The Data

Consists of the Billboard Top 100 songs in the US for each year from 1959 to 2023. The features included that we will be looking at:

• Lyrics
• Title, Artist
• Year
• Unique word count

The data was web scraped from https://billboardtop100of.com/ and the lyrics were pulled from Genius API (learn more here: https://genius.com/developers).

It was contributed to Kaggle by Brian Blakely and released under MIT license.

Text Processing

To build a good topic model, it was important to pre-process the text. The raw lyrics contained a large amount of stop words, which are common words like “the,” “and,” “is,” etc., which carry little/no semantic meaning. In this project, I also chose to filter out additional words that weren’t providing meaning, in an effort to improve the model.

The steps taken in pre-processing the text were as follows:

• Tokenization (splitting up the text into words)
• Lowercasing
• Removing punctuation
• Removing stop words (standard + custom list)
• Lemmatization (reducing words to their base, e.g. “running” to “run”)

The main thing to consider during this pre-processing step is how it will affect the model if you remove certain words. It will have a lot to do with your specific application, so take action accordingly.

In my case, I went through and iteratively chose words to remove by running the pre-processing, then creating a document-term matrix and examining the top ~30 words. From these words, I selected words that don’t provide semantic meaning to be added to my custom set of stop words.

Words were also added to this list after running the LDA algorithm and examining the topics, removing words that highlighted their lack of semantic meaning by appearing in every topic.

The following is the custom list I ended up using for the final model, as well as the code I used to create and examine the document term matrix. This code is built off of code provided by my Text Analytics professor, Dr. Anglin at the University of Connecticut.

stoplist = set(nltk.corpus.stopwords.words('english'))
custom_stop_words = {'na', 'got', 'let', 'come', 'ca', 'wan', 'gon',
                     'oh', 'yeah', 'ai', 'ooh', 'thing', 'hey', 'la',
                     'wo', 'ya', 'ta', 'like', 'know', 'u', 'uh',
                     'ah', 'as', 'yo', 'get', 'go', 'say', 'could',
                     'would', 'take', 'one', 'make', 'way', 'said',
                     'really', 'turn', 'cause', 'put', 'also',
                     'might', 'back', 'baby', 'ass' , 'girl', 'boy',
                     'man', 'woman', 'around', 'every', 'ever'}
stoplist.update(custom_stop_words)

# make lyric_tokens string of tokens instead of list for CountVectorizer
df["lyric_tokens_str"] = df["lyric_tokens_list"].apply(lambda x: " ".join(x))

vec = CountVectorizer(lowercase = True, strip_accents = "ascii")
X = vec.fit_transform(df["lyric_tokens_str"])

# X refers to the sparse matrix we saved as X. df is the original dataframe we created the matrix from.
matrix = pd.DataFrame(X.toarray(), columns=vec.get_feature_names_out(), index=df.index)

# top 10 most freq terms
matrix.sum().sort_values(ascending = False).head(10)

The following is the process_text function used to clean the data. It was modified for the purpose of including an argument for the custom stoplist.

def process_text(
    text: str,
    lower_case: bool = True,
    remove_punct: bool = True,
    remove_stopwords: bool = False,
    lemma: bool = False,
    string_or_list: str = "str",
    stoplist: set = None
):

    # tokenize text
    tokens = nltk.word_tokenize(text)

    if lower_case:
        tokens = [token.lower() if token.isalpha() else token for token in tokens]

    if remove_punct:
        tokens = [token for token in tokens if token.isalpha()]

    if remove_stopwords:
        tokens = [token for token in tokens if not token in stoplist]

    if lemma:
        tokens = [nltk.wordnet.WordNetLemmatizer().lemmatize(token) for token in tokens]

    if string_or_list != "list":
        doc = " ".join(tokens)
    else:
        doc = tokens

    return doc

An example of how this should work on “Sexy And I Know It” by LMFAO:

Raw: “Yeah, yeah, when I walk on by, girls be looking like damn he’s fly” Processed: [‘walk’, ‘look’, ‘damn’, ‘fly’]

Methods

As mentioned, we set up for the model by creating a bag of words for each document, and a list of BoWs for the corpus:

from gensim.corpora import Dictionary

gensim_dictionary = Dictionary(df['lyric_tokens_list'])
gensim_dictionary.filter_extremes(no_below=313, no_above=0.60)

# no_below 313 (out of 6292 for ~5% of the total corpus)
# no_above 0.60

# Create a list of BOW representations for the corpus
corpus = [gensim_dictionary.doc2bow(doc) for doc in df['lyric_tokens_list']]

We filter out extremes of 5% and 60%, meaning we are filtering out words that appear in less than 5% of songs and more than 60% of songs. These cutoffs were chosen iteratively, similar to the custom word list. This is another point where you might make a different decision based on the data.

In fitting the model, I used Gensim’s LdaModel and experimented with different amounts of topics (5 to 50). A for loop was used to make a model for 5, 10, 30 and 50 topics.

from gensim.models import LdaModel
from gensim.models import CoherenceModel

topic_range = [5, 10, 30, 50]

coherence_scores = []
lda_models = []

gensim_dictionary[0] # required to initialize

for num_topics in topic_range:
    lda_model = LdaModel(
        corpus=corpus,
        id2word=gensim_dictionary,
        num_topics=num_topics,
        random_state = 1)
    lda_models.append(lda_model)

    coherence_model = CoherenceModel(model = lda_model,
                                    texts = df['lyric_tokens_list'],
                                    dictionary=gensim_dictionary,
                                    coherence = 'c_v',
                                    processes = 1) # avoids weird addition error
    coherence = coherence_model.get_coherence()
    coherence_scores.append(coherence)
    print(f"Coherence score for {num_topics} topics: {coherence}")

A key decision here is the amount of topics you want to go with, and that is based on how many different models you create and evaluate. In this case, I fit 4 models, and chose among them.

We evaluate the models we fit using their coherence scores, which is a measure of semantic similarity among the top terms in a topic. In this case, the best performing model was the 30 topic model, with a coherence score of 0.408.

Results

Let’s investigate the contents of the generated topics below. I used the following block of code to create a dataframe of the selected final model, for ease of inspection.

list_of_topic_tables = []
final_model = lda_models[2]

for topic in final_model.show_topics(
    num_topics=-1, num_words=10, formatted=False
):
    list_of_topic_tables.append(
        pd.DataFrame(
            data = topic[1],
            columns=["Word" + "_" + str(topic[0]), "Prob" + "_" + str(topic[0])],
        )
    )

pd.set_option('display.max_columns', 500)
bigdf = pd.concat(list_of_topic_tables, axis=1)
bigdf

Reading through them, a lot of them seem to be picking up on different themes/ideas. We can explore more by visualizing the prevalence of the topics across the corpus. To do this, we’ll create a new data frame of our original data, but now with each song’s associated topic probabilities appended to it.

# Create a list of list of topic probabilities for each document
topic_probs = []
for document in corpus:
    document_topics = []
    for topic_prob in final_model.get_document_topics(document, minimum_probability=0):
        document_topics.append(topic_prob[1])
    topic_probs.append(document_topics)

topic_probs_df = pd.DataFrame(topic_probs)

doc_topics = df.reset_index().merge(topic_probs_df, left_index=True, right_index=True)
doc_topics['dominant_topic'] = topic_probs_df.idxmax(axis=1)
doc_topics.sample(3)

With this data frame set up, the following plot for the topic prevalence across the corpus was created:

topic_sums = topic_probs_df.sum().sort_values(ascending=False)
plt.figure(figsize=(12, 6))
sns.barplot(x=topic_sums.index, y=topic_sums.values)
plt.title("Topic Prevalence Across Corpus")
plt.xlabel("Topic #")
plt.ylabel("Sum of Probability")
plt.show()

The most prevalent topic (22) was defined by a high prevalence of the word love. I created a simple word cloud for this topic as well as others.

Topic 22 “Just Love”

With this in mind, I used the following code to plot the prevalence of topic 22 over time:

# convert 'Year' column to integer
doc_topics['Year'] = pd.to_numeric(doc_topics['Year'], errors='coerce').astype('Int64')

topic_22_by_year = doc_topics.groupby('Year')[22].mean()

plt.figure(figsize=(12, 6))
sns.scatterplot(data=topic_22_by_year)
plt.title('Prevalence of Topic 22 ("Just Love") Over Time')
plt.xlabel('Year')
plt.ylabel('Average Topic 22 Probability')
plt.grid(axis = 'y')
plt.show()

Songs that are mainly about love seem to have been on the decline. We might be seeing this due to a shift in love songs, as they could be incorporating more specific topics in addition rather than focusing only on love.

Topic 10 was the second most prevalent, and it looked to be pretty well defined as describing heartbreak:

Topic 10 “Heartbreak”

We see a slight downward trend, with a notable high point in 1963. This was a significant year in US History, as JFK was assassinated and the March on Washington occurred, with MLK Jr. delivering his famous “I Have a Dream” speech. These events could have played a role in the subject matter of the top songs of the year, although it could be coincidental. Regardless, like the decline in topic 22, it’s an interesting trend worthy of further exploration.

Perhaps most interesting for me was topic 21, which was defined by songs with a large proportion of explicit lyrics.

We see a huge increase in the prevalence of explicit lyrics over the past 20 to 30 years, and it will be interesting to see if this stagnates. Such a shift in our culture is really interesting and definitely worth further exploration.

One last point of note was the ‘unique word counts’ column, which we were able to visualize over time by taking the average for each year.

There is a clear upward trend, which is interesting since one might have thought that lyrics have been getting simpler over time.

Validation, Limitations and Conclusions

Validating the model mainly involved two things:

• Using the coherence score to get a quantitative measure of the amount of meaning captured by the generated topics
• Manual inspection of topics to determine if they describe a coherent theme

Further validation may include gathering expert opinions, bolstering the manual checking of the topics.

In closing, some strengths and weaknesses of the LDA model:

Strengths

• Unsupervised, finding themes in song lyrics without labeled data
• Able to identify trends in themes of lyrics over time
• Gives clues to more broad ideas of the evolution of American and global culture

Weaknesses:

• LDA ignores word order, which could be important in the context
• Large amount of generic words appearing across multiple topics, making it hard to distinguish topics
• Results are very dependent on pre-processing choices
• Pre-processing improvements to be desired, such as more custom stop words, and further optimizing the search space (number of topics, filtered extreme cutoff points)

Overall this was a really interesting application of topic modeling, and I enjoyed building and evaluating the model. It was fascinating to identify themes like heartbreak and explicit lyrics, and analyze their prevalence over time. Future work on this could include trying a BERTopic model rather than LDA, or utilizing more metadata such as genre or specific release dates.