How to scrape Tweets about Bitcoin with Python and Twint

Source: https://unsplash.com/photos/U6t80TWJ1DM

1. Introduction

In this article we are going to learn how to scrape Tweets about Bitcoin from Twitter by using the Twint Library. The objective is to create a dataset that contains Social Media Posts of Bitcoin within a specific time frame, so that it is possible to analyze the Sentiment.

2. Libraries

We are going to use the *twint and pandas *Libraries in this Tutorial, so make sure that you have them installed on your local machine.

pip3 install twint pandas

Twint is a great tool to scrape Tweets from Twitter since it has no API Limits and we therefore can scrape as many tweets as we want. Furthermore it is possible to filter our search to specific dates or languages.

IMPORTANT: If you follow this Tutorial with a JupyterNotebook you may run into problems when using Twint. To avoid this install the nest-asyncio library and import nest_asyncio to your Notebook.

!pip install nest-asyncio

import nest_asyncio

3. First Steps using Twint

In the code snippet of below you can see a function with which we can determine which topic we want to search, how many tweets we want to scrape and from which day the tweets should be. Selected dates need to be specified in the „YYYY-MM-DD HH:MM:SS“ format.

[embed]

4. Filter and Clean Tweets

Many tweets consist only of a URL that refers to another website. This has no information content in our case, which is why we want to filter tweets whose text consists mainly of a URL. The following function marks tweets whose text consists of more than 50% of a URL as “SPAM”, so that we can remove them from the dataframe later.

[embed]

Furthermore we want to filter out any Emojis.

[embed]

Unfortunately there are many Scam Tweets that are related to Bitcoin. However these Tweets always use specific words of which I created a list of. When we apply the following function to our dataset we will mark Tweets as “SPAM” if they contain any word of the defined list.

[embed]

Ultimately we are going to create a Function in which each of the previous functions will be called and in which we remove hashtags and URLs from the tweet.

[embed]

We then apply the text cleaning and filtering function on the column of the dataframe which contains the tweet and filter out any tweets that were previously marked as „SPAM“.

df["tweet"] = df["tweet"].apply(lambda row: clean_text(row))
df = df[df["tweet"] != "SPAM"]

It becomes evident that the size of the dataset shrank by a lot but when taking a look at the Tweets we now see that are barely and posts left that could be interpreted as Scam.

5. Create our Final Dataset

As mentioned earlier in the article we want to collect tweets about Bitcoin within a specific time frame so that we can detect patterns in the sentiment. I created a function that returns a list of strings in the required format so that we can specify our date range.

[embed]

We will then Loop over each date string and create a Dataframe of Bitcoin tweets that were posted on that specific date. We apply our previously created Text Cleaning and Filtering Function and will ultimately create one final dataframe.

[embed]

We now have successfully created a dataframe containing tweets about Bitcoin that were posted within the last week. This gives us the possibility to analyze the dataset further and to apply a Sentiment Analysis in order to detect trends and patterns in the popularity of the Cryptocurrency. Below you can find the complete Code. I hope you could learn something from reading this article and please feel free to contact me if you have any questions.

[embed]