Introduce To Web Scraping — Scraping Prayer Time In Indonesia With Python

Photo by David Monje on Unsplash

This article will learn how to retrieve data from a web or commonly called web scraping. In our case, this time, we will use the Beautifulsoup4 package and requests. If you don’t have the package in your python, you can install it first. The destination website that we will use is PKPU which contains prayer schedules from various regions in Indonesia.

Okay, for the first one, we will import the two packages. After we import, we create a variable called req, which contains requests with the get link that we have filled in. After that, we create a soup variable containing the Beautifulsoup package with the text parameter from req and the HTML parser as a parser. Now we can retrieve data from the web by calling the soup variable.

First, we want to retrieve the month on the web. Then we can inspect the element and find the location where the month is.

As you see in the picture above, the month’s description is in the b tag, inside the td tag with center align. We can directly scrap the data by creating a datt variable with the contents of the soup variable find that we defined earlier and entering the tag we want to search. The step is that we search for the outermost tag first, then find the tags inside, and add .text at the end of the code to retrieve text only.

Next, we want to take the location. If we inspect, then the location is here.

The method is the same as taking the month data above to retrieve text from that location. Namely, we find the soup variable, enter the outermost tag, and find another tag inside. At the end of the code, we give .text because we want to retrieve text data only. But here, we use the place variable not to be the same as the initial variable that we have defined.

Next, we want to retrieve the prayer schedule data for today. If we look at the HTML structure of the web, the latest date has a tr tag and a class, namely table_highlight.

So we search for all the elements in the tag, that is, by defining the scrap variable, which contains the soup variable that we find_all. Because the output of find_all is a list, we remove the data we want from the list by taking list number 0, which means we take all the list’s contents. But it will not work if more than one item is on the list. We will loop this variable later to retrieve the clock data sequentially.

And finally, we define a data variable with an empty data dictionary type whose function I will explain later.

For starters, this is the code we have created.

[embed]

Next, we want to retrieve prayer hours data according to the table, but if we look at the HTML structure on the web, it is only a sequence of hours. There is no specific tag indicating the first hour to enter what prayer schedule.

So what we can do is use looping and define the variable i = 0, which means we will loop if I am not more than 0. The data is a date. If the data is more than 0, then we do branching. If i = 1, then the data is morning prayer schedule, and so on. Then all the data is entered in a data variable in the form of a dictionary that we defined earlier. And don’t forget to add .get_text() at the end of each code because we want to retrieve text data only from the tag. And finally, at the end of the loop, the syntax i+=1 is given so that the variable i continues to grow. Like this loop, we have created.

[embed]

Then lastly, we call all the data that we have in the data variable in the form of a dictionary. We call it print syntax like this is the final result of our code.

[embed]

And we already have a program that can retrieve all prayer schedule updates. We only change the link to retrieve data from other regions. I’ve also put this code in the Github repository here. You can clone and then modify the code better to understand the use of package requests and Beautifulsoup4.