Evaluating the Performance of Pradhan Mantri Fasal Bima Yojana (PMFBY) by Scraping data from National Crop Insurance Portal (NCIP).

The Pradhan Mantri Fasal Bima Yojana (PMFBY) was launched in 2016 as a government-backed crop insurance scheme in India. The scheme aims to provide comprehensive insurance coverage to farmers, protecting them from potential financial losses due to crop failures caused by natural disasters, pests, or diseases. It represents a significant step toward ensuring the stability of farmers’ incomes and promoting sustainable agricultural practices across the country.

To streamline the management of PMFBY, the government launched the **National Crop Insurance Portal (NCIP)**. NCIP acts as a centralized platform where stakeholders, including farmers, insurance companies, and government agencies, can access detailed information on the implementation and performance of PMFBY.

It offers a ***real-time dashboard *with data on insured farmers, total area covered, premiums paid, claims settled, and more. This data is critical for policymakers, researchers, and industry experts looking to analyze the scheme’s efficacy over time. The [NCIP dashboard](https://pmfby.gov.in/adminStatistics/dashboard)** provides a wealth of data on PMFBY implementation. However, extracting this data manually can be time-consuming and prone to errors. Web scraping is an effective way to automate this process.

In this article, I will demonstrate how to extract data from the NCIP dashboard using web scraping tools like Selenium and Pandas. This data will help evaluate the performance of PMFBY over the last six years, shedding light on key trends and outcomes of the scheme.

Setting up Web Scraping with Selenium and Pandas

Prerequisites

Before diving into web scraping, ensure you have the following tools installed:

1. Python: The programming language we will use to write our scraping script.
2. Selenium: A popular web scraping tool that automates browser interaction.
3. Pandas: A powerful data manipulation library that allows us to clean, filter, and analyze data after extraction.
4. Chrome Driver: A driver that enables Selenium to control Google Chrome (ensure it’s compatible with your browser version).

You can install Selenium and Pandas via pip:

Step 1: Import necessary libraries:

import time
from selenium import webdriver
from getpass import getpass
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import Select
from time import sleep
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.chrome.service import Service as ChromeService
from webdriver_manager.chrome import ChromeDriverManager
from selenium.common.exceptions import StaleElementReferenceException
import pandas as pd
from io import StringIO

Step 2: Set up a web driver:

driver = webdriver.Chrome()
driver.get('https://pmfby.gov.in/adminStatistics/dashboard')

Step 3: Interact with links Filters: Depending on the structure of the NCIP dashboard, you may need to interact with dropdowns, buttons, or date filters to specify the data you want to scrape. Use Selenium’s methods like find_element_by_id, find_element_by_xpath, etc., to locate the elements on the page containing the desired data.

State_report = driver.find_element(By.XPATH,'//*[@id="app"]/div/div[1]/div[2]/div[1]/div/div/div/div/div/p/a[2]').click();
WebDriverWait(driver, 300).until(EC.invisibility_of_element_located((By.XPATH, "//div[@class='loader__loader___tH-dK']")))

Once you open the NCIP dashboard, you will see various options to filter data by year, Season, Scheme, and other parameters. For this article, I will focus on extracting data from the last six years (2018–2023). Create the list of year, season, and scheme. Season ‘01’ specifies Kharif and ‘02’ specifies Rabi whereas scheme ‘04’ specifies Yield based PMFBY scheme and ‘02’ specifies weather index based RWBCIS scheme.

year=['2023','2022','2021','2020','2019','2018',]
season=[ '01','02',]
scheme=['04','02',]

Step 4: Iterate over filters to collect data table for various Scheme — Season — Year combinations:

Once the relevant data is displayed on the screen, use Pandas to capture it. In many cases, the data will be in a tabular format, which makes it easy to scrape and store using Pandas.

for y in year:
    for s in season:
        for p in scheme:
            Year_menu =Select(WebDriverWait(driver,20).until(EC.visibility_of_element_located((By.XPATH, "(//*[@id='app']/div/div[1]/div[2]/div[2]/div/div[1]/div/div[2]/div/select)[1]"))))
            Year_menu.select_by_value(y)
            WebDriverWait(driver, 300).until(EC.invisibility_of_element_located((By.XPATH, "//div[@class='loader__loader___tH-dK']")))
            Season_menu =Select(WebDriverWait(driver,20).until(EC.visibility_of_element_located((By.XPATH, "(//*[@id='app']/div/div[1]/div[2]/div[2]/div/div[1]/div/div[3]/div/select)[1]"))))
            Season_menu.select_by_value(s)
            WebDriverWait(driver, 300).until(EC.invisibility_of_element_located((By.XPATH, "//div[@class='loader__loader___tH-dK']")))
            Scheme_menu =Select(WebDriverWait(driver,20).until(EC.visibility_of_element_located((By.XPATH, "(//*[@id='app']/div/div[1]/div[2]/div[2]/div/div[1]/div/div[4]/div/select)[1]"))))
            Scheme_menu.select_by_value(p)
            WebDriverWait(driver, 300).until(EC.invisibility_of_element_located((By.XPATH, "//div[@class='loader__loader___tH-dK']")))
            all_tables = pd.read_html(StringIO(driver.page_source))
            for table in all_tables:
                table["year"] = y
                table["Season"] = s
                table["Scheme"] = p
                table_list.append(table)

This script automates the task of navigating a web-based form with dropdown menus for year, season, and scheme. After selecting the values and waiting for the page to load, it scrapes any HTML tables on the page, adding contextual information (the year, season, and scheme), and stores them for further analysis. It repeats this process for all combinations of the three parameters.

Step 5: Store table data in a Pandas DataFrame:

table_list=[]
df=pd.concat(table_list,ignore_index=True)
df.head()

Step 6: Remove top row from multi column dataframe:

df.columns=[multicols[-1] for multicols in df.columns]

# Rename the last three columns
new_columns = df.columns.tolist()
new_columns[-3:] = ['year', 'Season', 'Scheme',]
df.columns = new_columns

df.head()

Step 7: Save and Analyze Data: After extracting the data, you can save it to a CSV file for analysis.

df.to_excel("NCIP_Data.xlsx",index=True)

Conclusion:

Using web scraping techniques like Selenium and Pandas, we can extract and analyze data from the NCIP dashboard to evaluate the performance of the Pradhan Mantri Fasal Bima Yojana over the past six years. This method allows us to gather a large amount of data efficiently and use it to uncover insights into the scheme’s effectiveness, identifying areas of improvement and success.

In the next part of this article, I will dive into a detailed analysis of the scraped data, highlighting key trends in crop coverage, claims settled, and farmer enrollment under PMFBY.