Getting started with PySpark

If you are a data scientist moving away from toy datasets and getting to work with mammoth volumes of real big data, Apache Spark is your savior.

Spark is written in Scala and if that is something you are comfortable with, this tutorial is not for you. On the other hand, if you are someone like me who is completely into Python, read on.

What is PySpark?

PySpark is an interactive Python interface to Apache Spark. It allows us to write Spark applications for distributed computing and unleash the power of various powerful features of Spark.

Image by Author

The goal of this short tutorial is to get you working with PySpark in a matter of minutes in two simple steps.

Step 1: Download docker desktop from here.

Step 2: Execute the following commands in your terminal

The first command makes the docker pulls all the configurations for Spark and Jupyter notebook.

The second one runs the Jupyter notebook in an interactive manner on port 8888 on the container.

Note- Copy the token you see on the terminal screen as you will need to enter it when you access Jupyter notebook via the local host 8888 on your browser.

The alternative to using a token authentication is to set up a password.

Thats it! You are all set up now to start coding up your first Spark application.

You can refer to Apache Spark documentation for simple examples to get started with and refer to the Apache Spark Github repo.

Pi Estimation using Monte Carlo Approach

This example is taken from Apache Spark examples (https://spark.apache.org/examples.html)

The example shows the use of Spark for compute-intensive tasks. We use the famous Monte Carlo sampling approach and the parallelize method to create 1000000 random datapoints(x ,y) in the unit square ((0, 0) to (1,1)) bounding our unit circle. We refer to these points as NUM_SAMPLES. For a unit circle(r=1), it is easy to see that the value of Pi is equal to the area of the circle (area=Pi*r²). Consider one quadrant of a circle, the points that satisfy the condition x²+y²<1 will lie within the circle. Using this criteria we filter and count the points in each quadrant of the unit square.

The total area of the circle is given by 4 count/NUM_SAMPLES. For a unit circle(r=1), it is trivial to see that the value of Pi is equal to the area of the circle (Pir²). Running the code below returns the approximate value of Pi.

Note- What I presented above is the simplest docker container approach to get Spark running on your machine for learning purpose. Setting up Spark clusters for production environment is a whole different beast and topic for another post.

I hope you found this tutorial useful and I look forward to your feedback on any misses. If you want me to cover any aspect of Machine Learning in Big Data please drop a line.

You can follow me here or connect with me on LinkedIn.