Spark, to my knowledge, Part 1

Industries are using Hadoop extensively to analyze their data sets. The reason is that the Hadoop framework is based on a simple programming model (MapReduce), and it enables a computing solution that is scalable, flexible, fault-tolerant, and cost-effective. Here, the main concern is to maintain speed in processing large datasets in terms of waiting time between queries and waiting time to run the program.

Spark was introduced by the Apache Software Foundation to speed up the Hadoop computational computing software process.

As against a common belief, Spark is not a modified version of Hadoop and is not really dependent on Hadoop because it has its own cluster management. Hadoop is just one of the ways to implement Spark.

Spark uses Hadoop in two ways: one is storage, and the second is processing. Since Spark has its own cluster management computation, it uses Hadoop for storage purposes only.

What is Apache Spark?

Apache Spark is a lightning-fast cluster computing technology designed for fast computation. It is based on Hadoop MapReduce, and it extends the MapReduce model to efficiently use it for more types of computations, which include interactive queries and stream processing. The main feature of Spark is its in-memory cluster computing, which increases the processing speed of an application.

The illustration depicts the different components of Spark.

SparkContext

• SparkContext is the entry point to any spark functionality. When we run any Spark application, a driver program starts, which has the main function and your SparkContext gets initiated.

source: www.tutorialspoint.com

class pyspark.SparkContext (
   master = None,
   appName = None, 
   sparkHome = None, 
   pyFiles = None, 
   environment = None, 
   batchSize = 0, 
   serializer = PickleSerializer(), 
   conf = None, 
   gateway = None, 
   jsc = None, 
   profiler_cls = <class 'pyspark.profiler.BasicProfiler'>
)

• master: Specifies the URL of the Spark cluster to connect to (e.g., local[*], spark://master:7077, yarn).
• appName: The name of your application, displayed in the Spark UI.
• sparkHome: The location of the Spark installation on cluster nodes. This is usually not required unless you’re using a custom Spark build.
• pyFiles: A list of Python files to be sent to the cluster and added to the PYTHONPATH.
• environment: A dictionary of environment variables to set on worker nodes.
• batchSize: The number of records per partition for RDDs.
• serializer: The serializer to use for RDDs (default is PickleSerializer).
• conf: A SparkConf object to set Spark properties.
• gateway: an existing Py4J gateway. Used internally.
• jsc: The underlying JavaSparkContext instance. Used internally.
• profiler_cls: The class to use for profiling Spark jobs.

Spark Cluster Mode

Spark applications run as independent sets of processes on a cluster, coordinated by the SparkContext object in your main program (called the driver program).

source: www.tutorialspoint.com

Alright, let’s explain this in a way that a 5-year-old can understand!

Imagine you have a big box of building blocks, and you want to build many different things with your friends. But instead of everyone building with the same blocks all at once and getting in each other’s way, you have a special way to organize it.

1. Big Boss (SparkContext): There’s a big boss who tells everyone what to do. This boss can talk to different types of organizers who manage the blocks.
2. Organizers (Cluster Managers): These organizers can be different people or systems (like Spark’s own organizer, Mesos, YARN, or Kubernetes). Their job is to give out blocks to different friends who are building.
3. Friends (Executors): Once the big boss talks to an organizer, the organizer gives out blocks to your friends. Each friend gets some blocks and they use these blocks to build something.
4. Building Instructions (Application Code): The big boss also gives your friends instructions on what to build. These instructions come in little packages (like JAR or Python files).
5. Tasks (tasks): Finally, the big boss sends specific tasks to each friend, telling them exactly what to build with their blocks.

Here are some important points about this setup:

• Separate Building Projects: Each friend works on their own project, so they don’t mix up their blocks with someone else’s. This way, everyone can work without disturbing each other.
• Different Organizers: It doesn’t matter which organizer is helping; as long as they can give out blocks and the friends can talk to each other, everything works smoothly.
• Communication: The big boss needs to keep talking to the friends while they build, so they need to be able to hear each other. It’s best if they are close by (like in the same room) so the big boss can easily give new tasks.

Apache Spark Core

Spark Core is the underlying general execution engine for spark platform that all other functionality is built upon. It provides In-Memory computing and references datasets in external storage systems.

Spark SQL

Spark SQL is a component on top of Spark Core that introduces a new data abstraction called SchemaRDD, which provides support for structured and semi-structured data.

Spark Streaming

Spark Streaming leverages Spark Core’s fast scheduling capability to perform streaming analytics. It ingests data in mini-batches and performs RDD (Resilient Distributed Datasets) transformations on those mini-batches of data.

Resilient Distributed Datasets:

GraphX

GraphX is a distributed graph-processing framework on top of Spark. It provides an API for expressing graph computation that can model the user-defined graphs by using the Pregel abstraction API. It also provides an optimized runtime for this abstraction.

Resilient Distributed Datasets (RDD)

RDD is a core data structure in Spark, representing immutable, distributed collections of objects. These datasets are split into logical partitions, which can be processed on different nodes within a cluster. RDDs can contain various types of objects in Python, Java, or Scala, including user-defined classes.

Formally, an RDD is a read-only, partitioned collection of records, created through deterministic operations on data in stable storage or other RDDs. They are fault-tolerant and can be operated on in parallel.

RDD Lineage

RDD lineage refers to the logical sequence of transformations that were applied to create an RDD from its source data or other RDDs. It represents the history of how data is derived and transformed, step by step, from its original form to the current RDD. The RDD lineage is used for fault tolerance. When a node fails, Spark can reconstruct lost partitions by reapplying the transformations specified in the lineage. Lineage information is recorded for each RDD, which enables Spark to recompute lost data efficiently without needing to store the entire dataset. The RDD lineage ensures that lost data can be recovered by re-executing only the necessary transformations.

Data Sharing is Slow in MapReduce

MapReduce is widely adopted for processing and generating large datasets in parallel.

distributed algorithm on a cluster. It allows users to write parallel computations

Using a set of high-level operators

Reuse intermediate results across multiple computations in multi-stage applications. The following illustration explains how the current framework works, while doing iterative operations on MapReduce. This incurs substantial overheads due to data replication, disk I/O, and serialization, which makes the system slow.

Programming

Spark provides an interactive shell − a powerful tool to analyze data interactively. It is available in either Scala or Python languages

Spark shell

Open Spark Shell

The following command is used to open Spark shell.

$ spark-shell

Create simple RDD

Let us create a simple RDD from the text file. Use the following command to create a simple RDD.

scala> val inputfile = sc.textFile(“input.txt”)

The output for the above command is

inputfile: org.apache.spark.rdd.RDD[String] = input.txt MappedRDD[1] at textFil