CouseSites

Complete PySpark Developer Course (Spark with Python) - Learn PySpark in depth with hundreds of Practical examples. Be a complete PySpark Developer. Set up a Hadoop Cluster.

This is a complete PySpark Developer course for Data Engineers and Data Scientists and others who wants to process Big Data in an effective manner. We will cover below topics and more:

Complete Curriculum for a successful PySpark Developer

Set up Hadoop Single Node Cluster and Integrate it with Spark 2.x and Spark 3.x

Complete Flow of Installation of Standalone PySpark (Unix and Windows Operating System)

Detailed HDFS Commands and Architecture.

Python Crash Course

Introduction to Spark (Why Spark was Developed, Spark Features, Spark Components)

Understand SparkSession

Spark RDD Fundamentals

How to Create RDDs

RDD Operations (Transformations & Actions)

Spark Cluster Architecture - Execution, YARN, JVM Processes, DAG Scheduler, Task Scheduler

RDD Persistence

Spark Shared Variables  - Broadcast

Spark Shared Variables  - Accumulators)

Spark SQL Architecture, Catalyst Optimizer, Volcano Iterator Model, Tungsten Execution Engine, Different Benchmarks

Difference between Catalyst Optimizer and Volcano Iterator Model

Spark Commonly Used Functions - Version, range, createDataFrame, sql, table, SparkContext, conf, read, udf, newSession, stop, catalog etc

DataFrame Built-in functions - new column functions, encryption functions, string functions, regexp functions, date functions, null functions, collection functions, na functions, math and statistics functions, explode functions, flatten functions, formatting and json functions

What is Partition,

What is Repartition

What is Coalesce

Repartition Vs Coalesce

Extraction - csv file, text file, Parquet File, orc file, json file, avro file, hive, jdbc

DataFrame Fundamentals

What is a DataFrame

DataFrame Sources

DataFrame Features

DataFrame Organization

DataFrame Rows,

DataFrame Columns

DataTypes. Practical examples.

Perform ETL Using DataFrame

    -- Extraction APIs

    -- Transformation APIs

    -- Loading APIs

    -- Practical Examples.

Optimization and Management - Join Strategies, Driver Conf, Parallelism Configurations, Executor Conf etc