🗄️Big Data

Big Data with Hadoop and Spark: full course guide

Distributed storage and processing: HDFS, YARN, MapReduce, Spark core, Spark SQL, streaming and the Hive and Kafka ecosystem, with runnable examples throughout.

94 pages6 modules30 topics

Beginner to Advanced · read it in your browser with Skiiling Pro.

📝 Evaluation25 Q🎙️ Mock Interview
Beta

Loading the reader…

What is inside this guide

01Big Data Fundamentals

  • What is Big Data?
  • The 5 Vs of Big Data
  • Distributed Computing Concepts
  • CAP Theorem
  • Batch vs Streaming Processing

02Hadoop Ecosystem

  • HDFS Architecture
  • MapReduce Programming
  • YARN Resource Manager
  • Hive & HiveQL
  • HBase & NoSQL Storage

03Apache Spark

  • Spark Architecture & RDDs
  • DataFrames & Datasets
  • Spark SQL
  • Transformations vs Actions
  • Spark Execution Model

04Spark Streaming

  • Structured Streaming Basics
  • Reading from Kafka
  • Windowing Operations
  • Stateful Processing
  • Checkpointing & Fault Tolerance

05Performance & Optimisation

  • Partitioning Strategies
  • Shuffle & Skew Handling
  • Caching & Persistence
  • Broadcast Joins
  • Spark UI & Debugging

06Cloud & Production

  • Running Spark on EMR / Dataproc
  • Delta Lake & Data Lakehouse
  • Data Pipeline Orchestration
  • Monitoring & Alerting
  • Cost Optimisation

How to read it

The reader above opens the complete guide in your browser. The contents panel on the left is built from the guide's own bookmarks, so every module and topic is one click away. You can search the full text, zoom, and highlight passages in four colours; highlights and your reading position are remembered on this device.

Reading the guides is part of Skiiling Pro, the one web subscription that also unlocks Role Prep and Practical Labs and Skill Plans. The syllabus above is free to browse. See pricing.