Určeno pro ...

• Software developers
• IT professionals

Přednáší ...

Lektoři z firmy: DataScript s.r.o.

Program kurzu, obsah studia ...

Goals Mastering Apache Spark is an extended workshop designed to familiarize attendees with Apache Spark - a well-known engine for fast Big Data processing. From zero to hero means that attendes do not need to have prior knowledge of how Apache Spark works, but after this workshop will be able to spin efficient, robust jobs running on Spark cluster starting from day one. They will not only gain general understanding of what Apache Spark is, what are its modules and how they work. Attendee will get detailed, in-depth insight of Apache Spark internals. This makes this workshop globally unique it prepares its graduates not only to start working with the framework, but also makes them right candidates to tackle any problems that might occur on production - A true heros that each distributed project needs.
Instructor Instructor has more then 10 years of profesional experience as software engineer - during which he worked on numerous number of projects for veriety of clients. He often
* Mastering Apache Spark is an extended workshop designed to familiarize attendees with Apache Spark - a well-known engine for fast Big Data processing. From zero to hero means that attendes do not need to have prior knowledge of how Apache Spark works, but after this workshop will be able to spin efficient, robust jobs running on Spark cluster starting from day one. They will not only gain general understanding of what Apache Spark is, what are its modules and how they work. Attendee will get detailed, in-depth insight of Apache Spark internals. This makes this workshop globally unique it prepares its graduates not only to start working with the framework, but also makes them right candidates to tackle any problems that might occur on production - A true heros that each distributed project needs.
Instructor Instructor has more then 10 years of profesional experience as software engineer - during which he worked on numerous number of projects for veriety of clients. He often call him self a veteran of early Apache Spark adoptions - his first big data project (based on Apache Spark) dates back to 2014. Back then Spark (version 0.
09.) was still under heavy development, where every problem encountered requried reading Spark s codebase - googling the question was pointless. That being a pain then, prove to be a great asset after a while. Experience gathered was invaluable, allowing Pawel to challange other Spark-based projects he worked on ever since. The knowledge he is now happy to share with you! Recently he was hired to write Apache Spark connector between HDFS & SlamData s Quasar project.
Outline

• Introduction
  
  • What is Apache Spark?
  • What was before?
  • Challenges, proble & issues with MapReduce
  • The Big Picture
  
  
• Spark Core
  
  • RDD
  • Transforamtions vs Actions
  • Partitions & Tasks
  • RDDs for key-value
  • Pipelining & Shuffeling
  • DAG & Stages
  • Resilience
  • Performance issues and how to handle them
    • Common pitfals (groupBy)
    • Deepr knowledge of how cluster works
    • Classpath & Serialization issues
    • Spark configuration
    • Spark UI & Spark history server
  • Caching & Checkpointing
  • Broadcasts & Accumulators
  • Joins
  
  
• Spark Core - internals
  
  • The 5 things definig RDD
  • Shuffeling algorithms
  • Spark Memory Model
  
  
• Spark SQL
  
  • Adventages of semi-structured data
  • SQL
    
    • Introduction
    • Joins
    • Hive
    
    
  • Dataframes
    
    • Introduction
    • Joins
    • JSON
    • Parquet
    • Avro
    
    
  • DataSets
    
    • Problems with Dataframes
    • Dataset to the rescue!
    
    
  • What makes Spark SQL run faster
    
    • Structure vs Expression
    • Catalyst Optimizer
    • Predicate Push Down
    • Project Tungsten
    
    
  
  
• Spark

Cíl školení / poznámka ke kurzu

Goals Mastering Apache Spark is an extended workshop designed to familiarize attendees with Apache Spark - a well-known engine for fast Big Data processing. From zero to hero means that attendes do not need to have prior knowledge of how Apache Spark works, but after this workshop will be able to spin efficient, robust jobs running on Spark cluster starting from day one. They will not only gain general understanding of what Apache Spark is, what are its modules and how they work. Attendee will get detailed, in-depth insight of Apache Spark internals. This makes this workshop globally unique it prepares its graduates not only to start working with the framework, but also makes them right candidates to tackle any problems that might occur on production - A true heros that each distributed project needs.
Instructor Instructor has more then 10 years of profesional experience as software engineer - during which he worked on numerous number of projects for veriety of clients. He often call him self a veteran of early Apache Spark adoptions - his first big data project (based on Apache Spark) dates back to 2014. Back then Spark (version 0.
09.) was still under heavy development, where every problem encountered requried reading Spark s codebase - googling the question was pointless. That being a pain then, prove to be a great asset after a while. Experience gathered was invaluable, allowing Pawel to challange other Spark-based projects he worked on ever since. The knowledge he is now happy to share with you! Recently he was hired to write Apache Spark connector between HDFS & SlamData s Quasar project.

Informace o rekvalifikaci

Tento kurz není rekvalifikační