[Hadoop] Hadoop Architecture

[WarrenLo]

Hadoop 基礎架構

Hadoop包含了四大基本模組:

• Hadoop Common: The common utilities that support the other Hadoop modules.
• Hadoop Distributed File System (HDFS™): A distributed file system that provides high-throughput access to application data.
• Hadoop YARN: A framework for job scheduling and cluster resource management.
• Hadoop MapReduce: A YARN-based system for parallel processing of large data sets.
• Others: Spark RDD

MapReducer Programming Model

MapReducer可使用多種程式語言，但Java最快速。

• 內部邏輯
  HDFS (Input) --> Splitting --> Mapping -->透過網路 --> Shuffling --> Reducing --> HDFS (Final Result)
  MapReducer分成Map與Reduce，中間必須透過網路傳輸。
  Shuffling: 負責資料彙整; 由YARN的Node Manager執行。

• Mapreducer 與 Spark RDD 比較
  MapReducer --> Disk Killer
  Spark RDD --> Memory Killer