HDFS Architecture

• Individual machines are known as nodes 
• A cluster can have as few as one node, as many as several thousands
• Two types of nodes: NameNode and DataNode 
• More nodes = better performance 
• HDFS is a filesystem written in Java 
• The NameNode daemon must be running at all times If the NameNode stops, the cluster becomes inaccessible 
• The NameNode holds all of its metadata in RAM for fast access 
• A separate daemon known as the Secondary NameNode takes care of some housekeeping tasks for the NameNode
• Although files are split into 64MB or 128MB blocks
• Blocks are stored as standard files on the DataNodes, in a set of directories specified in Hadoop’s configuration files 
• Without the metadata on the NameNode, there is no way to access the files in the HDFS cluster
• When a client application wants to read a file: It communicates with the NameNode to determine which blocks make up the file, and which DataNodes those blocks reside on. It then communicates directly with the DataNodes to read the data