Hadoop installation notes

Cluster planning:

• Small cluster (2-10 nodes): Clusters of three or more machines typically use a dedicated NameNode/ResourceManager, and all other nodes are workers. 
• Medium cluster (10-40 nodes): separate machines for master, usually dedicated edge node 
• Large cluster (> 40 nodes): occupies multiple racks, requires individual planning

Storage layer for HDFS:

• Cloud and virtualization solutions are very popular in the enterprise world 
• With Hadoop, focus on using JDOD (just bunch of disks) instead of SAN (storage area network) 
• Hadoop heterogeneous storage was introduced in Hadoop 2.5

Commodity hardware:

Balanced configuration for one worker recommended by vendors:

• 2-4 8-cores CPU 
• 128 GB RAM 
• 12 hard drives (1 or 2 TB each)

HDFS Architecture

• Individual machines are known as nodes 
• A cluster can have as few as one node, as many as several thousands
• Two types of nodes: NameNode and DataNode 
• More nodes = better performance 
• HDFS is a filesystem written in Java 
• The NameNode daemon must be running at all times If the NameNode stops, the cluster becomes inaccessible 
• The NameNode holds all of its metadata in RAM for fast access 
• A separate daemon known as the Secondary NameNode takes care of some housekeeping tasks for the NameNode
• Although files are split into 64MB or 128MB blocks
• Blocks are stored as standard files on the DataNodes, in a set of directories specified in Hadoop’s configuration files 
• Without the metadata on the NameNode, there is no way to access the files in the HDFS cluster
• When a client application wants to read a file: It communicates with the NameNode to determine which blocks make up the file, and which DataNodes those blocks reside on. It then communicates directly with the DataNodes to read the data

Курси по Big Data

http://newprolab.com/bigdata/?utm_source=datareview.info&utm_medium=referral&utm_campaign=interviewPetrov5.08.2015