Top 50 Hadoop Interview Questions and Answers

What is Hadoop?
Hadoop is an open-source framework for storing and processing large datasets in a distributed environment.
What are the core components of Hadoop?
- Hadoop Distributed File System (HDFS)
- MapReduce
- Yet Another Resource Negotiator (YARN)
What are the features of Hadoop?
Scalability, fault tolerance, flexibility, distributed storage, and cost-effectiveness.
What is HDFS?
HDFS is the storage layer of Hadoop, designed to store large datasets across multiple nodes.
What are the advantages of Hadoop?
- Handles big data
- Fault-tolerant
- Open-source
- Scalable and flexible

What is the role of NameNode in HDFS?
The NameNode manages metadata, such as file locations and permissions.
What is the role of DataNode in HDFS?
The DataNode stores actual data in blocks and communicates with the NameNode.
What is block size in HDFS?
Default block size is 128 MB (configurable). HDFS splits files into fixed-size blocks for distributed storage.
What is a Secondary NameNode?
It periodically saves a snapshot of the NameNode’s metadata but does not replace the NameNode.
What is the replication factor in HDFS?
It determines the number of copies of data stored across nodes. Default is 3.

What is YARN in Hadoop?
YARN is the resource management layer in Hadoop, managing cluster resources and scheduling tasks.
What are the components of YARN?
- ResourceManager: Manages resources and applications.
- NodeManager: Monitors resource usage on individual nodes.
- ApplicationMaster: Manages the lifecycle of individual applications.
What is a Container in YARN?
A container is a unit of resources (CPU, memory) allocated to run tasks.
What is the role of the ResourceManager?
It allocates cluster resources and schedules tasks across nodes.
What is the difference between YARN and MapReduce 1?
YARN decouples resource management from task scheduling, whereas MapReduce 1 combines both.

What is MapReduce?
A programming model for processing large datasets by dividing tasks into Map and Reduce phases.
What is the role of the Mapper?
The Mapper processes input data and generates intermediate key-value pairs.
What is the role of the Reducer?
The Reducer processes intermediate key-value pairs to produce the final output.
What is the difference between Combiner and Reducer?
The Combiner acts as a mini-Reducer, reducing data transfer between the Mapper and Reducer.
What is a Partition in MapReduce?
A partition determines how Mapper output is distributed to Reducers.

What is Apache Hive?
Hive is a data warehouse tool for querying and managing large datasets using SQL-like queries (HiveQL).
What is Apache Pig?
Pig is a scripting platform for analyzing large datasets using a language called Pig Latin.
What is Apache HBase?
HBase is a NoSQL database that runs on HDFS for real-time read/write access to large datasets.
What is Apache Spark?
Spark is a fast, in-memory data processing engine that integrates with Hadoop.
What is Apache Flume?
A tool for ingesting large volumes of streaming data into Hadoop.

How do you monitor a Hadoop cluster?
Use tools like Ambari, Ganglia, or the Hadoop Web UI.
What is Rack Awareness in Hadoop?
A mechanism to ensure data replication across different racks for fault tolerance.
What are the different modes in which Hadoop runs?
- Standalone Mode
- Pseudo-Distributed Mode
- Fully Distributed Mode
What are Hadoop’s configuration files?
- core-site.xml
- hdfs-site.xml
- mapred-site.xml
- yarn-site.xml
How is fault tolerance achieved in Hadoop?
Through data replication and the ability to re-run failed tasks.

What is a Hadoop Federation?
Federation allows multiple NameNodes to manage separate namespaces in a cluster.
What is Hadoop High Availability?
A setup with multiple NameNodes to ensure the system remains operational during failures.
What is speculative execution in Hadoop?
A mechanism to re-run slow tasks on different nodes to speed up job completion.
What is Hadoop Streaming?
A utility that allows MapReduce jobs to be written in any language, like Python or Perl.
What is the difference between InputFormat and OutputFormat?
- InputFormat defines how input data is split and read.
- OutputFormat defines how output data is written.

How do you optimize MapReduce jobs?
- Use Combiners.
- Optimize input splits.
- Use counters and profilers.
What is data locality in Hadoop?
The principle of processing data on the node where it is stored to minimize network overhead.
How do you handle small files in Hadoop?
Combine small files into larger files using tools like SequenceFile or HDFS Federation.
What is speculative execution?
A feature to re-execute slow tasks on another node to avoid bottlenecks.
What is the role of Shuffle and Sort in MapReduce?
It sorts and redistributes intermediate key-value pairs to Reducers.

How is security implemented in Hadoop?
Through Kerberos authentication, HDFS permissions, and network encryption.
What is Hadoop’s role-based access control (RBAC)?
A mechanism to assign permissions based on roles to ensure data security.
What is Kerberos in Hadoop?
A network authentication protocol used for securing Hadoop clusters.
How do you encrypt data in Hadoop?
Enable data encryption at rest and in transit using Hadoop security configurations.
What are Service Level Authorizations in Hadoop?
They ensure only authorized users can perform administrative tasks.

How do you troubleshoot a slow-running Hadoop job?
- Check the cluster resources.
- Examine task logs.
- Identify data skew or network bottlenecks.
How do you recover a failed NameNode?
- Use the Secondary NameNode metadata backup.
- Start a new NameNode and restore the checkpoint.
How do you scale a Hadoop cluster?
Add new nodes to the cluster and update configuration files.
How do you migrate data from RDBMS to Hadoop?
Use tools like Apache Sqoop.
How do you secure sensitive data in Hadoop?
- Use HDFS encryption.
- Mask sensitive data during ingestion.
- Implement Kerberos authentication.