Kafka Basics

1. What is Apache Kafka?
   Kafka is a distributed event streaming platform for high-throughput, fault-tolerant messaging, and real-time analytics.

2. What are the core components of Kafka?

   • Producer: Sends messages to topics.
   • Consumer: Reads messages from topics.
   • Broker: A Kafka server that stores and serves data.
   • Topic: A category to which messages are published.

3. What is a Kafka topic?
   A topic is a stream of data identified by a unique name in Kafka.

4. What is a partition in Kafka?
   Topics are divided into partitions for parallelism and scalability.

5. What is a Kafka cluster?
   A cluster is a group of brokers working together to distribute and manage data streams.

Kafka Architecture

6. What is a Kafka broker?
   A Kafka broker is a server in the Kafka cluster that stores and serves data.

7. What is ZooKeeper’s role in Kafka?
   ZooKeeper manages metadata, configurations, and leader elections in Kafka.

8. What is the role of a producer in Kafka?
   A producer sends data (messages) to Kafka topics.

9. What is the role of a consumer in Kafka?
   A consumer subscribes to topics and processes messages.

10. What is the leader and follower in Kafka?

    • Leader: Handles all read/write requests for a partition.
    • Follower: Replicates data from the leader and takes over if the leader fails.

Kafka Producers and Consumers

11. What is an offset in Kafka?
    A unique identifier for each message in a partition, tracking its position.

12. What is a consumer group?
    A group of consumers that share the load of reading from partitions of a topic.

13. What is the difference between at-most-once, at-least-once, and exactly-once delivery semantics in Kafka?

    • At-most-once: Messages are delivered once, possibly with loss.
    • At-least-once: Messages are retried until successfully consumed.
    • Exactly-once: Messages are delivered exactly once without duplication.

14. How do producers ensure message ordering in Kafka?
    By sending messages with the same key to a specific partition.

15. What is the difference between synchronous and asynchronous sends in Kafka?

    • Synchronous: Producer waits for broker acknowledgment.
    • Asynchronous: Messages are sent without waiting for acknowledgment.

Kafka Topics and Partitions

16. How do you create a Kafka topic?
    Use the kafka-topics.sh script with the --create option.

17. What happens if a partition becomes unavailable?
    Kafka promotes one of the followers to be the new leader.

18. How is partitioning done in Kafka?
    Based on the message key or round-robin if no key is provided.

19. What is a topic replication factor?
    The number of copies of a topic’s partitions across brokers.

20. What is ISR (In-Sync Replica)?
    ISR is a set of replicas that are fully synced with the leader.

Kafka Operations

21. How do you monitor Kafka?
    Use tools like JMX, Kafka Manager, or Prometheus with Grafana.

22. What is log retention in Kafka?
    A configuration that determines how long Kafka retains messages.

23. What is compaction in Kafka?
    A process to retain only the most recent messages for each key in a log.

24. How do you delete a Kafka topic?
    Enable topic deletion in the broker configuration and use kafka-topics.sh.

25. How do you scale Kafka?

    • Add more brokers.
    • Increase the number of partitions for topics.

Kafka Performance and Tuning

26. How can you improve Kafka performance?

    • Use batch processing.
    • Optimize producer and consumer configurations.
    • Tune broker settings.

27. What is message batching in Kafka?
    Combining multiple messages into one batch to reduce network overhead.

28. What is Kafka throughput?
    The number of messages processed per unit time.

29. How does Kafka achieve fault tolerance?
    Through replication and leader election for partitions.

30. What is Kafka’s backpressure handling?
    It prevents overwhelming consumers by slowing down producers when required.

Advanced Kafka Concepts

31. What is Kafka Streams?
    A client library for building real-time stream processing applications.

32. What is KSQL?
    A SQL-like language for stream processing on Kafka topics.

33. What is Kafka Connect?
    A tool for connecting Kafka with external systems like databases or file systems.

34. What is Schema Registry?
    A component for managing Avro schemas for Kafka messages.

35. What is the difference between Kafka and traditional message queues?
    Kafka supports distributed, scalable, and replayable logs, while traditional queues lack these features.

Kafka Security

36. How is security implemented in Kafka?

    • Encryption (SSL)
    • Authentication (SASL)
    • Authorization (ACLs)

37. What are Kafka ACLs?
    Access Control Lists for managing permissions to Kafka resources.

38. How do you encrypt Kafka data?
    Use SSL for encrypting data in transit.

39. What is Kafka’s authentication mechanism?
    SASL (Simple Authentication and Security Layer) or SSL certificates.

40. What is the principle of idempotent producers in Kafka?
    Ensures that a message is written exactly once, even if retries occur.

Fault Tolerance and Recovery

41. How does Kafka handle broker failures?
    Promotes ISR replicas to leaders for unavailable partitions.

42. What is the unclean leader election in Kafka?
    A mechanism to choose a new leader from non-ISR replicas in critical situations.

43. What happens if ZooKeeper fails?
    Kafka operations will halt until ZooKeeper is restored.

44. How do you recover a failed Kafka broker?
    Replace the failed broker and reassign partitions.

45. What is a dead-letter queue in Kafka?
    A special topic for storing messages that cannot be processed.

Scenario-Based Questions

46. How do you migrate data from one Kafka cluster to another?
    Use tools like Kafka MirrorMaker or Kafka Connect.

47. How do you handle large message sizes in Kafka?
    Increase the max.message.bytes configuration and use compression.

48. How do you troubleshoot a slow Kafka consumer?

    • Check consumer lag.
    • Optimize poll intervals and buffer sizes.
    • Monitor partition distribution.

49. What would you do if a Kafka partition is overloaded?
    Increase partitions or redistribute load by rebalancing consumers.

50. How do you integrate Kafka with a data pipeline?
    Use Kafka Connect for ingestion and Kafka Streams for real-time processing.