Apache Kafka Key Terminologies and Architecture Internal Working | Features | In Depth Explanation

Video Description

In this video of code decode we have kafka architecture which is very important in terms of kafka interview questions and answers Udemy Course of Code Decode on Microservice k8s AWS CICD link: https://openinapp.co/udemycourse Course Description Video : https://yt.openinapp.co/dmjvd Introduction to kafka : https://youtu.be/rl5PndGAq7s Kafka implementation with producer and consumer : https://youtu.be/WOPPG5tyApg Topic — A bucket of messages (or events in case of kafka) where services may place or read messages from. Messages are organized and durably stored in topics. A topic is similar to a folder, and these messages are the files in that folder. Topics are multi-producer and multi-subscriber. Messages can be read as many times as needed, unlike traditional messaging systems, msgs / events in kafka are not deleted after consumption. Instead, you can define for how long Kafka should retain those events/msgs. Kafka topics are divided into a number of partitions, which contain msgs/events in an unchangeable sequence. Each msg in a partition is assigned and identified by its unique offset. A topic can also have multiple partition. This allows multiple consumers to read from a topic in parallel. In Kafka, replication is implemented at the partition level. The redundant unit of a partition is called a replica. Each partition usually has one or more replicas over a few Kafka brokers in the cluster. Every partition (replica) has one server acting as a leader and the rest of them as followers. The leader replica handles all read-write requests for the specific partition and the followers replicate the leader. If the lead server fails, one of the follower servers becomes the leader by default. When a producer publishes a msg to a topic, it is published to its leader. The leader appends the msg to its offset and increments its msg offset. Summary Kafka has a sized component called a cluster. Inside a cluster, we’ve got several servers, also known as brokers (usually at least three brokers to provide enough redundancy.) when a message gets sent to a broker, it gets sent to a particular topic.Topics allow us to categorize data. The data can eventually get broken into several partitions. By splitting the topic into multiple partitions, we’re facilitating scalability, as each partition can be read by a separate consumer. Each broker is responsible for receiving messages from producers and committing those messages to disk. The broker is also responsible for answering consumers’ fetch requests and serving them. While designing a Kafka system, it’s always a wise decision to factor in topic replication. As a result, its topics’ replicas from another broker can solve the crisis, if a broker goes down. Broker1 has Topic 1 and 10 partitions , its replica is in Broker2, so on and so forth. It has got a replication factor of 2; it means it will have one additional copy other than the primary one. It is not possible to have the number of replication factor more than the number of available brokers. Moreover, exactly one consumer instance reads the data from one partition in one consumer group, at the time of reading. Since, there is more than one consumer group, in that case, one instance from each of these groups can read from one single partition. However, there will be some inactive consumers, if the number of consumers exceeds the number of partitions. Let’s understand it with an example if there are 8 consumers and 6 partitions in a single consumer group, that means there will be 2 inactive consumers. Kafka Cluster: A Kafka cluster is a system that comprises of different brokers, topics, and their respective partitions. Most Asked Core Java Interview Questions and Answers : https://youtube.com/playlist?list=PLyHJZXNdCXscoyL5XEZoHHZ86_6h3GWE1 Advance Java Interview Questions and Answers : https://youtube.com/playlist?list=PLyHJZXNdCXsexOO1VQ4vs-BM2-8CKrixd Java 8 Interview Questions and Answers : https://youtube.com/playlist?list=PLyHJZXNdCXsdeusn4OM33415DCMQ6sUKy Hibernate Interview Questions and Answers : https://youtube.com/playlist?list=PLyHJZXNdCXsdC-p2186C6NO4FpadnCC_q Spring Boot Interview Questions and Answers : https://youtube.com/playlist?list=PLyHJZXNdCXsexOO1VQ4vs-BM2-8CKrixd Angular Playlist : https://www.youtube.com/watch?v=CAl7RQSdq2Q&list=PLyHJZXNdCXsfxRtDwtGkDD_lLfTWc1g0i SQL Playlist : https://www.youtube.com/playlist?list=PLyHJZXNdCXse86eLuwy5uZohd_bddE9Ni GIT : https://youtube.com/playlist?list=PLyHJZXNdCXscpl6pxOnL2lRWJlzvzjyZE Subscriber and Follow Code Decode Subscriber Code Decode : https://www.youtube.com/c/CodeDecode?sub_confirmation=1 LinkedIn : https://www.linkedin.com/in/codedecodeyoutube/ Instagram : https://www.instagram.com/codedecode25/ #kafka #codedecode #kakfainterviewquestions