Explaining Diffs Between Apache Kafka and AWS Kinesis

The aim of this page📝 is to explain the differences and similarities between Kafka and AWS Kinesis based on the particular example of their operational models and architecture. I am just wrapping my head around various languages as we are using Kinesis a lot, but sometimes need to deploy data forwarding into Kafka.

• Kafka operates on a system of brokers, which are servers that store and process messages.
• Kafka gives more control to the operator in its configurability.
• Kafka is a distributed messaging system that provides scalability, reliability, and fault tolerance.
• Kafka brokers do write data to disk. However, the I/O operations are really carried out by the Operating System.
• In the case of Linux, it seems the data is written to the page cache until it can be written to the disk.
• Kafka has done its job of assigning the operating system the data to be written to the disk, but it is the operating system that decides when and how to write the data.
• This approach allows Kafka to provide high performance.
• The risk of data loss can be mitigated by replicating the data over more than one broker.

On the other hand,

• Kinesis is a managed service provided by AWS.
• Kinesis is built around a stream processor engine.
• By design, Kinesis will synchronously broker data streams and write and replicate ingested data into three different AWS machines.
• This replication cannot be reconfigured, influencing resource overhead such as throughput and latency.
• The machine configuration is abstracted away in Kinesis, which means you don’t have to manage any servers.

In terms of architecture,

• In Kafka, you have topics where all related events are stored. Similarly, in Kinesis, you have streams that are the equivalent of a Kafka topic.
• Kafka gives you complete freedom to define any number of partitions based on the cluster specifications. On the other hand, Kinesis uses shards.