As part of our Event-Driven Systems series, Ricky in the first part wrote about Apache Kafka as the most widely used message broker and why you should use Kafka. The second part was about how you can roll out an Apache Kafka in the cloud, this is how you do it, as Raffael showed us, namely with Strimzi. In the first part we looked at Apache Kafka and what Kafka is particularly suitable for.
The third part, which I would like to present to you today, is about whether there is not a cloud-native alternative to Kafka Apache and whether it shouldn’t be rolled out on your cloud, as a cloud-native alternative could be more resource-efficient and thus more cost-effective. The alternative is there and is called NATS. So we are jointly examining another message broker, which, like Apache, does not need an underlying JVM and is therefore much lighter.
NATS server
NATS is an open source (Apache 2.0) and cloud-native messaging system for adaptive edge and distributed systems. It was originally written in Ruby by Derek Collison and later ported to Go. NATS (or NATS Messaging) enables the exchange between applications and services. In doing so, data is packed into messages, which are then addressed by ‘Subject’ and not by IP or DNS name. This abstracts the underlying physical network layer. The data to be sent is encrypted and “packed” into a message by the sender. The message can then be received, decrypted and processed by one or more recipients.
Nowadays, there are more than 40 client API’s, like for example for Go, Java, JavaScript/TypeScript, Python, Ruby, Rust, C#, C, and NGINX.
It should be noted that NATS does not persist any messages. If a client is not available at the time of the message, this message is no longer visible to him.
NATS streaming
For this reason there is NATS Streaming. NATS Streaming implements message persistence and a message delivery guarantee.
Every connected client has to send a ACK, short for Acknowledge, when receiving the message.
If he does not do this, the message will be sent again after a certain time.
This means that it is possible that a message arrives twice at the recipient.
A NATS Streaming Server includes a NATS server. The messages are still received by the NATS server and forwarded to the streaming server.
The NATS streaming server can be made available securely and with high availability, but there are some restrictions due to the architecture.
-
NATS streaming is not a ‘work queue’, it is ‘message log based’. This means that messages are not deleted by
ACKs, only by limitations. -
Not horizontally scalable (#999)
-
Bad integration with NATS 2.0 / accounts / security concepts. No multi-tenancy (#1043)
-
Clients cannot pull messages themselves, messages are only pushed to them.
-
clients cannot register on specific channels (#1122)
NATS Jetstream
Release 2.2.0 of NATS was published in March 2021. With this release, NATS Jetstream was introduced, which fixed many problems with NATS Streaming (see NATS Streaming).
In addition, Jetstream has a number of features. We want to look at a very interesting one in detail.
Wildcards
At its core, NATS is responsible for sending and receiving messages. Sending and receiving are based on “subjects”, which assign messages in streams or topics. A subject is a simple case-sensitive string, which consists of alphanumeric characters and the “.” may exist.
The point has another useful function. You can use this to build a “Subject” hierarchy. For example, a structure could look like this. This is a logical structure.
|
|
NATS now offers us two wildcards that can be used by the recipient to listen to several subjects.
Matching A Single Token (*)
If a recipient wants to receive all the news from the internally used programming languages, he can subscribe to the following.
|
|
He now receives all the news from the Go and Java world.
However, the wildcard only means exactly one string.
So if you want to receive all the news about programming languages and frameworks, you cannot use this, since com.bova.developer.* would only select one additional level (so com.bnova.developer.language or com.bnova.developer.framework).
Matching Multiple Tokens (>)
There is another wildcard to select several levels.
|
|
Now all subjects under com.bnova.developer can be selected.
It is also possible to subscribe to all subjects using the following subject. Of course, you can only read those that you have access to. Can only be used at the end!
|
|
It is also possible to mix both wildcards.
|
|
would for example select com.bnova or ch.bnova.
NATS in action
Now let’s see how NATS works in practice. We start a local NATS server and use the official Docker image for this. So that we can use Jetstream directly, we have to specify -js as an argument.
|
|
Now we start a second container in which all NATS tools are already installed.
|
|
As we can see, we now receive a prompt and can now begin to interact with the server.
Streams
First, let’s create a stream. Streams define how messages are saved and retained. Streams consume normal NATS subjects. Every message found in these subjects is sent to the defined storage.
|
|
Let’s look at the configurations in detail
|
|
-
Subjects to consume =
com.bnova.>: Here we define which subjects we want to listen to -
Storage backend =
file: Should the stream be kept in the file or memory -
Retention Policy =
Limits: Only a certain number of messages should be saved -
Discard Policy =
Old: Old messages should be deleted when the maximum number of messages is reached -
Stream Messages Limit =
-1: Number of messages in the stream (-1 infinite) -
Message size limit =
-1: size of the entire message (-1 infinite) -
Maximum message age limit =
2m: Messages are kept for 2 minutes -
Maximum individual message size =
-1: Maximum size of a message (-1 infinite) -
Duplicate tracking time window =
2m: Time in which duplicates are checked -
Replicas =
1: Number of replicas
Write and read the first message
Now we can write our first message in the stream. To do this, we run the following command:
|
|
And look directly at what our stream looks like:
|
|
As we can see, there is now a message in our stream. We can now call this up for 2 minutes before it is deleted again by the Maximum Age configuration.
Let us now write a consumer with which we can read the message from the stream again.
|
|
Let’s take a look at the configuration again in detail:
|
|
-
Consumer name =
language-consumer: Name of the consumer -
Delivery target =
pull: Consumer pulls the messages from the sender -
Start policy =
all: All messages in the stream should be read -
Replay policy =
instant: Consumer will receive all messages as soon as possible -
Filter Stream by subject =
com.bnova.language: Read only from this subject -
Maximum Allowed Deliveries =
-1: ??? -
Maximum Acknowledgments Pending =
0: ???
Now we can read the message from the stream with our newly created consumer. To do this, we have to enter the following command:
|
|
So now you know what NATS is and how you can set up a NATS environment in your Kubernetes cluster. Stay tuned!
This text was automatically translated with our golang markdown translator.
