NATS Weekly #2

Week of November 22 - 28, 2021

🗞 Announcements, writings, and projects

A short list of announcements, blog posts, projects updates and other news.

⚡️ nats.py v2.0.0rc1 release

This release has graduated from the alpha2 release from last week.

📖 NATS.io documentation site updates

The documentation has been simplified and two notable sections have been added including Running a NATS service and Key/value Store.

We’ve updated our Docs!! 📚

Next time you visit you’ll notice a different flow & added content like “Running a NATS Service” & “Key/value Store”. Walkthroughs have also been added for core NATS functions & JetStream. https://t.co/6xPPF4YwGo— NATS.io (@nats_io) November 23, 2021

⚙️ Panini - A Python microframework for NATS

Checkout the Github repo: https://github.com/lwinterface/panini. Per the repo’s README:

Panini is a python microframework based on the nats.py library. Its goal is to offer developers an easy way to create NATS microservices with a lower barrier of entry. It provides a specific template for creating microservices, similarly to FastAPI, Aiohttp, or Flask.

🎞 GopherCon UK 2021: Adelina Simion - Using NATS

https://www.youtube.com/watch?v=AhnL5addsVo

A few highlights:

Form3 is a payment provider for financial institutions
Form3 replaced use of AWS SQS and SNS with NATS due to latency and scaling concerns
Emphasis is on the simplicity of the NATS protocol and control over a managed service
Core NATS demoed, noting the (intentional) lack of persistence
Core NATS is compared to UDP (at most once)
NATS Streaming highlighted as a response to the “lack of persistence”
JetStream introduced as the replacement for NATS Streaming going forward accounting for several limitations with NATS Streaming
JetStream is a subsystem of NATS where as NATS Streaming uses a sidecar model
JetStream compared to TCP (at least once)
Use pull consumers if you can’t “keep up” with JetStream
Leverage message producer de-duplication using the Nats-Msg-Id header within a configurable window (default is two minutes)
Why not Kafka? NATS is lighter weight, simpler, and just as battled tested 😉

One quick clarifying note: the first question (of the Q&A) asked about de-duplication as it pertains to consumer replay. This de-duplication mechanism is applicable on the producer side, specifically with JetStream accepting a message that was previously published (having the same ID within the window). If a duplicate message is delivered to a consumer (for whatever reason), this needs to be handled in the application code.

💡 Recently asked questions

Questions sourced from Slack, Twitter, or individuals. Responses and examples are in my own words, unless otherwise noted.

How can I add a user without restarting the NATS server?

There are two ways to manage users in NATS. There is a simple configuration-based approach, which defines user credentials in the server configuration, or a JWT model.

For the server configuration approach, the NATS server supports reloading the configuration using nats-server --signal reload as well as the standard SIGHUP syscall. This is an atomic operation and will not disrupt any work being done that the configuration does not impact.

The second approach to managing users relies on a decentralized JWT authentication mechanism. Setting this up is out of scope for this specific question, but once the operator and accounts are setup and synced with the server, users (specifically new user credentials) can be created at any time and used to authenticate with the server.

User credentials are created by being signed with an account. Accounts need to be synced with the server (and there are generally fewer accounts than users). When a client attempts to connect, they present the user credentials and the server simply checks the chain of trust. Specifically they ensure the user credentials are signed by the account, and the account is signed by the operator.

As a result of this user management model, server restarts/reloads are not necessary since the user auth is decoupled from the server configuration.

What is the difference between subjects and streams?

Last week I touched on this topic a bit. Every message that is published to NATS must have a concrete subject associated with it.

It could as basic as a single letter b or a verbose hierarchy like weather.us.pa.philadelphia.19104. Subject names are a design choice and an important one at scale. Having good segmentation will allow for distribution and concurrency on the consumption side.

The consumption side is where things get a bit more interesting. Although a published message requires a concrete subject, a subscription can use wildcards in the subject e.g. weather.us.pa.> to get all messages applicable to Pennsylvania in the US.

So what is a stream?

It is just a fancy, managed subscription :) When defining a stream, you must provide one or more subjects (wildcards allowed) and NATS will create an internal subscription that listens for published messages on those subjects.

Upon receiving those messages, it will reply with an acknowledgement (if the publisher is expecting one). The acknowledgment indicates the message has been sufficiently persisted with respect to the storage type (memory or file) and replication policy (N servers).

A good mental model is to think that these messages are effectively suspended in time and/or space relative to the streams retention policy. This allows for consumption of the messages at a later time without needing to be online when the messages are published.

Can you subscribe directly to a stream like you can a subject? Not quite, but for good reason.. this will come in a follow-up post.

Depending on your needs there are a couple options. If you simply want to view messages in the stream, you can use the command.

nats stream view <stream>

Ensure you either have the correct context set or you pass the correct --server option and credentials depending on which method authentication method you are using.You can use the --id option to specify a sequence ID to start from, such as 0 for all messages. Alternatively you can use --since which takes a time delta, like 2h for the messages in the past two hours. Use the --help option to see all other options that are available.

The behavior of this command is to paginate the stream and expects user input after each page to either continue or not.

There are a couple constraints to this command. First, a page size can be provided, but (at the time of this writing) it has an upper bound of 25 messages. For long streams, this could be painful to paginate through.

The second constraint is that a subscription to the stream is not maintained. The view command takes a snapshot of the last known sequence and reads messages up to that point even if new messages are written to the stream in the meantime.

If you want a real subscription you have two options depending on your need. If you simply want to monitor messages coming in, you can create a push consumer. Note, that you will need a subscription already created on the subject the consumer is pushing messages to such as nats sub monitor.ORDERS (per the example in the docs).

A pull consumer and a subscription can also be created on the command line:

# Add the consumer, follow the prompts

nats consumer add

# Subscribe to the consumer, follow the prompts

nats consumer sub

Depending on your use case for using the CLI for this purpose, one of these options may be appropriate. However, if your use case is to backup the stream, use nats stream backup.