NATS Weekly #34
Published on Jul 11th, 2022
Week of July 4 - 10, 2022
🗞 Announcements, writings, and projects
A short list of announcements, blog posts, projects updates and other news.
Official releases from NATS repos and others in the ecosystem.
Blog posts, tutorials, or any other text-based content about NATS.
- Distributed Message Streaming in Golang using Nats JetStream - by Ebubekir Yiğit
Github Discussions from various NATS repositories.
- nats.go - Fetch timeout
💡 Recently asked questions
Questions sourced from Slack, Twitter, or individuals. Responses and examples are in my own words, unless otherwise noted.
What exactly is the $G account?
NATS supports multi-tenancy using accounts. When no accounts are configured, a default account is created called
$G along with the
$SYS system account. This can be observed on the
accountz monitoring endpoint.
Likewise, if JetStream is enabled, you will see the
$G directory since stream/consumer data are organized and stored by account.
How can you check if a message has been re-delivered?
For JetStream which provides at-least-once quality of service (QoS), a message will be redelivered if the server doesn't receive an
TERM) from the client. This could be due to:
- the client fails to process the message within the ack wait duration
- the client crashed or had an error while processing the message
ACKmessage to the server was dropped due to a connection issue
Every JetStream message has metadata encoded in the reply subject for the server. Each client has a method or function to parse this metadata from a message. The metadata contains the field
NumDelivered which indicates the number of times the message has been delivered.
How large of a deduplication window is supported?
Thanks to Brent Dillingham for the question and analysis of what needs to be considered!
Streams can be configured to deduplicate messages when the
Nats-Msg-Id header is present. The docs state (emphasis mine):
The default window to track duplicates in is 2 minutes, this can be set on the command line using
--dupe-windowwhen creating a stream, though we would caution against large windows.
The takeaways from Brent include:
- Write performance may take a hit when the dedupe set is periodically pruned, but only if it’s a unique write; duplicate write detection would be unaffected because the mutex lock isn’t needed
- Write performance should otherwise be fairly constant since the Go map lookup is O(log(N)) and insertion is O(1)
- Server memory usage will increase
- Restoring streams on reboot will take longer because the dedupe data structures have to be rebuilt
So how large can the window be? Ultimately, it is a function of message rate and available server memory. Given a fixed size of memory, if the message rate is low, you can have a larger window. With a high message rate, the memory will grow faster so the window needs to be smaller.
The other consideration is the use case of the stream and/or what are the origin of the published messages being de-duplicated. In other words, ask "how likely is it for a duplicate message to be published after the initial attempt?" If you are simply wanting to handle the case of a retry after a brief network interruption, the default window is likely fine.
However, if the origin of the publish come from some upstream source that itself may contain duplicates (after a long period of time), then you may want to consider an alternative strategy for deduplicating.