NATS Weekly #29

Week of May 30 - June 5, 2022

🗞 Announcements, writings, and projects

A short list of announcements, blog posts, projects updates and other news.

️ 💁 News

News or announcements that don’t fit into the other categories.

⚡Releases

Official releases from NATS repos and others in the ecosystem.

nats.java - v2.15.2

⚙️ Projects️

Open Source and commercial projects (or products) that use or extends NATS.

ripienaar/nats-roundtripper - Go http.RoundTripper that uses NATS as a transport. This is not a new project, but wanted to make folks aware given a nice integration that Wally prototyped with the new Connect project.

🎬 Media

Audio or video recordings about or referencing NATS.

Building for the (Inevitable) Next Cloud Outage - by Pavel Nikolov
State of NATS: Core, Persistence, & Materialized Views - by Tomasz Pietrek

💬 Discussions

Github Discussions from various NATS repositories.

nats-server - Does jetstream work with the embedded test server?

💡 Recently asked questions

Questions sourced from Slack, Twitter, or individuals. Responses and examples are in my own words, unless otherwise noted.

How can I perform backups of stream data?

If you are hosting your own NATS server/cluster, it may be tempting to do a cp -r /jetstream /backup/ expecting a reliable backup, but this could result in corrupt copied files since the server would be in mid-write/sync if its running. Of course if the server is offline, then this method would be fine.

That said, for an online server, the standard way is using the backup command provided by the CLI. To do a full account backup, use the following:

nats account backup ./backup

And then to restore,

nats account restore ./backup

Note that this would need to be run per account. There is also a backup and restore at the stream level.

nats stream backup ORDERS ./ORDERS

and corresponding restore.

nats stream restore ./ORDERS

Note, there were previously nats backup and nats restore commands, but these have been deprecated in favor of the above per-account and per-stream ones.

One final highlight is to remember that the CLI is just another NATS client! So you can (and should) run it from a separate node. A good practice could also be to immediately upload it to object storage for long-term and low-cost storage.

In the NATS configuration, there is a [jetstream](https://docs.nats.io/running-a-nats-service/configuration#jetstream) section the enables you to set the server’s max file and max memory storage limits.

jetstream {

max_file: 500G

max_mem: 32G

store_dir: /opt/nats/jetstream/data

}

This is a shared limit across accounts and thus across all streams. Note, that if these values are not set explicitly, the memory will default to 75% of the available memory and up to 1TB of storage. These resources are not pre-allocated and can be changed if the underlying node resources change. In practice, they should be explicitly set to ensure consistency with a deployment since all nodes in a cluster should have the same resources allocated.

If accounts are being used, whether they are defined in the configuration file or using nsc, limits can be set. The sum of the limits across accounts must be less than or equal to the above store limits. Attempting to add another account or changing the size that exceeds the total limit would result in:

Can't start JetStream: Error enabling jetstream on configured accounts: insufficient storage resources available (10047)

Each account can have zero or more streams, each of which can have limits, including total size in bytes.

nats stream add --max-bytes=1G

Showing the explicit option, but this option would be prompted interactively if not set.The same behavior will result if the total size would exceed the account’s limit.

nats: error: could not edit Stream ORDERS: insufficient storage resources available (10047)

One important final note is in all of these cases, limits can be explicitly disabled. This includes the server store limits. The side effect of this is that streams and/or accounts could runaway and consume all the storage. This would result in a runtime “insufficient storage resources” once the server attempts to append and there is no disk space left, for example. So it is a good idea to set limits especially since they can be changed over time.