It’s critical that the output of tee is again redirected to stderr using >&2 — if you don’t do this, the text from the log will end up in the stdout that gets piped to the mbuffer, which will get written to tape. In that circumstance, tar will not be able to understand the archive when reading it back, since there will be spurious text data.
If you aren’t piping tar’s stdout through mbuffer, you can avoid the redirection problem because tar won’t be outputting to stdout at all. For example:
tar --label="backup-20230101-volume2" -b512 -cvf /dev/nst0 --exclude='.DS_Store' --files-from=/tmp/directories-list.txt 2> tee /tmp/tar-filelist.txt
I got a Tandberg LTO-6 drive off eBay recently as a way to have an offline, air-gapped third backup of data that normally lives on my NAS or backup storage server.
Although my NAS is already backed up daily to a ZFS pool on another server, all of these systems are networked—and therefore, vulnerable to ransomware, malware, sloppy sysadmin commands on the terminal, and even electric-surge-caused hardware malfunction. And although I do back up some data to cloud storage, not all data is worth the recurring monthly charges of S3/Glacier/Backblaze B2. Besides, playing with hardware is fun.
Magnetic tape, which can store as much as 2.5 TB uncompressed (in LTO-6, the generation I started with) or 12 TB uncompressed (in LTO-8, the current generation as of mid-2021), is a time-tested option that fits in perfectly.
Veeam Backup & Replication Community Edition works well with standalone tape drives. However, it’s a proprietary system that uses Microsoft Tape Format for the on-tape format—a format that is very challenging to recover yourself without using proprietary tools. Moreover, the tape backup mechanism in Community Edition (i.e., without using licensed NAS backup features) is not meant for backing up large volumes of general purpose files—it’s really designed for archiving VM backups from disk.
LTFS also works. However, my initial attempts to use it were foiled by a Microsemi HBA that doesn’t support TLR. Also, if you don’t use proprietary tape software, LTFS can actually perform more slowly for a bunch of reasons (e.g., multithreaded copying, large number of small files, etc.).
When using a Linux desktop, way more options are available using decades-old software that was designed for tape from the get-go.
This Tandberg drive seems to have the same guts as an HP LTO-6 drive. 256-bit encryption keys can be generated and loaded, but these drives require an extra flag (-a 1). The convenience advantage of enabling hardware encryption is that we can stream from tar directly to tape and back, and the encryption is all transparent to the applications.
stenc -g 256 -k keyfile.key -kd "optional key description"
stenc -f /dev/nst0 -e on -a 1 --ckod --protect -k keyfile.key
stenc -f /dev/nst0 --detail
stenc -f /dev/nst0 -e off -a 1
Bonus: Encoding a barcode into cartridge memory (aka LTO-CM or MAM) using IBM ITDT
We can try to read the attribute from the cartridge using ITDT:
.\itdt.exe -f \\.\tape0 readattr -p 0 -a 0806 -d 0806.bin
And we can try to encode it to the cartridge using ITDT:
.\itdt.exe -f \\.\tape0 writeattr -p 0 -a 0806 -s 0806.bin
Here’s the evidence that the barcode was properly encoded:
Appendix: Source Code
These are backups of the open source programs used above, providing some assurance that even if these programs end up disappearing from Linux distributions’ package repositories, I will still be able to access the data stored on these tapes. (There’s probably nothing to worry about here; it’s more likely LTO-6 drives will be EOL long before tar and mt-st disappear.)
I’ve been backing up some of my larger files to Bluray lately, instead of trying to upload them over a 10 Mbps uplink.
In the past, I used GPG (on a .tar or compressed .tar.xz) or Veracrypt (on a file container) to encrypt at rest, before burning those files onto a standard UDF/ISO9660 optical disc. Now that I use a Linux desktop, I wanted something slightly more native — a method that
protects the directory structure and filenames without needing to use an archive file (like .tar);
would be generally unintelligible on a Windows PC (this is a feature, not a bug); and
could be scripted on the command line for server backups, without requiring a GUI.
Based on some resources online, I settled on using LUKS.