Much easier way to encode LTO barcode into the tape cartridge memory

In a previous blog post, I used ITDT on Windows to write a barcode to an LTO6 tape’s 0x0806 attribute. I had to resort to that method because most Linux distros came with the sg_read_attr utility, but no way to write attributes.

But in sg3_utils 1.48 (released 20230801), a sg_write_attr utility was added!

  • sg_write_attr: new utility, Write Attribute is a SPC command that is used mainly by tape drives (ssc)

Now it is trivial to both read and write to the MAM attributes on the command line on a Linux machine connected to the tape drive. Use with caution!

Example of reading an attribute:

[frederick@tornado ~]$ sudo sg_read_attr -f 0x806 /dev/nst0
  Barcode: FJK663L6

Example of writing a barcode:

sudo sg_write_attr /dev/nst0 0x806=FJK683L6

Some other MAM attributes that can be accessed via sg_read_attr and sg_write_attr:

IDAttributeFormat
0x0002TapeAlert flagsBinary
0x0003Load countBinary
0x0005Assigning organizationASCII
0x0006Initialization countBinary
0x0008Volume identifierASCII
0x0009Volume change referenceBinary
0x0224Logical position of first encrypted blockBinary
0x0225Logical position of first unecrypted block after first encrypted blockBinary
0x0400Medium manufacturerASCII
0x0401Medium serial numberASCII
0x0404Assigning OrganisationASCII
0x0406Medium manufacurer dateASCII
0x0408Medium typeBinary
0x0800Application vendorASCII
0x0801Application nameASCII
0x0802Application versionASCII
0x0803User medium text labelText
0x0804Date and time last writtenASCII
0x0805Text localization identiferBinary
0x0806BarcodeASCII
0x0808Media poolText
0x0809Partition user text labelASCII
0x080BApplication format versionASCII
0x080CVolume coherency informationBinary
0x0820Medium globally unique identifierBinary
0x0821Media pool globally unique identifierBinary
0x1623Volume lockedBinary

And here is another open source project for reading useful LTO MAM attributes all at once: https://github.com/speed47/lto-info

Combining tar with logging and mbuffer

As a supplement to my last blog post on archiving to an LTO tape drive using conventional open source utilities like tar, this is how you can save a filelist of the tar output (same as would be outputted by tar -tvf /dev/nst0) to a text file, while also redirecting tar through mbuffer to the tape drive.

tar --label="backup-20230101-volume2" -b512 -cvf - --exclude='.DS_Store' --files-from=/tmp/directories-list.txt 2> >(tee /tmp/tar-filelist.txt >&2) | mbuffer -m 4G -P 80 -s 262144 -o /dev/nst0

It’s critical that the output of tee is again redirected to stderr using >&2 — if you don’t do this, the text from the log will end up in the stdout that gets piped to the mbuffer, which will get written to tape. In that circumstance, tar will not be able to understand the archive when reading it back, since there will be spurious text data.

If you aren’t piping tar’s stdout through mbuffer, you can avoid the redirection problem because tar won’t be outputting to stdout at all. For example:

tar --label="backup-20230101-volume2" -b512 -cvf /dev/nst0 --exclude='.DS_Store' --files-from=/tmp/directories-list.txt 2> tee /tmp/tar-filelist.txt

Adventures with single-drive backup to LTO tape using open source tools

I got a Tandberg LTO-6 drive off eBay recently as a way to have an offline, air-gapped third backup of data that normally lives on my NAS or backup storage server.

Although my NAS is already backed up daily to a ZFS pool on another server, all of these systems are networked—and therefore, vulnerable to ransomware, malware, sloppy sysadmin commands on the terminal, and even electric-surge-caused hardware malfunction. And although I do back up some data to cloud storage, not all data is worth the recurring monthly charges of S3/Glacier/Backblaze B2. Besides, playing with hardware is fun.

Magnetic tape, which can store as much as 2.5 TB uncompressed (in LTO-6, the generation I started with) or 12 TB uncompressed (in LTO-8, the current generation as of mid-2021), is a time-tested option that fits in perfectly.

Veeam Backup & Replication Community Edition works well with standalone tape drives. However, it’s a proprietary system that uses Microsoft Tape Format for the on-tape format—a format that is very challenging to recover yourself without using proprietary tools. Moreover, the tape backup mechanism in Community Edition (i.e., without using licensed NAS backup features) is not meant for backing up large volumes of general purpose files—it’s really designed for archiving VM backups from disk.

LTFS also works. However, my initial attempts to use it were foiled by a Microsemi HBA that doesn’t support TLR. Also, if you don’t use proprietary tape software, LTFS can actually perform more slowly for a bunch of reasons (e.g., multithreaded copying, large number of small files, etc.).

When using a Linux desktop, way more options are available using decades-old software that was designed for tape from the get-go.

Remember tar (tape archive)?

Turns out: tar is as usable in 2021 for tape as it was thirty years ago.

Piping a sorted list of filenames into tar

find 'Folder/' -type f | sort | tar -cvf /dev/nst0 --no-recursion -T -

Getting checksums before tar

find 'Folder/' -type f -print0 | sort -z | xargs -0 sha256sum | tee Folder.20210812.sha256sum

You can use sha256sum directly with a glob spec of files, but find will recurse through directories.

Buffering tar with mbuffer

tar --label="archive-name" -b512 -cvf - | sudo mbuffer -m 8G -P 80 -s 262144 -o /dev/nst0

tar -b512 sets the blocking factor to 512 so that each tar record matches the 262144-byte block size of the tape drive (512 × 512 = 262144).

mbuffer -P 80 tries to fill the buffer to 80% before starting to write out.

mbuffer -s 262144 matches the 262144-byte block size.

Verifying the contents of the tape archive

tar -tvf /dev/nst0

This only reads through the end of the file. You need to advance to the next file to read through another tape archive.

Advancing the tape

mt -f /dev/nst0 status
mt -f /dev/nst0 fsf 1
mt -f /dev/nst0 bsf
mt -f /dev/nst0 rewind
mt -f /dev/nst0 eject

Enabling hardware encryption (drive dependent)

This Tandberg drive seems to have the same guts as an HP LTO-6 drive. 256-bit encryption keys can be generated and loaded, but these drives require an extra flag (-a 1). The convenience advantage of enabling hardware encryption is that we can stream from tar directly to tape and back, and the encryption is all transparent to the applications.

stenc -g 256 -k keyfile.key -kd "optional key description"
stenc -f /dev/nst0 -e on -a 1 --ckod --protect -k keyfile.key
stenc -f /dev/nst0 --detail
stenc -f /dev/nst0 -e off -a 1

Bonus: Encoding a barcode into cartridge memory (aka LTO-CM or MAM) using IBM ITDT

The barcode is set in the RFID memory chip and is assigned attribute number 0806. HPE’s LTFS utilities can encode it as part of the LTFS format process, but I figured out how to do this when not using LTFS.

Every attribute is preceded by a 5-byte attribute header, which contains:

  • 2 bytes: the attribute number itself (hex 08 06)
  • 2 bytes: format—apparently ASCII (hex 01 00)
  • 1 byte: length—this has to be 32 decimal (hex 20)

The remaining 32 bytes should be padded with spaces. An example 37-byte binary file, when dumped using xxd (hexadecimal representation on the left, ASCII on the right) should look like this:

$ xxd 0806.bin
00000000: 0806 0100 2046 4a4b 3637 304c 3620 2020  .... FJK670L6
00000010: 2020 2020 2020 2020 2020 2020 2020 2020
00000020: 2020 2020 20

We can try to read the attribute from the cartridge using ITDT:

.\itdt.exe -f \\.\tape0 readattr -p 0 -a 0806 -d 0806.bin

And we can try to encode it to the cartridge using ITDT:

.\itdt.exe -f \\.\tape0 writeattr -p 0 -a 0806 -s 0806.bin

Here’s the evidence that the barcode was properly encoded:

Screenshot of HPE Library and Tape Tools showing barcode field

Appendix: Source Code

These are backups of the open source programs used above, providing some assurance that even if these programs end up disappearing from Linux distributions’ package repositories, I will still be able to access the data stored on these tapes. (There’s probably nothing to worry about here; it’s more likely LTO-6 drives will be EOL long before tar and mt-st disappear.)