Calculating surcharges

Let’s say a credit card processor charges a variable rate plus a fixed fee on any credit card transactions. For example, Stripe’s current merchant fee in the US is 2.9% + $0.30. This means that, for a $100 amount charged, the seller is getting only $100 − ($100 × 2.9% + $0.30) = $96.80 after the fee is deducted. In some contexts, sellers will want to make up for the difference by passing the fee on to the customer.

What if the seller wants to add a surcharge to recoup the amount of the processing fee? They can’t just tack on 2.9% + $0.30 to the subtotal, because that surcharge itself will incur the processing fee, and the seller will still lose money.

So the formula instead is:

\begin{aligned}
p_v &=\text{the variable portion of the processing fee (e.g., 0.029)} \\
p_f &=\text{the fixed portion of the processing fee (e.g., 0.30)} \\
x &=\text{the subtotal} \\
f &=\text{the surcharge needed to recoup the processing fee} \\
\end{aligned} \\
f=\frac{x+p_f}{1-p_v}-x

This follows from a simple derivation. The amount on which the processing fees will be charged is x+f. This means the processing fees are

(x+f)p_v+p_f

What the seller earns after processing fees, then, is the total minus the above fee:

(x+f)-((x+f)p_v+p_f) \\
=(1-p_v)(x+f)-p_f

Our goal is for the expression above (what the seller earns after processing fees are deducted) to end up equal to the subtotal. Now solve for f that makes that true:

\begin{equation*}
\begin{aligned}
x &= (1-p_v)(x+f) - p_f \\
x+p_f &= (1-p_v)(x+f) \\
\frac{x+p_f}{1-p_v} &= x+f \\
f &=\frac{x+p_f}{1-p_v}-x \\
\end{aligned}
\end{equation*}

Opt out of the arbitration agreement in Zoom’s Terms of Service now

Zoom April 1, 2023 email to customers about Terms of Service

This post represents the views of the undersigned author only. You should consult a lawyer of your own if you want advice on your situation. See disclosures.

On April 1, 2023, Zoom sent an email to users stating that it had updated its Terms of Service. Purportedly those changes had an effective date of March 31, 2023—ponder how that works for formation of a valid contract, and just how much “bargaining” power consumers really had in that.[1]

If you are a Zoom user, you should opt out to preserve your rights in case a future dispute arises and you want to be able to sue them in real court (or benefit from a federal class action brought by others). Here’s how to do it:

27.11 Opt-Out. You may reject this Arbitration Agreement and opt out of arbitration by sending an email to opt-out@zoom.us within (i) thirty (30) calendar days of April 1, 2023 if you are an existing user, or (ii) thirty (30) calendar days of the date you created your account if you are a new user. Your opt-out notice must be individualized and must be sent from the email address associated with your individual Zoom account. An opt-out notice that purports to opt out multiple parties will be invalid as to all such parties. No individual (or their agent or representative) may effectuate an opt out on behalf of other individuals. Your notice to opt-out must include your first and last name, address, the email address associated with your Zoom account, and an unequivocal statement that you decline this Arbitration Agreement. If you do decide to opt out, that opt out will apply to this Arbitration Agreement and all previous versions thereof, and neither party will have the right to compel the other to arbitrate any Dispute. However, all other parts of this Arbitration Agreement will continue to apply to you, and opting out of this Arbitration Agreement has no effect on any other arbitration agreements that you may enter into in the future with us.

The new arbitration agreement is incredibly long—and different from what used to be standard arbitration provisions.

Unlike normal arbitration agreements of years past, which (simply) banned class actions and forced people to proceed individually and confidentially in arbitration, this version dictates what happens if a lot of customers individually sue in arbitration.

As explained below, the strategy of foreclosing class actions using mandatory arbitration backfired: highly motivated plaintiffs will still file their disputes individually. And paying lawyers and arbitration fees to fight against 12,500+ individual plaintiffs on 12,500+ individual cases is way more costly than litigating against one aggregated opponent, even if the ultimate recovery (e.g., damages award) being paid out is smaller.

So these contractual changes try to get back some of the efficiency of aggregation, while still denying people a chance to have that fight in real courts—courts that are accustomed to supervising class actions and deciding claims on an aggregate basis. Ask yourself: what’s the point? Why try so hard to avoid going to a real federal or state court, unless this works out better for the company?

“The [mandatory arbitration] system is set up to produce skewed, pro-defendant results, because defendants are buyers in the market for private judging: the initial choice of private judging over public dispute resolution belongs solely to the defendants. … This means that arbitration vendors’ biggest marketing incentive is to make arbitration attractive to defendants—more attractive than courts.”

David S. Schwartz, Mandatory Arbitration and Fairness, 84 Notre Dame L. Rev. 1247, 1338 (2009)

The new provisions say, in effect, that if more than 50 people represented by the same lawyers/organizations have similar claims against Zoom, 16 of them get hand-picked and decided first, while everyone else’s claims get paused. Once those 16 “bellwether” claims are decided, everyone else is then forced to mediate (i.e., negotiate), under the theory that the parties will likely settle based on how the bellwether cases fared.

Yet at the same time, the agreement provides separately that the outcome of any one arbitration (e.g., if a consumer wins against the company) shall have “no preclusive effect” in any other one—a rule that, if enforceable, is net helpful to the company because adverse rulings can’t be held against it and have to be re-fought each time. So if the claimants want to pursue their case, they still face the procedural hurdles of arbitration, like limited discovery, almost no option to appeal, and the unavailability of meaningful injunctive relief.[2]

Consumers already don’t read terms of service when they click through a sign-up online. Indeed, many “consumers are unaware of forced arbitration clauses … sometimes hidden inside of envelopes, delivery boxes, and privacy policies”—yet courts have enforced arbitration agreements against consumers who had no actual awareness of the provisions.[3] But how many people understand what this agreement is doing and what it means for their rights? Under what theory of the formation of a valid contract can we really say that a consumer—other than a savvy lawyer—knew what they were “agreeing” to?

This quote shows how long some of the new provisions in Zoom’s arbitration agreement are—and it’s just an excerpt:

27.6 Mass Action Waiver. You and Zoom agree that, except as specified in Section 27.7 below, each of us may bring claims against the other only on an individual basis and not on a class, collective, representative, or mass action basis, and the parties hereby waive all rights to have any Dispute be brought, heard, administered, resolved, or arbitrated on a class, collective, representative, or mass action basis. Subject to this Arbitration Agreement, the arbitrator may award declaratory or injunctive relief only in favor of the individual party seeking relief and only to the extent necessary to provide relief warranted by the party’s individual claim. Notwithstanding anything to the contrary in this Arbitration Agreement, if a court decides by means of a final decision, not subject to any further appeal or recourse, that the limitations of this Section 27.6 are invalid or unenforceable as to a particular claim or request for relief (such as a request for public injunctive relief), you and Zoom agree that that particular claim or request for relief (and only that particular claim or request for relief) shall be severed from the arbitration and shall be pursued in the state or federal courts located in San Jose, California. This subsection does not prevent you or Zoom from participating in a class-wide settlement of claims.

27.7 Bellwether Arbitrations. To increase the efficiency of administration and resolution of arbitrations, you and Zoom agree that if there are fifty (50) or more individual arbitration demands of a substantially similar nature brought against either party by or with the assistance of the same law firm, group of law firms, or organizations within a one hundred and eighty (180) day period (“Mass Filing”), the parties shall select sixteen (16) individual arbitration demands (eight (8) per side) for arbitration to proceed (“Bellwether Arbitrations”). Only those sixteen (16) arbitration demands shall be filed with the arbitration provider, and the parties shall hold in abeyance, and not file, the non-Bellwether Arbitrations. Zoom will pay the arbitration provider’s costs for the sixteen (16) Bellwether Arbitrations. The statutes of limitation, including the requirement to file within one (1) year in Section 27.10 below, shall remain tolled when non-Bellwether arbitration demands are held in abeyance. While the Bellwether Arbitrations are adjudicated, no other demand for arbitration that is part of the Mass Filing may be processed, administrated, or adjudicated, and no filing or other administrative costs for such a demand for arbitration shall be due from either party to the arbitration provider. If, contrary to this provision, a party prematurely files non-Bellwether Arbitrations with the arbitration provider, the parties agree that the arbitration provider shall hold those demands in abeyance.

All parties agree that arbitration demands are of a “substantially similar nature” if they arise out of or relate to the same event or factual scenario and raise the same or similar legal issues and seek the same or similar relief. Any party may request that the arbitration provider appoint a sole standing administrative arbitrator (“Administrative Arbitrator”) to determine threshold questions such as (i) whether the Bellwether Arbitration process is applicable or enforceable, (ii) whether particular demand(s) are part of a Mass Filing, and (iii) whether demands within a Mass Filing were filed in accordance with this Agreement, including Section 27.3. In an effort to expedite resolution of any such dispute by the Administrative Arbitrator, the parties agree that the Administrative Arbitrator may set forth such procedures as are necessary to resolve any disputes promptly. The Administrative Arbitrator’s costs shall be paid by Zoom.

The parties shall work in good faith with the arbitrator to complete each Bellwether Arbitration within one hundred and twenty (120) calendar days of its initial pre-hearing conference. The parties agree that the Bellwether Arbitration process is designed to achieve an overall faster, more efficient, and less costly mechanism for resolving Mass Filings, including the claims of individuals who are not selected for a Bellwether Arbitration.

Following resolution of the Bellwether Arbitrations, the parties agree to engage in a global mediation of all remaining arbitration demands comprising the Mass Filing (“Global Mediation”). The Global Mediation shall be administered by the arbitration provider administering the Bellwether Arbitrations. If the parties are unable to resolve the remaining demands for arbitration comprising the Mass Filing within thirty (30) calendar days following the mediation, the remaining demands for arbitration comprising the Mass Filing shall be filed and administered by the arbitration provider on an individual basis pursuant to the arbitration provider’s rules, unless the parties mutually agree otherwise in writing. Any party may request that the arbitration provider appoint an Administrative Arbitrator to determine threshold questions regarding the newly filed demands.

The parties agree to cooperate in good faith with the arbitration provider to implement the Bellwether Arbitration process, including the payment of filing and administrative costs for the Bellwether Arbitrations, deferring any filing costs associated with the non-Bellwether Arbitration Mass Filings until the Bellwether Arbitrations and subsequent Global Mediation have concluded, and cooperate on any steps to minimize the time and costs of arbitration, which may include: (i) the appointment of a discovery special master to assist the arbitrator in the resolution of discovery disputes; and (ii) the adoption of an expedited calendar of the arbitration proceedings. This Bellwether Arbitration provision shall in no way be interpreted as authorizing a class, collective, or mass action of any kind, or an arbitration involving joint or consolidated claims under any circumstances, except as expressly set forth in this provision. The statutes of limitation applicable to each arbitration demand within a Mass Filing, including the requirement to file within one (1) year in Section 27.10 below, shall remain tolled from the time a party makes a Pre-Arbitration Demand to the time when that party files the arbitration demand with the arbitration provider.

27.8 Settlement Offers and Offers of Judgment. At least ten (10) calendar days before the date set for the arbitration hearing, you or Zoom may serve a written offer of judgment upon the other party to allow judgment on specified terms. If the offer is accepted, the offer with proof of acceptance shall be submitted to the arbitration provider, who shall enter judgment accordingly. If the offer is not accepted prior to the arbitration hearing or within thirty (30) calendar days after it is made, whichever occurs first, it shall be deemed withdrawn, and cannot be given as evidence in the arbitration. If an offer made by one party is not accepted by the other party, and the other party fails to obtain a more favorable award, the other party shall not recover their post-offer costs and shall pay the offering party’s costs from the time of the offer (which, solely for purposes of offers of judgment, may include reasonable attorneys’ fees to the extent they are recoverable by statute, in an amount not to exceed the damages awarded).

The parties agree that any disputes with respect to settlement offer(s) or offer(s) of judgment in a Mass Filing are to be resolved by a single arbitrator to the extent such offers contain the same material terms. For arbitrations involving represented parties, the represented parties’ attorneys agree to communicate individual settlement offer(s) or offer(s) of judgment to each and every arbitration claimant or respondent to whom such offers are extended.

27.9 Arbitration Costs. Except as provided for in a Mass Filing (see Section 27.7), your responsibility to pay any filing, administrative, and arbitrator costs will be solely as set forth in the applicable arbitration provider’s rules. If you have a gross monthly income of less than 300% of the federal poverty guidelines, you may be entitled to a waiver of certain arbitration costs.

27.10 Requirement to File Within One Year. To the extent permitted by applicable Law, and notwithstanding any other statute of limitations, any claim or cause of action under this Agreement (with the exception of disputes under Section 27.2(2)) must be filed within one (1) year after such claim or cause of action arose, or else that claim or cause of action will be permanently barred. The statute of limitations and any arbitration cost deadlines shall be tolled while the parties engage in the informal dispute resolution process required by Section 27.3 above.

One other change of note: Whereas the older, short arbitration agreement elected to use the American Arbitration Association to decide claims, the new contract selects National Arbitration & Mediation (NAM).

Think about how the changes to the arbitration agreement reveal the imbalance in bargaining power on this supposed “agreement.” Zoom had the ability to purportedly replace its entire earlier arbitration agreement (“This Arbitration Agreement supersedes all prior versions.”)—but individual consumers have no such ability to do that. At any time, Zoom purportedly can change not just how claims are arbitrated but even who presides over the arbitration—but you, you normal person, you don’t get to do that.

In fact, Zoom even included a provision that you can’t even have an agent send your opt-out of arbitration on your behalf. You don’t get to use a lawyer. You don’t get to opt out for your minor child.

Your opt-out notice must be individualized and must be sent from the email address associated with your individual Zoom account. An opt-out notice that purports to opt out multiple parties will be invalid as to all such parties. No individual (or their agent or representative) may effectuate an opt out on behalf of other individuals.

Section 27.11.

To be clear, Zoom isn’t the only company making these changes. In recent months, Showtime (updated January 12, 2023), HBO Max (updated December 20, 2022), have all added similar mass-claim provisions that switch to using NAM as the arbitration provider. Query why they are all flocking away from AAA and JAMS to NAM, and what that suggests about what kind of incentives there are for NAM to differentiate itself from the other providers by being… more friendly to defendant companies?

And Showtime’s terms, most egregiously, even purport to limit discovery—making it that much harder for a claimant to get the internal evidence proving their claim.[4] And these agreements have suspiciously similar wording, suggesting that maybe the same lawyers are responsible for putting these changes in.

How we got here

Historically, class actions in courts allowed for aggregation of many claims (sometimes claims from people nationwide) in one case. Those rules were intended to enable efficient and fair resolution of disputes that share a common legal cause. That’s important where the actions of one national or multinational company could easily affect millions of users—and simultaneously harm millions of people with a common legal claim against the company.

To avoid mass torts and class actions, companies started putting in arbitration agreements to try to foreclose that type of lawsuit, achieving their own form of private tort reform. It is conventional for arbitration agreements to force claims to be arbitrated on an individual basis.

But clever plaintiff’s lawyers figured out how to game the system: if they had 1000 clients who were arguably harmed, they could file 1000 demands for individual arbitration, all following the express terms of the arbitration agreement. Since many agreements required the company to pay for the cost of filing the arbitration,[5] this strategy could rack up a giant bill for the company, to proceed with the arbitration.

12,500 Uber drivers did exactly that: filing many similar (even verbatim) claims individually against the company. That caused Uber to get in a legal fight with the arbitration provider, JAMS! Uber tried to weasel out of paying the arbitration fees by arguing that JAMS could aggregate the claims or work more efficiently on common issues.

Meanwhile, other companies that faced over 75,000 individual arbitration claims, decided to end their arbitration agreements altogether and revert back to going to court, as the New York Times reported. Again, query how exactly these contracts of adhesion can be imposed on customers—where the giant company can unilaterally decide to stop using arbitration by editing its terms, but individual customers have no choice.

Legislative reform on the horizon?

This all should be banned. And lawyers who are trying to take advantage of the system this way should really question whether what they are doing is good for their clients or good for the system. Egregious abuse of mandatory arbitration is a surefire way to attract the attention of legislators.

The 117th Congress took some action. The House of Representatives passed H.R. 963 (the FAIR Act of 2022) to end mandatory consumer arbitration. But that bill never made it through the Senate during that term.

The House Report observed:

Over the past several decades, forced arbitration clauses have become virtually ubiquitous in everyday contracts. Often buried deep within the fine print of employment and consumer contracts, forced arbitration deprives millions of Americans of their day in court to enforce state and federal rights. Because arbitration lacks the transparency and precedential guidance of the justice system, there is no guarantee that the relevant law will be applied to these disputes or that fundamental notions of fairness and equity will be upheld in the process.

Unlike the judicial system—in which courts’ decisions are generally public and, by building on precedent, cumulatively create a body of law—the results of arbitration disputes are often kept secret. …

Additionally, the company imposing arbitration often selects the presiding arbitrator or arbitration provider, creating a conflict of interest in which the purportedly neutral arbitrator may be motivated by the prospect of obtaining repeat business from the company rather than focused on fairly assessing the claim. …

As a result of the decline of enforcement of state and federal statutory protections, forced arbitration makes it more likely that corporate harms and abuse will go unchallenged. … as Professor Gilles observes, “forced arbitration is not an alternative regime for resolving claims, it is a means of suppressing legal claims altogether.” Judge William G. Young, who was appointed by President Ronald Reagan, likewise stated that the proliferation of forced arbitration clauses means that “business has a good chance of opting out of the legal system altogether and misbehaving without reproach.”

Forced Arbitration Injustice Repeal Act of 2022, H.R. Rep. No. 117-270, at 4-5 (2022).

It’s time for Congress to take up this bill again. Mandatory arbitration is systematically unfair to consumers, and these recent changes show just how far companies will go to press their advantage in this private system.

Footnotes

Footnotes
1 Section 15.1 says it’s your responsibility as a customer to monitor their contractual changes, and “If you continue to use the Services after the effective date of the Changes, then you agree to the revised terms and conditions.” Seems dubious if the effective date was before it told customers.
2 “An arbitration award shall have no preclusive effect in another arbitration or court proceeding involving Zoom and a different individual.” Section 27.4. On its face, this could cut both ways, both helping the company avoid preclusive unfavorable rulings, and harming the company where favorable rulings should also apply to later claims. But under ordinary preclusion doctrines, some decisions that are favorable to the company can’t be held against future claimants who had no ability to participate in the earlier dispute or whose facts differ, whereas some decisions that are unfavorable to the company can be held against it despite the difference in the identity of the later claimant. On balance, this sort of rule probably skews slightly in favor of the company.
3 H.R. Rep. No. 117-270, at 8-9 (2022).
4 Section 1.3 (“Discovery During Arbitration”) in Showtime’s terms provides: “The parties shall each be limited to a maximum of one (1) fact witness deposition per side… Document requests shall be limited to documents that are directly relevant to the matter(s) in dispute or to its outcome; shall be reasonably restricted in terms of time frame, subject matter and persons or entities to which the requests pertain; shall not include broad phraseology such as ‘all documents directly or indirectly related to’; and shall not be encumbered with extensive ‘definitions’ or ‘instructions.’ … Electronic discovery, if any, shall be limited as follows. Absent a showing of compelling need: (i) electronic documents shall only be produced from sources used in the ordinary course of business, and not from backup servers, tapes or other media; (ii) the production of electronic documents shall normally be made on the basis of generally available technology in a searchable format that is usable by the requesting party and convenient and economical for the producing party; (iii) the parties need not produce metadata, with the exception of header fields for email correspondence; (iv) the description of custodians from whom electronic documents may be collected should be narrowly tailored to include only those individuals whose electronic documents may reasonably be expected to contain evidence that is material to the dispute; and (v) where the costs and burdens of e-discovery are disproportionate to the nature of the dispute or to the amount in controversy, or to the relevance of the materials requested, the arbitrator may either deny such requests or order disclosure on the condition that the requesting party advance the reasonable cost of production to the other side, subject to the allocation of costs in the final award.” How many consumers understand what the heck this is about?
5 But see section 27.9 of Zoom’s arbitration agreement that now shifts those back to claimants.

Combining tar with logging and mbuffer

As a supplement to my last blog post on archiving to an LTO tape drive using conventional open source utilities like tar, this is how you can save a filelist of the tar output (same as would be outputted by tar -tvf /dev/nst0) to a text file, while also redirecting tar through mbuffer to the tape drive.

tar --label="backup-20230101-volume2" -b512 -cvf - --exclude='.DS_Store' --files-from=/tmp/directories-list.txt 2> >(tee /tmp/tar-filelist.txt >&2) | mbuffer -m 4G -P 80 -s 262144 -o /dev/nst0

It’s critical that the output of tee is again redirected to stderr using >&2 — if you don’t do this, the text from the log will end up in the stdout that gets piped to the mbuffer, which will get written to tape. In that circumstance, tar will not be able to understand the archive when reading it back, since there will be spurious text data.

If you aren’t piping tar’s stdout through mbuffer, you can avoid the redirection problem because tar won’t be outputting to stdout at all. For example:

tar --label="backup-20230101-volume2" -b512 -cvf /dev/nst0 --exclude='.DS_Store' --files-from=/tmp/directories-list.txt 2> tee /tmp/tar-filelist.txt

Reformatting WD Red Pro 20TB (WD201KFGX) from 512e to 4Kn sector size

WD Red Pro marketing photo

TL;DR: it’s easy using Seagate’s openSeaChest utility. Jump down to How to do the reformat using openSeaChest_Format below for the guide.

I recently purchased a pair of 20TB hard drives to replace the array of 8TB drives from almost 4 years ago (one of which had failed last year, and had been replaced under warranty). The 4×8TB array uses as much as 34 watts in random read and idles at 20 watts.[1] The new 2×20TB would use about 13.8 watts when active, and 7.6 watts idle.[2]

A ZFS mirror would provide ~20TB usable capacity — an increase of 4TB along with a power consumption savings of up to 59%.

The only wrinkle was that these drives, unlike Seagate’s, are advertised only as 512e drives that emulate a 512-byte sector size, without any advertised capability to reformat in 4Kn. Many newer Seagate Exos drives have this capability built in (advertised as “FastFormat (512e/4Kn)”), whereas interestingly WD’s spec sheets for Red Pro drives no longer mention the sector size.

Exos spec sheet screenshot showing FastFormat (512e/4Kn)
Exos X16 spec sheet screenshot showing FastFormat (512e/4Kn)

This distinction between using emulated 512e and native 4K sector size doesn’t make much of a practical difference in 2022 in storage arrays, because ZFS typically writes larger blocks than that. But I still wanted to see whether I could.

FastFormat using Seagate’s openSeaChest

Usually it is a bad idea to use one vendor’s tools with another’s. There were a lot of forum posts suggesting that the right utility is a proprietary WD tool called “HUGO,” which is not published on any WD support site. Somebody made a tool for doing this on Windows too: https://github.com/pig1800/WD4kConverter .

But I am using these WD Red Pros in a NAS enclosure running Linux, not Windows.

Seagate has one of the leading cross-platform utilities for SATA/SAS drive configuration: SeaChest. I think I’ve even been able to run one of these on ESXi through the Linux compatibility layer. Seagate publishes an open-source repository for the code under the name openSeaChest, available on GitHub: https://github.com/Seagate/openSeaChest , and thanks to the license, vendors like TrueNAS are able to include compiled executables of openSeaChest on TrueNAS SCALE.

openSeaChest includes a utility called openSeaChest_Format (sometimes compiled with the executable name openSeaChest_FormatUnit):

# openSeaChest_FormatUnit --help
==========================================================================================
 openSeaChest_Format - openSeaChest drive utilities - NVMe Enabled
 Copyright (c) 2014-2022 Seagate Technology LLC and/or its Affiliates, All Rights Reserved
 openSeaChest_Format Version: 2.2.1-2_2_1 X86_64
 Build Date: Sep 26 2022
 Today: Sun Dec  4 13:49:42 2022	User: root
==========================================================================================
Usage
=====
	 openSeaChest_Format [-d <sg_device>] {arguments} {options}

Examples
========
	openSeaChest_Format --scan
	openSeaChest_Format -d /dev/sg? -i

*** excerpted ***
	--setSectorSize [new sector size]	
		This option is only available for drives that support sector
		size changes. On SATA Drives, the set sector configuration
		command must be supported. On SAS Drives, fast format must
		be supported. A format unit can be used instead of this
		option to perform a long format and adjust sector size.
		Use the --showSupportedFormats option to see the sector
		sizes the drive reports supporting. If this option
		doesn't list anything, please consult your product manual.
		This option should be used to quickly change between 5xxe and
		4xxx sector sizes. Using this option to change from 512 to 520
		or similar is not recommended at this time due to limited drive
		support

		WARNING: Set sector size may affect all LUNs/namespaces for devices
		         with multiple logical units or namespaces.
		WARNING (SATA): Do not interrupt this operation once it has started or 
		         it may cause the drive to become unusable. Stop all possible background
		         activity that would attempt to communicate with the device while this
		         operation is in progress
		WARNING: It is not recommended to do this on USB as not
		         all USB adapters can handle a 4k sector size.

I had a feeling openSeaChest could be used for my purposes, based on this information on StackExchange, indicating that the WD drives use ATA standards-defined “Set Sector Configuration Ext (B2h)” and “Sector Configuration Log.” Since that user was able to effect the reformat on a WD Ultrastar DC 14TB drive using camcontrol on FreeBSD, that was a good sign that the reformat should be possible. After all, the larger WD Red Pro drives are basically rebadged/binned Ultrastar drives.

This was confirmed later using openSeaChest_Format , but I also saw an encouraging sign in the drive info shown by openSeaChest_SMART -d /dev/sdX --SATInfo:

# openSeaChest_SMART -d /dev/sdf --SATInfo

*** excerpted ***
ATA Reported Information:
	Model Number: WDC WD201KFGX-68BKJN0
	Serial Number: REDACTEDSERIAL
	Firmware Revision: 83.00A83
	World Wide Name: REDACTEDSERIAL
	Drive Capacity (TB/TiB): 20.00/18.19
	Native Drive Capacity (TB/TiB): 20.00/18.19
	Temperature Data:
		Current Temperature (C): 44
		Highest Temperature (C): 45
		Lowest Temperature (C): 24
	Power On Time:  3 days 15 hours 
	Power On Hours: 87.00
	MaxLBA: 39063650303
	Native MaxLBA: 39063650303
	Logical Sector Size (B): 512
	Physical Sector Size (B): 4096
	Sector Alignment: 0
	Rotation Rate (RPM): 7200
	Form Factor: 3.5"
	Last DST information:
		DST has never been run
	Long Drive Self Test Time:  1 day 12 hours 6 minutes 
	Interface speed:
		Max Speed (Gb/s): 6.0
		Negotiated Speed (Gb/s): 6.0
	Annualized Workload Rate (TB/yr): 7437.88
	Total Bytes Read (TB): 33.87
	Total Bytes Written (TB): 40.00
	Encryption Support: Not Supported
	Cache Size (MiB): 512.00
	Read Look-Ahead: Enabled
	Write Cache: Enabled
	SMART Status: Unknown or Not Supported
	ATA Security Information: Supported
	Firmware Download Support: Full, Segmented, Deferred, DMA
	Specifications Supported:
		ACS-5
		ACS-4
		ACS-3
		ACS-2
		ATA8-ACS
		ATA/ATAPI-7
		ATA/ATAPI-6
		ATA/ATAPI-5
		ATA/ATAPI-4
		ATA-3
		ATA-2
		SATA 3.5
		SATA 3.4
		SATA 3.3
		SATA 3.2
		SATA 3.1
		SATA 3.0
		SATA 2.6
		SATA 2.5
		SATA II: Extensions
		SATA 1.0a
		ATA8-AST
	Features Supported:
		Sanitize
		SATA NCQ
		SATA NCQ Streaming
		SATA Rebuild Assist
		SATA Software Settings Preservation [Enabled]
		SATA In-Order Data Delivery
		SATA Device Initiated Power Management
		HPA
		Power Management
		Security
		SMART [Enabled]
		DCO
		48bit Address
		PUIS
		APM [Enabled]
		GPL
		Streaming
		SMART Self-Test
		SMART Error Logging
		EPC
		Sense Data Reporting [Enabled]
		SCT Write Same
		SCT Error Recovery Control
		SCT Feature Control
		SCT Data Tables
		Host Logging
		Set Sector Configuration
		Storage Element Depopulation

Drive information before the reformat

The WD201KFGX drive comes formatted in 512e by default.

openSeaChest is able to discover the drive’s capability to support 4096 sector sizes using the --showSupportedFormats flag.

# openSeaChest_FormatUnit -d /dev/sdf --showSupportedFormats

*** excerpted ***
/dev/sg6 - WDC WD201KFGX-68BKJN0 - REDACTEDSERIAL - ATA

Supported Logical Block Sizes and Protection Types:
---------------------------------------------------
  * - current device format
PI Key:
  Y - protection type supported at specified block size
  N - protection type not supported at specified block size
  ? - unable to determine support for protection type at specified block size
Relative performance key:
  N/A - relative performance not available.
  Best    
  Better  
  Good    
  Degraded
--------------------------------------------------------------------------------
 Logical Block Size  PI-0  PI-1  PI-2  PI-3  Relative Performance  Metadata Size
--------------------------------------------------------------------------------
               4096     Y     N     N     N                   N/A            N/A
               4160     Y     N     N     N                   N/A            N/A
               4224     Y     N     N     N                   N/A            N/A
*               512     Y     N     N     N                   N/A            N/A
                520     Y     N     N     N                   N/A            N/A
                528     Y     N     N     N                   N/A            N/A
--------------------------------------------------------------------------------

How to do the reformat using openSeaChest_Format

The command you want is --setSectorSize 4096

An example of this command is shown below, but you need to manually add --confirm this-will-erase-data to actually make it happen.

You must be certain you are acting on the correct drive! Use openSeaChest_Info -s to identify all connected drives.

To add just a tiny bit of friction to prevent drive-by readers from simply copying-and-pasting this command without thought, potentially wiping out the contents of their hard drive, I’m excluding it in the line below so you need to add it yourself. You also need to specify the correct drive instead of sdX.

# openSeaChest_FormatUnit -d /dev/sdX --setSectorSize 4096 --poll

In my example, it took only a few seconds, and the command provided confirmation that it was successful. Depending on your selected level of verbosity (-v [0-4]) you may see more detail about the ATA commands issued.

*** excerpted ***

Setting the drive sector size quickly.
Please wait a few minutes for this command to complete.
It should complete in under 5 minutes, but interrupting it may make
the drive unusable or require performing this command again!!

*** excerpted ***

Command Time (ms): 499.89

Set Sector Configuration Ext returning: SUCCESS

Successfully set sector size to 4096

After the instant reformat of the sector size, it is critical that you unplug and reinsert the hard drive to reinitialize it in the new format.

Drive information after the reformat

openSeaChest_SMART -d /dev/sdX --SATInfo shows this information:

*** excerpted ***

ATA Reported Information:
	Model Number: WDC WD201KFGX-68BKJN0
	Serial Number: REDACTEDSERIAL
	Firmware Revision: 83.00A83
	World Wide Name: REDACTEDSERIAL
	Drive Capacity (TB/TiB): 20.00/18.19
	Native Drive Capacity (TB/TiB): 20.00/18.19
	Temperature Data:
		Current Temperature (C): 41
		Highest Temperature (C): 45
		Lowest Temperature (C): 24
	Power On Time:  3 days 17 hours 
	Power On Hours: 89.00
	MaxLBA: 4882956287
	Native MaxLBA: 4882956287
	Logical Sector Size (B): 4096
	Physical Sector Size (B): 4096

And hdparm -I /dev/sdX confirms:

*** excerpted ***

ATA device, with non-removable media
	Model Number:       WDC WD201KFGX-68BKJN0                   
	Serial Number:      REDACTEDSERIAL            
	Firmware Revision:  83.00A83
	Transport:          Serial, ATA8-AST, SATA 1.0a, SATA II Extensions, SATA Rev 2.5, SATA Rev 2.6, SATA Rev 3.0
Standards:
	Supported: 12 11 10 9 8 7 6 5 
	Likely used: 12
Configuration:
	Logical		max	current
	cylinders	16383	16383
	heads		16	16
	sectors/track	63	63
	--
	CHS current addressable sectors:    16514064
	LBA    user addressable sectors:   268435455
	LBA48  user addressable sectors:  4882956288
	Logical  Sector size:                  4096 bytes [ Supported: 2048 256 ]
	Physical Sector size:                  4096 bytes
	device size with M = 1024*1024:    19074048 MBytes
	device size with M = 1000*1000:    20000588 MBytes (20000 GB)
	cache/buffer size  = unknown
	Form Factor: 3.5 inch
	Nominal Media Rotation Rate: 7200

Success! Let me know if this worked for you, and what model of hard drive it worked on.

Footnotes

Footnotes
1 Based on 8.4 W maximum power draw of ST8000NM0206 and 5 W idle power, according to Seagate’s spec sheet.
2 Based on 6.9 W “read/write” average, compared to 3.8 W when idle but loaded, according to WD’s spec sheet.

Adventures with single-drive backup to LTO tape using open source tools

I got a Tandberg LTO-6 drive off eBay recently as a way to have an offline, air-gapped third backup of data that normally lives on my NAS or backup storage server.

Although my NAS is already backed up daily to a ZFS pool on another server, all of these systems are networked—and therefore, vulnerable to ransomware, malware, sloppy sysadmin commands on the terminal, and even electric-surge-caused hardware malfunction. And although I do back up some data to cloud storage, not all data is worth the recurring monthly charges of S3/Glacier/Backblaze B2. Besides, playing with hardware is fun.

Magnetic tape, which can store as much as 2.5 TB uncompressed (in LTO-6, the generation I started with) or 12 TB uncompressed (in LTO-8, the current generation as of mid-2021), is a time-tested option that fits in perfectly.

Veeam Backup & Replication Community Edition works well with standalone tape drives. However, it’s a proprietary system that uses Microsoft Tape Format for the on-tape format—a format that is very challenging to recover yourself without using proprietary tools. Moreover, the tape backup mechanism in Community Edition (i.e., without using licensed NAS backup features) is not meant for backing up large volumes of general purpose files—it’s really designed for archiving VM backups from disk.

LTFS also works. However, my initial attempts to use it were foiled by a Microsemi HBA that doesn’t support TLR. Also, if you don’t use proprietary tape software, LTFS can actually perform more slowly for a bunch of reasons (e.g., multithreaded copying, large number of small files, etc.).

When using a Linux desktop, way more options are available using decades-old software that was designed for tape from the get-go.

Remember tar (tape archive)?

Turns out: tar is as usable in 2021 for tape as it was thirty years ago.

Piping a sorted list of filenames into tar

find 'Folder/' -type f | sort | tar -cvf /dev/nst0 --no-recursion -T -

Getting checksums before tar

find 'Folder/' -type f -print0 | sort -z | xargs -0 sha256sum | tee Folder.20210812.sha256sum

You can use sha256sum directly with a glob spec of files, but find will recurse through directories.

Buffering tar with mbuffer

tar --label="archive-name" -b512 -cvf - | sudo mbuffer -m 8G -P 80 -s 262144 -o /dev/nst0

tar -b512 sets the blocking factor to 512 so that each tar record matches the 262144-byte block size of the tape drive (512 × 512 = 262144).

mbuffer -P 80 tries to fill the buffer to 80% before starting to write out.

mbuffer -s 262144 matches the 262144-byte block size.

Verifying the contents of the tape archive

tar -tvf /dev/nst0

This only reads through the end of the file. You need to advance to the next file to read through another tape archive.

Advancing the tape

mt -f /dev/nst0 status
mt -f /dev/nst0 fsf 1
mt -f /dev/nst0 bsf
mt -f /dev/nst0 rewind
mt -f /dev/nst0 eject

Enabling hardware encryption (drive dependent)

This Tandberg drive seems to have the same guts as an HP LTO-6 drive. 256-bit encryption keys can be generated and loaded, but these drives require an extra flag (-a 1). The convenience advantage of enabling hardware encryption is that we can stream from tar directly to tape and back, and the encryption is all transparent to the applications.

stenc -g 256 -k keyfile.key -kd "optional key description"
stenc -f /dev/nst0 -e on -a 1 --ckod --protect -k keyfile.key
stenc -f /dev/nst0 --detail
stenc -f /dev/nst0 -e off -a 1

Bonus: Encoding a barcode into cartridge memory (aka LTO-CM or MAM) using IBM ITDT

The barcode is set in the RFID memory chip and is assigned attribute number 0806. HPE’s LTFS utilities can encode it as part of the LTFS format process, but I figured out how to do this when not using LTFS.

Every attribute is preceded by a 5-byte attribute header, which contains:

  • 2 bytes: the attribute number itself (hex 08 06)
  • 2 bytes: format—apparently ASCII (hex 01 00)
  • 1 byte: length—this has to be 32 decimal (hex 20)

The remaining 32 bytes should be padded with spaces. An example 37-byte binary file, when dumped using xxd (hexadecimal representation on the left, ASCII on the right) should look like this:

$ xxd 0806.bin
00000000: 0806 0100 2046 4a4b 3637 304c 3620 2020  .... FJK670L6
00000010: 2020 2020 2020 2020 2020 2020 2020 2020
00000020: 2020 2020 20

We can try to read the attribute from the cartridge using ITDT:

.\itdt.exe -f \\.\tape0 readattr -p 0 -a 0806 -d 0806.bin

And we can try to encode it to the cartridge using ITDT:

.\itdt.exe -f \\.\tape0 writeattr -p 0 -a 0806 -s 0806.bin

Here’s the evidence that the barcode was properly encoded:

Screenshot of HPE Library and Tape Tools showing barcode field

Appendix: Source Code

These are backups of the open source programs used above, providing some assurance that even if these programs end up disappearing from Linux distributions’ package repositories, I will still be able to access the data stored on these tapes. (There’s probably nothing to worry about here; it’s more likely LTO-6 drives will be EOL long before tar and mt-st disappear.)

Removing useless Windows 10 preinstalled apps

Based on https://community.spiceworks.com/topic/1408834-removing-windows-10-apps-gpo, with the additional refinement of a filter that removes only Store apps and not system apps or frameworks, using PowerShell:

1. List the apps that would be uninstalled.

The -AllUsers flag requires an elevated PowerShell run on an administrator account. Omit the -AllUsers flag if running as a nonadministrator for the current user.

Get-AppxPackage -AllUsers | where-object {$_.IsFramework -eq $false -And $_.name -notlike "*store*" -And $_.name -notlike "*calc*" -And $_.SignatureKind -eq "Store"} | select Name

On a 1711 newly installed VM, this resulted in this list:

Name
----
Microsoft.MicrosoftOfficeHub
Microsoft.Microsoft3DViewer
Microsoft.ZuneVideo
Microsoft.WindowsMaps
Microsoft.WindowsFeedbackHub
Microsoft.BingWeather
Microsoft.Messaging
Microsoft.MicrosoftStickyNotes
Microsoft.XboxIdentityProvider
Microsoft.XboxSpeechToTextOverlay
Microsoft.Print3D
Microsoft.GetHelp
Microsoft.WindowsSoundRecorder
Microsoft.Getstarted
Microsoft.WindowsCamera
Microsoft.3DBuilder
Microsoft.Xbox.TCUI
Microsoft.People
Microsoft.RemoteDesktop
Microsoft.XboxGameOverlay
Microsoft.Office.Sway
Microsoft.Windows.Photos
Microsoft.MSPaint
Microsoft.SkypeApp
Microsoft.XboxApp
Microsoft.DesktopAppInstaller
Microsoft.WindowsAlarms
Microsoft.OneConnect
Microsoft.Wallet
Microsoft.ZuneMusic
Microsoft.Office.OneNote
microsoft.windowscommunicationsapps
Microsoft.MicrosoftSolitaireCollection

2. Actually uninstall them.

Get-AppxPackage -AllUsers | where-object {$_.IsFramework -eq $false -And $_.name -notlike "*store*" -And $_.name -notlike "*calc*" -And $_.SignatureKind -eq "Store"} | Remove-AppxPackage

Certain apps that cannot be uninstalled might be listed in the output.

Creating a LUKS-encrypted DVD/BD data disc

I’ve been backing up some of my larger files to Bluray lately, instead of trying to upload them over a 10 Mbps uplink.

In the past, I used GPG (on a .tar or compressed .tar.xz) or Veracrypt (on a file container) to encrypt at rest, before burning those files onto a standard UDF/ISO9660 optical disc. Now that I use a Linux desktop, I wanted something slightly more native — a method that

  1. protects the directory structure and filenames without needing to use an archive file (like .tar);
  2. would be generally unintelligible on a Windows PC (this is a feature, not a bug); and
  3. could be scripted on the command line for server backups, without requiring a GUI.

Based on some resources online, I settled on using LUKS.

Continue reading “Creating a LUKS-encrypted DVD/BD data disc”