Managing Multi-Worker Solo Mining: Stability and Recovery Playbook

The Night We Couldn't Find the Dead Worker

Somewhere around worker forty, an alert fired for a rig with zero shares in the last ten minutes. Good catch — except the alert just said "worker-023," and we had three racks with rigs numbered independently by whoever set them up that week. It took almost twenty minutes to physically locate the machine. That's the moment naming discipline stops being a nice-to-have and becomes the difference between a two-minute fix and a half-hour outage.

Below a certain worker count, you can run on memory. Above it, every failure turns into a coordination problem before it's a technical one — you're not just fixing a rig, you're figuring out which rig, who touched it last, and whether it's isolated or the first sign of something bigger.

Naming That Tells You Something

We stopped using sequential numbers years ago. A worker name should tell you rack, row, and position without opening a spreadsheet — something like r3-b2-14 for rack 3, row B, position 14. When an alert comes in at 2 a.m., you want to know where to walk before you've finished reading the message.

Alongside naming, every worker gets a baseline hashrate and an alert threshold set relative to that baseline, not a flat number across the fleet. A machine pushing 95 TH/s and a newer unit pushing 200 TH/s shouldn't share the same alert floor — one dropping 10% is normal noise, the other dropping 10% is worth a look.

What Recovery Actually Looks Like

When something goes sideways, the sequence matters more than the tools:

1. Spot the anomaly from share recency or a jump in reject rate — usually before a full disconnect shows up.
2. Isolate by naming convention: is it one rig, one rack row, or everything on one switch?
3. Trigger failover to the backup endpoint and confirm shares are flowing again within 10-15 minutes — if it's not back by then, something else is wrong.
4. Close the incident with a written cause and a one-line prevention step, even if that step is just "replace this switch, it's the third time."

That last step is the one people skip, and it's the one that actually cuts down repeat incidents. An incident log that just says "fixed" teaches you nothing six months later.

We keep the log dead simple: timestamp, worker name, symptom, root cause, time to recovery, and one line on what changes next. After a few months that log starts telling you things a dashboard never will — which rack row keeps losing connectivity, which switch is on borrowed time, which PSU batch is failing early. None of that shows up in a single night's data. It only shows up once you've got a few dozen incidents logged the same way, side by side.

Growing the Fleet Without Breaking It

New workers go on in small batches — five to ten at a time, not fifty in an afternoon — with a health check after each batch before the next one goes live. It's slower than we'd like some weeks. But uncontrolled onboarding is exactly how a naming convention falls apart and monitoring thresholds start lying to you, because you added twenty rigs on the same night and can't tell which config change caused which reject spike.

Honest caveat: this discipline costs you deployment speed. If you need forty rigs online by Friday, batch onboarding will frustrate you. We still do it, because the alternative is spending the following week untangling which of forty simultaneous changes broke what.

A repeatable recovery workflow beats fast, uncontrolled growth every time reject rates start climbing at 3 a.m.

Related Reading

For endpoint failover setup specifically, see the Solo Mining Playbooks on MySoloPool.