Mine would be when I had a 48 bay disk array / JBOD fail on me… badly. After a storm, it killed the larger card that allowed for me to get many of the drives into a PCIe 16x slot, and I was relegated down to only getting ~8 disks per box made from spare hardware. A single box I got 16 going. Add to this mix an SSD for mid-line caching.
These were all running bcache on top of mdraid… One single mount.
Yes I understand how obnoxiously stupid it was to run RAID6 on a 48 disk volume. It was almost all just stuff I could re-acquire over time, not irreplaceable things.
I just HAD to solve this one though.
In come several spare chassis / mobo etc… get a bunch of drives powered and on /dev/ , move to the next.
A couple spare gigabit switches…
several gigabit NICs…
two explicit paths for each machine…
a bit of iSCSI magic, and one machine now had the physical disks all exposed to it… mdadm --assemble blah blah, bit of UUID chaos…
It’s surprising that while a bit speed limited (I think I got just around 110MB/sec reads), it was nicely performant for what a huge mess of wires and disks just strewn out around my rack.
Managed to evacuate all I needed without much issue once I got that going. Now, I try to keep my arrays under 16 drives at a time, or keep a very rigid policy of “I can lose this and don’t care” vs “this box gets RAID10 and/or offsite backups nightly”.
Mine would be when I had a 48 bay disk array / JBOD fail on me… badly. After a storm, it killed the larger card that allowed for me to get many of the drives into a PCIe 16x slot, and I was relegated down to only getting ~8 disks per box made from spare hardware. A single box I got 16 going. Add to this mix an SSD for mid-line caching.
These were all running bcache on top of mdraid… One single mount.
Yes I understand how obnoxiously stupid it was to run RAID6 on a 48 disk volume. It was almost all just stuff I could re-acquire over time, not irreplaceable things.
I just HAD to solve this one though.
In come several spare chassis / mobo etc… get a bunch of drives powered and on /dev/ , move to the next.
A couple spare gigabit switches…
several gigabit NICs…
two explicit paths for each machine…
a bit of iSCSI magic, and one machine now had the physical disks all exposed to it… mdadm --assemble blah blah, bit of UUID chaos…
It’s surprising that while a bit speed limited (I think I got just around 110MB/sec reads), it was nicely performant for what a huge mess of wires and disks just strewn out around my rack.
Managed to evacuate all I needed without much issue once I got that going. Now, I try to keep my arrays under 16 drives at a time, or keep a very rigid policy of “I can lose this and don’t care” vs “this box gets RAID10 and/or offsite backups nightly”.
back up your critical stuff people!