|
My 5x8tb ZFS RAIDZ2 pool went offline and did a scrub and is now showing 3 CKSUM errors on one of the drives. A little reading through the linked documentation ( http://zfsonlinux.org/msg/ZFS-8000-9P ) indicates I should investigate whether that drive had actual issues or if was just a random error. What is the best way to go about that in Linux. It's Proxmox, which is based on Debian, if that helps.
|
#
?
Sep 19, 2020 05:29
|
|
|
|
| # ? Apr 19, 2024 22:58 |
|
|
I would run dmesg and see if there's any errors for the drive in question (drive ready / reset errors) in the log, and smartctl -A to see what it has to say.
|
|
#
?
Sep 19, 2020 10:53
|
|
|
Checksum errors indicate that the disk returned data that didn't match what ZFS knows the checksum (stored in the hashtree-parent record) to be, which is different from a traditional read or write-error as those shows up in the console messages (ie. dmesg and/or /var/log/messages (or whereever Linux stores logs nowadays)). Which means, in addition to checking logs and S.M.A.R.T attributes (which you should be graphing with collectd, prometheus, zabbix, librenms, or something else), you should also reseat the cable at both ends (ie. controller/socket as well as the disk) and/or replace the cable if you have a spare.
|
|
#
?
Sep 19, 2020 12:38
|
|
|
I'd say the majority of errors my ZFS pool has thrown on the last 8 years have been solved by shutting down and reseating the cables. I've got a drawer of Monoprice SATA cables, plus a few spare SAS breakouts for the HBA, and when I get checksum errors or any other errors during a scrub, swapping the cable normally fixes it. I don't think I'm buying garbage cables, but literally everything in my setup is at least 6-7 years old except the 4 12TB drives I just swapped in.
|
|
#
?
Sep 19, 2020 15:03
|
|
|
If you chart S.M.A:R.T attributes and can link ZFS checksum errors to increases in ECC/CRC error rate/count without seeing an increase in uncorrectable errors, it's usually a sign that there's something transient affecting the disk firmware/controller - something that's usually also solved by the power-cycle involved in reseating a cable. Similarily, transient problems affecting the controller (whether on a HBA on a daughterboard, on the motherboard, or integrated into the CPU/SoC) also have a good chance of being dealt with by the power-cycle. So reseating the cable is definitely one of the first trouble-shooting steps if you have physical access to the computer.
|
|
#
?
Sep 19, 2020 19:51
|
|
|
Less Fat Luke posted:I hate the amount of waste shucking creates, like goddamn just sell the same binned bare drives with the smaller warranty for slightly less I use the shells to make my own easystores with the drives its replacing. and they would never do what you said about selling normal drives for less. it's lovely but they want to hit the market of people that don't know how to install hardware to have more storage.
|
|
#
?
Sep 19, 2020 21:30
|
|
|
B&H has some discounts on WD drives for another ~3.5 hours FYI. https://www.bhphotovideo.com/c/buy/Dealzone/di/6573 Don't get the 4GB internal Red, though! Edit: Be sure to look at the coupon price. For example the Internal Red 10TB is $90 off so $239
|
|
#
?
Sep 23, 2020 01:11
|
|
|
So I've run into never-ending weird "power sense ERR ERR BLAH BLAH BLAH!" errors with an LSI 9260i card -> SAS expander -> ZFS pool hanging on by a thread that makes me want to throw my case off my balcony into traffic. My 20 drive RAIDZ2 pool in my dual Xeon ESXi server is my baby so I came here for HBA recommendations and any massive case recommendations. If I have to test a drive again with a full scan, it shows healthy, then errors out on the rebuild in a minute again, I'm going to throw a loving fit. If I reseat any cables again like blowing on old NES cartridges... sorry, ranty tonight. I've worked with Supermicro cases a lot from the beginnings of my IT career so I'm looking at them, but if you have experience with any cheaper ones that are still good, I'm all ears.
|
|
#
?
Sep 23, 2020 02:18
|
|
|
Do you have enough cooling in your case? I was getting the power sense errors and bus resets using an LSI HBA until I cranked up the case fan minimum RPM to get a lot more air moving through the system. Most HBAs just have a passive heatsink and rely on the server chassis super-high RPM fans to push air over them.
|
|
#
?
Sep 23, 2020 02:36
|
|
|
SAS HBAs are designed for high static-pressure fans (not just high rpm) mounted in the case to provide the airflow necessary for cooling the hot bits - and the same is true of CPU heatsinks, other HBAs (especially fiber NICs), memory (ever notice how they're aligned with the airflow on server motherboards).
|
|
#
?
Sep 23, 2020 08:43
|
|
|
OpenZFS Developer Summit 2020 looks interesting.
|
|
#
?
Sep 23, 2020 19:00
|
|
|
Less Fat Luke posted:Do you have enough cooling in your case? I was getting the power sense errors and bus resets using an LSI HBA until I cranked up the case fan minimum RPM to get a lot more air moving through the system. Most HBAs just have a passive heatsink and rely on the server chassis super-high RPM fans to push air over them. I thought I did; I have a little bracket thing with two somewhat high 80mm RPM fans positioned over the card on the expansion slots pushing air inwards with the side off. This thing, basically. I'll have to do some digging to see what this card's airflow requirements are and if the fans have a chance of delivering it. They certainly are not the the jet turbine 80 mm fans in server chassis in datacenter. I'll have to look and see if there is some contraption to allow the bracket to position the fans in the same airflow channel as the CPUs' and exhaust fan, not perpendicular on the side. If some Rube Goldberg contraption and some weedwhacker fans will save me from a rebuild costing a few grand, so be it. 
|
|
#
?
Sep 23, 2020 23:54
|
|
|
Can also add a fan right to any HBA: https://www.reddit.com/r/DataHoarder/comments/8u7syj/how_i_cooled_my_lsi_sas92118i_card_with_a_noctua/
|
|
#
?
Sep 24, 2020 00:38
|
|
|
That's unnecessarily fancy install. When the fan died on my old GPU I just ziptied a 12cm around the card. Worked just fine.
|
|
#
?
Sep 24, 2020 17:34
|
|
|
Most people seem to just reapply paste then wedge screws in the heat sink https://www.google.com/amp/s/josephlo.wordpress.com/2018/04/25/dell-h310-hba-cooling/amp/ 40mmx10mm fans from Noctua are like $16. There is a dude who 3D printed a bracket for his h310 but that’s probably overkill https://www.reddit.com/r/DataHoarder/comments/77nryf/i_designed_and_3d_printed_a_fan_mount_for_my_dell/
|
|
#
?
Sep 24, 2020 17:46
|
|
|
Hey folks! I'm trying to upgrade FreeNAS (or more accurately, install a whole new version - latest 11.3U4.1) on my HP N40L, and I'm encountering an error that I'm not sure how to deal with. When installing (from a Kingston USB drive) to my boot drive (a Samsung Fit 32GB), I get "Operation not permitted dd: /dev/da0: operation not permitted The FreeNAS installation on da0 has failed". The closest thing I can find online is either that the iso is bad (checked the checksum, it matches) or, if I'm interpreting this link correctly, maybe it's already installed? I guess I'm not sure what I'm supposed to see once it's successfully installed, so I don't know that it's correctly reading the installer from the Kingston drive and trying to write it to the Samsung - if it boots up successfully and is installed, does it still just look like the installer media? Without the Kingston drive in, I just see a blinking cursor in the top left. Just for funsies, tried connecting to it via IP (the old IP, and scanning for it using arp -a, no dice). Krakkles fucked around with this message at 00:08 on Sep 28, 2020 |
|
#
?
Sep 28, 2020 00:00
|
|
|
console will look like this: Are you sure your boot drive has space? phosdex fucked around with this message at 00:52 on Sep 28, 2020 |
|
#
?
Sep 28, 2020 00:50
|
|
|
I probably need to invest in some large external storage later this year. I need at least 12tb but probably willing to go up to 16tb depending on price . Has to be a single drive, USB and external power (obviously at that size its external) I've seen some western digital at 12tb and some at 14tb for not that much more, but I'm always wary of those odd sizes (like some of the Seagate that fail more often are 1.5tb and 3tb) Should I be worried about a 14tb? What drives in the 12-16tb range would you guys recommend? At this point it sounds like they're all decent and just price tends to be an issue these days. For what I'm using it for, it's Gonna be used with a Playstation 5 but I'll wait to see confirmation on what limits they have. I'm just curious on the difference between 12-16tb drives currently
|
|
#
?
Sep 28, 2020 01:38
|
|
|
phosdex posted:console will look like this: It's definitely big enough, but it occurs to me that my first attempt to install may have caused the drive to not appear empty. I'll reformat it Reformatting worked, it's up and running, but I can't get in to the webUI. U:admin/P:freenas, U:admin/P:<what I entered during install>, U:root/P:<what I entered during install> all don't work. Edit: And, changing the password via the console, root/that changed pass don't work Krakkles fucked around with this message at 02:37 on Sep 28, 2020 |
|
#
?
Sep 28, 2020 02:06
|
|
|
Username is root, I think default password is freenas.
|
|
#
?
Sep 28, 2020 03:13
|
|
|
Oops, I knew I missed one in my post, sorry. That one didn't work either. It’s not quite PEBKAC, but close: the keyboard I used wasn’t sending ‘e’ correctly. Entering the password without that worked. And now ada3 is reporting SMART errors. Is there anything I should do other than shut it down, and start shopping for drives? Are WD Reds still the go to? Krakkles fucked around with this message at 04:18 on Sep 28, 2020 |
|
#
?
Sep 28, 2020 03:20
|
|
|
Krakkles posted:Oops, I knew I missed one in my post, sorry. That one didn't work either. No on reds unless you're buying 10TB+ sizes, 8TB reds and below are cursed with SMR and should be avoided for NAS use. Think the red pros are supposed to be SMR-free but lol at trusting WD, they sell 7200rpm drives that they market as 5400s honestly at this point I'm considering just going back to seagate, they do SMR too but at least they'll tell you
|
|
#
?
Sep 28, 2020 15:32
|
|
|
It's 2-6TB that's SMR, not 8TB.
|
|
#
?
Sep 28, 2020 17:37
|
|
|
Good to know. I'm probably going for 4TB (I figure I'm at 2 now, if I double it, by the time these are failing I'll be thinking about going up again). Looking at seagates!
|
|
#
?
Sep 28, 2020 18:02
|
|
|
So I'll be building out my new network in the next few weeks as the equipment comes in and I'm realizing now is a perfect time to establish a theme for naming conventions of devices. Wanted to see if anyone had really good ideas. So far I've thought of the following areas - Sopranos (Tony, Melfi, Paulie, Big Pussy, etc.) - Babylon 5 (Agamemnon, White Star, Black Star) - The Expanse (Rocinante, Anubis, Barbapiccola, Canterbury, Nauvoo, Donnager, etc.) - Guitarists (Hendrix, Vaughan, Clapton, King, Vai, Satriani, Bonamassa) Maybe stratify within each of them. Network appliances are named after the mafia/OPA/UN ships, User PCs are the cops/UN/etc. But way open to other suggestions! Want to have some fun with it and lean even more into my geekiness here.
|
|
#
?
Sep 28, 2020 18:25
|
|
|
TraderStav posted:So I'll be building out my new network in the next few weeks as the equipment comes in and I'm realizing now is a perfect time to establish a theme for naming conventions of devices. Wanted to see if anyone had really good ideas. So far I've thought of the following areas
|
|
#
?
Sep 28, 2020 18:26
|
|
|
Charles posted:It's 2-6TB that's SMR, not 8TB. You're right, mixed up the two. Anyway, this is a relatively current list of drives that have SMR: https://www.ixsystems.com/community/resources/list-of-known-smr-drives.141/
|
|
#
?
Sep 28, 2020 20:08
|
|
|
Matt Zerella posted:As someone who just worked on a project where servers were named after lethal weapon characters... don't do this.
|
|
#
?
Sep 28, 2020 20:21
|
|
|
I named my hypervisors after some of our cats that passed on 
|
|
#
?
Sep 28, 2020 20:58
|
|
|
How are SeaGate Ironwolf drives? Googling seems to suggest they’re the new NAS drive, and the price looks fine. Pro vs non-pro?
|
|
#
?
Sep 29, 2020 09:31
|
|
|
They're fine. And all CMR, so no awkward WD Red situations. Jumping from IronWolf regular to IronWolf Pro gets you a higher workload rating, 2 more years of warranty, 2 years of free data recovery, and — the main feature for most users — higher spindle speeds for some capacities. If you just want to shovel some data on a small NAS, regular will do the job and be a decent chunk cheaper. The non-Pros are also generally quieter/cooler/lower-power than WD's new fake 5400 rpm drives, if any of those factors are a concern. Molten Llama fucked around with this message at 15:34 on Sep 29, 2020 |
|
#
?
Sep 29, 2020 15:30
|
|
|
Awesome, thank you! I don't think I need the higher spindle speeds - of course "faster is better", but like you said, shovel data on a NAS. The other features, though, sound worth the minimal additional cost. When it gets here in ~a week, I'll probably be back with questions about how to not screw up swapping & resilvering it in FreeNAS - thank you for all of your patience and help!!!
|
|
#
?
Sep 29, 2020 16:19
|
|
|
Earlier this year I bought a ds918+, 2 6tb ironwolves, and a 480gb m.2 ssd for cache. I've moved all my photos and music to it, and have started building up a plex library. I love it. Every night it sends an encrypted backup diff to backblaze. I don't backup my tv or movies, so the total backup is pretty small. I am a computer toucher at work, and have no interest in touching them unless being paid to do so (computers are bad!!), so I appreciate the kind of "set it and forget it" experience synology gave me. Last night I got another 6tb ironwolf and plopped it in. I can't believe how easy it was. It's still "expanding", but whatever, I assume it will finish at some point. I have no questions, comments, nor concerns. Just appreciating how easy it was to get into it. Wayne Knight fucked around with this message at 17:28 on Sep 29, 2020 |
|
#
?
Sep 29, 2020 17:25
|
|
|
Google Backup & Sync is such a turd of a program. I'm on a fresh install of windows so I thought I'd see whether I can restore my files from Google as a test. That went along the lines of: 1. Download & install Backup & Sync 2. Set Documents, Pictures, and Videos folders to be backed up 3. As expected a new entry for "My Computer" shows up on drive.google.com with the three new empty folders. So far so good. 4. In the web UI, selected contents of Documents folder under "Old Computer" and dragged it into the Documents folder under "New Computer" 5. Let the sync run...and it only synced empty folders. No actual files. Of course there are the usual dead ends of community forum posts from users describing similar issues. I also have a scheduled SyncToy run that I use to backup these 3 folders to my NAS, so to resolve my situation I just restored from that copy. This experience definitely makes me want to ditch Backup & Sync though. I will continue to backup to my NAS, which in turn is backed up to b2. I'd prefer to have these most irreplaceable files backed up on a second cloud provider as well though. The main thing I liked about Google Backup & Sync was that it made it easy to see those files on my phone, automatically identifying subjects in the pictures, or being able to find a photo by searching "turtle", and any new pictures I take on my phone are automatically backed up with the same capabilities. Crashplan has a lovely client...Google has a lovely client... why is this poo poo so hard to get right. Maybe I should just stick these three folders inside the "Google Drive" folder, since that seems way easier to sync down to a new machine. Then just change location of those folders in windows.
|
|
#
?
Sep 30, 2020 10:27
|
|
|
fletcher posted:Google Backup & Sync is such a turd of a program. I'm on a fresh install of windows so I thought I'd see whether I can restore my files from Google as a test. That went along the lines of: IIRC Backup and Sync doesn't put files back into directories you modify on the web.
|
|
#
?
Sep 30, 2020 17:20
|
|
|
I ordered some SATA cables and moved my encased drives on that the other day. I’ve been pleasantly impressed with the speed gains, but it would be nice for a little extra snappiness for dealing with photos. Is there a way to set up caching on the SSD I have sitting larger unused like Synology apparently does with NVMe? Also anyone have a reasonably priced case recommendation so I can start not having these drives just sitting on a table?
|
|
#
?
Sep 30, 2020 17:27
|
|
|
Buff Hardback posted:IIRC Backup and Sync doesn't put files back into directories you modify on the web. What's the right way to handle that scenario with Backup & Sync then? The only posts I could find about it recommended the steps I took.
|
|
#
?
Sep 30, 2020 19:06
|
|
|
fletcher posted:What's the right way to handle that scenario with Backup & Sync then? The only posts I could find about it recommended the steps I took. The "cleanest" (it isn't really clean at all): just download the folders and put them back manually where they're supposed to go.
|
|
#
?
Sep 30, 2020 19:07
|
|
|
Google Drive File Stream works a bit more like Google Drive used to, but you have to be a Gsuite subscriber.
|
|
#
?
Sep 30, 2020 21:37
|
|
|
|
| # ? Apr 19, 2024 22:58 |
|
|
Is there any reason to not run FreeNAS or whatever under ESX so I can use the overhead to do other things?
|
|
#
?
Oct 4, 2020 19:16
|
|