Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us $3,400 per month for bandwidth bills alone, and since we don't believe in shoving popup ads to our registered users, we try to make the money back through forum registrations.
  • Post
  • Reply
IOwnCalculus
Apr 2, 2003





Kingnothing posted:

I thought it was 4 drives per SFF8087 not 3? All the breakout cables seem to have 4 SATA connectors.

On a "dumb" breakout cable, yes. If you get into expanders everything changes.

I have ~20 drives hanging on a single 8087 right now.

Adbot
ADBOT LOVES YOU

Kingnothing
Aug 2, 2007

Domo Arigato, Archer Roboto


Armed with my new info, and knowing I can plug an h310 into an open 4x, I was looking at clearance for my GPU. It seems as though the h310 would sit against the 3080 FE I plan on getting and block a shitload of intake, plus the h310s supposedly get crazy hot.

So I had a dumb idea because earlier today I saw GPU flex cables for vertical GPU mounting. What if I ran one of those and put the h310 at the bottom of the case or vertical?

It seems like they make x1-x8 or x4-x8/16 for mining rigs. Just in case anyone is scrolling through and doesnít have an open end 4x but wants an 8x card. Maybe itís still a dumb idea and Iím wasting my time researching, but hey who knows?

Hadlock
Nov 9, 2004





H110Hawk posted:

Ssh is the gold standard on the internet.

Yep. Not using SSH is liable to get you fired for gross incompetency in a lot of tech shops.

The only other method to look at might be using something like gcloud's oauth stuff, but unless you're buying a turnkey solution from a 3rd party vendor, implementing it in software is not a lot of fun. If oauth is used internally at Google with no breaches, it's probably good enough for you

catspleen
Sep 12, 2003

I orphaned his children. I widowed his wife.

This LSI 9211-4i is 4x in case it still matters to you.

https://docs.broadcom.com/doc/12352061

Looks like they are going for <$30 on eBay right now.

DrDork
Dec 29, 2003
commanding officer of the Army of Dorkness

Kingnothing posted:

Armed with my new info, and knowing I can plug an h310 into an open 4x, I was looking at clearance for my GPU. It seems as though the h310 would sit against the 3080 FE I plan on getting and block a shitload of intake, plus the h310s supposedly get crazy hot.

So I had a dumb idea because earlier today I saw GPU flex cables for vertical GPU mounting. What if I ran one of those and put the h310 at the bottom of the case or vertical?

It seems like they make x1-x8 or x4-x8/16 for mining rigs. Just in case anyone is scrolling through and doesnít have an open end 4x but wants an 8x card. Maybe itís still a dumb idea and Iím wasting my time researching, but hey who knows?

Riser cables like that are, indeed, A Thing, and are a core component of a lot of SFF cases because it's the only way they can jam a video card into the space. They normally work pretty well, but you can occasionally get ones that were made so cheaply that the signal degradation is enough to negatively impact things: sometimes devices simply won't show up, other times it'll force a fallback to PCIe 2.0 or whatever. So if you get weird behavior it's probably a lovely riser card, not because the idea isn't reasonable.

wolrah
May 8, 2006
what?


cr0y posted:

I am still in the pondering phase, anything else I might want to consider other than SSH? It just seemed like the least hassle, spin up a VM, give it storage from my array and isolate it as much as I can from anything else on my network since I don't want to make my family members end points on my VPN.
The only alternative I'd consider would be something like SyncThing, which is basically like a private P2P Dropbox. If you've ever heard of Bittorrent Sync, this is basically an open source implementation of the same concept.

SSH file transfers have the major advantage of being well supported and well documented on basically every platform that matters though, where SyncThing is easy on the major desktop platforms and then can be run on a lot of NASes but may be quirky or unofficial in those cases.

If you go with the SSH route though, something I'd add to what's already been said is to only open your SSH port to as little of the world as you can practically get away with. Ideally your firewall supports looking up DNS names for whitelisting and you could set up Dynamic DNS clients at the remote sites so you could have it be literally only exposed to the relevant addresses. If that's not practical for one reason or another, then try to whitelist the smallest range(s) you can that will still cover the dynamic range you expect to see them connecting from. Sometimes that might be a single /24 (common with smaller cable and fixed wireless providers), sometimes you might need to whitelist an entire ISP (common with national DSL or cellular providers).

Either way, you severely limit the potential sources of attack to only those loosely network-adjacent to your legitimate users. If there's no legitimate reason someone should be connecting from an ISP on the other side of the planet then they shouldn't be able to connect even if they had all the other information.

D. Ebdrup
Mar 13, 2009
Probation
Can't post for 4 hours!


IOwnCalculus posted:

On a "dumb" breakout cable, yes. If you get into expanders everything changes.

I have ~20 drives hanging on a single 8087 right now.
Quite a few SAS HBAs can handle 1024 drives per controller with enough daisy chaining and expanders.

priznat
Jul 7, 2009

Let's get drunk and kiss each other all night.

Those pcie riser cables are fine as long as it isnít getting too crazy long (10-24Ē) usually and good enough for gen3. Iíve used ones that are 8Ē long that were only officially rated for gen2 and they were fine at gen3.

Matt Zerella
Oct 7, 2002


I love UnRAID but my god I loving hate how it handles VMs. No snapshots, no cloning.

Smashing Link
Jul 8, 2003

I'll keep chucking bombs at you til you fall off that ledge!

Grimey Drawer

Matt Zerella posted:

I love UnRAID but my god I loving hate how it handles VMs. No snapshots, no cloning.

Agree.

Nulldevice
Jun 16, 2006


Toilet Rascal

Couldn't you just handle those functions from the CLI using virsh? Assuming they use KVM for virtual machines. And if you could, would those cloned machines show up in the inventory in the web interface?

Matt Zerella
Oct 7, 2002


Nulldevice posted:

Couldn't you just handle those functions from the CLI using virsh? Assuming they use KVM for virtual machines. And if you could, would those cloned machines show up in the inventory in the web interface?

I want it in the GUI. Proxmox does it. It's pretty ridiculous paid software like UnRAID can't but then again their CEO just said bit rot doesn't exist so who knows where this platform is headed.

Once I can easily expand ZFS I might be jumping ship.

D. Ebdrup
Mar 13, 2009
Probation
Can't post for 4 hours!


Matt Zerella posted:

Once I can easily expand ZFS I might be jumping ship.
It's progressing.

Axe-man
Apr 16, 2005

The product of hundreds of hours of scientific investigation and research.

The perfect meatball.


Clapping Larry

Yeah, i love my synology, but I am building a linux server to teach myself some devops stuff on the side, and ZFS expansion by adding does seem to be very complicated compared to MDADM.

If they can fix that, i would love to have my new server using it, and expand it in the future.

Right now every uses ubuntu though so no BSD os on this one, (in my area at least)

Matt Zerella
Oct 7, 2002


I basically wrote an Ansible pull bootstrap and pull repo so I can at least get a baseline up on the new VM. It's still incredibly irritating.

UnRAID has some extremely stupid shortcomings to it that I never used to care about but now that I'm WFH they're becoming more apparent when I need scratch VMs.

No API. So I can't use packer, the problems I pointed out above.

Cmon guys there are free solutions out there with more features.

CommieGIR
Aug 22, 2006

If Godzilla can do it, you know I can deliver!

Pillbug

Axe-man posted:

Yeah, i love my synology, but I am building a linux server to teach myself some devops stuff on the side, and ZFS expansion by adding does seem to be very complicated compared to MDADM.

If they can fix that, i would love to have my new server using it, and expand it in the future.

Right now every uses ubuntu though so no BSD os on this one, (in my area at least)

ZFS is so much more mature, MDADM has some cool features but I got really fed up with it fast.

Paul MaudDib
May 2, 2006

"Tell me of your home world, Usul"


unraid/btrfs/mdadm/SHR etc all suck but until the end of time weíre going to have people coming in here and being all ďwell I donít want to have to buy 4 drives at once!Ē and then going and losing their data

like, if you care so little about data loss/if itís all easily replaceable you can just add drives one at a time with ZFS as well, plus then you get protection against corruption as well

and the Linus torvalds thing is just lol, heís actively out of touch these days, see also: his AVX rant

THF13
Sep 26, 2007

Keep an adversary in the dark about what you're capable of, and he has to assume the worst.


Has anyone in this thread ever actually lost data due to corruption that ZFS could actually recover from?
The protections while nice seem to cover problems a lot less likely than losing your array in a fire, theft, ransomware attack, software (looking at you Emby) deleting libraries, or user error.

Matt Zerella
Oct 7, 2002


Personally I'm just sick of how hacky UnRAID is. It's slow as poo poo, and they've made some weird choices.

I'm not worried about data integrity on zfs, I'm more interested in snapshots, compression, mountpoints, etc.

Now that I'm a PROFESSIONAL LINUX TOUCHER I'm much more comfortable setting up a Linux machine with docker and compose and getting essentially the same features as UnRAID without dumb poo poo like only being able to log into ssh as root (which is incredibly awful).

I still maintain if you don't care about VMs and want to roll your own NAS/media server then nothing beats it but yeah once you need more advanced features and/or speed without relying on a weird cache setup, it falls short.

H110Hawk
Dec 28, 2006


Paul MaudDib posted:

and the Linus torvalds thing is just lol, heís actively out of touch these days, see also: his AVX rant

Huh. Thanks for that rabbit hole. This was interesting: https://blog.cloudflare.com/on-the-...quency-scaling/

Paul MaudDib
May 2, 2006

"Tell me of your home world, Usul"


H110Hawk posted:

Huh. Thanks for that rabbit hole. This was interesting: https://blog.cloudflare.com/on-the-...quency-scaling/

the long and short of it is that

(a) this is primarily due to Intel having two AVX-512 units on every core, there's nothing implicitly wrong with the instruction set (and in fact a lot is right, it fixes a lot of problems with AVX2 and provides useful extensions, thus Torvalds is completely wrong). The same problem would exist if you put like 4 256-bit AVX2 units on a single core too. You could conceivably do like AMD did with AVX2 on Zen1 and Zen+ and run AVX-512 at half-rate on a 256-bit unit over 2 cycles.

(b) this primarily affects Intel server (Skylake-SP) only. Xeon-W downclocks substantially less, and on the consumer Skylake-X you can control how much it downclocks completely. And in fact on Ice Lake it only downclocks 100 MHz from peak in one specific scenario, so it mostly won't affect future processors at all.

it's not something that is particularly useful inside the kernel, so there's no direct reason he would care, but Torvalds is just getting old and out of touch as far as the broader computing world outside his project.

IOwnCalculus
Apr 2, 2003





THF13 posted:

Has anyone in this thread ever actually lost data due to corruption that ZFS could actually recover from?
The protections while nice seem to cover problems a lot less likely than losing your array in a fire, theft, ransomware attack, software (looking at you Emby) deleting libraries, or user error.

Yes, a long loving time ago:

IOwnCalculus posted:

Somewhat unrelated, for reasons I still don't know, my trusty old Linux mdraid RAID5 array decided to dump all of the metadata on all but one drive, and md refused to rebuild the array in any meaningful manner. Since RAID is not a backup I have all of the data backed up elsewhere, so I took this as a sign to spin up a Nexenta Community Edition VM and rebuild the array as a RAIDZ2 (I can stand to pare some data off). Currently restoring my data over the internet at about 20Mbps.

I've been fully on ZFS since and at least twice it has saved me from full array restorations when I had two drives chuck errors in the same raidz vdev.

cr0y
Mar 24, 2005

IRONKNUCKLE PERMBANNED! READ HERE


Matt Zerella posted:

I love UnRAID but my god I loving hate how it handles VMs. No snapshots, no cloning.

This is my main gripe with it, I would really like to get down to a single for server in my house for everything but I still have to keep my ESX NUC humming along because of how well it manages VMs, plus Veeam etc. I have most of my stuff running in docker containers but I still need some VMs for this and that.

I even tried to nest ESX in an unraid VM but no matter what I could not get ESX to detect the virtual hard drive I gave to it ☹️

Hughlander
May 11, 2005



THF13 posted:

Has anyone in this thread ever actually lost data due to corruption that ZFS could actually recover from?
The protections while nice seem to cover problems a lot less likely than losing your array in a fire, theft, ransomware attack, software (looking at you Emby) deleting libraries, or user error.

Zfs cow and snapshots should save you from Emby shouldnít it?

D. Ebdrup
Mar 13, 2009
Probation
Can't post for 4 hours!


THF13 posted:

Has anyone in this thread ever actually lost data due to corruption that ZFS could actually recover from?
The protections while nice seem to cover problems a lot less likely than losing your array in a fire, theft, ransomware attack, software (looking at you Emby) deleting libraries, or user error.
Absolutely, several times.
Professionally I've seen UREs during traditional raid rebuilds that resulted in needing to restore the entire array (and which resulted in the companies moving to ZFS when I brought up that ZFS was designed to deal with this exact scenario), and also professionally I've had to use ZFS' ability to go back to known-good transaction groups when a txg doesn't get finished properly before an improper shutdown and ZFS couldn't recover without a bit of help (something that traditional RAID can't do at all).

You're not the first, nor last, to make this mistake, but you have to realize that ZFS, and all RAID for that matter, isn't about reliability in the mainframe RAS sense - it's about the other part of the reliability-availability-servicability tripod: availability; ie. the idea that the data remains available even if a certain subset of the system fails.
Real reliability for storage layers comes from having two controllers connected to one disk via two cables (multipathing, a feature of SAS) and having multiple machines configured in active-active failover mode.

ZFS also has features like snapshots (that Hughlander also brought up) which, while not unique to ZFS (as they exist in UFS too, for example) are designed to be atomic and so lightweight that they can be performed at zero-cost, thereby making it trivial to recover from cryptolockers and accidental deletions, that other RAID systems struggle to deal with unless you implement something on top of it like Shadow Copies on Windows (which ZFS also integrates with).
EDIT: An example of this is that companies I've worked for have had zfs snapshot frequencies as low as every 15 minutes - which results in over 42k snapshots a month.

D. Ebdrup fucked around with this message at 11:57 on Sep 14, 2020

Farmer Crack-Ass
Jan 2, 2001

~this is me posting irl~


I can't remember, did ZFS get around to improving performance on resilvering? I seem to recall that was another sticking point for awhile.

Paul MaudDib
May 2, 2006

"Tell me of your home world, Usul"


ZFS never changed the default which is to resilver slowly and leave a large amount of performance available for on-line transactions, since thatís a sensible default for its primary role for storage appliances. But you can tune it to use the whole disk and it will.

IOwnCalculus
Apr 2, 2003





There was some behavior in 0.8 that changed that improved things significantly, I can't recall what the actual name of it is.

vvv Yep, that. It's still a super long process on my array but it's faster than it used to be.

IOwnCalculus fucked around with this message at 03:25 on Sep 15, 2020

Less Fat Luke
May 23, 2003

Just the tip!


Exciting Lemon

Sequential scrub and resilver

Axe-man
Apr 16, 2005

The product of hundreds of hours of scientific investigation and research.

The perfect meatball.


Clapping Larry

Paul MaudDib posted:

unraid/btrfs/mdadm/SHR etc all suck but until the end of time weíre going to have people coming in here and being all ďwell I donít want to have to buy 4 drives at once!Ē and then going and losing their data

like, if you care so little about data loss/if itís all easily replaceable you can just add drives one at a time with ZFS as well, plus then you get protection against corruption as well

and the Linus torvalds thing is just lol, heís actively out of touch these days, see also: his AVX rant

I think Linus is an interesting fella but never really saw him as a scion of the computer world that some do.

I think for me, I need to stop thinking of hardware expansion without wiping out the raid and recovering from backups as the rule rather than the exception.

I have to admit it is easy sysadmin work to throw a few hdds in a existing array and call it good.

edit: tbf i have multiple backups of everything in multiple locations and forms and use RAID 6 which has lower bit flip rebuild failure rate

Axe-man fucked around with this message at 03:42 on Sep 15, 2020

D. Ebdrup
Mar 13, 2009
Probation
Can't post for 4 hours!


Farmer Crack-rear end posted:

I can't remember, did ZFS get around to improving performance on resilvering? I seem to recall that was another sticking point for awhile.
Aside from sequential I/O support which has already been mentioned, there's more to it.
On FreeBSD, which defaults to a kernel tickrate of 1000 (despite being essentially borderline soft-realtime/tickless, even down to interrupt handling), both scrubs and resilvers are much faster than they are on any other untuned system, for the simple reason that scrubs and resilvers are tied to the tickrate (in FreeBSD, this is controlled by a pair of sysctl values in the vfs.zfs OID).
I believe every other Unix-like that ZFS is implemented on also has a fluid tickrate support, so that leaves at least two values to tweak, in case it's something you want to play with - although I can't remember how it's done on Illumos-derivatives, and don't know how it's done on Linux, if you use that.

ZFS resilvering speed easily exceeds traditional hardware RAID resilvering - I've seen some of the latter struggle to get above 10MBps on production systems, and never seen anything resilver at the speeds ZFS is capable of even on fairly meager hardware with spinning rust, let alone NVMe SSDs on a fast machine.

Less Fat Luke
May 23, 2003

Just the tip!


Exciting Lemon

Also, can I just add on the subject of the Unraid guy saying just believe the disks in regards to URE errors - how can a software developer have trust that drive firmwares are bug-free? The firmware is getting more and more complex as the industry adds features like SMR and there's no way that the controlling software is formally validated (if that's even possible at this scale). Like the drive industry tried to implement drive encryption and the efforts were so poor that Bitlocker now defaults to software encryption over any existing drive firmwares encryption feature.

Anyways thanks for listening to my TED talk.

H110Hawk
Dec 28, 2006


Less Fat Luke posted:

Also, can I just add on the subject of the Unraid guy saying just believe the disks in regards to URE errors - how can a software developer have trust that drive firmwares are bug-free? The firmware is getting more and more complex as the industry adds features like SMR and there's no way that the controlling software is formally validated (if that's even possible at this scale). Like the drive industry tried to implement drive encryption and the efforts were so poor that Bitlocker now defaults to software encryption over any existing drive firmwares encryption feature.

Anyways thanks for listening to my TED talk.

I agree disk firmware is not bug free, I also believe in the futility of arguing with the firmware when it throws URE. Or did I miss the point of your TED talk?

Less Fat Luke
May 23, 2003

Just the tip!


Exciting Lemon

H110Hawk posted:

I agree disk firmware is not bug free, I also believe in the futility of arguing with the firmware when it throws URE. Or did I miss the point of your TED talk?
No I'm saying defense in depth; if the disk says there's a problem rely on the redundant data but also use checksums to make sure that you're catching bad data from the disk. Layer on ECC RAM if you can afford it and so on.

I'm saying the disk can give you faulty data that it doesn't consider in error, which filesystem checksumming protects against (which is what this guy says is a waste of time. I should have been clearer so it was more of a TED-X I guess.

Kingnothing
Aug 2, 2007

Domo Arigato, Archer Roboto


Can anyone recommend a software to pool a couple drives under a single letter? I have a bunch of smaller drives I intent to use as basically a scratch disk for my server and I wanna lump em into together for ease of use.

Really would like something free if possible. I donít need any fancy features such as redundancy or file backup/duplication, literally just to pool the space into one larger lump.

Paul MaudDib
May 2, 2006

"Tell me of your home world, Usul"


Kingnothing posted:

Can anyone recommend a software to pool a couple drives under a single letter? I have a bunch of smaller drives I intent to use as basically a scratch disk for my server and I wanna lump em into together for ease of use.

Really would like something free if possible. I donít need any fancy features such as redundancy or file backup/duplication, literally just to pool the space into one larger lump.

go to Administrative Tools in the windows control panel > Computer Management > Disk Management

delete the partitions on the disks, right click "disk X" on one of the disks (doesn't matter which), Create Spanned Volume (or simple volume?), check the drives that you want to use.

note that losing any disk will trash the whole spanned volume though, so back up anything you care about

Paul MaudDib fucked around with this message at 17:14 on Sep 15, 2020

Kingnothing
Aug 2, 2007

Domo Arigato, Archer Roboto


Paul MaudDib posted:

go to Administrative Tools in the windows control panel > Computer Management > Disk Management

delete the partitions on the disks, right click "disk X" on one of the disks (doesn't matter which), Create Spanned Volume (or simple volume?), check the drives that you want to use.

note that losing any disk will trash the whole spanned volume though, so back up anything you care about

Is there something thatíll do it without taking down the whole thing if one drive kicks? Iíve heard of drivepool but I wasnít sure if it was lovely.

spincube
Jan 31, 2006

I spent so I could say that I finally figured out what this god damned cube is doing. Get well Lowtax.


Grimey Drawer

As in, take a bunch of internal hard drives and combine them into one? Windows has a tool called Storage Spaces built-in that'll allow you to do that.

bobfather
Sep 20, 2001

I will analyze your nervous system for beer money

Kingnothing posted:

Is there something thatíll do it without taking down the whole thing if one drive kicks? Iíve heard of drivepool but I wasnít sure if it was lovely.

DrivePool is the best way to do what everyone in this thread does (have lots of storage), but instead theyíre obsessed with free software like FreeNAS or other paid options like Unraid. The only downside to DrivePool is that youíll want the whole suite because all of Stablebitís softwares are excellent.

Please donít use Storage Spaces because Microsoft cannot be trusted to keep it stable.

Adbot
ADBOT LOVES YOU

Less Fat Luke
May 23, 2003

Just the tip!


Exciting Lemon

Stablebit Drivepool does exactly what you want but unfortunately isn't free. It'll pool the drives with or without redundancy into one logical view, and if one drive dies it'll only take the files on it.

e;f,b

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply