|
necrobobsledder posted:Most people's onboard SATA ports are Intel ICH controllers and will be fine for most home server needs. Depending upon how many PCI-e lanes are allocated for the SATA controllers you may want something else for a 1000 MBps throughput, 500k+ sequential IOPS build. If you're trying to do some boot-from-LAN type of scenarios that's nearly unheard of in home scenarios then it may make some sense. The coolest thing I've done at home is setup Starwind Virtual SAN, with an Adaptec RAID controller, 4x4tb drives in RAID10; with SSD cache on 10gbe. The bottleneck between that thing and my primary desktop is the 850Evo in the desktop.  Seriously awesome; but I do a _ton_ of vmware lab stuff for work, so it's worth it. The synology on 1gbe gets all my media storage and is entirely sufficient.
|
#
?
Oct 14, 2016 01:13
|
|
|
|
| # ? Apr 29, 2024 11:27 |
|
|
necrobobsledder posted:If a big honking enterprise class SAN can experience disruptive data corruption on a shelf of drives because of a bad controller, you can lose your whole array's worth of data at home because of it. We are actually dealing with this at work right now. I don't know the details but we have a ZFS-based SAN that's been slowly making GBS threads itself to death over the past year. My cheapass boss won't approve the hardware purchases because reasons. We're doing performance testing on this system to optimize for the production instance so that's fun.
|
|
#
?
Oct 14, 2016 01:17
|
|
|
IOwnCalculus posted:Onboard (more specifically, on-chipset) controllers are generally well supported. Intel/AMD and the open source community all have a very big interest in making sure that Linux / FreeBSD / other OS here are rock-solid stable on the most common SATA controllers. This same mindset seems to extend to server-grade SATA controllers. The older LSI controllers are a particular favorite because they're everywhere in the server world, and are far from bleeding edge. They're well supported in pretty much every OS and they can be had cheaply because pretty much every server manufacturer has used them at some point. What's the general maximum number of SATA ports allowable on a PCIe card? I usually see a max of four.
|
|
#
?
Oct 14, 2016 02:19
|
|
|
Greatest Living Man posted:What's the general maximum number of SATA ports allowable on a PCIe card? I usually see a max of four. Depends on how much you're willing to pay. You can get LSI cards with two SFF break-out ports for 8 total SATA ports pretty cheap, though.
|
|
#
?
Oct 14, 2016 03:02
|
|
|
Greatest Living Man posted:What's the general maximum number of SATA ports allowable on a PCIe card? I usually see a max of four. If you want to maintain good performance over all of the drives the general rule of thumb is one drive per PCIe lane.
|
|
#
?
Oct 14, 2016 05:41
|
|
|
DrDork posted:Depends on how much you're willing to pay. You can get LSI cards with two SFF break-out ports for 8 total SATA ports pretty cheap, though. And then you can take advantage of SAS multipliers. You start losing peak throughout at that point.
|
|
#
?
Oct 14, 2016 07:15
|
|
|
I'd also replace the Q1900 motherboard in the original post with the more modern Braswell equivalent: the ASRock J3710. I've just switched from the Bay Trail J1800 to the Braswell J3710. It's not quite as much of a leap in benchmarks as I was expecting, but it does feel much slicker in daily use, can run full disk encryption with the same responsiveness as my J1800 (which ran with non-encrypted disks) and has HEVC advertised on the box. I'm not transcoding with mine, but it's handy if you might want to use Plex. It's ever so slightly more power hungry than my J1800 (13 watts at the wall on idle, as opposed to 12 watts) but they are super light on power to start with.
|
|
#
?
Oct 14, 2016 12:14
|
|
|
Krailor posted:If you want to maintain good performance over all of the drives the general rule of thumb is one drive per PCIe lane. But a PCI x16 card is 16 PCI lanes, right?
|
|
#
?
Oct 14, 2016 14:53
|
|
|
Doesn't' mean that there's 16 lanes on the board that are physically there. Most SAS controllers top out at 8 lanes anyway and lots of servers will only put 8 lanes onto each 16x physical slot in favor of more slots.
|
|
#
?
Oct 14, 2016 15:18
|
|
|
I'm going to experiment this weekend with a mirror pair in my home server. The noteworthy part of this story is that one drive is a 2.5" WD blue and the mirror is an SSD. Will Ubuntu poo poo the bed? Will it be like a three legged race, with Usain Bolt tied to Barry White? apropos man fucked around with this message at 16:07 on Oct 14, 2016 |
|
#
?
Oct 14, 2016 16:02
|
|
|
apropos man posted:I'm going to experiment this weekend with a mirror pair in my home server. The noteworthy part of this story is that one drive is a 2.5" WD blue and the mirror is an SSD. Will Ubuntu poo poo the bed? Assuming by "mirror pair" you mean RAID 1, you are dumb and you'll only get the speed on the WD Blue and the size of the SSD. Ubuntu won't poo poo the bed, since the entire idea of RAID is to allow the OS to not care about the underlying details of each physical drive, but you're putting yourself in a worst-case scenario here. Just run the SSD as your main drive and make a rsync or other backup script/program to mirror onto the Blue nightly if you want redundancy.
|
|
#
?
Oct 14, 2016 17:38
|
|
|
Calm down. I said it was an experiment. I actually posted my intention to see if there's anything inherently wrong with it. I guess it won't result in data corruption or anything: I'll merely be adding 'wait cycles' to any writes to the array, while the filesystem waits for the mechanical drive to be ready?
|
|
#
?
Oct 14, 2016 18:07
|
|
|
No, unless there's some hardware failure along the line, the setup should work. You very probably will have slow writes, as you expect; many RAID 1 implementations do not consider a write operation complete until both drives have committed the change, so the overall effect to you as the user will be as if you were always writing to the WD Blue (because you are). Some implementations will allow it to be "complete" when either drive finishes, and then tries to sync them, but in practice that still ends up limiting you to the buffer size of the slower drive anyhow, so it's usually not much of a help. You may also have slow reads, too, since depending on how the RAID is implemented, it very well may not respect that the SSD should be the "primary" read source, and may therefore tap the WD Blue with 50% of your read operations. I mean, you can do what you want, but there's literally no reason to do what you've proposed, as it eliminates the vast majority of the reasons to have a SSD in the first place.
|
|
#
?
Oct 14, 2016 18:21
|
|
|
I know it's a waste and kind of stupid. I feel like practising setting up RAID1. Today I went into CeX (a kind of junk shop for second hand phones, DVD's and computer parts here in the UK) and I was gonna throw £15 at a drive to pair with the WD Blue but they had nothing that looked decent, only a really scratched looking Samsung 250GB. I have a spare OCZ Arc that I'm not really using and that was gonna end up in the pair. I might have a look in town tomorrow for a semi-decent mechanical drive.
|
|
#
?
Oct 14, 2016 19:12
|
|
|
apropos man posted:I know it's a waste and kind of stupid. I feel like practising setting up RAID1. Today I went into CeX (a kind of junk shop for second hand phones, DVD's and computer parts here in the UK) and I was gonna throw £15 at a drive to pair with the WD Blue but they had nothing that looked decent, only a really scratched looking Samsung 250GB. If you want to play with RAID levels and assembling volumes, you could instead look at learning ZFS. You can instead of using physical drives, create a vdev backed by a file on disk. You can then assemble / do stuff with RAID arrays with the whole array running from separate files on the same disk. You could also just use VMs, but a RAID 1 of a hard disk and an SSD sounds like a bad time.
|
|
#
?
Oct 14, 2016 19:23
|
|
|
Creating a load of vdevs across a couple of disks sounds like an idea. I assume that this has to be planned carefully so that if a disk dies then data retention is maintained?
|
|
#
?
Oct 14, 2016 19:36
|
|
|
apropos man posted:Creating a load of vdevs across a couple of disks sounds like an idea. I assume that this has to be planned carefully so that if a disk dies then data retention is maintained? Can always dig up a fistful of thumb drives too.
|
|
#
?
Oct 14, 2016 19:44
|
|
|
apropos man posted:Creating a load of vdevs across a couple of disks sounds like an idea. I assume that this has to be planned carefully so that if a disk dies then data retention is maintained? The above suggestion of using vdevs is just for play. Obviously, you gain nothing by having a RAID 0/1/5/6/10 made up from from a bunch of files on the same disk.
|
|
#
?
Oct 14, 2016 19:51
|
|
|
Skandranon posted:The above suggestion of using vdevs is just for play. Obviously, you gain nothing by having a RAID 0/1/5/6/10 made up from from a bunch of files on the same disk. Hmm. Gonna look for guides on RAID1 with ZFS I think...
|
|
#
?
Oct 14, 2016 19:54
|
|
|
The primary reasons for setting up RAIDZ variants with multiple vdev uses on the same disk include: 1. Testing configurations 2. Expecting to expand to other drives. I had 4 drives for a while before that I was doing some testing with and to stagger failure likelihoods a bit, I waited a couple months before ordering from another supplier. In the meantime, I made an 8 partition RAIDZ2 with my 4 drives and ran them through some tests. Expanding zpools is a bit awkward if you haven't done it before and you may want to practice managing them conceptually before committing a bunch of time and money.
|
|
#
?
Oct 14, 2016 20:31
|
|
|
So I've installed the WD Blue on its own, wiped it and created two 40GB files to use as vdevs. Then created a zfs mirror array, so I have this: code:If I buy another cheap drive tomorrow I'm interested in doing the same as I did tonight, but at a device level. I'm also interested in encryption, since my other drives are encrypted. If I apply luks to the actual pool (ztank0) I take it that zfs will still mirror everything correctly? So if everything is working the checksum of my file0 and file1 vdevs should always be matching? What about more esoteric RAID arrangements with zfs? How is encryption deployed on a zfs array with 5 drives or n drives? Are there special considerations depending on the arrangement? EDIT: going to bed now. Will read in the morning.
|
|
#
?
Oct 14, 2016 23:07
|
|
|
Anyone have experiences or opinions on running ZFS or Btrfs on LVM volumes. I'm interested on having checksumming filesystem, but I wouldn't want to give up the flexibility that LVM and mdadm provide. I remember reading years ago that you should use ZFS only with raw devices, not even partitions, but I don't remember the justification for that.
|
|
#
?
Oct 15, 2016 14:35
|
|
|
While I have no real experience with LVM volumes, the desire to use ZFS on raw devices comes from one of two places, depending on what you mean by "raw." If you mean "raw" as in without gpt partitions, that was mostly an effort to get 4k alignment correct to maximize performance. Most drives these days don't need any futzing anymore to get aligned properly, so that's mostly old hat. If you mean "raw" as in not abstracted/virtualized/behind some other intermediate layer, then yeah, ZFS strongly prefers direct access to the drives because it assumes it has direct access. So when ZFS goes to write something, and the subsystem comes back and tells it it's done, it assumes it's telling the truth. Depending on your subsystem, this may or may not be correct. You also lose some of ZFS's features: it (generally) no longer can read SMART data to detect failing drives, it (generally) can no longer package read/writes into transactions, so performance can be degraded, it can usually still detect file/checksum errors, but generally cannot correct them anymore, and generally any disk-based failure runs a much higher risk of somehow borking the entire pool when you try to replace the drive. That's a lot of generallyies because each subsystem acts a bit different, but it's never an ideal situation.
|
|
#
?
Oct 15, 2016 17:25
|
|
|
It's similar to the reasoning why you should generally not use RAIDZ on hardware RAID backed vdevs - ZFS is built with assumptions around not being virtualized as well as mutual exclusion that can impact data safety in the event of failures (although the strangest recommendation for use on VMware managed storage to use virtual compatibility mode rather than physical like you would with MS clustering is bizarre to me). It's peculiar to want to use LVM with ZFS given that one of the tooling usage aims of ZFS was literally to avoid all the hassles of LVM's various commands (pvs, vgs, lvs, etc.), so now you'd be looking at a possible scenario of worst-of-both-worlds instead of best-of-both-worlds. If you really want to though, I would use LVM on top of ZFS zvols for your block devices.
|
|
#
?
Oct 15, 2016 17:58
|
|
|
Can anyone offer a suggestion to nail down a bottleneck? I have a TS140 with an Adaptec 6405e RAID card, and 4x 3tb 7200k drives in RAID10. I just moved this from my desktop too the TS140 and went from ~300mbps r/w to about 120. The two systems are connected iSCSI at 10gbe and it maxes out the network until the TS140 RAM cache is full, then dies down to 120ish. I've benchmarked this on the desktop and TS140 and it's pretty consistently different in iometer. I'm thinking it has to be a bottleneck somewhere on the TS140 but it's a relatively modern PCIe port. What am I missing? Config: TS140 Quad core Xeon 20gb RAM Intel X540-T2 10gbe Adaptec 6405e 4x 3tb 7200k drives RAID 10 Server 2016 Star winds virtual SAN The performance is the same local, network, and iSCSI. I just don't know what would bottleneck it..
|
|
#
?
Oct 15, 2016 23:38
|
|
|
Were the drives connected to the same card when they were in your desktop? The bottleneck looks like the controller writing to the drives. 120MByte/s is about what you'd expect writing to a single drive. Theoretically you could expect better in RAID10. Check your config to see how the card is configured to cache etc.
|
|
#
?
Oct 16, 2016 01:20
|
|
|
SynMoo posted:Were the drives connected to the same card when they were in your desktop? Async writes; same card in both. It's really, truly rather stange.
|
|
#
?
Oct 16, 2016 03:52
|
|
|
necrobobsledder posted:It's peculiar to want to use LVM with ZFS given that one of the tooling usage aims of ZFS was literally to avoid all the hassles of LVM's various commands (pvs, vgs, lvs, etc.), so now you'd be looking at a possible scenario of worst-of-both-worlds instead of best-of-both-worlds. I've used LVM with mdadm to deal with the scenario where you have 4 drive RAID-6 and you want to convert that to a 5 drive RAID-6, or convert a 6 x 1TB RAID-6 into a 4 x 3TB RAID-6. Even if I tried to create a similar system that I have currently, split the drives into several partitions, create separate RAIDZ2 vdevs and pool them it wouldn't work, since apparently it's impossible to empty vdev of data and remove it from a pool. I guess this could be done with LVM-on-zvols, but that seems too advanced setup for my first dable in ZFS. ZFS-on-LVM-block device sounds simpler and easier to understand setup, but I guess the abstraction layer is a problem.
|
|
#
?
Oct 16, 2016 19:15
|
|
|
What sort of chassis has 4+ 2.5" 15mm bays? http://www.pcworld.com/article/3130234/storage-drives/seagate-drops-the-worlds-largest-tiny-hard-drive.html PCWorld is reporting that 5TB 2.5" drives are going to be $85, which is suspect. Even if the pricing is a wash, I'd prefer 2.5" drives if there were a case that could make a 4-8 drive home NAS a good bit smaller.
|
|
#
?
Oct 19, 2016 20:41
|
|
|
The DS416slim only takes 12.5mm drives. Rats.
|
|
#
?
Oct 19, 2016 21:01
|
|
|
Moey posted:The DS416slim only takes 12.5mm drives. Rats. Exactly! Who are these things for? Not laptops, not small enclosures, what's the usage case for a 15mm 2.5" drive?
|
|
#
?
Oct 19, 2016 21:03
|
|
|
15mm 2.5" drives are what a lot of SAS drives fall under (lower latency without needing to price into enterprise SSD territory is still valuable, and short-stroking via smaller platters is one approach). For those announced drives, I'd look for 2.5" SATA backplanes or maybe 2U or 3U servers with 2.5" bays.
|
|
#
?
Oct 19, 2016 21:15
|
|
|
Twerk from Home posted:What sort of chassis has 4+ 2.5" 15mm bays? I don't think you'll find many directly, buuuutttt you can absolutely find 5.25" -> 2.5" adapters and get 3 or 4 drives in there. So you could probably get a SFF box with two or three 5.25" bays and cram a gently caress-ton of storage in there. Getting 60TB of space out of a Shuttle or similar box would be nuts.
|
|
#
?
Oct 19, 2016 22:56
|
|
|
What's the consensus of using 2.5" drives for casual storage vs 3.5"? They're gonna run quieter and more economical, but are they as reliable? I'm talking for home file storage and serving up movies.
|
|
#
?
Oct 19, 2016 22:56
|
|
|
apropos man posted:What's the consensus of using 2.5" drives for casual storage vs 3.5"? They're gonna run quieter and more economical, but are they as reliable? I'm talking for home file storage and serving up movies. More economical? 3.5" drives are cheaper per gb and should run cooler.
|
|
#
?
Oct 19, 2016 23:31
|
|
|
You can grab external 2.5" drives and shuck them for around $110 per 4 TB drive. Good 4 TB drives are at least $140 regardless of size. I wouldn't expect them to run particularly well just because the original drives' warranties are maybe 2 years tops, but for home storage of Linux ISOs it really doesn't matter IMO. Drive prices have been rather slowly going down and my strategy of upgrading capacity every so often before the drive warranty on everything expires. My plan going forward is to primarily use 2.5" drives for bulk storage and if I need a lot of fast, low latency disks and I expect to use it for less than 6 months then I should probably just shell out money to run something in a cloud. Heat / power is primarily associated with the RPMs and number of platters of the drive rather than the physical size of the drive.
|
|
#
?
Oct 20, 2016 01:15
|
|
|
5tb for 85 dollars is well within "too good to be true" territory especially on a 2.5 enterprise form-factor drive, hot-to-the-touch Toshiba 3.5in 5tbs only go as low as $120. Unless they mean production costs, then that's actually a believable figure.
|
|
#
?
Oct 20, 2016 02:06
|
|
|
necrobobsledder posted:Heat / power is primarily associated with the RPMs and number of platters of the drive rather than the physical size of the drive. That makes sense. The only time I've noted the number of platters before purchase was on an old Seagate Momentus XT: I bought the single platter 250gb version because I obsessively figured that a single platter would load Windows from the outside edge and be therefore faster as an OS drive. How obsessively embarrassing!
|
|
#
?
Oct 20, 2016 07:25
|
|
|
apropos man posted:That makes sense. The only time I've noted the number of platters before purchase was on an old Seagate Momentus XT: I bought the single platter 250gb version because I obsessively figured that a single platter would load Windows from the outside edge and be therefore faster as an OS drive. How obsessively embarrassing! We all did stupid nerd poo poo in the aughts. I used Windows Server 2003 as my gaming machine for a few years because somebody, somewhere said that it has less bloat and was therefore faster than XP.
|
|
#
?
Oct 21, 2016 08:25
|
|
|
|
| # ? Apr 29, 2024 11:27 |
|
|
The whole XP memory management thing was such a clusterfuck of exploits that often went unnoticed and unreported that switching to any server version was probably a great idea even for games. At least up until SP1/2 or whenever they finally started thinking about security and it sunk in that the web probably wasn't going away.
|
|
#
?
Oct 21, 2016 15:11
|
|
