Enterprise Storage Megathread: Why is my NAS a SAN?

The Something Awful Forums > Discussion > Serious Hardware / Software Crap > Enterprise Storage Megathread: Why is my NAS a SAN?

EnergizerFellow: Oct 11, 2005; More drunk than a barrel of monkeys

A few idle observations to cheaply improve the Backblaze box:

- LVM/RAID6/JFS? Get those bad boys on OpenSolaris with RAID-Z2 and ZFS.

- Boot from CompactFlash via a direct-plug IDE adapter. CF+adapter can be had for <$30/ea. Run OS in memory from tmpfs. Flash is vastly more reliable than a spindle, if you keep the write cycle count down.

- 110V @ 14A. Seriously 110V? These boxes need to be on 208V ASAP. Better efficiency from the PSs and AC lines too.

- Bump the motherboard budget to get something with ECC memory, PCIe x4 slots, and multiple NICs that you can channel bond (802.1ax). Something like an ASUS P5BV-M will run you ~$150/ea.

- 2x 4-port SATA PCIe x4 cards for ~$60/ea and run one of the SATA backplanes off the motherboard SATA port. Fewer chips to fail in the box, eliminate the very over saturated PCI SATA card, and one of the PCIe controllers.

- Run the numbers on low-power CPUs and 5400/7200 RPM 'green' drives. Given the large number of boxes, additional component cost can be offset in power consumption savings and datacenter BTU.

# ¿ Sep 6, 2009 22:53

Adbot: ADBOT LOVES YOU

# ¿ May 22, 2013 21:02

EnergizerFellow: Oct 11, 2005; More drunk than a barrel of monkeys

ragzilla posted:

They mentioned that on the page- the onboard SATA ports have issues with multipliers. Of course that may not be an issue on a different mobo.

I noticed that too and from looking into the issue, their post is the only reference I can find on modern Intel chipsets having issues with Silicon Image multipliers. From their wording, I would also infer they tried running all/most of the backplanes from the Intel ICH controller. Under the idea I had, only one of the backplanes would be off the motherboard.

Another issue they alluded to was the SATA controller chipset. All low-end multipliers on the market are from Silicon Image, near as I can tell, and the Marvell chipsets didn't support multipliers until rather recently (~1-2 year ago). Also looks like all commodity 4-port SATA PCIe chips on the market are from Marvell, while the 2-ports are Silicon Image, thus their odd card choice, I'm guessing. They probably standardized their layout before the current Marvell chips where available.

H110Hawk posted:

I wonder if there are other factors preventing this, such as disk supply or other inefficiencies in the system they aren't showing us.

I do wonder that as well, such as why they have a seemingly high-speed CPU. I wonder if they have single-thread performance issues.

# ¿ Sep 7, 2009 19:13

EnergizerFellow: Oct 11, 2005; More drunk than a barrel of monkeys

16TB (minus 4 KB?) is a hard limit under the Microsoft iSCSI initiator, regardless of x86 or x64.

# ¿ Oct 30, 2009 20:38

EnergizerFellow: Oct 11, 2005; More drunk than a barrel of monkeys

crazyfish posted:

So I'd be able to sidestep the issue with a separate HBA? I think we have a Qlogic HBA laying around somewhere that I could try.

I theory, that should do it. I assume you're creating the NTFS filesystem by hand from the CLI with 8 KB or 16 KB clusters?

As for some limits information straight from Microsoft:
http://support.microsoft.com/kb/140365
http://www.microsoft.com/whdc/devic...ge/GPT_FAQ.mspx

Any technical reason for a filesystem this large? Dealing with an unclean mount will be a bear.

# ¿ Oct 30, 2009 21:46

EnergizerFellow: Oct 11, 2005; More drunk than a barrel of monkeys

EoRaptor posted:

Minimum 2TB
NFS, maybe iSCSI
Data De-duplication, and support for backing up deduped data (NDMP?).
Snapshots
Multi-path I/O (nice to have, but not critical)
Expandability, both additional I/O and Additional Disks.
AD/LDAP integration for user permissions

With the deduplication need, I'd look into the NetApp FAS2040 (w/ DS4243 tray if you need >12 spindles), which is the only 2000-series worth looking at (otherwise go to the FAS3140, but that's closer to $70K with a DS4243 tray). FAS3140 does have a lot more CPU horsepower and expansion slots, however.

The EMC AX4-5i is down in your price range, but, AFAIK, (still) doesn't have thin provisioning or deduplication.

As mentioned, deduplication on ZFS still hasn't made it out of development builds, AFAIK.

# ¿ Mar 19, 2010 23:00

EnergizerFellow: Oct 11, 2005; More drunk than a barrel of monkeys

Price of entry for clustering Sun storage hardware?

# ¿ Mar 20, 2010 01:54

EnergizerFellow: Oct 11, 2005; More drunk than a barrel of monkeys

Insane Clown Pussy posted:

What the hell is up with NetApp's pricing? Their "Windows bundle" that includes the VSS-aware snapshot tech is $3k per controller on the 2020 or $16k per controller on the 2040 for the exact same feature-set. I was leaning towards EMC until their AX4 pricing doubled when they added the VSS software.

Yeah, NetApp's list prices are comical and not in a good way. You need to go through a VAR and openly threaten them with real quote numbers from EqualLogic, EMC, Sun, etc. The pricing will suddenly be a lot more competitive. Welcome to the game.

quote:

At this rate we're either going to go for Equallogic or bite the bullet and get some more HP/Lefthand boxes. Compellent are dropping by next week though, so we'll see what they say.

Let us know about their pricing and SAS tray availability. I'm curious what they're up to these days. Might as well get 3PAR in there for the humor value.

# ¿ Mar 20, 2010 03:27

EnergizerFellow: Oct 11, 2005; More drunk than a barrel of monkeys

Zerotheos posted:

Netapp was and still is very appealing. In fact, probably the main reasons that we didn't end up buying their solutions weren't even anything to do with the technology. It was their seemingly uncaring attitude towards prospective customers, their absolutely crack-smoking pricing and a refusal to distribute their simulator unless you're already a customer.

Was that through NetApp directly or a specific VAR? A lot of that could be alleviated with some VAR bingo.

# ¿ Mar 31, 2010 23:56

EnergizerFellow: Oct 11, 2005; More drunk than a barrel of monkeys

Zerotheos posted:

VAR. Tried to go directly through Netapp, but they kept referring us away. I couldn't get anything that even remotely resembled competitive pricing.

Oracle/Sun, on the other hand, told us not to bother with VARs and that they'd help us directly. Their direct pricing beat the hell out of the VAR.

Dell/Equalogic was incredibly eager about making a sale. Visited us 3 times, brought cookies, muffins, lunch, demo units. Unbelievably, they wouldn't give me any manual/user guide documentation on the PS6000 series. They were upset to the point of nearly being unprofessional when we decided against their product, even through we ended up buying all the server hardware for the project from them anyways.

Nexenta, I emailed with my first contact quote request at about 7:30 in the evening and they provided a quote within 3 minutes. Being a software solution with a very usable free version, it allowed me to build a full system in testing before I even bothered contacting their sales.

Those have largely been my experiences as well, especially with Dell. The meeting where we told Dell we where buying a 3-digit quantity of Cisco UCS blades instead of their M1000e/M610 blades and no more EqualLogics was 'fun' to say the least.

You ever get Compellent and/or 3PAR in? I'd like to hear some more experiences with them. They've sure been hitting up the trade shows lately.

# ¿ Apr 1, 2010 21:06

EnergizerFellow: Oct 11, 2005; More drunk than a barrel of monkeys

The double-parity thing has more to do with spindles only having an error rate of 10^14 or 10^15 and the chances of corrupt data is quite high when you're rebuilding a multi-TB RAIDed datasets.

http://blogs.zdnet.com/storage/?p=162

# ¿ Apr 2, 2010 01:51

EnergizerFellow: Oct 11, 2005; More drunk than a barrel of monkeys

TobyObi posted:

We're already saturating 4Gb FC links, and we're migrating to 8Gb with this upgrade that is being worked on.

Sadly, single gig links aren't going to cut it. If they did, my life would be easy...

What are you up to if you're needing 8G FC? If performance is a real concern, you may want to look into NFS over 10G ethernet. We're pushing out a bunch of 10G NFS right now, so feel free to drop me a line.

# ¿ Apr 10, 2010 00:11

EnergizerFellow: Oct 11, 2005; More drunk than a barrel of monkeys

Maneki Neko posted:

From what I recall during discussions with our resellers when we were picking up our 5000s + fabric extenders, Nexus 2000 is all 1gig, no 100 meg.

EDIT: Apparently the 2248s do 100/1000.

Correct on the 2148T.

We just built out a new datacenter extension with 2148Ts all over the place and people where wondering why the DRACs didn't work... Good times. At least the 2248TP is launching next month, which does support 100 Mbit.

# ¿ May 20, 2010 21:01

EnergizerFellow: Oct 11, 2005; More drunk than a barrel of monkeys

lilbean posted:

Sooo, I'm in the enviable position of looking at SSD-based disk arrays right now.

But in all seriousness, what part of your current storage isn't up to snuff? Latency on reads and/or writes? Lack of IOps in reads/writes? Throughput? Not enough of the right cache types? Of that 400GB, how much is the typical working set?

quote:

I'm looking at the Sun F5100, and it's pretty expensive but seems completely solid (and loving awesome).

While quick, it's basically four logical groups across 16 4-channel SAS ports, plus not very big. You can definably use it for ZIL and L2ARC if you're already deep into the Sun clustered storage world.

quote:

Anyone have experience with other SSD arrays?

If you're already invested in an existing SAN infrastructure that will let you use 3rd party FC trays, look into the Texas Memory Systems RamSan. TMS is a NetApp partner company.

TobyObi posted:

Don't fill a standard disk array with flash disks. You will want to stab yourself in the face as they thrash the poo poo out of each other. If the controller of the SSD has any onboard TRIM style algorithm, they will go nuts in a RAID group. The controller will not be able to handle the burst I/O that the SSD can produce, so you end up speed limiting the entire thing.

Pretty much, yep. You need to think of SSDs as big, super fast cache to front lots and lots of 15K spindles. May as well got SAS (and iSCSI/NFS) unless you have a huge investment in an FC infrastructure.

EnergizerFellow fucked around with this message at May 21, 2010 around 04:05

# ¿ May 21, 2010 03:51

EnergizerFellow: Oct 11, 2005; More drunk than a barrel of monkeys

lilbean posted:

What's the budget we're working with? Also, pretty much this...

Misogynist posted:

Do you need to grow your capacity? Your speeds? For how long? Where is your capacity/performance now? Where do you anticipate it in five years? If you want a solution that still scales out half a decade from now, you're going to be paying for it. You need to look at what it is you really need, and what the real business case is for it.
[...]
From a storage perspective, we need to know what kind of IOPS (I/O transfers per second) you're looking for, and what the basic performance profile is of your applications (random vs. sequential I/O, reads vs. writes, and where your bottlenecks are). We also need to know what kind of reliability you're looking for, what your plans are regarding replication and disaster recovery features, whether you need to boot from SAN, and how you plan to back all of this up.

# ¿ May 21, 2010 19:14

EnergizerFellow: Oct 11, 2005; More drunk than a barrel of monkeys

lilbean posted:

The budget for the storage improvement is 50K.

That's not much in the storage world, sadly. Something like a Netapp FAS3100 w/ a pair of Flash Cache (PAM II) cards and 2+ 15K SAS trays will set you back $100K+, easily. Something cheaper would be an EqualLogic PS6010XV or PS6510X.

lilbean posted:

I did a bit of testing though for the ZIL and it seems the two controllers in the array soak up the log writes to their cache anyways, and the system didn't get a big boost from moving the ZIL to SSDs.

Sounds about right on the ZIL. Realistically, all the ZIL buys you is faster confirmation from the NAS/SAN on synchronous writes, i.e. cached synchronous writes. If you aren't limited on synchronous wrights now, the ZIL indeed won't buy you much.

quote:

Right now the setup is ZFS on a single 2530. We do hardware RAID10 on two groups of six disks, assign one group to each controller and then use ZFS on the host to concatenate the pools. Additionally we have two SSDs (Sun's rebranded X25E drives) that we're going to use for the L2ARC, a few hot Oracle partitions and possibly the ZIL.

Why not striped for the host-level combination?

EnergizerFellow fucked around with this message at May 22, 2010 around 01:55

# ¿ May 22, 2010 00:58

EnergizerFellow: Oct 11, 2005; More drunk than a barrel of monkeys

oblomov posted:

IMO, NetApp FS3100 with PAM card is gonna run you a lot more then $100K. Hell, a shelf of 450GB fiber drives is going to run over $30K.

Yeah, more like ~$200K. Nothing like overkill.

quote:

However, a 2150 (or whatever the hell is the equivalent right now) with a fiber shelf can probably be had for around $50K. NFS license is pretty cheap for 2000 series too.

Yeah, the FAS2040 is a much more reasonable option. Some back of the envelope numbers would put a FAS2040 w/ DS4243 tray, 15K spindles, and NFS option at ~$70K and ~5500 IOPS.

The FAS2040 is the only model I'd touch in that whole lineup. The FAS2020 and FAS2050 are effectively a major generation behind and I'd expect to get an EOL notice on them any day now. I'd also expect a FAS2040 variant with 10G and more NVRAM to show up soonish as well.

# ¿ May 22, 2010 04:24

EnergizerFellow: Oct 11, 2005; More drunk than a barrel of monkeys

oblomov posted:

For memory throughput especially, the 75xx series is really really good.

I was going to say just that. You're looking at a lore more money (and probably IBM Power) if you want a DB box that's faster than the new Intel Xeon 7500-series (aka Beckton or Nehalem-EX). Much, much more memory bandwidth than a desktop i7, if that's seriously what you're thinking of benchmarking.

EnergizerFellow fucked around with this message at May 23, 2010 around 05:38

# ¿ May 23, 2010 05:34

EnergizerFellow: Oct 11, 2005; More drunk than a barrel of monkeys

adorai posted:

They were quoting us around $300k for the initial 3140 HA pair and 10tb + 10tb sata until we told them we were going to go with Sun. Then they price matched. As far as shelves, the pricing I am quoting is pretty standard.

Exactly this. You need to openly threaten NetApp with competition from Sun, plus a little EqualLogic for good measure.

# ¿ Oct 29, 2010 00:57

EnergizerFellow: Oct 11, 2005; More drunk than a barrel of monkeys

1000101 posted:

We generally hear nothing but positive feedback about 3par in the field as well. In fact the only real negative thing I've heard is that its hard to find people that really understand 3par/know it inside and out for services.

That's been my second-hand experience with 3par as well. That and people bitching about their high-end product needing a custom cabinet.

1000101 posted:

I hope they're forking out good amounts of cash to keep the engineers on for a couple more years. I just can't fathom much good coming out of that merger.

How much does Compellent really have though? Their storage systems where basically Supermicro and Xyratex commodity hardware running Windows Storage Server or Nexenta with some block-level auto-migration tiering glue logic and a nice GUI.

I see a lot of software/patents being migrated to the EqualLogic group and existing Compellent customers moved to the next-gen high-end EqualLogic, which will move to Dell-designed/made NetApp-style hardware. Maybe, know knows.

EnergizerFellow fucked around with this message at Dec 14, 2010 around 21:29

# ¿ Dec 14, 2010 21:25

EnergizerFellow: Oct 11, 2005; More drunk than a barrel of monkeys

Linux Nazi posted:

What would you guys recommend for a simple to setup 50TB storage device? Performance really isn't an issue as it is basically just warehousing data, and will really only be accessed by one host.

If you don't care about performance and just want lots of dirt cheap storage, look into a Dell PowerVault MD3200i iSCSI array with 2-3+ MD1200 trays hanging off it, fully loaded with 2TB 7K drives. The MSRP is expensive enough (~$45K) that you will want to call a Dell sales rep to get competitive commercial pricing.

# ¿ Jan 25, 2011 22:40

EnergizerFellow: Oct 11, 2005; More drunk than a barrel of monkeys

wang souffle posted:

Anyone have experience with Oracle/Sun's Unified Storage Systems? Looking into the 7120 model at the moment as an alternative to a NetApp FAS2040.

What kind of usable capacity do you need? What kind of uptime? Disk IOps? Connectivity?

# ¿ Feb 7, 2011 17:27

EnergizerFellow: Oct 11, 2005; More drunk than a barrel of monkeys

three posted:

VMFS can only be up to 2TB, so there is one reason.

RDM is limited to 2TB as well due to the 32-bit SCSI emulation layer. You're stuck presenting NFS or iSCSI to the guest for >2TB. iSCSI also lets you do things like clustering in Windows.

# ¿ Mar 30, 2011 05:53

EnergizerFellow: Oct 11, 2005; More drunk than a barrel of monkeys

For the VMware side of things, but applicable generally:
http://media.netapp.com/documents/tr-3916.pdf
http://media.netapp.com/documents/tr-3808.pdf
http://www.vmware.com/files/pdf/vsp...e-protocols.pdf

Vanilla posted:

-- With iSCSI it's still OK to just use the standard servers NICs but you need an iSCSI TOE card if you want to boot from iSCSI SAN?

Yep.

Vanilla posted:

-- All you need with iSCSI is the MS Initiator, which is free.

Yep.

Vanilla posted:

-- Generic IP switches are fine - or do they need certain 'things?

Flow control support is the biggie here. If you're using a single switch pair for a complete VMware cluster vertical, watch the switch's backplane bandwidth and latency. Think 3560X and up in Cisco land. Make sure you get a 1:1 mapping of initiators to physical NICs for iSCSI multi-pathing. Jumbo frames are a complete waste of time on modern NICs and switches, in addition to causing latency issues.

http://www.vmware.com/pdf/vsphere4/...csi_san_cfg.pdf

Vanilla posted:

Anyone know some free, accepted tools for copying LUNs off of DAS onto arrays? Robocopy still liked?

Any general rules for someone moving from just DAS onto an iSCSI array?

Yeah, you can just present additional datastores to the servers, use a local copy, then cut over the point points. Pretty straightforward stuff.

# ¿ Jun 1, 2011 19:58

EnergizerFellow: Oct 11, 2005; More drunk than a barrel of monkeys

Misogynist posted:

Don't forget that many commodity integrated NICs shipped with even low-end servers are perfectly capable of booting from iSCSI. For example, IBM's xSeries servers with UEFI can boot from iSCSI using the commodity Broadcom NetXtreme II NICs that they ship with.

TCP Segment Offload (TSO) is the one particular piece of TOE that generally causes issues with jumbo frames. It's generally possible to disable TSO without affecting the rest of your TOE acceleration.

See the links I put out earlier, which covers a modern build of VMware and hardware. The following goes into how frame size affects latency and throughput:

http://media.netapp.com/documents/tr-3808.pdf

Absolute storage throughput is very rarely an issue outside of backup media servers and streaming media concentrators. It's usually all about latency, latency, IOPS, IOPS, and IOPS. Did I mention IOPS?

Misogynist posted:

Don't forget that many commodity integrated NICs shipped with even low-end servers are perfectly capable of booting from iSCSI. For example, IBM's xSeries servers with UEFI can boot from iSCSI using the commodity Broadcom NetXtreme II NICs that they ship with.

This is par for the course these days. Only exception I can think of are the Cisco UCS B-series blade mezzanine cards, which have intentionally broken iSCSI boot code (so they can sell you licensed FCoE ports...).

EnergizerFellow fucked around with this message at Jun 2, 2011 around 18:06

# ¿ Jun 2, 2011 17:56

Adbot: ADBOT LOVES YOU

# ¿ May 22, 2013 21:02

EnergizerFellow: Oct 11, 2005; More drunk than a barrel of monkeys

what is this posted:

The fact that in this day and age MBR is still commonplace, and required for Win7 boot on most computers, does not give me great confidence.

Yes, partition table isn't file system, and the EFI/BIOS thing is closely wrapped up with MBR/GPT, but seriously guys? You couldn't figure something out?

Apple switched cleanly from OpenFirmware to EFI without most people noticing any change...

If it's any consolation, UEFI is now virtually everywhere as of the 32mn Intel chips (Sandy Bridge, Westmere). Windows itself has supported EFI booting of GPT partitions on x64 since Vista SP1, aka Server 2008. VMware guest containers are EFI as of 5.0 as well.

# ¿ Feb 3, 2012 17:42

The Something Awful Forums > Discussion > Serious Hardware / Software Crap > Enterprise Storage Megathread: Why is my NAS a SAN?