Enterprise Storage Megathread: Why is my NAS a SAN?

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > Enterprise Storage Megathread: Why is my NAS a SAN?

«‹›8 »

adorai: Nov 2, 2002; 10/27/04 Never forget; Grimey Drawer

ExileStrife posted:

I was working on one of the storage teams at a very large company, though not working directly on the technology (mostly just workflow improvements). Never had to get my hands dirty with this stuff, but one of the side projects that I would hear about occasionally was bringing in three DMX 4's to move 1.4 PB onto. Since each DMX 4 can handle 1 PB alone, what kind of factors would drive the decision to get three? Future capacity management seems like an odd answer to me, since the forecast was not anywhere near that in for the near future because other data centers were going up. Might this be for some kind of redundancy? Is it possible for one of those DMX's to completely fail? Is it seriously like one singular, monster storage array?

it was probably for failover. if you have two units with 700TB on each, and one fails, then if the max is 1PB then you will exceed it with the failover. So you add an additional unit and then you can failover properly.

# ¿ Aug 30, 2008 15:05

Adbot: ADBOT LOVES YOU

# ¿ Dec 6, 2023 15:08

adorai: Nov 2, 2002; 10/27/04 Never forget; Grimey Drawer

echo465 posted:

Anyone using a Promise SAN? We're using a Promise VTrak M610i, which is a 16-SATA, iSCSI box for around $5k. Populate it with 1TB hard drives (that are really only 930MB or so), and you've got 13TB of RAID-6 storage for around $7500.

i'd rather run a whitebox openfiler than some poo poo rear end promise nas.

# ¿ Sep 6, 2008 22:48

adorai: Nov 2, 2002; 10/27/04 Never forget; Grimey Drawer

Alowishus posted:

Just out of curiosity, in an HP environment, how does one add their own 1TB LFF SATA drives? Is there a source for just the drive carriers? Because using HP's 1TB SATA drives @~$700/ea makes this quite an expensive solution.

I looked into this once (u320 drives), and the trays were a couple hundred a piece second hand, couldn't find them new.

# ¿ Sep 18, 2008 23:02

adorai: Nov 2, 2002; 10/27/04 Never forget; Grimey Drawer

Catch 22 posted:

Cool, I will get on that Monday.

FWIW, I love my netapp filers. I have no idea how many iops I am really using on my iSCSI storage backend for vmware, but I am running about 15 app servers off of it. I have one shelf dedicated to VM storage, 14 15k rpm FC drives, probably cost $20k new for just that one shelf. A new head will run you like $30k, maybe a little less, so you are looking at $50k for around 1.5TB of high iops storage. Then you spend $10k on a sata shelf for bulk storage and you are at your $60k budget, but you've got a netapp which is very very easy to work with.

# ¿ Sep 20, 2008 04:50

adorai: Nov 2, 2002; 10/27/04 Never forget; Grimey Drawer

BonoMan posted:

One of our companies has a FC SAN that has Stornext or whatever and it's like 3500 per computer. Do Dell EqualLogic's that are iSCSI require licenses per user?

Sun or NetApp both have unlimited iscsi initiators.

# ¿ Jun 9, 2009 21:15

adorai: Nov 2, 2002; 10/27/04 Never forget; Grimey Drawer

Can someone tell me what the practicality of using a Sun 7210 w/ 46 7200 rpm disks as the backend for approximately 40 VMware ESX guests is? On the one hand, I am very afraid of using 7200rpm disks here, but on the other hand there are 46 of them.

I realize that without me pulling real IOPS numbers this is relatively stupid, but I need to start somewhere and this seems like a good place.

# ¿ Jun 13, 2009 14:47

adorai: Nov 2, 2002; 10/27/04 Never forget; Grimey Drawer

Misogynist posted:

You're also not telling us what kind of workload you're trying to use here. I've got close to 40 VMs running off of 6 local 15K SAS disks in an IBM x3650, but mostly-idle development VMs have very different workloads than real production app servers.

They are real production VMs. Here is a quick list of what I have running currently.

6 DCs
1 500 user exchange box (about half of which are for part time employees who use it very very lightly)
1 BES for roughly 30 blackberries
4 light-medium duty database servers
4 light webservers
1 citrix / TS license server
5 Application / Terminal Servers

This is what we currently have running on an older IBM SAN, 16 15k FC drives and 8 10k FC drives.

In addition to that workload, I want to add about 15 application servers, with workloads that I haven't even started to measure. All are currently running 2 to 6 10k disks.

# ¿ Jun 13, 2009 15:42

adorai: Nov 2, 2002; 10/27/04 Never forget; Grimey Drawer

We are currently looking at replacing our aging ibm san with something new. The top two on our list are a pair of Netapp 3020s and a 2050 for our offsite or a pair of EMC Clarions. I am interested in looking at a dual head Sun Unified Storage 7310 and a lower end sun at our offsite. The numbers seem to be literally half for the Sun solution, so I feel like I have to missing something on it.

For usage, the primary purpose will be backend storage for about 100 VMs, some cifs, and some iSCSI/fibre storage for a few database servers.

Any thoughts from you guys?

# ¿ Aug 25, 2009 23:24

adorai: Nov 2, 2002; 10/27/04 Never forget; Grimey Drawer

optikalus posted:

Edit: also, anyone remember that whitepaper about SATA drives reaching 2TB in a RAID setting will almost always be guaranteed to fail upon rebuild due to the size of the drive exceeding the drive's own bit error MTBF?

The URE rate on a 1TB SATA drive is high enough that it becomes likely you will encounter one while rebuilding a large array containing them. If you are running RAID5, that means you will probably lose the array. RAID6 is still safe.

# ¿ Sep 2, 2009 22:46

adorai: Nov 2, 2002; 10/27/04 Never forget; Grimey Drawer

StabbinHobo posted:

its absolutely insane for anyone without two very niche aspects to their storage needs:
- can be slower than poo poo sliding up hill
- can afford a 67TB outage to replace one drive

I think I'd trust mogilefs's redundancy policies more than linux's software raid6.

since it's apparently commodity hardware, i think you could probably run opensolaris w/ raidz2 on it if you wanted to.

# ¿ Sep 3, 2009 00:27

adorai: Nov 2, 2002; 10/27/04 Never forget; Grimey Drawer

For that much storage sun is probably the cheapest outside of a roll your own type solution.

# ¿ Sep 17, 2009 03:15

adorai: Nov 2, 2002; 10/27/04 Never forget; Grimey Drawer

shablamoid posted:

Does anyone have any experience with Open-E's DSS v6?

Seems like a waste of money. if you are going to roll a software solution on commodity hardware, there are plenty of other solutions out there, specifically Opensolaris.

# ¿ Oct 31, 2009 22:06

adorai: Nov 2, 2002; 10/27/04 Never forget; Grimey Drawer

On our NetApp 3140, when running the command: priv set diag; stats show lun; priv set admin

I am seeing a large percentage of partial writes. There does not seem to be a corresponding number of misaligned writes, as you can see in the below output:

lun:/vol/iscsivol0/lun0-W-OMCoT2A9Iw:write_align_histo.0:72%
lun:/vol/iscsivol0/lun0-W-OMCoT2A9Iw:write_align_histo.1:0%
lun:/vol/iscsivol0/lun0-W-OMCoT2A9Iw:write_align_histo.2:0%
lun:/vol/iscsivol0/lun0-W-OMCoT2A9Iw:write_align_histo.3:0%
lun:/vol/iscsivol0/lun0-W-OMCoT2A9Iw:write_align_histo.4:0%
lun:/vol/iscsivol0/lun0-W-OMCoT2A9Iw:write_align_histo.5:0%
lun:/vol/iscsivol0/lun0-W-OMCoT2A9Iw:write_align_histo.6:0%
lun:/vol/iscsivol0/lun0-W-OMCoT2A9Iw:write_align_histo.7:0%
lun:/vol/iscsivol0/lun0-W-OMCoT2A9Iw:read_partial_blocks:0%
lun:/vol/iscsivol0/lun0-W-OMCoT2A9Iw:write_partial_blocks:27%

Should I be worried about this? All of my vmdks are aligned (i went through every single one to double check), plus there are no writes to .1-.7 so the evidence also shows no alignment issue. I submitted a case to have an engineer verify there is no issue, but I was wondering if anyone else has seen partial writes like this on a VMware iSCSI lun. The partial writes typically hover around 30%, but can vary. They were at 75% at one point today.

# ¿ Dec 24, 2009 02:45

adorai: Nov 2, 2002; 10/27/04 Never forget; Grimey Drawer

1000101 posted:

This is odd....

Are you using flexclone or ASIS or anything like that? When you allocated the LUN, which LUN type did you set?

ASIs yes, flexclone no, esx lun type

# ¿ Dec 25, 2009 05:51

adorai: Nov 2, 2002; 10/27/04 Never forget; Grimey Drawer

Cultural Imperial posted:

Are you using Operations Manager? If so, can you create a graph for iSCSI latency? That should tell us how well your iSCSI SAN is performing. When you created your iSCSI LUNs, did you use snapdrive or did you create them manually? If you created them manually did you remember to align the VMFS formatting? Also, do you have the ESX host utilities kit installed?

NetApp support got back to me, and they said it's nothing to worry about.

To answer your questions, I could create the graph, but am pretty lazy. As far as lun creation, it was all done with snapdrive or by selecting the proper lun type when creating it. And I do have the host utilities installed.

# ¿ Jan 3, 2010 23:41

adorai: Nov 2, 2002; 10/27/04 Never forget; Grimey Drawer

Nomex posted:

sacrifice performance for space.

Our secondary site has enough storage to hold all of the data, and just enough performance for our critical apps to keep running, so we have all sata disk on a single controller at our DR site. We are comfortable with letting our non critical apps be down so long as the data is intact.

# ¿ Jan 10, 2010 00:13

adorai: Nov 2, 2002; 10/27/04 Never forget; Grimey Drawer

you need logzillas and readzillas to get any kind of performance out of the sun gear.

yzgi posted:

3) Misalignment of the LUNs. I've read, here and elsewhere, about aligning the LUNs and VMFS/VMDK partitions but I'm a bit confused about when it matters. I know we never did any alignment on the EMC.

You should align both the VMFS volume and the VMDKs. Otherwise, a single 4k write could require 2 reads and 3 writes, instead of just one write. By aligning all of your data you will likely see a 10% to 50% performance improvement.

edit: about the alignment. Here is your unaligned data:

code:

VMDK                    -------111111112222222233333333
VMFS             -------11111111222222223333333344444444
SAN       -------1111111122222222333333334444444455555555

Each set of numbers is a 4k block, and each ------- is the final 3.5k of your 63 sector (31.5k) offset. Notice how to write the block of 2s at the VMDK level, you have to write to both the 2s and the 3s of the VMFS level, which will require you to write to the 2s, 3s, and 4s at the SAN level. More importantly, a partial write requires you to read the existing data first. The problem is amplified at the single block level, with larger datasets the impact is less dramatic however it still exists. Here is what it would look like with an extra 512 bytes (aligned at the 32k boundary):

code:

VMDK                      ------- 111111112222222233333333
VMFS              ------- 11111111222222223333333344444444
SAN       ------- 1111111122222222333333334444444455555555

A write of the 2s at the VMDK level requires you to write the 3s of the VMFS level, which only requires a write of the 4s at the SAN level, and does not require ANY reads of the SAN.

adorai fucked around with this message at 23:23 on Jan 13, 2010

# ¿ Jan 13, 2010 23:09

adorai: Nov 2, 2002; 10/27/04 Never forget; Grimey Drawer

Cowboy Mark posted:

Personally I'd have got VMWare high availability and moved virtual machines around instead of this clustering lark, but I think it's a bit late for that now. Does anyone have any advice? Have Dell screwed us over and sold us something not fit for purpose?

It'll fit, because you can run Hyper-V on the clustered role. It can be active/active and since you have enterprise edition, you can run 4 VMs on each host (not sure how the licensing compliance angle works out if one fails). Even if you decided to pick up VMware at this point, your windows server enterprise licenses aren't wasted, because they allow 4 VMs each on a single host.

# ¿ Feb 3, 2010 12:53

adorai: Nov 2, 2002; 10/27/04 Never forget; Grimey Drawer

If you don't need HA, you might want to take a look at the lower end 7000 series devices from sun. They are the only vendor that won't nickel and dime you on every little feature.

# ¿ Mar 2, 2010 23:20

adorai: Nov 2, 2002; 10/27/04 Never forget; Grimey Drawer

Nukelear v.2 posted:

I have a setup similar to your description running on Dell MD3000i's. Dual controller, 15x450GB SAS 15k RPM drives. Was a bit over $13k.

The Sun 7110's do look fairly nice, not very familiar with their kit. Are they basically PC's or does it have any sort of dual controller redundancy?

you need to move up to the 7300 series for HA with shared storage.

# ¿ Mar 3, 2010 22:49

adorai: Nov 2, 2002; 10/27/04 Never forget; Grimey Drawer

if HA is not a requirement, a 7100 series from sun with 2TB raw can be had for around $10k.

# ¿ Mar 19, 2010 22:49

adorai: Nov 2, 2002; 10/27/04 Never forget; Grimey Drawer

EnergizerFellow posted:

As mentioned, deduplication on ZFS still hasn't made it out of development builds, AFAIK.

It will be, and it will be an in place upgrade. The Netapp isn't going to cut it, NFS is a $5k add on on a 2050, I can't imagine it's much cheaper on a 2040.

# ¿ Mar 19, 2010 23:15

adorai: Nov 2, 2002; 10/27/04 Never forget; Grimey Drawer

EnergizerFellow posted:

Yeah, NetApp's list prices are comical and not in a good way. You need to go through a VAR and openly threaten them with real quote numbers from EqualLogic, EMC, Sun, etc. The pricing will suddenly be a lot more competitive. Welcome to the game.

We paid less than half of list after we got a quote from Sun. Our EMC and Compellent quotes only got us about a third off.

# ¿ Mar 20, 2010 13:27

adorai: Nov 2, 2002; 10/27/04 Never forget; Grimey Drawer

Insane Clown Pussy posted:

Do the 8 port HP Procurve 1810G-8 switches support simultaneous flow control and jumbo frames? HP have give me two different answers so far. If not, are there any other "small" switches that support both flow control and jumbo? I don't have a huge datacenter so I'd rather not drop another few grand on a couple switches if I can avoid it. The OCD in me would also make me cringe every time I saw a switch with only 25% of the ports in use by design.
Leaning towards going with Compellent at the moment, with Equallogic a close 2nd.

We run procurve managed switches at work, and I have a 1810G-8 at home. It's great. I would never use an 1810G to connect tens of thousands, if not hundreds of thousands of gear. Seriously, for our pair of NetApp 3140s alone we are using 7 ports per switch. Add in a total of 6 ports per switch for redundant interswitch connections, and a pair of 24 port switches would be running low already.

To answer your question, I can select both options, I can't actually say whether it works or not.

# ¿ Mar 24, 2010 22:45

adorai: Nov 2, 2002; 10/27/04 Never forget; Grimey Drawer

EnergizerFellow posted:

You ever get Compellent and/or 3PAR in? I'd like to hear some more experiences with them. They've sure been hitting up the trade shows lately.

I'll tell you this much about compellent. We were looking at them late last year, and at the time, they did not support raid6. I thought that was kind of silly, and I inquired about it, and their pre-sales engineer told me I was dumb and that there was no chance of data loss in raid 5.

That was pretty much the end of the compellent discussion.

# ¿ Apr 1, 2010 22:22

adorai: Nov 2, 2002; 10/27/04 Never forget; Grimey Drawer

StabbinHobo posted:

So if you have a "4+1" RAID-5 setup on a 3par array for instance, what that really means is that there are four 256mb chunklets of data, per one 256mb chunklet of parity, but those actual chunklets are going to be spread out all over the place and mixed in with all your other LUNs effectively at random.

They use raidsets, of whatever size, in your example 4+1. If you have, for example, 16 disks (i don't know how big a compellent shelf is, but we'll use 16 disks for the example), it will be broken up into 3 raidgroups of 4+1. If you lose 2 disks from 2 seperate raid groups, you are fine. If you lose 2 disks from the same raid group, you are going to be restoring from backups. It's not voodoo magic, if you have 1 parity disk in your raid group, losing 2 disks from it is just as bad as losing all five. The odds of that happening aren't super high, and I am sure it is mitigated by aggressive sparing, but that doesn't change the fact when talking about sata backed tier 2 or tier 3 storage the (published) chance of disk failure is relatively high, and there is no end to end ECC that you find in FC or SAS arrays, which means raid 5 is not reliable enough for anything other than archival in the enterprise.

# ¿ Apr 2, 2010 01:16

adorai: Nov 2, 2002; 10/27/04 Never forget; Grimey Drawer

StabbinHobo posted:

wait, you lost me here, is that because its important to spend money on buzzphrase technology you don't understand in the enterprise, or is that because archival storage is ok to lose data?

It's because you can generally afford some downtime on your archive while you restore from your replication partner or from tape.

# ¿ Apr 2, 2010 01:31

adorai: Nov 2, 2002; 10/27/04 Never forget; Grimey Drawer

TobyObi posted:

To do either NFS or iSCSI, it's time to fork out for 10Gb ethernet infrastructure, otherwise it's just pissing into the wind.

I think you would be very surprised by how much utilization you would see with iscsi. None of our VMware hosts come even close to saturating a single gigabit link with iscsi traffic. Even without 10Gb ethernet, I think it's worthwhile to consider the benefits of iscsi, which pretty much comes down to port cost and management.

# ¿ Apr 9, 2010 22:39

adorai: Nov 2, 2002; 10/27/04 Never forget; Grimey Drawer

TobyObi posted:

We're already saturating 4Gb FC links, and we're migrating to 8Gb with this upgrade that is being worked on.

Sadly, single gig links aren't going to cut it. If they did, my life would be easy...

Are you saturating 4Gb links on the SAN side or on the host side? By using trunking or a few 10Gb ports for your SAN you can do it cheaply. Our SANs obviously generate a LOT more iscsi traffic than any individual host.

# ¿ Apr 10, 2010 00:13

adorai: Nov 2, 2002; 10/27/04 Never forget; Grimey Drawer

Cyberdud posted:

What do you suggest for a small company who wants around 2-4 Tb that can survive one drive failure.

If you do not require HA I would buy one of these. You can pick one up for well under $10k. You can then get a second one and put it in a colo or even in someone's basement and replicate your data to it. You'll also want to get a gigabit (probably managed) switch if you don't already have one, which I would get something like this. You could probably come in right at $10k for your primary site plus another $8k for your replication partner if you wanted to go that route.

# ¿ Apr 22, 2010 22:20

adorai: Nov 2, 2002; 10/27/04 Never forget; Grimey Drawer

lilbean posted:

Sooo, I'm in the enviable position of looking at SSD-based disk arrays right now. I'm looking at the Sun F5100, and it's pretty expensive but seems completely solid (and loving awesome). Anyone have experience with other SSD arrays? Should I even be thinking about filling a standard disk array with flash disks (probably X25-Es?)

It would be useful to know the intended purpose and budget.

# ¿ May 21, 2010 01:53

adorai: Nov 2, 2002; 10/27/04 Never forget; Grimey Drawer

lilbean posted:

The budget for the storage improvement is 50K. We already have a pair of Sun 2530s with 12x600GB 15K SAS disks, and three HBAs to put in each host (active-passive).

Can you just get ahold of sun and pick up a few zeus ssds for your ZIL and l2arc on the existing 2350s?

edit: I see that the 2350s are apparently not using zfs. If you don't need HA you could try a 7200 series unit from sun.

# ¿ May 21, 2010 22:02

adorai: Nov 2, 2002; 10/27/04 Never forget; Grimey Drawer

soj89 posted:

I want to build 3 identical file servers and have 2 in the office and one doing off-site replication from the CEO's home using Crashplan.

I don't know what crashplan is, but windows server can replicate data all on it's own, transferring only block level changes (and looking for already existing blocks it can copy before putting it on the wire) to keep any number of file shares in sync.

# ¿ May 31, 2010 18:08

adorai: Nov 2, 2002; 10/27/04 Never forget; Grimey Drawer

soj89 posted:

Bottom line: What's the best RAID type to put in place? What about the controller type? The more I've been reading, it seems like Raid 1+0 is preferred over Raid 5 in any case. Would an i7 quad with 8 gb be overkill for the main file server? Especially since it's not particularly i/o intensive.

I was referring to DFS, in response to the previous post. As for what RAID level to use, you can get better performance from raid10, however, most large disk arrays will use raid5 or raid6. I would go with raid6 for any array with over 5 disks if IO was secondary to capacity, personally.

# ¿ May 31, 2010 23:03

adorai: Nov 2, 2002; 10/27/04 Never forget; Grimey Drawer

Maneki Neko posted:

What other options out there are actually worth considering?

Short term: stick with opensolaris; long term linux + btrfs.

# ¿ Jul 16, 2010 04:01

adorai: Nov 2, 2002; 10/27/04 Never forget; Grimey Drawer

GanjamonII posted:

Oracle is reporting average request latency of 35-50+ms for some of the database files, whereas our storage team reports average request latency on the filer is something like 4ms. So seems there is something going on between oracle and the filer. CPU usage on the servers is low, there isn't any network issues we're aware of, though we're checking into it.

Two things I would try/check: Unteam the NICs, use one for iSCSI and one for normal traffic, and verify that your LUNs are properly aligned.

# ¿ Jul 29, 2010 00:34

adorai: Nov 2, 2002; 10/27/04 Never forget; Grimey Drawer

i always see how many nics people use for iSCSI and wonder, are they going overkill or are we woefully underestimating our needs. We use 3 nics on each host for iscsi, 2 for the host iscsi connections and 1 for the guest connections to our iSCSI network. We have 6 total nics on each of filers, setup as 2 3 nic trunks in an active/passive config. We have roughtly 100-120 guests (depends if you include test or not) and don't come close to the max throughput of our nics on either side.

# ¿ Aug 12, 2010 02:03

adorai: Nov 2, 2002; 10/27/04 Never forget; Grimey Drawer

Cultural Imperial posted:

It's worth your while to install Cacti or Nagios somewhere to monitor the switches your filers are plugged into. Are we talking about NetApp filers here? If so you can also check autosupport for network throughput graphs.

We do monitor our switchports with cacti, sampling every minute. We don't come close to saturation.

# ¿ Aug 12, 2010 21:52

adorai: Nov 2, 2002; 10/27/04 Never forget; Grimey Drawer

Syano posted:

I know this has to be an elementary question but I just wanted to throw it out there anyways to confirm with first hand experience. We have our new MD3200i humming along just nice and we have the snapshot feature.Before i start playing around though I assume that snapshotting a LUN that supports transactional workloads like SQL or Exchange is probably a bad idea right?

I don't know anything about the md3200i, but if it is a copy on write snapshot, it's fine, however you'll want to put it into hot backup mode first so that you know your snapshot is clean.

# ¿ Aug 31, 2010 22:02

Adbot: ADBOT LOVES YOU

# ¿ Dec 6, 2023 15:08

adorai: Nov 2, 2002; 10/27/04 Never forget; Grimey Drawer

Misogynist posted:

Keep in mind that there may be a substantial performance penalty associated with the snapshot depending on how intensive, and how latency-sensitive, your database workload is. Generally, snapshots are intended for fast testing/rollback or for hot backup and should be deleted quickly; don't rely on the snapshots themselves sticking around as part of your backup strategy. The performance penalty scales with the size of the snapshot delta.

are speaking specifically to the md3200i or snapshots in general?

# ¿ Sep 1, 2010 01:17

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > Enterprise Storage Megathread: Why is my NAS a SAN?

«‹›8 »