Enterprise Storage Megathread: Why is my NAS a SAN?

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > Enterprise Storage Megathread: Why is my NAS a SAN?

«‹›2 »

Intraveinous: Oct 2, 2001; Legion of Rainy-Day Buddhists

OK, time for my fun time, I've read all 5 pages of the threads and learned quite a bit, and I'll preemptively thank rage-saq for all the help he's given me in the past as well... I apologize in advance for this book, and fully realize that I'm going to need to bring in consultants. I'm just the kind of guy who likes to fully wrap his head around everything before I do that so I can talk semi-intelligently to them and know if they're feeding me a line of BS or cutting corners. What it comes down to is I have to manage all of this day to day, so anything I can do to make myself better at doing that, I try to do.

I work for state government, all of the hardware in our data centers is managed by two people, myself, and a colleague. Both of us were kind of dropped into the "Data Center Admin" position from previous positions as Unix/Linux Sysadmins, and now we're expected to be the jack of all trades for everything from networking to SAN to Windows/Linux/AIX/VMware admin to backups... They decided to put us in this position after years of changing landscapes, and no one person who had any idea of what was going on managed everything. So we wind up with a lot of "I got what I needed for my project at the time" dropped in our lap. I'm trying to get everything into a centralized mode with knowledge of what's coming down the pipe, and planning ahead for it.

90% of what is set up now was the way it is when we started being responsible for it. Currently we have the following SAN related stuff in our primary DC (and you can assume that just about everything is replicated at our backup DC):

HP EVA4000 supporting a 4 node HP Itanium2 cluster running Oracle RAC on RHEL 4.5, mapping two 400GB RAW LUNs for the database, and a few other smaller RAW and ext3fs LUNs for rman, voting, etc.
- These 4 servers each have 2 dual port 2Gig FC HBAs
- The HBAs connect up to two HP 2Gbps FC Switches, one interface from each card to each switch
- We have four connections from the switches to the controllers (HSV200) for the EVA (one connection to each controller from each switch)
- EVA has two disk drawers, each with 14 73GB 15K FC drives (28 total)

The storage system is dedicated ONLY to the Oracle RAC system at this time, but I'd like to get to more truly centralized storage if I can. Currently we don't do any snapshotting/cloning/replication with this sytem, and instead use Oracle Dataguard to push redo logs to our backup DC. Both systems are technically live/production, however. We run the public access from our backup DC and internal access at our main site. If I can do this with some type of delta based replication in the future, that would be OK with me. We currently have 100Mbps to our backup site, with talk of upgrading it to redundant 1Gbps links early next year.

Next we have a Gateway rebranded Xyratex 5412E (I think - I base this on the look and specs from Xyratex's website as I can't get any information from Gateway/MPC on what it actually is and if Xyratex firmware/perf tools/etc will work on it.)
This system has two controllers in the main encloser, each with 2 4Gb FC interfaces, and a second enclosure which is connected by SAS link. The main enclosure has 12 300GB 15K SAS drives, secondary has 6 300GB 15K SAS. System is set to a monolithic array for each enclosure (Array 0 = the 12 300GB, Array 1 = the other 6, and vDisks are split from here)
This system currently supports the following through a pair of 12 port (upgradeable to 16 port) Brocade Silkworm 200E 4Gb FC switches:
- 3 node Lotus Domino/Notes cluster on RHEL 4.6 500GB ext3 LUN for each node 1 single port FC HBA per node
- 2 additional nodes for SMTP and Domino Admin tasks with 100GB LUN for each, 1 single port FC HBA per node
- 2 node VMware ESX 3.5.1 cluster with a total of 6 LUNs - 3 250GB LUNs for VMs, a 30GB for Swap, a 100GB LUN that I use for staging P2V migrations, and a large 150GB LUN for a file server VM. There are currently 12 production VMs, with plans for more in the future. Systems are PE2950 and each has two single port FC HBAs.
- 1 Windows Storage server 2003 box which runs VCB and has a 200GB LUN for backup to disk of a couple of other standalone servers.

I also have a FC native dual drive LTO4 IBM tape library for tape backups.

I'd like to start consolidating the few dozen standalone servers I have to use shared storage as they come up for replacement, but I want to make sure I have a storage system that can support it. Right now, there's only about 4TB of total data at each site, but a document management project for "less paper" is in contract negotiations right now, and will probably need 8-12TB in the first year or two, so expandability is important, even though I know I'm still talking about small potatoes.

I like FC because it's what I've used before and I have some current infrastructure, most importantly the tape libraries, but I realize that I'm going to need some updates to the infrastructure to be able to add 20+ more hosts in the next couple of years.

I guess my questions are:
- Is it feasible to have a high-avail, fairly high volume Oracle DB on the same storage system as the rest of my systems, or should I keep it segregated as it is now?
- If keep it segregated, can my EVA4000 be upgraded to 4Gb FC when I replace the servers later this year?
- Can I have all of my storage systems on a single fabric to allow for over the SAN backup with Netbackup 6.5? (My plan is to keep the Xyratex, use the existing SAS drives as a Disk to Disk to Tape for important data, and add another drawer of SATA drives and make a disk backup staging area for everything else once I get a new central storage system)
- If I have mixed-vendor storage on the same fabric, is that going to cause problems with LUN mappings?
- Am I looking for something that's going to cost a half million dollars to do (I'll need two of whatever I end up with - one for each site)

Thanks in advance for reading this far, and for any insight you can offer!

# ¿ Oct 1, 2008 17:23

Adbot: ADBOT LOVES YOU

# ¿ Apr 20, 2024 01:20

Intraveinous: Oct 2, 2001; Legion of Rainy-Day Buddhists

Catch 22 posted:

Yes (soon). Check the Virtualization megathreads last page, read about VMware FT.

While VMware FT sounds great and cool, I have no intentions at all whatsoever to virtualize my Oracle RAC system. It's beastly, and right now the 3 year old, up for replacement later this year system it runs on is Four nodes at each site, each with 4 Itanium2 processors, and 32GB of RAM. I'm thinking I'll probably replace it with a similar number of IBM P/Series boxes with multiple Power 6 chips and 48-64GB of ram per box. Itanium's not my cup of tea, though perhaps if I were on HP-UX instead of RHEL, I'd be in better company. Sadly, I've been told that when it was purchased, Oracle RAC 10g hadn't been certified on the similarly priced/targeted P/Series systems (it was 3 weeks after the PO was cut). All of our other mid-end boxes are P/Series on AIX.

But I digress. What I'm more asking is can I have the LUNs for my Oracle boxen on the same storage system (still on a segregated array, I assume) as my less performance intensive things like File Servers, AD DC VMs, DHCP, support databases, etc for easier management. (One disk system to buy and support rather than two)

Thanks for the pointer back to that thread though, I've not looked at it since before VMworld, so there's been some interesting reading.

# ¿ Oct 1, 2008 19:37

Intraveinous: Oct 2, 2001; Legion of Rainy-Day Buddhists

OK...

We've run into an insane amount of problems on our crappy array (Gateway Branded Xyratex 5402E) with VMware. The basics of the setup are on page 5 if you're interested. Basically, the problem is that if the array gets busy at all, it starts ignoring commands, LUNs wind up remounting RO or locking completely, and everything goes to hell.

Regardless of that, we'll have a proposal for a new array in the next couple of weeks. The new solution will be expected to house a smallish (compared to some) production VMware environment (Probably 3-5 ESX hosts with up to 60 or 70 VMs), as well as a couple of fairly high-use Oracle databases, and the data store for an Enterprise Content Management system (scans of documents stored as PDFs/TIFFs.) Total disk space for this will be in the neighborhood of 15TB, and we'll need some overhead for expansion.

I have reason to believe that the vendor will be specifying an HP EVA system. That's not necessarily a problem for me, since we have an EVA4000 already for our Oracle RAC environment, and I'm fairly comfortable managing it, but I really really really need to make sure we avoid getting something that we're going to have a bunch of problems with again.

The vendor was selected for the ECM system project thru the RFP process, and that included the hardware to support it. I'm not confident that they have anyone who's a SAN or VMware expert, and I don't really trust my own experience enough to feel completely comfortable. There are two of us that take care of everything from x86, Itanium, and Power hardware; Windows, Linux, and AIX management; Storage; Backups; etc. Because we have to be jacks of all trades, as much as I'd love to, I haven't been able to really specialize in much of it. I'm going to recommend we bring in a 3rd party consultant to help us out with evaluation, but I'm worried that (lack of) budget might kill that.

My understanding is that the EVA (and maybe everyone else now) spreads I/O across every disk you have assigned to the disk pool, regardless of individual vdisk settings. Do I need to worry about creating separate disk pools for Oracle and VMware, or will creating one large pool for the Tier 1 disks (15K FC) actually give me better performance even though it'll be different workloads? I think that the PDF/TIFF storage will probably be on lower tiered disk (probably some 7.2K FATA, since access time on it will be a lot less critical than on the VMware and Database sides). Am I just over thinking/completely out of my depth with these worries because of the problems I've had recently?

Would I be better served pushing for a different solution, ie NetApp or EMC?
I've heard a lot of good things about VMware on NetApp NFS, and a few things about Oracle over NFS as well (though I don't know whose).

Any help anyone can offer would be great.

EDIT, fixed some stuff I said that didn't apply.

Intraveinous fucked around with this message at 00:48 on May 8, 2009

# ¿ May 7, 2009 22:49

Intraveinous: Oct 2, 2001; Legion of Rainy-Day Buddhists

Is there a site out there somewhere that has reviews and information on various enterprise storage products? I've done some googling around and found mostly reviews of consumer level products.

My next project to be piled on is to replace 9 small file servers that were originally deployed out in field offices. Because they were standalone, 5U Dell PE 2800 servers were purchased, and an LTO2 tape drive installed in each. About a year ago, it was determined that the network connections to these remote offices had been upgraded enough that the servers could be moved back to our central computer room. Now they're about to come off of maintenance, and the renewal cost is high enough that I think I should have no problem getting a NAS box to replace all 9 of them.

I'm looking at the HP DL185G5 or DL380G5 storage servers. Each of the 9 servers has only ~200GB of shared storage, so it shouldn't be any problem at all to put something together. I haven't really used Windows 2003 Storage server, how does it stack up for a NAS OS? I'm leaning toward it because the current 9 are all standard Windows boxes, so the changes in setup should be minimal.

Since most of the users have connections between 1.5Mbps and 5Mbps, I'm thinking I could probably get away with 7200RPM SATA or MDL SAS for the drives, so long as I give myself enough redundancy, thinking RAID6.

I've got a wide range of brands already, but we've been moving toward standardizing on HP for most of our x86 and storage stuff. That doesn't mean I wouldn't consider anything non-HP if they're still a tier 1 provider and have a good product at a good price.

I also just installed an EVA 4400, would I be smarter to buy a disk shelf and some FATA drives for it and then get a SAN gateway box or just set up another Windows File Server on VMware?

# ¿ Jul 16, 2009 14:52

Intraveinous: Oct 2, 2001; Legion of Rainy-Day Buddhists

1000101 posted:

Clearly this is the case but I'm not sure how Oracle handles it. Is Oracle going to acknowledge every write? If so then yeah 70ms would make the DBA want to blow its brains out. Or does it just ship logs on some kind of interval? More of an Oracle question than a storage question but it would be helpful in deciding if you should use array based replication or application based.

Typically when we're doing offsite replication and the RTT is >10ms we tend to use async replication but it's often crash consistent. Exceptions are when you use tools like SnapManager to do snapmirror updates as part of your DR process. It's a larger RPO but you're going to be application consistent on the other side.

Knowing little about Oracle what I might do is something like this on an hourly schedule:

1. Place oracle in hot backup mode
2. Dump an oracle DB backup (if feasible, may not be depending on DB size)
3. Snapshot array
4. Take oracle out of hot backup mode
5. Replicate recent snapshot offsite.

Step 2 may be completely redundant though. This is not unlike how something like NetApp Snapmirror works (kick off exchange VSS writers, kick off array snapshot, turn off VSS writers and tell the array to update snapmirror which sends the recent snap offsite.)

Bandwidth requirement is basically whatever it takes to replicate the difference between each snapshot. So if you're ready heavy you could probably use less than 128kb or if you're write heavy it could get pretty insane. It is definitely something to keep an eye on.

Apologies in advance, as I'm way behind in this thread. If this has already been answered months ago, just feel free to ignore me.

The way we replicate our Oracle Databases is using Oracle Dataguard. Basically, you have a temporary LUN set up on the DR/Receiving side that receives a log ship on a set interval. Your shipping interval can be as short or long as you like, and since we've got 100Mbit connection to DR, we go on a constant (real time) shipping schedule, and if something makes it get behind, such is life. Our SLA is for DR to be within 5 minutes of Production. Main DB writes out logs as changes are made and ships them out to the DR site, where they are applied. Whole thing is handled nicely in Oracle Grid Control.

IANADBA, so I'm not sure if this requires RAC in order to work this way, or not, but that is what we are running currently. 4x node Integrity for our production systems, and 2x node for DR, running Oracle Database Enterprise and RAC 10g.

# ¿ May 25, 2010 02:52

Intraveinous: Oct 2, 2001; Legion of Rainy-Day Buddhists

OK, I'm finally caught back up. This is such a great thread in general, so thanks to everyone contributing so far.

My question is if anyone has any experience with the FusionIO line of enterprise PCI-Express SSDs, AKA HP StorageWorks I/O Accelerator Mezzanine cards for the C-Series blades. I believe IBM OEMs their standard form factor PCIe cards as well, but I don't know what they call them.

Basically, I have a fairly small (~30GB) index db sitting in front of a much larger Oracle RAC db. This index handles the majority of queries from a website that gets about 15 million hits a month, and only when a user drills down into a query does it fetch from the main RAC db.

The index is running right now on a fairly ancient (We're about to have a tenth birthday party for it) IBM RS/6000 box and a SSA attached 6 x 9GB disk (4+1, 1 hot spare) RAID 5 array that was set up long before I was around. It sits at 100-105% utilization in topas 24x7, pulling between 700 and 1000 IOPS of 99% random small reads.

AFAIK, nothing says I can't replace this box with non-IBM Power hardware, so I'm thinking about dumping it on a BL460/465c blade (CPU licensing costs will likely skew things in Intel's favor since I should still be able to get dual core 550x cpu) with one of the 80GB SSDs. FusionIO and HP have been claiming north of 100K IOPS, and 600-800MB/sec read rates from this kit.

I'm sure once I eliminate the disk I/O bottleneck, I'll find another, but this seems like the perfect use for the part. Considering that I was looking at 5-10x more money, wasted space (both disk and rack unit), plus a bunch of extra power to short stroke an array to get even 3-5K IOPS, I'm having a hard time finding a fault, even if I only get 25% of advertised performance.

My one big worry would be fault tolerance. The data is pretty static, generated at timed intervals by a script from the larger database, so I'm not worried about the data loss as much as the downtime if it fails. A half-height blade would (in theory) let me put two of them in (if I didn't need any other expansion at all) and do a software mirror, but am I being stupid? I'm not going to be able to hot-swap a mezzanine card no matter what I do.

I'd have another blade at our DR site that could be failed over to in that case, but if I can avoid that problem as much as possible, that would be ideal.

So anyway, please tell me I've found exactly the right thing for this job, or that I'm an idiot. Although please, if it's the latter, tell me why and suggest something else to look into.

Intraveinous fucked around with this message at 18:31 on May 25, 2010

# ¿ May 25, 2010 17:50

Intraveinous: Oct 2, 2001; Legion of Rainy-Day Buddhists

H110Hawk posted:

Assuming you have some kind of HA way to failover intra-datacenter and inter-datacenter you could do just what you are suggesting. I'm adverse to blades, but whatever makes you happy. I would grab 4 of the cheapest 1u Dell/HP/IBM/Whatever you can find with 5520 cpus in them, fill them with memory and a boot disk. When it gets there, velcro in an intel SSD to one of the empty bays. It doesn't need cooling, you could even leave the plastic spacer in there as a caddy.

Use two in your live datacenter and two in your DR. Have your generate-index script write to the live server's hot spare, fail over, write to the new hot spare, then write to the hot-spare-datacenter servers serially. Remember to detect write failures so you know when your SSD dies and call it a day.

I would suggest a full commodity hardware solution but I guess that wouldn't go over well. Instead of an off the shelf intel ssd you could use one of those PCI-E SSD's you were looking at as well.

If I needed to fail-over, it would be a manual process of changing the db-links on the Websphere servers that host the site to point at the hot-spare server at the backup datacenter. Total downtime would be in the neighborhood of 3-4 minutes, which is kind of ugly, but considering there is currently ZERO redundancy for the 10 year old IBM box, it's a step up. Funny how things have a way of getting too big for their britches a lot faster than people expect them to, but then still have problems getting funding to "do things right(tm)".

As for Blades vs anything else, I share your apprehension, but we have standardized on HP kit, and HP only offers the PCIe SSD as a mezzanine card. Price point on the 80GB is around $4400 list, so figure ~$3500 after discount. The 60GB hot plug SATA "Midline" SSD is around $1500 list. Either would be such an improvement, people would probably soil themselves, but I tend to think the PCIe attached would give me more leeway in case I don't get budget to replace this thing for another 8-10 years (yikes).

I'd love to roll my own, and even considered the possibility of loading the DB into RAM, but decided the cost difference wouldn't be worth it. Sadly, roll your own solutions tend to mean whoever rolled it is stuck supporting it forever by themselves, at least around here.

Thanks for the info.

# ¿ May 26, 2010 15:22

Intraveinous: Oct 2, 2001; Legion of Rainy-Day Buddhists

Nomex posted:

If you're worried about fault tolerance, you might want to go with an sb40c storage blade and 6 of the MDL SSDs in RAID 10. That would give you about 60k random read IOPS and ~15k writes.

This was the way we ended up getting approval for. BL460c + SB40c with SSDs. Now that I'm getting down to actually buying things, I wondered about using something other than HP's MDL SSDs. Performance numbers for them aren't the greatest, and although I'll be dramatically increasing the performance no matter what, I can't help but worry about using midline drives with a 1 year warranty in a production box. For the price point on the HP 60GB MDL SSDs, I can get 100GB (28% Overhead) "Enterprise" SSDs from other vendors. Examples would be the recently announced Sandforce 1500 controller based offering from OCZ, Super Talent, etc. The SF1500 allows MLC, eMLC, or SLC flash to be used, has a super capacitor for clearing the write buffers in case of a power outage (these will be on UPS and generators, but still nice in case someone does something stupid), Promises read/write rates up to near the limits of SATA 2, and come with 3-5 year warranties, vs HP's puny 1 year.

Being such new stuff, I'm a little hesitant to put prod on Sandforce, and in an "unsupported" configuration, but I'm also hesitant to spend the money on HP's drives which aren't rated for high end workloads, have a shorter warranty, and are slower all around.

HP is supposed to be releasing their "Third Generation Enterprise SSDs" some time in the next few months, but I can't really wait around any longer, as the performance problems are getting more and more common on the current kit.

TL;DR version:
Making an array of 6x SSDs in Storage Blade, stick with supported, but slower, lower rated midline SATA HP drives, or go balls out and bleeding edge with unsupported SandForce 1500 based enterprise SSDs like the OCZ Deneva Reliability/Vertex 2 EX, or Super Talent TeraDrive FT2 for about the same cost.

# ¿ Jul 23, 2010 17:51

Intraveinous: Oct 2, 2001; Legion of Rainy-Day Buddhists

Nukelear v.2 posted:

This thread needs a bump.

Building up a smallish HA MSSQL cluster and my old cheap standby MD3000 is definitely looking long in the tooth. So I'm going back to my first love, the HP MSA and I must say the P2000 G3 MSA looks very tempting. Anyone use either the FC or SAS variants of this and have any opinions on it? I've also been reading that small form factor drives are the 'wave of the future' for enterprise storage, logically it seems to be better but I haven't really heard too much about it, so I'm also trying to decide if the SFF variant is the better choice.

We have a couple of the P2000 MSAs and have been happy with them by and large. Ours are the SAS versions. One is used for a small SQL server install, and the other for a remote web server farm that allows uploads of documents into our doc mgmt system.

# ¿ Jul 23, 2010 18:03

Intraveinous: Oct 2, 2001; Legion of Rainy-Day Buddhists

Nomex posted:

You could go with 6 Intel X25-E drives instead. They're still unsupported, but they have a 5 year warranty and use SLC flash. Also they're rated for 35,000/3,300 read/write IOPS each. They might be older tech, but pretty reliable.

On a side note, I've got a customer who's going to be stacking 10 Fusion IO drives in a DL980 as soon as the server is released. I can't wait to run some benchmarks on that.

I figured if I were doing an unsupported config anyway, may as well take advantage of the additional speed offered by the new drives (285/275MBps R/W, 50K IOPS Aligned 4k random write), though I would be using something with more known reliability in the X25-E.

On the 10X Fusion IOs, how do they plan to stack them? Software RAID? The Infiniband attached chassis with IO Drives that Fusion used for one of the National Laboratories look insanely nice.

# ¿ Jul 23, 2010 19:38

Intraveinous: Oct 2, 2001; Legion of Rainy-Day Buddhists

Nebulis01 posted:

So I have a stupid question. I'm new to administering a SAN so please bear with me.

We just recently got a Dell Equalogic PS4000X. I configured it out of the box for RAID6 since it's mainly going to be used for read access. However I was exploring some of the volume options and it appears you can assign a raid type for the volume, but it won't change any of the member drives to fit this configuration.

My question is this:

Is it possible for a member to have more than one raid type or do all 16 drives have to be allocated to the same raid type? It seems like that would be a stupid design but I've not found anything in the documentation to answer it, or I'm blind and can't locate it.

Any help would be much appreciated.

I don't have specific experience with the Equalogic kit, but the way it works in the EVA world is that your disk group (made up of all or some of the disks you have) is turned into a large RAID volume. You can choose "Single drive failure protection", similar to RAID 5, or "Double drive failure protection", similar to RAID 6. From there, each LUN (vDisk) you create on that disk group has a "vRAID" level that you assign when you create it. The EVA secret sauce then splits things out for that vDisk and spreads them around the larger disk group. So say you have a Double Protection disk group, and you create a vdisk with vRAID1. You acheive that by having each block stored on two different disks of the disk group. vRAID 5 is similar, except you have each block striped across multiple disks, and then have a parity block stored as well.

I'm not sure if that's how EQL works it or not, but I hope it wasn't a waste of time.

# ¿ Aug 6, 2010 14:07

Intraveinous: Oct 2, 2001; Legion of Rainy-Day Buddhists

Any recommendations on RAID controllers for a small array of SSDs (4-6). Will be in either Raid 10 or RAID 5/6 if the controller is good enough at it. Read heavy database (95%+), so R5/6 write penalty shouldn't be too big an issue.

The HP Smart Array P400 results below:

code:

[root@JBCMI ssd]# /opt/iozone/bin/iozone -b ~/4k25g4t.xls -r 4k -s 25g -t 4 -i 0 -i 1 -i 2
        Iozone: Performance Test of File I/O
                Version $Revision: 3.347 $
                Compiled for 64 bit mode.
                Build: linux-AMD64

        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                     Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
                     Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
                     Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
                     Fabrice Bacchella, Zhenghua Xue, Qin Li.

        Run began: Mon Aug 30 12:09:57 2010

        Record Size 4 KB
        File size set to 26214400 KB
        Command line used: /opt/iozone/bin/iozone -b /root/4k25g4t.xls -r 4k -s 25g -t 4 -i 0 -i 1 -i 2
        Output is in Kbytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
        Throughput test with 4 processes
        Each process writes a 26214400 Kbyte file in 4 Kbyte records

        Children see throughput for  4 initial writers  =  253416.93 KB/sec
        Parent sees throughput for  4 initial writers   =  229461.66 KB/sec
        Min throughput per process                      =   61416.07 KB/sec
        Max throughput per process                      =   64604.90 KB/sec
        Avg throughput per process                      =   63354.23 KB/sec
        Min xfer                                        = 24924492.00 KB

        Children see throughput for  4 rewriters        =  259375.90 KB/sec
        Parent sees throughput for  4 rewriters         =  234136.11 KB/sec
        Min throughput per process                      =   63879.16 KB/sec
        Max throughput per process                      =   65675.30 KB/sec
        Avg throughput per process                      =   64843.97 KB/sec
        Min xfer                                        = 25497648.00 KB

        Children see throughput for  4 readers          =  490873.09 KB/sec
        Parent sees throughput for  4 readers           =  490830.09 KB/sec
        Min throughput per process                      =  119007.65 KB/sec
        Max throughput per process                      =  124878.35 KB/sec
        Avg throughput per process                      =  122718.27 KB/sec
        Min xfer                                        = 24984912.00 KB

        Children see throughput for 4 re-readers        =  477533.65 KB/sec
        Parent sees throughput for 4 re-readers         =  477503.03 KB/sec
        Min throughput per process                      =  115802.55 KB/sec
        Max throughput per process                      =  121579.46 KB/sec
        Avg throughput per process                      =  119383.41 KB/sec
        Min xfer                                        = 24973364.00 KB

        Children see throughput for 4 random readers    =   35728.62 KB/sec
        Parent sees throughput for 4 random readers     =   35728.53 KB/sec
        Min throughput per process                      =    8926.97 KB/sec
        Max throughput per process                      =    8937.35 KB/sec
        Avg throughput per process                      =    8932.16 KB/sec
        Min xfer                                        = 26183936.00 KB

        Children see throughput for 4 random writers    =   23527.42 KB/sec
        Parent sees throughput for 4 random writers     =   20701.37 KB/sec
        Min throughput per process                      =    5757.43 KB/sec
        Max throughput per process                      =    6035.68 KB/sec
        Avg throughput per process                      =    5881.86 KB/sec
        Min xfer                                        = 25011236.00 KB



"Throughput report Y-axis is type of test X-axis is number of processes"
"Record size = 4 Kbytes "
"Output is in Kbytes/sec"

"  Initial write "  253416.93

"        Rewrite "  259375.90

"           Read "  490873.09

"        Re-read "  477533.65

"    Random read "   35728.62

"   Random write "   23527.42


iozone test complete.

# ¿ Aug 30, 2010 23:35

Intraveinous: Oct 2, 2001; Legion of Rainy-Day Buddhists

Cultural Imperial posted:

That's impressive! What's your rationale for RAID10? What's the database?

Rationale for RAID10 is the old SAME (Stripe And Mirror Everything) rule of thumb on Oracle DB storage, but since SSDs make many of the concerns moot, and I shouldn't have to worry too much about rebuild time on a 50-60GB SSD, I thought that RAID6 might be feasible. I'm still just a little bit afraid of RAID5. With all of the drives being installed at the same time, I'll have pretty similar write levels on all of them. With SSD, if I were to hit the write boundary causing multiple cells to fail at the same time on multiple drives, having the ability to lose 2 of them without array loss would be nice.

P400 controller however, is somewhat bad at RAID5/6 implementation, as in, it's much slower at offloading the parity calculation than say the P410, PERC/5, or PERC/6. Since the db is a 95/5 mix of reads/writes, I ended up deciding that I didn't care too terribly much and stayed with the P400.

Database is an index for an OLTP system running on Oracle 11gR2. Should be interesting, just waiting to find the next big bottleneck now that disk i/o won't be it.

# ¿ Sep 16, 2010 23:08

Intraveinous: Oct 2, 2001; Legion of Rainy-Day Buddhists

Misogynist posted:

We've had remarkably few problems with our hardware (other than the ill-fated Xyratex SA-48 enclosures which have been replaced by LSI/DDN), but BlueArc's handling of firmware is really idiotic and not what I would expect of a large-scale software vendor.

Could you expand at all on your ill-fated Xyratex adventures? They seem to be the OEM for just about everything that's not LSI made or small/big enough to be made in house, but I keep hearing about problems with their stuff.

We have a couple of small arrays (24 spindle SAS, 4Gb FC backed) that were purchased by someone unawares of storage technology and have been nothing but problems. The name stamped on the outside of the Xyratex kit, Gateway, doesn't exist anymore since the demise of MPC.

Generally, if you try to put any kind of load, at all, whatsoever on them, the controllers will crash. Sometimes it's just a SCSI Bus reset, other times the controller locks up completely, but appears to be up to any multipathing software (with the exception of vSphere, which catches it), it just won't service any requests.

Xyratex hasn't been terribly willing to help me directly, saying they're just the OEM, and any service is the responsibility of the company that sells the box. That's fine with me, I understand that, and in fact, we found another company that resells the stuff that would sell us a basic service contract that covers dead drives and whatnot, but haven't gotten very far in troubleshooting the larger problems.

# ¿ Sep 17, 2010 17:14

Intraveinous: Oct 2, 2001; Legion of Rainy-Day Buddhists

Misogynist posted:

We have constant, unending controller failures. When the controllers work, they perpetually spit errors back to the controllers managing them even though they're actually in a completely fine state. A few weeks ago, what pushed me over the edge was having to take over our Exchange environment for a little while and delete 500,000 messages that the Xyratex enclosure had politely emailed us over the weekend.

Weird, you could have been explaining what I go through all the drat time (with the exception of the 500K emails). Mine's not nice enough to email me even if it has a legitimate failure. 1 more second with the thing would be too long. I thought that I could relegate it to a backup to disk target once I finally got all the production data off of it, but no, it can't even do that well. Apparently backups put too much strain on the drat thing.

# ¿ Sep 17, 2010 20:14

Intraveinous: Oct 2, 2001; Legion of Rainy-Day Buddhists

Corvettefisher posted:

I thought I would make 2 raid 1 arrays, then go to raid 0 and add both arrays but to no avail

Could it be like some of the HP onboards where you just select RAID1, then select more than 2 disks, and it automagically changes itself to 10/1+0?

# ¿ Nov 17, 2010 05:35

Intraveinous: Oct 2, 2001; Legion of Rainy-Day Buddhists

Noghri_ViR posted:

So did anyone pay attention to the emc new product announcements today. What are your thoughts on them? I was about to rule them in out new purchase but now I'm going to take a second look

I didn't bother with it, but I heard something about them stuffing a bunch of hook^H^H^H^H dancers in a Mini Cooper, and jumping a motorcycle over a bunch of Symmetrix cabinets... Sounds like a lot of fluff with not much substance to me.

That said, having not watched it, I don't know if there was any substance, but if I'm going to take my time to watch a webcast, I want to know what the product is, what it does, and how it does it. If I wanna watch stunts, I'll do that another time.

In other news, has anyone recently done a comparison between Compellent and 3PAR?

I haven't gotten even budgetary quotes from either of them yet, but both seem to have nice feature sets. Once I get some base numbers, if they're affordable, I'll probably ask for a demo unit from each of them.

Intraveinous fucked around with this message at 16:14 on Jan 24, 2011

# ¿ Jan 24, 2011 16:09

Intraveinous: Oct 2, 2001; Legion of Rainy-Day Buddhists

Misogynist posted:

Speaking of benchmarking, what's everyone using? I'm trying to use iozone to benchmark a few particular RAID controllers, and I'm having a hell of a time with it. I'm getting performance differences of up to 300% on certain benchmarks on identical runs.

I did a lot of testing with iozone last summer and had pretty consistent results across identical runs. What options are you running it with?

My standard baseline run is:
iozone -b baseline.xls -r 1m -s 8g -t 6 -i 0 -i 1 -i 2

Obviously adjust your -r for block size you want to test and -s to make sure it's large enough to take cache out of the picture. I've seen the most variance with different -t settings, since it simulates the queue depth, e.g., I was able to see large difference at high queue depth between drives that were otherwise identical with the exception of having SATA or SAS connections. I tested identical arrays made up of Seagate Constellation ES 1TB drives, 6x SATA and 6x SAS. Only difference between them was that the SAS version supported 6Gb SAS, but I was using a 3Gb SAS/SATA RAID card, so that shouldn't matter. There was a very definite drop off of performance on the SATA drives above a certain setting for -t. I don't have the results handy to know what that number was, sorry. Even then though, I'd see very similar results between runs, all else being equal.

HTH

# ¿ Feb 7, 2011 16:49

Intraveinous: Oct 2, 2001; Legion of Rainy-Day Buddhists

evil_bunnY posted:

Your EVA doesn't self-report? For shame.

HP hounded me until I'd let them come in and install their silly SIM Remote Reporting service, since they were shutting down the old phone-home systems. Once I let them come and set it up, it's far worse off than it ever was before. It loses connectivity with their servers for at least a few days every week, and likes to do things like ignore failed disks when they happen, then submit 5 duplicate tickets for the failure a few days after I manually log a case, get the replacement and install it. It usually decides to do this at 3AM on a Tuesday so I get phone calls from India until I wake up, log in, check the EVA, and call them back to say it's a false report.

So if it's anything like mine, the EVA is *supposed* to self report, but doesn't do a reliable job of it.

Meeting with HP/3PAR, NTAP, and Compellent this week to look at some different storage. Should be interesting.

# ¿ Feb 15, 2011 16:38

Intraveinous: Oct 2, 2001; Legion of Rainy-Day Buddhists

InferiorWang posted:

HP makes getting competitive pricing, at least with the lefthand units but probably with all their storage, impossible. Butt hurt VARs refuse to give me quotes because they're not the preferential partner on the project and I can't make a purchase without competitive quotes. I hate VARs.

Better than the alternative, where the different VARs throw fits and want you to call HP and tell them you don't wanna work with the preferential partner. I've gotten put in the middle of 'registered opportunity' spats a few times in the recent past. While I understand that the registration process is there for a reason, to protect the VARs, it should be transparent to the customer. I also understand the VAR not wanting to spend time to provide you what will basically amount to list pricing, which pretty much guarantees they won't be selected to fill the order.

That said, if you're having problems with it, see if CDW or PC Connection or one of the ginormous resellers will give you a quote, or call your HP rep and explain the situation. I'd be they can figure something out to get you the quotes you'll need.

# ¿ Mar 2, 2011 21:12

Intraveinous: Oct 2, 2001; Legion of Rainy-Day Buddhists

So, what's the preference on arrays where you create several different groups of spindles, set a RAID protection level on that array, then present luns from it -vs- the ones that do a large, per tier, disk group and then virtual raid at some stripe level on top of it?

I've dealt mostly with HP EVA, which does the latter, and we're looking at different options now to replace an EVA 4000. 3PAR and Compellent both do things similar to the EVA, while NetApp and EMC(?) does the former.

The Netapp we're looking at is a V32xx, which we could use to virtualize and manage the EVA 4400 we'll still be using for at least a few years. So cutting it down to only doing stuff in OnTap would cut some of the management tasks.

Right now I've got budgetary quotes in hand from 3PAR and NTAP, and expect a Compellent one soon. Haven't talked to EMC yet. Anyone else I should be talking to? Am I making too big a deal out of the differences in management?

# ¿ Mar 4, 2011 22:22

Intraveinous: Oct 2, 2001; Legion of Rainy-Day Buddhists

blk posted:

My questions for you:

Should I get one processor or two? Dual core or quad core?

Is it worth investing in hard drives as SSDs continue to evolve and come down in price?

Would I really need SAS drives for this kind of use, or could I get away with SATA?

If the money's only available till the end of the month, and nothing else requires replacement now, I'd spend the money on a new server. Nothing says you have to toss the old one, maybe get some more RAM and use it to set up a lab using free VMware/Xen/HyperV?

All of the rest depends on what kind of usage you're seeing now, use perfmon to graph a few days of CPU and memory usage, as well as HDD read/write rates and latencies.

For a fileserver, I rarely see any benefit in having more than 1 CPU, and have sometimes even gone with the lower end SMB servers that use Xeon 3xxx or desktop processor. Whether this will work, or whether to get dual or quad depends heavily on what you learn from the perfmon reports. If you're seeing high CPU usage on the existing 4 cores, one modern quad core Xeon 55xx/56xx will have enough instruction per clock speed increase to handle everything no problem. If you're seeing minimal CPU usage the majority of the time (which I'd suspect), you might be able to get away with a Xeon 3xxx/Core CPU, either dual or quad, depending on costs. In the 5xxx series Xeons, you don't get a whole lot of price reduction on the dual cores, but you do get feature reductions which can put them below the Xeon 3xxx series.

SSDs main benefit is high speed, which you're rarely if ever going to need on a general user file server, especially on 100Mb network. You'll spend several times the money for the same amount of storage with SSDs, and with no need for the speed, it will just be wasted money.

I'm partial to SAS, personally, but there may not be much reason to use it here, for many of the same reasons SSDs won't do much good. One thing to consider if you go with SATA (or nearline 7.2K SAS) is that you may need to rethink your RAID level on the higher capacity drives. If you're using 1 or 2TB 7.2K drives with RAID 5 and lose a drive, you've got quite an increased chance of a second drive failing before rebuild finishes on the first drive. That means you lose the entire array. Not sure if Dell's current PERC cards include R6 support by default, or if it's a feature you have to license (you do on some HP cards). You'd also need to include the cost of additional drives for the higher RAID levels: 2TB usable would quadruple your current storage, and requires 3 1TB drives with R5, but 4 1TB drives with R6.

I threw together an R310 with a Xeon X3430, 8GB of RAM (2x4GB RDIMM), PERC H700 with 512MB write cache, 4x 1TB Near-line SAS drives, DVD, and Win 2008 R2 SP1 with 5 CALs. List price was around $4250 on the Dell SMB site. Being a non-profit, I'm sure you'd get significant discounts, so that seems to be well within your budget.

HTH

# ¿ Jun 6, 2011 18:41

Intraveinous: Oct 2, 2001; Legion of Rainy-Day Buddhists

FISHMANPET posted:

Thoughts on Compellent? We just had a meeting a Dell rep. It all sounds pretty awesome, if it works.

Odds are we're going to go with a JBOD attached to a server because we're dumb, but it looks cool.

We just took delivery about 6 weeks ago of two Series 40 controllers and 5 disk trays. So far it's been great.

Copilot support is fantastic, great getting someone who knows what's going on on the second ring. I'll be heading up to Eden Prairie sometime this summer to go through their training, but the majority of everything I've touched thus far has been very intuitive.

The data progression seems to work pretty well, though we're still in the process of getting everything up and into production use. We'll be getting a second array for our DR site in the next few months, and setting up array replication.

So far, I'm really happy with it.

# ¿ Jun 9, 2011 16:19

Intraveinous: Oct 2, 2001; Legion of Rainy-Day Buddhists

amishpurple posted:

The PO for our Compellent just went out today. We chose them over NetApp and EMC and I can't wait to get it in. We got 156 spindles coming!

Enjoy it, we've got 88 (64x146GB 15K, 24x2TB 7.2K) and so far, it's speedy as all hell. 3PAR went all ball baby when we mentioned Compellent. Netapp reseller immediately started talking bad about their technology, then came back with a new quote that was 30% below their previous "best and final", but we still ended up with the CML.

Intraveinous fucked around with this message at 20:22 on Jun 9, 2011

# ¿ Jun 9, 2011 20:10

Intraveinous: Oct 2, 2001; Legion of Rainy-Day Buddhists

Vanilla posted:

So with all the big vendors the salesman is totally in control of the price. The price he will submit depends heavily on the situation and you should never believe the 'best and final' line - It's bullshit.

Are you a huge Netapp fan with money? Hell you'll get a little discount.

Are you a non-Netapp account and in the middle of a refresh which may feature a 'price price price'vendor like Dell? You'll get a great discount. Netapp are under a lot of pressure - they have one revenue stream, a huge growth rate to meet and wall street to please. They will drop the price.

When Dell come up against EMC / Netapp they know they don't have the feature set to match so they just drop the price because often price is the biggest factor.

Some common sense tips:

- Research into when end of quarter is for certain vendors. You will get a better price the closer you leave it to their EoQ.
- Involve multiple vendors in the process and even inform the vendors who you are looking at (but share nothing more).
- When you've reached the point where you think you can't get the price down any more that's fine - move onto other things. You got a great deal - why not secure the same discount levels for your upgrade costs for the next 3 years? This is actually really easy to do because sales people don't see / care that far ahead. They just want that deal.
- Don't believe the 'only valid for this quarter' crap. If you turned up three days into a new quarter and said 'I want to buy this now' they're not going to risk you going elsewhere. If they play hard, play hard back.

All of this is true, and I didn't mean to make it out as me bashing Netapp or 3PAR's tech, all of them had pros and cons, and I'm sure all of them would have been very capable of doing the job we were needing the array for. There were definitely things about each that I liked better than the others, but all were very capable.

# ¿ Jun 10, 2011 15:34

Intraveinous: Oct 2, 2001; Legion of Rainy-Day Buddhists

KS posted:

I'm glad to hear that everyone else is having good experiences with Compellent too. We bought a pair of them about six months ago to replace some Hitachi arrays and they have been nearly perfect. We are incredibly happy with the performance we're getting, which is good, because I recommended them.

We did have an array go down last month, which was scary, and I might as well throw this out there in case it helps anyone else: if your controllers have 8gig emulex cards in them and you're using brocade switches with 6.3+ firmware, there is a TSB out on Brocade's site warning of incompatibility. It was a very tense Sunday as the array gradually lost connectivity to 50+ servers. The problem appears after a controller reboot, and it took Compellent swapping the cards for Qlogics (in <3 hours) to get us back on our feet.

Do you have a link to the TSB? I recently upgraded my brocade switch firmwares, and since then, I'll randomly lose one MPIO link at a time.

Since we're not fully in production on it yet, that hasn't been a problem as of yet, but I'm fairly certain we've got four 4port 8gb Emulex cards.

# ¿ Jun 13, 2011 17:18

Intraveinous: Oct 2, 2001; Legion of Rainy-Day Buddhists

KS posted:

I don't have a link. You need brocade support THROUGH BROCADE to have access to them, and ours is through Hitachi.

This might help though, our Compellent case # for the outage was REDACTED. You may be able to reference this and get them to deliver the TSB, or at least point them in the right direction to fix your issue.

(if you get the tsb, send it to me please!)

YGPM. I'm not finding anything specifically listing Compellent or Emulex card incompatibility at this point.
I'll give CoPilot a call and reference your case # to see if they can give me the TSB #.

Thanks for the heads up.

Intraveinous fucked around with this message at 22:52 on Jun 13, 2011

# ¿ Jun 13, 2011 21:46

Intraveinous: Oct 2, 2001; Legion of Rainy-Day Buddhists

Syano posted:

This may be better suited to the virtualization thread but whatev I will give it a go here: We are setting up a Xenserver environment on top of HP Lefthand storage. Reading through the best practice guide published by HP they recommend as best practice having 1 VM per 1 volume per 1 storage respository. Is there any reason why?

Not sure how Lefthand specifically does it, but often the recommendation for low numbers of VMs per volume/LUN is to avoid issues with SCSI locks during metadata updates. I really wish best practice guides would get into the "Why this is a best practice, and what might happen if you don't follow it" realm more often.

# ¿ Jun 15, 2011 22:49

Intraveinous: Oct 2, 2001; Legion of Rainy-Day Buddhists

FISHMANPET posted:

Any thoughts on Sun's 7000 series, specifically the 7120 or the 7320 (as I imagine that's all that would be in our price range? Most of us are on the Compellent side, but one guy is super on the Oracle side, so we basically have to fight against Oracle for this. Would something like that work well as a VMWare Image store? How much access to the OS does the 7000 series give?

The word "Oracle" just leaves a bad taste in my mouth in general. I know it's Sun kit, but once you buy it, you're at the whim of Oracle deciding whether or not to continue supporting it, and honestly, you're not getting much of anything you can't get with a white box (besides Oracle's "outstanding" support [/sarcasm]. I like ZFS, and run white box ZFS implementations both at home and work. I know you said you'd been burned by white box implementations before, so really the questions are: How much time are you going to be able to spend learning the ins and outs to really know the system, and how much turnover do you have in your IT department? If you'll be the primary one supporting the system, and it will be your primary function (so you'll know it like the back of your hand) there's less to fear. If it will be dumped off on a contractor or something, even something with Oracle's support will be better.

I'm not trying to play fanboy for Compellent...

DCIG did a "Midrange Array Buyers Guide" last year. Ranked the Compellent Series 20/30 as "Recommended" or "Excellent" in all fields, and the Sun 7310 as "Entry Level" in all but Software, which it got a "Good".

You can grab it here: https://dcig.wufoo.com/forms/dcig-2010-midrange-array-buyers-guide/?utm_source=&utm_medium=&utm_campaign= (I'd make sure to have a trash email to give them, since they'll spam the hell out of you.)

I don't know how much stock you put in those kinds of things, but sometimes the glossy marketing stuff is enough to convince the person with the purse strings that you're right.

# ¿ Jun 16, 2011 15:40

Intraveinous: Oct 2, 2001; Legion of Rainy-Day Buddhists

FISHMANPET posted:

...it's just a matter of convincing the big boss that the Oracle solution probably isn't very good. We've also been burned recently by Oracle support, so really not sure why we would even consider it. If the 7000 series is a blackbox, that's a knock against it.

The big boss is basically deciding between Compellent or some Dell servers with MD1200 arrays connected to them, running ZFS. The need to discredit Oracle comes from the third party that has way too much influence.

That White Paper looks good, even if it took some serious drug deals to get a copy of it. The link you posted didn't work, so I snipped the end off and got a registration page, registered and got a link, and that just gave me a header png hosted on dropbox. Googled for the paper, found the link on the DCIG site, registered with NEXSAN, the link they gave me 404ed, googled the file name, and found a copy: http://www.adn.de/media/CMS_Bilder/Nexsan/2010_DCIG_Midrange_Array_Buyers_Guide.pdf

I'll have to keep an eye out for the 2011 buyer's guide.

Crap... sorry about that. I should have just uploaded it somewhere.

Before too much longer, the Compellent array will be just that: a couple of Dell Servers with MDxxxx drive shelves. Currently, the controllers are 3U Supermicro boxen. The shelves are Xyratex OEM, who make quite a few different array vendor's shelves. The difference will be that the secret sauce is the Storage Center software rather than the ZFS software.

If you have an experienced admin who can spend time keeping it running/tuned to the workload; advantage may go to ZFS whitebox due to the ability to use cheaper SSDs for L2ARC/ZIL. I'd still advocate using "enterprise" SSD over a Vertex 2, but at least you could use ~$700 SSDs that are approved/supported by the server manufacturer, vs the $10K+ most array vendors seem to charge per SSD.

If array maintenance is going to just be another task on a list of CJ duties, having someone you can call at 3AM to help you through recovery from a problem is obviously a plus. The advantage goes to Compellent there. CoPilot is outstanding, especially for the company that can't afford a dedicated storage admin.

This assumes iSCSI connection, since last time I set up any ZFS/commodity hardware based "SAN", you didn't have an option for presenting it out over fibre channel. That may have changed by now, so I might be wrong.

Best of luck to you, I'm sure whatever you end up with will work fine, and it'd be (it was) a harder decision against anyone other than Oracle.

# ¿ Jun 20, 2011 17:19

Intraveinous: Oct 2, 2001; Legion of Rainy-Day Buddhists

Honky_Jesus posted:

The newer models are petty spiffy; I touch myself thinking about the SL8500.
I like tape.

A place across the street just ripped out two SL8500s (I believe the somewhat joke was that they were going to just put em down by the river for the homeless folks to live in.)

Went to an IBM, not sure what model it is... It's smaller tho, and I guess it works better with TSM, which was their reason for replacing it. I'm not willing to believe that a flagship library like an SL8500 had any problems with TSM that STK/SUN/Oracle engineers wouldn't fall all over themselves to fix, so I'm thinking there may have been something else going on.
I told them I'd take them off their hands if I'd had the floor space.. They seemed at least partially serious about just dumping them. Made me sad.

# ¿ Jul 25, 2011 19:43

Intraveinous: Oct 2, 2001; Legion of Rainy-Day Buddhists

Vanilla posted:

So speaking quite honestly an NX4 box is basically as low as it gets and is very old. Support is likely the same.

VNX support is much better, VMAX / Symmetrix support is a country mile beyond both of those. The VNX is a world away from the NX4 - the NX4 is from what, 2006?

Compellent is more VNX range so you're talking a different EMC ball game. With Compellent Dell are doing great but there is one teensy weensy problem with Compellent:

Compellent is still a 32 bit OS which means the maximum cache is 4GB. This is a 'welcome to 2003' roadblock for Compellent and likely offers so more cache than you use today. That's the kind of cookie that isn't going to get solved without a painful software upgrade and the usual 'buy a ton of memory'' offering (assuming you can upgrade a 32bit array to a 64 bit array - likely you are buying the last lemons off the truck).

Other arrays, such as the VNX and Netapp arrays can offer you far more cache on the controller and also through the use of SSD drives or PAM cards. These make a world of difference.

We've got a couple of Compellent arrays that we installed last year. So far, I haven't seen any issues with the 4GB of cache. We're using them behind a 10 node ESX environment, as well as an Oracle RAC database cluster on IBM AIX.

So far, I've had nothing but good experiences. Latency has been kept in good shape, and the data progression has saved me a bunch of both space and time, since I don't have to do anything to move things.
I've been told that the upgrade to 64bit OS will be coming later this year, and shouldn't provide any more headaches than a normal controller software upgrade. The series 40 controllers already came loaded with 6GB of RAM, so if RAM is used as cache, it should be possible to see an immediate increase. Since they're just commodity supermicro xeon servers, adding cache beyond that should be as simple as adding memory.

I know that there is a cache card currently, and if they're not using memory as of now, then it would take a hardware upgrade to use more than 4GB of cache.

It seems to me that the "They're 32BIT!!! Only 4GB of CACHE!!!!" has become a rallying cry against Compellent mostly since NTAP finally got 64 bit code running last year. That said, I haven't really seen any issues with having "only" 4GB of cache. Your workload may vary of course.

Our biggest reason for going with the Compellent was that we have a very small staff (4) that is responsible for a very large number of systems (everything in the data center). We don't have the luxury of having a network admin team, a storage admin team, a windows team, a unix team, a VMware team, etc etc etc. We are all of those teams. Therefore, the Compellent having a bunch of features that are done for me, in the background, without me having to mess with anything, was a big selling point. I still CAN mess with stuff if I need to, but I can let the majority of it happen automatically.

# ¿ Jan 23, 2012 18:22

Intraveinous: Oct 2, 2001; Legion of Rainy-Day Buddhists

Bluecobra posted:

I disagree. My definition of a whitebox is to use your own case or barebones kit from a place like Newegg to make a desktop or server. If Compellent chose to use a standard Dell/HP/IBM server instead, it would be miles ahead in build quality. Have you ever had the pleasure of racking Compellent gear? They have to be some of the worst rail kits ever, thanks to Supermicro. The disk enclosure rail kits don't even come assembled. It took me about 30 minutes to get just one controller rack mounted. Compare that with every other major vendor, racking shouldn't take more than a couple of minutes due to railkits that snap into the square holes.

The Supermicro controllers were the easier of the parts to rack. The Disk Enclosures are OEM'd by Xyratex, who make enclosures for a wide array of different vendors out there. Info here (yeah, it's a few years old and things have likely changed) http://www.theregister.co.uk/2009/03/02/xyratex_sff_arrays/.

It was definitely an annoyance at first, but once I'd done one, the others weren't that hard. I don't base the worth of something on how easy it is to rack.

EDIT: I was behind a bit in the thread and didn't notice SC 6.0 release had already been talked about.

Intraveinous fucked around with this message at 23:48 on Jan 23, 2012

# ¿ Jan 23, 2012 22:29

Intraveinous: Oct 2, 2001; Legion of Rainy-Day Buddhists

NippleFloss posted:

Not really germane to your argument, but NTAP has supported more than 4GB of ram for quite some time. The 64-bit upgrade that you're talking about was to Aggregates, not to the actual code base. It allowed larger pools of disk but did not provide any additional overhaul of OnTAP to support larger memory pools, as it was not required.

Cache size is one of those questions that customers ask because it's a nice easy measuring stick number that you can use to compare two arrays, but it's really only a valid question in the context of a specific workload and the overall design of the controller. If your cache is appropriately matched to your disk on the back end, and the application on the front end, then you'll be fine.

That said, I have seen workloads that will absolutely thrash the 32GB per controller in the FAS6080 nodes, so there are certainly instances where more cache that 4GB would be requisite, at least on a NetApp controller. I'm not conversant enough with Compellant and how they use their cache to say, but I'd guess 4GB isn't enough for high end workloads on a single node of their gear either.

Yeah, you're correct that I wasn't saying that Ontap 8 being 64 bit was what allowed them to address more memory or more cache, just that since NTAP started using a 64bit OS the poo poo-slinging against Compellent for using a 32bit OS has increased several fold. Absolutely true on the match cache to back end disk to workload or you'll be sorry. I doubt most people who had workloads that would thrash 32GB of cache on a FAS6xxx would be looking at Compellent to begin with. For my environment, I was easily able to hit my storage IOPS and latency requirements with 3-4 shelves of 2.5" 15Krpm disks, and my size requirements with 1-3 shelves of 7.2K 3.5" 2TB disks.
We looked very hard at 3PAR, and couldn't justify the 4x raise in price over the Compellent. There were nice features that I liked better on each. We also had NTAP come in, and while their kit was quite nice, and I liked the PAM Cards and the price, it really came down to their resellers not listening to what I told them I wanted and needed on multiple occasions. They were trying to sell me a V3xxx array to stick in front of my existing EVA4400. While that's an interesting concept, they kept balking when I told them that I wanted this array to be able to stand on its own and handle the full load by itself. Being able to snap and dedupe and whatever else on EVA4400 LUNs would have been a nice value-add, but I couldn't get them to give me a straight up quote for a FAS/Vxxxx and enough disk to handle the whole load by itself.

On the upgrade to Storage Center 6.0, I don't know if the memory in the controller is used as a cache or not, but I suspect it's not. I know that there's a separate flash backed cache card installed into the controller as well.

# ¿ Jan 24, 2012 00:03

Intraveinous: Oct 2, 2001; Legion of Rainy-Day Buddhists

szlevi posted:

New, Dell-based Compellent boxes are on their way, slated for Summer release.

Yeah... I had expected that to happen sooner, to be honest. That said, our TAM said SC 6.0 *will* enable larger read cache via the onboard memory, and that there is a kit to upgrade the controllers from 6GB to 12GB of RAM to take full advantage of that. He said that if we're insistent, we might be able to get SC 6.0 earlier, but currently it's less than 20 systems running it, and all of them are dev/test systems.

# ¿ Jan 24, 2012 23:32

Intraveinous: Oct 2, 2001; Legion of Rainy-Day Buddhists

what is this posted:

The fact that in this day and age MBR is still commonplace, and required for Win7 boot on most computers, does not give me great confidence.

Yes, partition table isn't file system, and the EFI/BIOS thing is closely wrapped up with MBR/GPT, but seriously guys? You couldn't figure something out?

Apple switched cleanly from OpenFirmware to EFI without most people noticing any change...

While I agree with you fully in principle, the fact that Apple was able to make a change isn't really all that telling. Since Apple has complete control over the entirety of the environment on both hardware and software, it should be a lot easier for them to do than someone like MS trying to make sure it will work on every possible configuration out there from umpteen vendors.

ReFS is the one that was supposed to be in Windows 2008, then 2008 R2, and now Windows 8 (if it doesn't get cut again), right?

# ¿ Jan 26, 2012 17:18

Intraveinous: Oct 2, 2001; Legion of Rainy-Day Buddhists

Awesome, thanks for the cited history refresher! I watched some of that unfold as it happened, just stopped thinking about it so long ago... Also, on the bickering between Cairo/WinFS/ReFS, etc: I asked if this new ReFS that was linked was the one that was supposed to come out ages ago, (Cairo and/or WinFS) since I haven't been following it. szlevi gave a very nice list of articles informing me of the actual history of what I was asking about, so thanks. No need to run in the special olympics over it.

szlevi posted:

Yeah, Apple by default treats its customers like poo poo so they really didn't care what they think. At all.

Nah, it's WAAAAAAY older than that.
The story starts in the 90s, with a supposedly information-centered next-gen Microsoft OS, dreamed up by none other than Gates himself, code named Cairo if I remember correctly... and it was to be built on a brand new object-based file system - I remember hearing about a new, next-gen fs in a class back in the 90s that relies on object metadata, supposedly coming in the next NT, version 5 (today known as Windows 2000)...

...which never happened.

But the file system idea did stick and it's got a budget and was subsequently named WinFS and promised that even though it won't come in the soon-to-be-released first unified new OS (Windows XP, that is), it will be in the next one, called Longhorn, along with Microsoft Business Framework (MBF) etc: http://blogs.msdn.com/b/theil/archive/2004/05/24/139961.aspx

http://weblogs.asp.net/aaguiar/archive/2004/08/28/221881.aspx
Then we heard specs and features have been really scaled back...

...it was merged with ObjectSpaces: http://www.alexthissen.nl/blogs/main/archive/2004/05/22/word-is-out-objectspaces-to-be-merged-with-winfs.aspx

...and then rumors about being delayed again, nothing in Longhorn and a lot of denial.
Shortly after the denials, of course, eventually came the admission that indeed, nothing will debut in Longhorn: http://www.microsoft.com/presspass/press/2004/Aug04/08-27Target2006PR.mspx

Despite a lot of nerd talk about future this was obviously a death sentence for WinFS and MBF - and couple of years everything was scrapped.
MBF was gone in as little as one year:
http://www.microsoft-watch.com/content/operating_systems/microsoft_scuttles_plans_for_standalone_microsoft_business_framework.html

WinFS died in June 2006, very unceremoniously, in a simple blog post: http://blogs.msdn.com/b/winfs/archive/2006/06/23/644706.aspx

Whatever useful left was scrapped and worked into SQL Server 2008, that's it.

A classic MS-sized fuckup, spanning a decade or more.

Intraveinous fucked around with this message at 18:18 on Jan 30, 2012

# ¿ Jan 30, 2012 18:10

Intraveinous: Oct 2, 2001; Legion of Rainy-Day Buddhists

ozmunkeh posted:

Is there anything wrong with NL-SAS drives for a small VMware installation (max 3 hosts, typical AD, Exch, SQL environment)?

We're looking seriously at an Equallogic PS4100X with 24 x 600GB 10K NL-SAS drives.

The capacity in fine, and the estimated IOPS look more than adequate but I'm not entirely convinced about the bit where everything is stored on a bunch of 2.5" SATA drives. It doesn't feel quite right. Anyone got one of these in production?

You say "stored on a bunch of 2.5" SATA drives", but then you say you're looking at an array with 600GB 10K NL-SAS. Unless things have changed a lot, 10K is usually not nearline, and if it's SAS, it's SAS...

Most configs I've seen with nearline drives are 7.2K high capacity drives, eg 1TB+.
They can be SAS or SATA, with SAS giving you better command queueing in some workloads and the possibility of being dual-pathed on the SAS backend. As far as I know, the fastest 2.5" 600GB drives you'll find are 10K. I've only seen 15K drives up to 146 or 300GB in SFF (2.5") drives.

I didn't really answer your question, but I wanted to make sure I knew what you were talking about for sure before attempting it.

# ¿ Feb 24, 2012 03:08

Intraveinous: Oct 2, 2001; Legion of Rainy-Day Buddhists

Wonder what it would sound like?

Reminds me of a time in school. I had been working at a store that took trade ins of old computer equipment, and someone brought us an IBM 3340 or something "Hard Disk Drive" that was about the size of a large dorm fridge. It had several 14" platters and stored some kind of "huge" amount of data (probably under 1GB). I paid them $50 for the nostalgia of the thing, and figured I'd probably be able to get that out of the scrap metal if I really wanted to. I brought it in to show my professor, and he said that he'd written a program once to spin up/down the platters to different speeds, creating music. We messed with the controllers enough to get very basic control, but the semester ended before we got it to play full musical compositions. I did end up scrapping the thing later, after it was in a basement that flooded, but I still have one of the head arm control bars. It's about 3/4 inch wide and a 14-18 inches long and solid steel. I keep it in the car next to my driver's seat as an emergency self defense tool.

# ¿ Mar 2, 2012 17:14

Adbot: ADBOT LOVES YOU

# ¿ Apr 20, 2024 01:20

Intraveinous: Oct 2, 2001; Legion of Rainy-Day Buddhists

evil_bunnY posted:

Cause of death: random seek to the back of the head.

Now that's a head crash!

*crickets*

error1 posted:

Storage arrays singing actually sound more like this

I got to hear that sound last week for similar reasons (replacing a UPS), though not quite on that scale, since my EVA is only 1 rack.

Intraveinous fucked around with this message at 21:34 on Mar 5, 2012

# ¿ Mar 5, 2012 21:27

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > Enterprise Storage Megathread: Why is my NAS a SAN?

«‹›2 »