Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
Mierdaan
Sep 14, 2004

Pillbug
Thanks for the thread, 1000101.

Anyone out there using Dell's MD3000i box? We're just getting to the point where we need something low-end, due to increasing interest in virtualization and an unwillingness on my part to replace an ancient Proliant ML350 file server with another traditional file server. We don't have anything terribly IOPS-intensive we'd be putting on it; probably just Exchange transaction logs SQL transaction logs + DB for 200-person company setup, so I don't think iSCSI performance issues are worth worrying about for us.

It's a 15 spindle box, so we're thinking about carving it up thusly:

1) RAID 10, 4x 300GB 15K RPM SAS drives: SQL DB
2) RAID 10, 4x 146GB 15K RPM SAS drives: SQL transaction logs
3) RAID 1, 2x 73GB 15K RPM SAS drives: Exchange transaction logs
4) RAID 5, 4x 450GB 15K RPM SAS drives: VMimages, light file server use
(with 1 spare drive for whatever RAID set we decide most needs it)

That'd take care of our immediate needs and give us some room to expand with additional MD1000 shelves (2 can be added to the MD3000i, though the IO doesn't scale at all). We're a small shop and have no experience with SANs, so I could definitely use some feedback on this idea.

Mierdaan fucked around with this message at 20:04 on Aug 29, 2008

Adbot
ADBOT LOVES YOU

Mierdaan
Sep 14, 2004

Pillbug

Mierdaan posted:

Thanks for the thread, 1000101.

Anyone out there using Dell's MD3000i box? We're just getting to the point where we need something low-end, due to increasing interest in virtualization and an unwillingness on my part to replace an ancient Proliant ML350 file server with another traditional file server. We don't have anything terribly IOPS-intensive we'd be putting on it; probably the most IOPS would be Exchange transaction logs, SQL transaction logs + DB for 200-person company setup, so I don't think iSCSI performance issues are worth worrying about for us.

It's a 15 spindle box, so we're thinking about carving it up thusly:

1) RAID 10, 4x 300GB 15K RPM SAS drives: SQL DB
2) RAID 10, 4x 146GB 15K RPM SAS drives: SQL transaction logs
3) RAID 1, 2x 73GB 15K RPM SAS drives: Exchange transaction logs
4) RAID 5, 4x 450GB 15K RPM SAS drives: VMimages, light file server use
(with 1 spare drive for whatever RAID set we decide most needs it)

That'd take care of our immediate needs and give us some room to expand with additional MD1000 shelves (2 can be added to the MD3000i, though the IO doesn't scale at all). We're a small shop and have no experience with SANs, so I could definitely use some feedback on this idea.

Floating this question again :)

Mierdaan
Sep 14, 2004

Pillbug

optikalus posted:

I've been looking at the MD3000i as well, mainly because it is the only iSCSI filer that Dell sells that doesn't use insanely expensive disks.

Have you looked at any vendors outside of Dell? The reseller we work with is pretty big on Dell, so I don't know how biased they are; they're claiming wonderous things about the 3000i.

"optikalus posted:

Those disks should yield a healthy 170 IOPS per disk (vs. ~130 for 10k) so that would be ideal for a database. If your database can live fine on 680 IOPS, its probably good enough.
Time to dig up some performance counters.

"optikalus posted:

That extra drive should be used as an universal hot-spare, though.
I wish; apparently it can only be a hot spare for one RAID set, not any RAID set. I was thinking as long as the spare was at least as large as the largest spindle in a RAID set, it could be universal, but I'm told that's not the case.

Mierdaan
Sep 14, 2004

Pillbug

Auslander posted:

Oh man, this thread is a welcome sight.

I'm just about to get a Clariion AX4 configured w/ 11 400GB 10k SAS drives and 2 heads to replace our crap-rear end Ubuntu NFS server that's currently providing backend storage to 4 ESX frontends.

Any setup tips on carving this thing up once it ships to us? How should i connect VM -> storage? Expose a LUN for each VM then use RDM in vmware? One giant VMFS volume stored to a RAID-10 on the AX4? A mix of both? Nothing? OH GOD THE CHOICES ARE KILLING ME :smith:

Haha, I'm right where you are. We're looking at the Dell MD3000i and the AX4 right now, and the jump from crappy DAS to a real (if low-end) SAN is just imposing.

Mierdaan
Sep 14, 2004

Pillbug

Misogynist posted:

And you ain't kidding, the performance on the MD3000 is really not spectacular

What do you mean by not spectacular? I mean, the most IOPS-intensive thing we're throwing onto it is Exchange 2007 install for 250-ish users pushing about 600 IOPS. Are we going to be unhappy? :(

Mierdaan
Sep 14, 2004

Pillbug

1000101 posted:

That EMC box is a fine piece of equipment. I have a customer who just installed one and is pretty happy with it.

Your concerns about NetApp are certainly valid.

What reasons can I use to convince management to go with the AX4-5 instead of the MD3000i? The Equallogic box is cheaper, and I fear that's all they're looking at right now :(

Tell me tales of woe, SHSC.

Mierdaan
Sep 14, 2004

Pillbug

Catch 22 posted:

Connectivity- The AX4s are optional and can be upgraded later to Fiber, EQL is iSCSI only.
Agreed, this is a potential bonus though I don't see us moving to FC anytime soon.

Catch 22 posted:

Controllers-The EMC works in a Active/Active mode, meaning they both do work and should one fail the other takes on the full load. This gives it better performance over the EQL Active/Passive controllers because you only get to use one controller (thus 3 ports even though it has 6 total)

Host ports- EMC has 4, 2 per controller and you can use all 4 when both controllers are working. EQL is 3
Where are you getting that? I think the controllers can be active/active on the Equallogic as well, but may not be that way by default. Also each controller on the Equallogic has 2 gigE ports for iSCSI and 1 gigE for management, which is exactly the same as the AX4 as far as I know...

Catch 22 posted:

Max Host- EMC is 64 and the EQL is 512. *gasp* EQL wins? Not exactly, if you want to try and pound out 512 hosts on that thing, be my guest. EMC scaled the hosts correctly for environments the AX4 is applicable for, don't let the Dell rep talk it up because it has a higher number.
Yeah, either of these numbers is overkill for us, so it doesn't matter much now.

Catch 22 posted:

Drive options- EMC lets you mix and match in the same array enclosure and lets you span LUNs over multiple DAEs, EQL is a all or nothing in one enclosure and it can only be one mass array. NO LUNS. This means your applications are all trying to get a bit of the pie, unlike a dedicated performance LUN. Ask the EQL guy about LUNs, he will avoid it or try to say the "array" in the box is like a LUN.
I know the spanning between enclosures issue is true (yes on EMC, no on EQL) but I'm pretty sure you can mix drive types in an enclosure with either. I'll check into the monolithic array, I hadn't heard that and our reseller surely didn't mention it.

Catch 22 posted:

Capacity-EMC 584 GB to 60 TB, You can scale it one drive at a time. EQL will make you drop a cash load to buy another fully populated DAE.
EQL maybe, but how does this work buying from Dell? I didn't seem to get the impression I couldn't just pop drives in as needed to expand.

Catch 22 posted:

Cache-EMC, 1 GB per Storage Processor (2GB), EQL-1 GB per controller
So, identical assuming a dual controller EQL box?

Catch 22 posted:

Tell me abit more about your environment and I can help more.

I'm not trying to pick apart your post; I really want there to be good reasons to go with the AX4 as I keep hearing ghost stories about horrible performance etc with the MD3000.

Our environment is pretty simple for right now; we'd be running, at most, an ERP and Exchange 2007 for a 200-person company, light file sharing, and potentially an PE2950 ESXi box hosting a few light-work Server2003/2008 VMs.

Mierdaan
Sep 14, 2004

Pillbug
Thanks Catch 22 and Vanilla, I'll do some more reading :)

For reference, the EQL box we were quoted was
- Dual controller
- 4x 300gb 15k SAS
- 4x 146gb 15k SAS
- 2x 73gb 15k SAS
- 5x 450gb 15k SAS

for $14914. Since that comes to 15 spindles total, we'd need a DPE+DAE AX4 setup. An identical combination of drives (+4 for the vault software) would leave us 5 empty drive slots, but the first quote we got (different resellers here) was $23432. So the price difference is definitely non-trivial :(

Mierdaan
Sep 14, 2004

Pillbug
I was originally talking about the Dell MD3000i; I don't know what EQL product number that is (it is EQL hardware, right? Not Dell-manufactured?), I only know it under the Dell line. Their configurator tool on the Dell site lets me pick any of these drives:

73GB 15K RPM Serial-Attach SCSI 3Gbps 3.5-in HotPlug Hard Drive [$209]
146GB 15K RPM Serial-Attach SCSI 3Gbps 3.5-in HotPlug Hard Drive [$244]
300GB 10K RPM Serial-Attach SCSI 3Gbps 3.5-in HotPlug Hard Drive [$454]
300GB 15K RPM Serial-Attach SCSI 3Gbps 3.5-in HotPlug Hard Drive [$489]
400GB 10K RPM Serial-Attach SCSI 3Gbps 3.5-in HotPlug Hard Drive [$524]
1TB 7.2K RPM Universal SATA 3Gbps 3.5-in HotPlug Hard Drive [$482]
450GB 15K RPM Serial-Attach SCSI 3Gbps 3.5-in HotPlug Hard Drive [$559]
500GB 7.2K RPM Universal SATA 3Gbps 3.5-in HotPlug Hard Drive [$244]
750GB 7.2K RPM Universal SATA 3Gbps 3.5-in HotPlug Hard Drive [$328]
1TB 7.2K RPM Near-Line SAS 3Gbps 3.5-in HotPlug Hard Drive [$489]
500GB 7.2K RPM Near-Line SAS 3Gbps 3.5-in HotPlug Hard Drive [$307]
750GB 7.2K RPM Near-Line SAS 3Gbps 3.5-in HotPlug Hard Drive [$391]
1TB 5.4K RPM Energy Efficient SATA 3Gbps 3.5-in HotPlug Hard Drive [$482]

Mierdaan
Sep 14, 2004

Pillbug

rage-saq posted:

No, the MD3000i is a box from LSI. Equallogic are totally different products, but I wouldn't buy either. They are both rather immature products in a large number of ways (web management, strange limitations and non-competitive price/performance).

Thanks rage-saq. What would you say is the best bang for your buck in the $15-$25k range? As previously mentioned we don't currently need anything with crazy good performance, but we don't want to handicap ourselves later. Only need about 3-4TB usable space, with the ability to scale up later.

Mierdaan
Sep 14, 2004

Pillbug

rage-saq posted:

What kind of features? How many hosts? What kind of disk configuration?
HP has a really great lineup with the MSA2000. There is an FC model, iSCSI model and SAS (still a SAN, think FC without FC) model and they are all pretty awesome from a management and price/performance standpoint.

Snapshots are really all we'd be looking for right now. Don't need replication as I'm still working on getting the first device in the door :(

20 or so hosts would be reasonable for the forseeable future.

Disks, we could realistically go for a mix of large SATA and smaller faster SAS drives, but our first stab at carving up 15 spindles in the MD3000i was:
1) RAID 10, 4x 300GB 15K RPM SAS drives: SQL DB
2) RAID 10, 4x 146GB 15K RPM SAS drives: SQL transaction logs
3) RAID 1, 2x 73GB 15K RPM SAS drives: Exchange transaction logs
4) RAID 5, 4x 450GB 15K RPM SAS drives: VMimages, light file server use
(with 1 spare drive for whatever RAID set we decide most needs it)

Mierdaan
Sep 14, 2004

Pillbug

Catch 22 posted:

EMC 2U SAS/SATA 12 DRIVE DAE
EMC SECOND SPS OPTIONAL AX4-5SPS
EMC FACTORY CONFIG SERVICES AX4-5 DP
EMC AX4-5 EXPANSION PACK AX4-5EXPAN
EMC 4 - 146 GB 15K SAS W VAULT SOFTW
EMC 146GB 15K SAS DRIVE AX-SS15-146
EMC 750GB 7200RPM SATA II DRIVE

EMC NAVI MANAGER FOR AX4-5
EMC SNAPVIEW FOR THE AX4-5
EMC 2U DUAL SP DPE FC FRONT END W 1U
EMC MIRRORVIEW ASYNC FOR THE AX4-5
Sub-Total $26K

This is one of 3 vendors pricing, I am supposed to get better pricing from Dell/EMC but I am waiting on the quote...4 days now. Slowasses.

What's the quantity on those lines? That's close to what we were quoted, would love to compare prices ;)

Mierdaan
Sep 14, 2004

Pillbug
How does the Snapview SKU work for the AX4? We didn't have it on our quote, but were assured we had the capability to do snapshots. Does Snapview get you some additional functionality we wouldn't have, or is our reseller just including but not as a line item?

Mierdaan
Sep 14, 2004

Pillbug

rage-saq posted:

Additionally, the MSA2000 series is a block-level virtualized storage system, so you don't necessarily have to carve up disk groups based off I/O patterns like you do with traditional disk-stripe arrays.

Can you explain this a little more? I've always felt like I was missing something by carving up disk groups like I did before, so I'm glad to know I am. I just don't understand quite what :)

Mierdaan
Sep 14, 2004

Pillbug

Mierdaan posted:

rage-saq posted:

Additionally, the MSA2000 series is a block-level virtualized storage system, so you don't necessarily have to carve up disk groups based off I/O patterns like you do with traditional disk-stripe arrays.

Can you explain this a little more? I've always felt like I was missing something by carving up disk groups like I did before, so I'm glad to know I am. I just don't understand quite what :)

Reposting this question on a new page :) Is rage-saq just talking about the ability to automatically move more frequently-used data to faster disks? That makes sense if you have slower spindles in your device, but if you cram the whole thing full of 15k RPM SAS drives I'm not quite sure what this accomplishes. Again, totally missing something :(

Mierdaan
Sep 14, 2004

Pillbug
rage-saq, Vanilla, thanks a lot for the explanations. I think I see the light now ;)

Mierdaan
Sep 14, 2004

Pillbug

H110Hawk posted:

To diverge from the thrilling debate about some Dell bottom of the barrel disk enclosure...

Sorry we don't all have 23 thumpers to brag about. Seriously dude, you're a loving dick sometimes.

Mierdaan
Sep 14, 2004

Pillbug
Didn't see this posted yet, so, HP acquires LeftHand for $360M.

Mierdaan
Sep 14, 2004

Pillbug
With the FAS2020, do we need to pay for the CIFS and NFS software licenses if we only intend to use it as an iSCSI target initially? Understandably we'd lose the NAS functionality, but we can add it back in later.

We have a PC Mall sales engineer claiming we need at least CIFS to allow Windows-based hosts to use the FAS as an iSCSI target, which just seems odd to me.

edit: nevermind, I asked him the question again via email and he recanted. Huzzah!

Mierdaan fucked around with this message at 20:01 on Nov 19, 2008

Mierdaan
Sep 14, 2004

Pillbug
Is there a good online resource for finding out what Netapp's different software licenses actually allow you to do? Their online doc on SnapDrive is horrible, horrible marketing garbage.

Mierdaan
Sep 14, 2004

Pillbug

Cultural Imperial posted:

Unfortunately the only good source is the NOW support site. What would you like to know?

I'm not even sure, really. I wanted to read through some different documentation on the various Snap* and Flex* products, just for my own edification, but wasn't finding much.

Mierdaan
Sep 14, 2004

Pillbug
Well, we've been hosed.

We went through a lengthy selection process between HP (MSA 2012i), Netapp (FAS2020) and EMC (AX4). We decided on Netapp, and were planning on buying it through PC Mall. Midway through the process, though, we started getting frightened by how little our PC Mall rep actually knew the product line (e.g. told us the FAS2020 had 20 drive bays instead of 12, told us it didn't do deduplication when it does, etc), so we took our quote from PC Mall to our trusted local VAR, played matchmaker with him and Netapp (they previously had a bad relationship) so that we could get the same pricing we were getting through PC Mall, but buy it from someone who wasn't an idiot.

So all is well and good, our local VAR gets the same pricing and we buy the unit. It arrived yesterday, we unboxed it and found out... it only has one controller in it. Despite a dual-controller configuration being talked about in all of our conversations with PC Mall, their quotes had always listed the following part numbers

code:
FAS2020-R5
DOC-2020-C
FAS2020-NS-12-X287A-R5-C
FAS2020-NS-BASE-R5-C
X5518A-R6-C
X800E-R6-C
SW-T1-ISCSI-C
SW-T1-SRESTORE-S-C
CS-A-INST-4R
SW-SDRIVE
SW-SDR-WIN
SW-SSP-SDR-WINDOWS
Apparently the lack of a -C on the first line means single-controller. PC Mall screwed this up, their Netapp distributor (Arrow, who we talked to for a few hours on the phone and clearly knew we wanted a dual-controller unit) screwed this up, and the error transferred over to our local VAR uncaught when he recreated the quote and matched the price.

So, the question is, what do we do now? We went with the Netapp for good reasons, and don't want to return it, but it isn't the device we thought it would be. Netapp has said they'll sell our VAR a second controller for $7400, and we could get it for that price, but I'm not even sure if that's a good deal. Anyone with access to Netapp pricing know if we're getting hosed there again, or if they're actually trying to bend over backwards for us like they assure us they are?

Mierdaan
Sep 14, 2004

Pillbug
Yeah, I know. Thankfully we're not the kind of shop that loses billions of dollars for every hour of downtime or anything, so the idea of running with a single-controller unit doesn't keep me up at night. We probably wouldn't even see any performance boost from a second controller unless we add more shelves, correct?

Mierdaan
Sep 14, 2004

Pillbug
It was a strange situation. Our VAR is normally very good, and when we initially approached them they stated outright that if we wanted Netapp, we should probably go somewhere else, as he felt like his pricing through his existing supplier wasn't at all competitive.

In the middle of the process, when we wanted to ditch PC Mall, we got contacted by a local Netapp rep. We put that guy in contact directly with our VAR so they could get better pricing to our VAR, and that was most of our VAR's work on this. We basically gave him our (already hosed-up) PC Mall quote, and said "get this configuration with this price and we'll buy it through you." So that's what he did, the error just propagated unnoticed.

I know our VAR isn't putting a markup on the controller at the price he gave us; that's what Netapp is selling it to him for.

Mierdaan fucked around with this message at 19:42 on Jan 29, 2009

Mierdaan
Sep 14, 2004

Pillbug

1000101 posted:

It supports clustering. We had to upgrade one of our client's 2020's to support it but it does work. The only fault with the 2020 is that its basically a dead end platform. The 2050 is generally a much better fit for people and has a lot more expandability.

Worst case you can turn a 2020 head into a dumb shelf of disks for a 2050, I'm pretty sure. We were moving to this from a few ProLiant ML350s with four 72GB drives in RAID5, on a shoestring budget, so... it may not seem expandable to you, but it's like the Promised Land for us.

Mierdaan
Sep 14, 2004

Pillbug
Performance question.

I've got an ESXi server with a VMFS LUN on our Netapp FAS2020. I need to create a file server VM which needs to serve up two shares, 500G each. I can't cram all of this inside the VMFS LUN because the A-SIS engine on the FAS2020 won't run against a volume larger than 500G, so I'm stuck separating at least the shares out in some way. Will I see any performance benefit by creating these as VMDKs in additional VMFS LUNs, or by just hooking the Server 2008 VM directly (ISCSi) to the LUNs and letting it format them with NTFS? What's best practice here?

Thanks storage goons.

Mierdaan
Sep 14, 2004

Pillbug
Thanks. I found this study on VMware's website that seems to indicate it doesn't make too much of a difference, and honestly this isn't a high IO file server. I'm probably worrying too much.

Mierdaan
Sep 14, 2004

Pillbug

rage-saq posted:

I'd advise against getting any kind of advice on important system design from what is essentially the best buy of product fulfillment centers. Call in a consultant and engage with them in a discovery and project assessment for virtualizing your infrastructure.

Check out my post history in this thread if you want to see why you shouldn't do this.

Mierdaan
Sep 14, 2004

Pillbug
Netapp question:

We have a FAS2020 connected to an entirely segregated iSCSI network, and then through the BMC for ssh management. Is there any way to force Autosupport emails to go through the BMC's connection, since the other network it's connected to is completely isolated?

Mierdaan
Sep 14, 2004

Pillbug

Number19 posted:

My VAR was talking about putting my ESX datastores in NFS instead of iSCSI. I believe he mentioned something about it being easier to resize NFS volumes. We didn't go into a lot of details about it as this was just a get to know you" kind of deal.

Another good reason to do this is so that ESX can see the savings from deduplication. We have our ESXi stores done over iSCSI and while the filer happily dedupes, ESXi isn't aware that its stores are a good 60% less full than it thinks they are.

edit: but that's what we get for not paying for NFS/CIFS licenses. Stupid shoestring budgets.

Mierdaan
Sep 14, 2004

Pillbug

KoeK posted:

If you do not need any performance at all you could look at the Netgear RNR4410-100EUS, should be between $1700 - $2500. It is rack mountable and has 4 1Tb disks. The drawbacks are the absolute terrible speed, the annoying web interface and the crappy build quality.

I have one of these I think, from when they were made by Infrant and mine only has 4x 500GB drives. It is absolutely terrible and I wouldn't recommend it to anyone.

Mierdaan
Sep 14, 2004

Pillbug

Misogynist posted:

Their support might suck but don't be a chauvinist douche

Post/username combo, right here.

content: For a FAS2020, is there anything I'm doing wrong that forces me to spend half my time rebooting these damned BMCs? Pretty frequently, when I try to ssh to them I get 'server unexpectedly closed connection' and NetApp's answer was to just reboot the BMC. Works fine, but it's happening frequently enough that I'm this close to using the filerview command line, and ugh.

Mierdaan
Sep 14, 2004

Pillbug
I've been getting Autosupport alerts all night about failing power supplies and spare disks for a FAS3170 at a credit union in California. I've never heard of this credit union; I have no clue why I'm listed as a contact for it.

I hope their actual storage admin knows :ohdear:

Mierdaan
Sep 14, 2004

Pillbug

three posted:

Are you bob@bob.com ?

Edit that out please, I don't want people knowing my email address...

Seriously, no I have a pretty specific email address that you wouldn't get by hammering on a keyboard. I'm also already in Netapp's system for my own filers, so clearly some bits got lost somewhere.

edit: Ed, from the credit union's mailroom just called me to let me know I have a package waiting. I told him to give it to the IT department.

Mierdaan
Sep 14, 2004

Pillbug
You might get an answer here, but you'd probably have better luck in the Home Storage megathread.

Mierdaan
Sep 14, 2004

Pillbug

Timdogg posted:

Also, Dell is trying to push their "iSCSI Optimized" switches, but I have limited experience with their switches and am hesitant to jump in now. Anyone recommend them?

We got one of these pieces of poo poo - terrible decision. It's sitting unused on a shelf now, I'm trying to convince my boss to let me run it over with my car since we replaced it with a 3560.

Mierdaan
Sep 14, 2004

Pillbug

Dreadite posted:

We're looking at the possibility of getting a SAN for our office of ~25 users. We wouldn't need more than 2TB of space, or anything particularly fast, but we'd like one for all the cool features that come with a SAN.

The problem seems to be price, as we're only looking to spend 10-12k. Is getting something to meet our modest needs under 12k completely unrealistic? What brands should we be looking at? I've gotten a quote for 15k for a NetApp 2020, but we'd really like to spend less than that so it doesn't cut into our budget for new servers.

I think Misogynist's question from above applies to you too; we picked up a FAS2020 as well when we were getting into low-end enterprise storage, but I think the VNX line is a better bang for your buck now.

Misogynist posted:

Have you looked at EMC's lower-end VNX line?

Mierdaan
Sep 14, 2004

Pillbug
Seeing the Unisphere video made me realize how bad I have it; are there any good management tools for NetApp? We're still on 7.3.3 unfortunately, since our 2020 won't run 8.x code.

Mierdaan
Sep 14, 2004

Pillbug

madsushi posted:

I like NetApp's System Manager, their 2.0 product (in open beta) is actually pretty sharp.

Oh; I tried that in 1.1 and it was terrible, I'll give it another shot!

Adbot
ADBOT LOVES YOU

Mierdaan
Sep 14, 2004

Pillbug
My boss had the misfortune of trying to buy some Compellent kit right as the acquisition was wrapping up - it was a pretty high-priority purchase for him, and they couldn't get anyone from Compellent on the phone because they'd all hosed off to the Bahamas for two weeks or something.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply