|
luminalflux posted:This is also true of HP LeftHand systems. That is, the VAR claims that that will happen, but the HP RSSWM stuff is kinda finicky. Back when our Lefthand stuff was still under warranty (sigh) the first we would know of an issue with the array was getting a notification from Exchange telling us the information store had dismounted. We'd see weird errors or an unresponsive CMC and open a support ticket. They would then ask us to run the script that would send then the last few days of logs. That poo poo was supposed to run daily but would fail all the time. Sometimes months would go before it was apparent it had failed. They didn't ever seem surprised or give a poo poo that their systems had stopped reporting. I'm sure it's different now but there's pretty much zero chance of us spending on anything Lefthand-related again.
|
#
?
Dec 19, 2011 02:24
|
|
|
|
| # ? Apr 24, 2024 11:19 |
|
|
luminalflux posted:Sounds like what happened to Tieto in Stockholm a few weeks ago, causing outages and unrecoverable data for a load of big customers. Yeah, super interested in hearing more about this if possible! It's a bit beyond the pail that a single SSD failure can down you're enterprise storage.
|
|
#
?
Dec 19, 2011 03:01
|
|
|
The email reporting is a lot better in SAN/iQ 9 than 8. Before each and every alarm had to be configured to email, now you just select "mail critical alarms to blah@foobar.com" and Bob's your uncle. So far I'm pretty happy with my P4300 system, but I don't do any snapshots or anything, it's just 6 iSCSI volumes available for VMware. Only really had troubles with LACP not distributing network traffic evenly over both ports, which got better by using ALB. edit: marketingman posted:Yeah, super interested in hearing more about this if possible! It's a bit beyond the pail that a single SSD failure can down you're enterprise storage. Tried to find english accounts of that, but everything is in Swedish unfortunately and I don't really know anyone involved so I can't get it from the horses mouth. Try running this article through Google Translate and it might give you something useful (plus links to other articles about the whole incident). It was a combination of SSD in FAST cache failing and some upgrade procedures not working, along with the recovery not going as planned. luminalflux fucked around with this message at 03:10 on Dec 19, 2011 |
|
#
?
Dec 19, 2011 03:03
|
|
|
luminalflux posted:Sounds like what happened to Tieto in Stockholm a few weeks ago, causing outages and unrecoverable data for a load of big customers. luminalflux posted:Tried to find english accounts of that, but everything is in Swedish unfortunately and I don't really know anyone involved so I can't get it from the horses mouth. Try running this article through Google Translate and it might give you something useful (plus links to other articles about the whole incident). It was a combination of SSD in FAST cache failing and some upgrade procedures not working, along with the recovery not going as planned. You would not imagine the amount of good will EMC's lost over this in Sweden. Spamtron7000 posted:I'll get a full writeup from EMC this week. 
evil_bunnY fucked around with this message at 03:26 on Dec 19, 2011 |
|
#
?
Dec 19, 2011 03:21
|
|
|
marketingman posted:Yeah, super interested in hearing more about this if possible! It's a bit beyond the pail that a single SSD failure can down you're enterprise storage. I'll get a full writeup from EMC this week. Luckily we didn't lose any data but there's no way a FAST cache failure should cause such mayhem. What's worse is that I "escalated" the issue to my EMC account rep this morning at 8am to try and get the replacement part sooner than 10:30am tomorrow. It's 10 hours later and I still haven't even gotten a response from him. Their support has gone to poo poo.
|
|
#
?
Dec 19, 2011 03:22
|
|
|
I say keep reporting the EMC issues. I was holding my tongue, too, worried that maybe it was just my implementation. If it's gonna take a customer rebellion and exodus for them to lose their hubris and get back to making the kickass storage they know how to make, let it begin in this thread. Speaking of broken EMC gear, anyone have any background or tips on rehydrating a neglected Centera? I need that thing out of my datacenter and I help the server team a lot if I can unlock that data.
|
|
#
?
Dec 19, 2011 04:02
|
|
|
evil_bunnY posted:Are you going to claim back the support expenses? That'd get them going It's funny, I've had PAMII cards fail in filers to no effect beyond slower performance, and I've been able to non-disruptively replace them. There's more I'd say but NDAs and all... Spamtron7000 posted:I'll get a full writeup from EMC this week. Luckily we didn't lose any data but there's no way a FAST cache failure should cause such mayhem. What's worse is that I "escalated" the issue to my EMC account rep this morning at 8am to try and get the replacement part sooner than 10:30am tomorrow. It's 10 hours later and I still haven't even gotten a response from him. Their support has gone to poo poo. I've had NetApp drag their heels before on similarly crucial parts (not similar like HDDs, more like motherboards), I think it's just something we have to live with when it comes to support. Having said that, it's pretty rare to not have the replacement part within a few hours. Muslim Wookie fucked around with this message at 04:59 on Dec 19, 2011 |
|
#
?
Dec 19, 2011 04:55
|
|
|
luminalflux posted:This is also true of HP LeftHand systems. Same true for Dell EqualLogic: it phones home automatically and you'll get the replacement part within hours (provided you have decent support, not just the free NBD) as well as a cal lto arrange service technician (if you need it.) That being said mine is small IT team and this "phone home" is off but we pay attention and catch everything - using other monitoring tools (it might also helps that I arranged all the rack cabinets behind a glass wall facing the main entrance corridor so when people come and go they see al the lights...  )
|
|
#
?
Dec 19, 2011 05:12
|
|
|
bort posted:I say keep reporting the EMC issues. I was holding my tongue, too, worried that maybe it was just my implementation. If it's gonna take a customer rebellion and exodus for them to lose their hubris and get back to making the kickass storage they know how to make, let it begin in this thread. Hi Centera buddy. Do you have the pea file? Pea is the key.
|
|
#
?
Dec 19, 2011 05:20
|
|
|
Spamtron7000 posted:I'll get a full writeup from EMC this week.
|
|
#
?
Dec 19, 2011 06:26
|
|
|
Spamtron7000 posted:I'll get a full writeup from EMC this week. Luckily we didn't lose any data but there's no way a FAST cache failure should cause such mayhem. What's worse is that I "escalated" the issue to my EMC account rep this morning at 8am to try and get the replacement part sooner than 10:30am tomorrow. It's 10 hours later and I still haven't even gotten a response from him. Their support has gone to poo poo. The write up is called an RCA (Root Cause Analysis). If I were you i'd get my CTO/CIO to call as high up as possible in EMC to report your data-down issue, the faulty part and the lack response from the account rep. Part of his job is to be available when poo poo like this happens. If you hit a bug it's acceptable for time to be taken to fix, but it's not acceptable to simply take that length of time to get a spare part. Every city has a parts dump and it can't get to the stage where poo poo isnt there.
|
|
#
?
Dec 19, 2011 07:42
|
|
|
bort posted:
Find the application that was writing to the Centera and rehydrate using that. It's going to take a while and you'll need more space on the production volume...
|
|
#
?
Dec 19, 2011 08:04
|
|
|
bort posted:I say keep reporting the EMC issues. I was holding my tongue, too, worried that maybe it was just my implementation. If it's gonna take a customer rebellion and exodus for them to lose their hubris and get back to making the kickass storage they know how to make, let it begin in this thread. Also, just while i'm thinking about it. After every support call or once a quarter a lot of accounts are sent a Customer Satisfaction Survey (CSAT) and Support Survet (STS I think?). These are administered by a third party so that all information gets reported fairly and there is no massaging. These are religiously looked at and obsessed over. If not satisfied make sure to vent using these forms, these are how EMC know's what's going on and they're taken very seriously. For example, Customer Services Managers are paid based on the STS results. A number of poor results and they don't get their objectives bonuses (but do remember the STS is around the service you received upon an issue, not the actual issue). CSAT: Dear EMC your poo poo isn't as good as it used to be. Look what occurred due to a single bad SSD. What is it 1992? STS: Dear EMC, when a storage processor goes down my day gets real peachy when a support engineer reboots the good SP instead of the bad one. Please do it again. I'd love a reason to kill you with my bare hands.
|
|
#
?
Dec 19, 2011 09:42
|
|
|
I never received any of those. It took months to get EMC support to stop calling the guy who'd left the position and to start calling me. Now that I have third-party support, they still call me even though my contract has expired. Vanilla, you have to be on some kind of different support tier than the rest of us. The tier I was always on was crowded Asian call centers and idiots who fail the active instead of the standby. The onsite guys who'd work on the Celerras wouldn't know any Linux at all (I saw a guy screw up an ls command line and have to look at Powerlink). I don't think EMC considers support improvement a priority even a little bit, or they would be improving and that's not happening. It's unraveling further.
|
|
#
?
Dec 19, 2011 13:02
|
|
|
luminalflux posted:The email reporting is a lot better in SAN/iQ 9 than 8. Ours are on 7. 
|
|
#
?
Dec 19, 2011 15:49
|
|
|
Misogynist posted:Just walked into an environment where the management is considering a substantial investment in an HDS stack. I've never worked with or really heard much from them, besides the BlueArc stuff we've had that long predates the Hitachi acquisition. Can anyone give me their impressions on the technology? My current employer "upgraded" from an AMS 500 to an AMS 2300 shortly before I started. While the 500 management is OK, the java UI on the 2300 is so horrendously godawful that I want to murder the team responsible. We're talking 60+ second wait times just to hit the "edit host group" button and have the info appear. Feature-wise, my impression coming from a mid range HP shop was it seemed horribly outdated compared to EVAs, but maybe we didn't have the right licenses. I tried using them as D2D backup targets after retirement and just could not get the performance where it should be, even on big sequential writes. Spamtron7000 posted:Just to spite them I'm going to replace the DataDomains with Quantum DXi's. Got a pair of 6702s about 2 months ago and they are loving awesome. I like backup targets that can saturate 10gb connections.
|
|
#
?
Dec 19, 2011 15:52
|
|
|
Cavepimp posted:Well I just ordered an EMC VNXe 3300 right before everyone started bitching about EMC in here. I couldn't go Netapp though (previous bad experience here, no way it would fly), and EMC beat HP for our circumstances/budget. Return it and get an Equallogic SAN. If you're not a storage guy and you are going to be supporting the SAN, stay the hell away from EMC.
|
|
#
?
Dec 19, 2011 20:34
|
|
|
bort posted:Vanilla, you have to be on some kind of different support tier than the rest of us. No, I'm not on any tier. I work for EMC in Pre-Sales, so I spend all day working with the tech and customers. A lot of the stuff in the thread is shocking and i'm always the first to say 'call high' because what's being seen is not normal or acceptable. On the support side every (EVERY) account has a Customer Support Manager whom you can escalate issues and bad support to. I very much want CSAT and STS forms filled out. Better support means a better life for me too as we get pulled into the more serious support issues (not to fix, just to support communication, speak to the custoemr management to let them know what is being done and not to beat up the sys admin who is busy trying to work with support, etc). Although, I have to admit, going back a few years there's also been a few shocking support issues in this thread that when I track them down (out of interest) turn out to be major PEBCAK errors......but whatever, usual drill - it's just easier to blame the vendor.
|
|
#
?
Dec 19, 2011 22:38
|
|
|
Vanilla posted:Although, I have to admit, going back a few years there's also been a few shocking support issues in this thread that when I track them down (out of interest) turn out to be major PEBCAK errors......but whatever, usual drill - it's just easier to blame the vendor. The difference is that on the medium or small business side, these "PEBCAK" errors are due to the lack of ease of use. I don't get to just yell at my users when they don't know something above their experience. It is my job to make their jobs easier, and to help them when I cannot make it easy enough for them to do. When EMC tries to come into the SMB space and sell units to compete with the likes of Equallogic, and even has impressive demos of "You can set up this SAN in 15 minutes! Just enter the number of Exchange users and away you go!", it's complete bullshit. The VNX may be easier to use than the only Clariions and Celerras, but it is not by any stretch of the imagination easy to use. You have to hold your tongue a certain way, dive in to the command line, then click this window 3 times while holding shift and you should be good to go. I'll give you an example of a problem I am having right now. I'm sure it's a simple one, but after the awesome support I've had I really don't feel like calling them up again. Trying to set up "Connect Home", which for those of you that don't know is EMCs service where the SAN sends off alerts to EMC and then EMC comes out and fixes the problem for you. I have the SMTP address set up, I have our site ID in, I have Encryption disabled because that should make it easier. I disabled FTP, Modem, Monitoring Station HTTPS - basically just the bare bones. I go to do a test, I select Test Type: Primary Email(email_1), which is my email config, then I hit test. The error message I get is below. Severity: Error Brief Description: ------------------------------------------------------------------- ConnectEMC: Product Version: 2.6.0-30 Build Date: Thu May 21 13:23:21 EDT 2009 Build Platform: RedHat5.3 Copyright (C) EMC Corporation 2003-2009, all rights reserved. ------------------------------------------------------------------- Reading configuration file: ConnectEMC.ini. Error, But No Error Message Error 14504231222: Test failed for the Primary Email. Full Description: No additional information is available. Recommended Action: No recommended action is available. For more information and assistance: 1. Log into http://powerlink.emc.com and go to Support > Search Support. 2. Use the message ID or text from the error message's brief description to search. Message ID: 13690601492 ******* And of course when I go to Powerlink to look up either the Error ID or the Message ID there is nothing. And this is by far from the first error message like this that I have run into.
|
|
#
?
Dec 19, 2011 23:07
|
|
|
I can't believe you don't know how to fix Error 1450423122. Nub.
|
|
#
?
Dec 19, 2011 23:13
|
|
|
"Slot 3 reason code of 0 is stale" meant I had a bad fiber jumper on my Celerra. I'm a PEBCAK.
|
|
#
?
Dec 20, 2011 02:15
|
|
|
bort posted:"Slot 3 reason code of 0 is stale" meant I had a bad fiber jumper on my Celerra. I'm a PEBCAK.
|
|
#
?
Dec 20, 2011 13:39
|
|
|
I mean, I am not trying to attack anyone. Vanilla has been extremely helpful to me in this thread and through PMs. EMC has some pretty great technologies, and I am sure if you set them up the right way they work very well. But I cannot in good conscious recommend them to someone without a dedicated storage admin, or if it is their first dive into centralized storage / virtualization. For an IT guy at an SMB shop, there is so much else to worry about.
|
|
#
?
Dec 20, 2011 15:05
|
|
|
Internet Explorer posted:I mean, I am not trying to attack anyone. Vanilla has been extremely helpful to me in this thread and through PMs. EMC has some pretty great technologies, and I am sure if you set them up the right way they work very well. But I cannot in good conscious recommend them to someone without a dedicated storage admin, or if it is their first dive into centralized storage / virtualization. For an IT guy at an SMB shop, there is so much else to worry about.
|
|
#
?
Dec 20, 2011 15:32
|
|
|
Misogynist posted:It certainly sounds like you're doing a great job of worrying about it yourself instead of having the EMC support contract you're paying for do that worrying. Like I said, I support a medium sized company and I handle pretty much everything, from SAN implementation to printer jams. The times I have contacted support took me about 4-5 hours of jumping from tech to tech and then piecing together what they all had to say, because none of them could figure it out. One of my coworkers has been out for 2 weeks and with the holidays, things are a bit busier and I don't have 4-5 hours to play with. Honestly, it sounds like you have no idea what you are talking about. But don't let that stop you from hopping in for a quick quip.
|
|
#
?
Dec 20, 2011 15:58
|
|
|
Well it's too late for me to change my mind now anyway since there was very much an "end of the year tax considerations" component to getting the purchase signed off on. I'm not under any illusions that it's going to be as simple as the salespeople wanted me to think, but your experiences with the VNX line are probably different than what I'll run into with the VNXe. They are quite different in a lot of ways.
|
|
#
?
Dec 20, 2011 16:07
|
|
|
That is true. And if you are only doing NAS or only doing iSCSI then it should simplify things quite a bit.
|
|
#
?
Dec 20, 2011 17:06
|
|
|
EMC is fine. They just don't seem to have that "jump through hoops" mentality that they used to have. They also don't have the same level of competitive advantage over other vendors that allowed them to charge a premium in the past. If you've already bought the VNXe, don't worry - it's not going to turn into a pumpkin. It's a great little platform to support your VMs.
|
|
#
?
Dec 20, 2011 18:23
|
|
|
Cavepimp posted:Well it's too late for me to change my mind now anyway since there was very much an "end of the year tax considerations" component to getting the purchase signed off on. It's a nice box by all accounts. It does seem to have some quirks that I've read of, but I think some of that comes down to if you've never used a SAN vs. if you have so have certain expectations. Things I can think of off the top of my head that I've read, no idea how valid but.. Management can only be done from the same subnet as the storage controllers 2tb limit on block volumes Dedupe only works on CIFS/NFS and not on block storage RAID is preconfigured depending on what "disk pack" you add For most SMB's those likely aren't issues.
|
|
#
?
Dec 20, 2011 18:31
|
|
|
Internet Explorer posted:Like I said, I support a medium sized company and I handle pretty much everything, from SAN implementation to printer jams. The times I have contacted support took me about 4-5 hours of jumping from tech to tech and then piecing together what they all had to say, because none of them could figure it out. One of my coworkers has been out for 2 weeks and with the holidays, things are a bit busier and I don't have 4-5 hours to play with. That's clearly nonsense, I agree. You paid for your support, it shouldn't take more than one support person to handle your case. When we have a problem with our EqualLogic box and I'm out, my sysadmin-in-training (no clue about storage systems beyond what I taught him) simply calls Dell Enterprise Support, ask for an EQL tech and he gets someone on the line who can give him proper support incl. hands-on via Webex. If the tech needs an analyst then he sends our diag output to one and he gets back to us in an hour or so but if it's an emergency he immediately gets someone on the line straight from New Hampshire and they start working on it via Webex. If a part is needed it's usually here in less than two hours, followed by a tech to install it in less than an hour (if I'm here we cancel this), all arranged by someone calling us back all the time (email updates are always coming.) The tech who opened the case (took the first call) remains engaged all the way to the end, he's the single point of contact, he follows through with all updates and requests feedback until we say it's OK to close the case. I don't remember the full price but I don't think this "3-y 24/7 4-hr on-site mission critical" (or whatever it's called) package was more than a grand or two... ...and yes, setting up the array from zero to connecting to the first LUN via iSCSI took me literally 40-45 minutes back when I received it (yes, I did wire up all the switches, hosts etc beforehand.) I haven't tested anything from EMC since our long-gone Symmetrix monsters but as far as I heard it's still not the case with current EMC units, right? I realize that ours is a smaller setup (PS6510 + a two-node 2008 R2 96-core cluster w/ 15-20 VMs and two node Storage Server 2008 cluster for file servers, all running on 10GbE) but a bigger, more complex setup also means more support costs so resources for support should scale up in a linear fashion. quote:Honestly, it sounds like you have no idea what you are talking about. But don't let that stop you from hopping in for a quick quip. Aye, it's quite normal as far as I can tell...
|
|
#
?
Dec 20, 2011 19:29
|
|
|
Internet Explorer posted:Return it and get an Equallogic SAN. If you're not a storage guy and you are going to be supporting the SAN, stay the hell away from EMC. THIS. Downside is that if he wants NAS features he still needs a server on front of the EQL SAN (unless he buys an EQL FS7500 NAS box as well.)
|
|
#
?
Dec 20, 2011 19:34
|
|
|
I was going to avoid posting this, because I feel like I am just making GBS threads on EMC at this point, but it was too amusing not to post. So I went to EMC's support site to report the problem I was discussing earlier with the Connect Home settings. I went to Live Chat, filled out all our information, gave a detailed description of the problem, and then hit chat. "Sorry, Live Chat support is unavailable! Please call one of these numbers:!" And of course the form reset, erasing all the info that I filled out. Fair enough, let me just submit a Support Request through their online portal. Fill it all out, click submit, same thing! Forms reset, "please call this number!" And this is from the world leader in storage and virtualization? Can't figure out how to check to see if support is available before presenting the form, or at the very least preserving the form so I can resubmit later. Awesome. Maybe I should just be posting in the poo poo That Pissing You Off Daily thread.
|
|
#
?
Dec 20, 2011 20:22
|
|
|
Internet Explorer posted:I was going to avoid posting this, because I feel like I am just making GBS threads on EMC at this point, but it was too amusing not to post.
|
|
#
?
Dec 20, 2011 20:43
|
|
|
Me, too. Evidently during the storage array chaos this weekend, one of our Centeras sent an alert as well. We missed it in the fun of trying to get our environment back online and QA'd. As a result of the alert, EMC India called my on-call engineer at 12:15am local time to ask him for a status on the ticket. My engineer didn't know what EMC was talking about so he started talking about the storage array. It took them 5 minutes to figure out neither one of them was on the same page. Once my engineer figured out that EMC called him to wake him up about a Centera alert that had happened 36 hours earlier he was livid. Seriously, EMC. Everybody who outsources support has to think of this poo poo. I have to manage staff in India and I know it's a huge challenge, but I do it and my customers never have to deal with it. Seems like EMC has screwed up its support big-time.
|
|
#
?
Dec 20, 2011 21:35
|
|
|
As a compliment to EMC, I just had a tech call me back after about an hour and a half on the problem I was posting about earlier. I put it in as a low priority, so that is pretty impressive. I called them up and submitted a ticket over the phone. The tech Webexed in, saw exactly what the problem was, and knew what the solution would be. On top of that, it was the same tech that helped me with a problem a month ago. So I am not sure if that was chance, which I doubt, or if they are kind of prioritizing support requests to techs that have already helped the user, which is great. My problem? Turns out I needed to enable SMTP relay for either EMC's domain on my Exchange server, or enable SMTP relay from the VNX Control Station IP address. Absolutely a PEBKAC issue, but from a usability standpoint would it have killed the message to say "Unable to Relay message, please check the following settings:" I asked the tech if we could have just put a valid email address from our domain, which is what I would normally do, and he said no. Had to be coming from an EMC email address. Considering pretty much every mail server deploys with relaying disabled by default, you would think this would cut down on number of tickets significantly. Spamtron7000 posted:Me, too. Evidently during the storage array chaos this weekend, one of our Centeras sent an alert as well. We missed it in the fun of trying to get our environment back online and QA'd. As a result of the alert, EMC India called my on-call engineer at 12:15am local time to ask him for a status on the ticket. My engineer didn't know what EMC was talking about so he started talking about the storage array. It took them 5 minutes to figure out neither one of them was on the same page. Once my engineer figured out that EMC called him to wake him up about a Centera alert that had happened 36 hours earlier he was livid. I meant to reply to you yesterday, but got side tracked. You had a ton of downtime due to FAST Cache failure? How did that happen? I thought they were supposed to be RAIDed anyways? Did you lose two drives at once or something? I only ask because I think I read about something similar that happened to another company, but I for the life of me cannot find the article and I was curious as to exactly what happened. Internet Explorer fucked around with this message at 21:51 on Dec 20, 2011 |
|
#
?
Dec 20, 2011 21:48
|
|
|
Internet Explorer posted:I meant to reply to you yesterday, but got side tracked. You had a ton of downtime due to FAST Cache failure? How did that happen? I thought they were supposed to be RAIDed anyways? Did you lose two drives at once or something? I only ask because I think I read about something similar that happened to another company, but I for the life of me cannot find the article and I was curious as to exactly what happened. Only one SSD failed out of 4 configured in a RAID1 set. I'm not sure why, but immediately afterward a 0x740a event stating that the entire enclosure was faulted, following by a bunch 0x720e event codes went out saying that the various hosts no longer have a working physical connection. My guess is that the disk fault somehow triggered the system to think the entire enclosure faulted - basically taking all of the FAST Cache offline and causing mayhem with the storage processors. Beyond that, it's just guessing - EMC said they'll have an engineer working on an RCA once the part is replaced and FAST cache is re-enabled.
|
|
#
?
Dec 20, 2011 22:28
|
|
|
That's pretty crazy. You would think a disk failing in a RAID array, especially a RAID1, wouldn't cause that type of problem. Glad we are not using FAST Cache yet. Hopefully by the time I can convince the higher ups to but a few SSDs, they'll have that particular bug ironed out.
|
|
#
?
Dec 20, 2011 22:30
|
|
|
Internet Explorer posted:That's pretty crazy. You would think a disk failing in a RAID array, especially a RAID1, wouldn't cause that type of problem. Glad we are not using FAST Cache yet. Hopefully by the time I can convince the higher ups to but a few SSDs, they'll have that particular bug ironed out. It sucks because FAST cache has been awesome so far. Every single one of our environments is lightning fast since we upgraded.
|
|
#
?
Dec 20, 2011 22:36
|
|
|
Anyone with knowledge about Compellent's Data Progression feature? How does it work, policies/period/chunk etc?
|
|
#
?
Dec 20, 2011 23:06
|
|
|
|
| # ? Apr 24, 2024 11:19 |
|
|
szlevi posted:Anyone with knowledge about Compellent's Data Progression feature? How does it work, policies/period/chunk etc? It marks used blocks and progresses them either up or down tiers (sometimes just a raid level) It usually kicks off around 7-10pm depending on how you set it up. I've worked with their equipment for around 5 years, know several of their support staff and am currently upgrading to some newer controllers, so I can likely answer any of your questions about their equipment.
|
|
#
?
Dec 21, 2011 05:43
|
|
