|
Interestingly, it appears that the iSCSI initiator in Server 2008 R2 is much faster than in Server 2003...we upgraded a box that was connected to our Netapp via 10Gbe and measured throughput jumped from 100MB/sec to 400MB/sec. Has anyone else noticed this?
|
#
¿
Feb 14, 2012 22:18
|
|
|
|
| # ¿ Apr 28, 2024 17:53 |
|
|
So...how hosed are we?director of IT posted:HAY GUIZE IMMA MOVE ALL OUR FILES TO THIS UNUSED NAS BECAUSE WE NEED MORE SPACE WHAT COULD POSSIBLY GO WRONG HAHAHAHA ...we really did need more space. I expressed concerns about whether it was advisable to make this move without extensive testing. I was laughed off. So, he did it anyway. Our entire file storage (except for databases and some other stuff like Exchange) got moved over the weekend to a Reldata appliance. This included shared network folders used by many people, as well as every user profile in the entire company (about 4,000 employees). To my great lack of surprise, the NTFS permissions on all of those folders and files (millions of them) essentially got put through a wood chipper/meat grinder/Blendtec blender/insert appropriate metaphor of destruction here. No one can get into their stuff. We can't fix it, because we don't have permissions to modify the folders. Admittedly I don't know much about how the appliance works but I'm guessing that it has its own filesystem and provides NTFS emulation of some sort. Poking around ACLs on folders I noticed "NODE-C\Administrators" which seems mighty suspicious to me. They're on the phone with Starboard right now trying to unfuck this. We are highly reliant on centralized user profiles (everyone's path is \\fileserver\profiles\username) because the vast majority of our users are Citrix users, which means NONE OF THEIR loving APPS WORK. This has been going on for days and it still isn't fixed. I want to die.
|
|
#
¿
Jun 19, 2012 22:09
|
|
|
He used robocopy. I have no idea what happened. It's too late to revert; we're committed to the new structure. Also his title is sorta meaningless as he's really the chief sysadmin but also happens to be the director of IT. He and one other guy do essentially all of the SA work for the entire company.
|
|
#
¿
Jun 19, 2012 22:29
|
|
|
Additional details have emerged: - I'm not sure what model Reldata this is. It's one of the ones based on Supermicro chassis with vertically mounted drives. - It has come to light that apparently the particular appliance we have will not support the user load they are putting on it. I have no idea why they didn't figure this out BEFORE the migration. - I'm not sure if it's being used in iSCSI target mode or NAS mode but I speculate it is the latter. - The reason why I think that is ACLs aren't working properly. Even when permissions appear to be set correctly on a particular folder, it doesn't work the way it should. Specifically it seems to have issues with individual users, although groups tend to work ok. Essentially at this point it appears that it is in fact impossible for them to fix it because it won't even do what they're trying to make it do. They just figured this out today, and the migration happened on 6/15. The best part? We have no other options. We don't have any other hardware we can move it to and we can't roll back either because the person responsible is too proud to admit his mistake or because he did it in such a way that rolling back is now impossible. He actually tried to blame the helpdesk (my department) by saying that robocopy and the ACLs didn't work right because we left our computers on and had files open. Uh...that's not how file locking works. The most we would have had open is a folder window, not any actual files, and we certainly weren't locking anyone's user profile files. If I had something else to fall back on I would resign without notice tomorrow, because this is ridiculous. One person brought a $700 million company to a grinding halt.
|
|
#
¿
Jun 20, 2012 01:15
|
|
|
Actually he did (sort of) test it Friday night by having us check permissions on the copy and there were problems. He thought he had them fixed and went merrily on his way. It is of note that he said, and I quote, "This really scares me." He won't be fired. He's been at the company since it was formed in 2004 and he's only in his position as a result of nepotism.
|
|
#
¿
Jun 20, 2012 14:39
|
|
|
paperchaseguy posted:For a $700 million company, two sysadmins seems a pretty small number. YOU ARE CORRECT SIR. They're trying to hire another. No luck so far. I wonder why?  I asked in a completely neutral way if the Reldata appliance we have could also be used as an iSCSI target, and wouldn't that take SMB/CIFS out of the equation? He responded by saying, well, it's a SAN with a built-in NAS head. He doesn't even know how the loving thing works. 
|
|
#
¿
Jun 20, 2012 16:43
|
|
|
Internet Explorer posted:It sounds like a lot of other companies these days, honestly. Staff and budgets are cut to the point where you have no other choice but to fly by the seat of your pants. 2 sysadmins with 5,000 users in a company with 700m annually? Sounds like they are probably overworked like hell. First to go is the change management, then the documentation, then the testing, the day-to-day maintenance... It gets to the point where the only things you have time to do is put out fires and try to ram some improvements through, so you don't accrue so much technical debt that you'll never climb out of it. You just described exactly how the IT department here runs. Another awesome example: last year, they needed to upgrade their Exchange infrastructure but didn't have the skills to do it. So they paid Dell something like $300,000 for hardware and consulting and Dell set up Exchange 2010 for them. There is absolutely no documentation on what they did. It only exists in the senior sysadmin's head.
|
|
#
¿
Jun 20, 2012 20:06
|
|
|
paperchaseguy posted:That's pretty inexcusable for anywhere but a startup. And they could have (should have) paid Dell to give them documentation for a small amount extra. COO just came over and asked what the flying gently caress is going on. He kinda got the brushoff. So. YOTJ, etc. Also, why is it I can't find any Reldata hardware for sale anywhere? And why is it so hard to find documentation? Is Starboard trying to pretend Reldata never existed?
|
|
#
¿
Jun 20, 2012 21:15
|
|
|
Uh...yeah, about that. We actually have a Netapp FAS3420 with two shelves. It's being used for our "really" mission critical stuff, but originally that stuff was on Reldata hardware as well. Is the FAS3420 supposed to sound like a jet taking off all the time, by the way?
|
|
#
¿
Jun 20, 2012 21:37
|
|
|
spoon daddy posted:Depends on cooling but in general, they aren't quiet. Hmm, well... It's the loudest goddamn thing I've ever heard in a data center. If I stand behind it it blows my hair around. It is being used very heavily (two 10GbE links I think, and it runs a database that has extremely heavy traffic), but still. This is the appliance itself I'm talking about, not the shelves.
|
|
#
¿
Jun 20, 2012 23:23
|
|
|
NippleFloss posted:This is true and problems like the one PTM is having are large scale organizational failures so it's not entirely fair to blame one dude for all of it. But I spent a couple of years as a lead for a group of sysadmins and despite providing examples of what I expected I don't think I ever received an acceptable project plan from any of them. He has a book about ITIL on his desk. The irony is hysterical. Edit: Sorry for the derail, I feel like I'm talking about drama now. I'll stop.
|
|
#
¿
Jun 21, 2012 01:32
|
|
|
So I verified that the appliance in question is a Reldata 9400 12TB. Does anyone know about the performance of such a system (note in particular that I'm pretty sure it has 7200rpm drives) when used as a CIFS NAS? They are gradually getting the permissions issues fixed but as that happens the usage level of the appliance is also increasing and I'm wondering how long it will take before it self-destructs from the strain.
|
|
#
¿
Jun 21, 2012 15:06
|
|
|
Dumbass IT Director: "Yeah I think we're gonna roll back and abandon the Reldata..." Starboard Support: "NO WAIT WE HAVE A HOTFIX" Dumbass IT Director: "A HOTFIX YOU SAY WELL I NEVER" Briefly, my life was looking up. A ray of sunlight was visible, as it appeared they realized they needed to jettison the Reldata and get a real filer. I then heard, as from very far away, a sound that was very much like a giant, $700 million toilet flushing. Previous to this Starboard kept going "DAFUQ" and had no idea what the problem was. Apparently something is going on with communication between the NAS and active directory when ACLs are interrogated, I don't fully understand it. What it really boils down to is load, and this filer cannot handle the load that is being put on it. Even if they get it "working" it's going to eventually turn into a very expensive pool of molten metal on the data center floor. I hate my life. 
|
|
#
¿
Jun 22, 2012 01:04
|
|
|
You know, I think if your job is so stressful and so much pressure is being put on you that you start to ponder , that is a bad sign. That Careerbuilder commercial is on repeat in my mind.
|
|
#
¿
Jun 22, 2012 17:53
|
|
|
What's that you say? ANOTHER hosed UP SHARE?  His boss is back in town tomorrow. I can only hope...
|
|
#
¿
Jun 25, 2012 17:15
|
|
|
What if they were SSDs? Just curious, I don't know much about SSD vs 15k performance.
|
|
#
¿
Jun 25, 2012 22:03
|
|
|
Lingering permissions issues continue. "Yeah, we don't actually support auditing, mmmkay..." They patched the loving kernel in their customized Linux OS on the appliance with custom code written just for us. Oh, and it's a live patch. If the appliance reboots for any reason we're hosed. Weirdness like losing permissions that were already set continues. Every time I talk to him about it he digs himself into a deeper hole and there's no sign of a replacement that will be more reliable. Uh, the NAS, that is, not him. I have no idea about his status although I've heard that management isn't exactly happy and some people caught on to his responsibility in the matter and have been complaining. The company is undergoing a huge PR crisis right now so fortunately for him, that has overshadowed things a bit. Supposedly a more permanent patch is coming but they keep finding new issues to add to it. Oh, and the patching is pretty loving stupid considering that the product was advertised as capable of doing all the things that it can't do. TL;DR gently caress RELDATA/STARBOARD DO NOT BUY THEIR lovely PRODUCTS GET A REAL NAS (NETAPP).
|
|
#
¿
Jun 27, 2012 02:32
|
|
|
The fun just never ends, guys. Starboard Storage finally pulled a permanent patch out of their collective Russian asses (did you know the company is run by Russians now? I have no problem with Russians, just thought it was interesting), and it made things worse, not better.  I am Jack's complete lack of surprise.
|
|
#
¿
Jul 9, 2012 15:46
|
|
|
I can confirm that Synology's products are excellent and so is their support. We are rolling them out for on-site software repository purposes at 130+ sites. It's only a single-drive model but it did some very specific things that no other device we found would do (primarily with FTP access for Wyse Device Manager, which we use to patch/image Wyse thin clients).
|
|
#
¿
Jul 18, 2012 21:22
|
|
|
"Device not ready" is not a message you want to get in DMC from an iSCSI target that is being used as the quorum disk on a rather large and important SQL cluster. On the other hand it's not my responsibility, so:
|
|
#
¿
Jul 30, 2012 16:25
|
|
|
Boss is leaning towards getting some Dell/Equallogic poo poo because I guess they're in the far upper right of Gartner's magic quadrant for SAN/NAS vendors. Is the stuff any good?
|
|
#
¿
Aug 16, 2012 17:43
|
|
|
Some of you may recall my posts last year about the lovely Reldata NAS system I had to work with. I no longer work at that company, thank god, but I've kept in touch with some of my former coworkers. It was working sort of OK except for lingering, weird permissions issues (like one particular AD user that had Full Control on the entire filesystem who shouldn't have and they never could seem to get rid of it), but I knew after looking at the specs of the system vs what it was being used for that they were pushing it way, way too hard. I predicted it would eventually fail catastrophically. Earlier this week, it did exactly that. The sole file storage for every user profile and shared file in a company with 4,000+ employees just plain quit. If you're going to put all your eggs in one basket, at least don't use a basket made out of toilet paper.
|
|
#
¿
Jan 13, 2013 20:59
|
|
|
evil_bunnY posted:What happened after? I don't give a gently caress, but I haven't heard yet. They were on conference calls with Starboard the whole week trying to resurrect it.
|
|
#
¿
Jan 17, 2013 17:52
|
|
|
I've been working with a NetApp FAS3240 at my new job and I have to say that I love it. I was pleasantly surprised by how easy to use the management tools are. Snapmanager for Exchange, on the other hand, can kiss the darkest part of my rear end in a top hat.
|
|
#
¿
Jan 19, 2013 06:05
|
|
|
The biggest issue I've had with SME is that it seems if you make any sort of change to your mailbox server (change the name of a database, create a new database, move a database, etc), you have to run the configuration wizard again and you have to throw out your backup job and create a whole new one. If I'm mistaken about this, please enlighten me, but NetApp has a knowledgebase article that says pretty much exactly what I just said.
|
|
#
¿
Jan 28, 2013 17:33
|
|
|
madsushi posted:I think the idea is to set it up once and then forget it. In theory you are supposed to be able to set it up and forget it. In practice, if you make any of the changes I mentioned, it breaks the backup job, even if it is only one DB out of several that doesn't work. It still breaks the entire job. The error is something about the path not matching when it attempts to do a VSS copy, and when I gave that information to NetApp they sent me to the aforementioned KB article...which says that you have to run the config wizard again, and if it is an existing DB and you changed the name, you have to move it to another LUN and then move it back. Yeah.
|
|
#
¿
Jan 28, 2013 18:11
|
|
|
|
| # ¿ Apr 28, 2024 17:53 |
|
|
I'm using Exchange 2007, and we don't have the budget to upgrade (probably for the entire year), so all that lovely stuff that 2010 adds doesn't apply to me. I'm also running my mailbox server on Server 2003R2, not sure if that makes a difference.
|
|
#
¿
Jan 28, 2013 21:20
|
|