|
Um... wow. I'm not sure what's more surprising, either a) that he was a director and didn't know permission issues after an NTFS/CIFS move was a possibility, or b) that he was a director and was performing the move himself. Did he not use robocopy or similar migration tool? hope you got good backups lol!
|
#
?
Jun 19, 2012 22:26
|
|
|
|
| # ? Apr 25, 2024 11:20 |
|
|
He used robocopy. I have no idea what happened. It's too late to revert; we're committed to the new structure. Also his title is sorta meaningless as he's really the chief sysadmin but also happens to be the director of IT. He and one other guy do essentially all of the SA work for the entire company.
|
|
#
?
Jun 19, 2012 22:29
|
|
|
Powdered Toast Man posted:So...how hosed are we? Can you restore the data to its original location with a snapshot? What happened to the original data? Did he bother to do a full backup before the move? If you have to restore the permissions manually, you'll need to weigh security vs. access. when you get control of the files back you can do a blanket domain users read all so at least people can get back to work, then start setting the permissions manually. We had a guy take ownership of about 2k directories and it literally took months to get it sorted. Nomex fucked around with this message at 22:52 on Jun 19, 2012 |
|
#
?
Jun 19, 2012 22:48
|
|
|
If the source is still intact, you can use a tool like SetACL to mirror the permissions from the source to your destination.
|
|
#
?
Jun 19, 2012 22:52
|
|
|
Anyone here have any experience with Dot Hill? We have about a 20k budget for this project and have looked at EMC's offering already (VNXe3300). No room in the budget for a second SAN right now, but we do have monies for a DR project later on this year. I might be able to cut out some of the software off of the quote, but I hear it is more expensive later on. We are looking at using it for a small SAP upgrade running on VMWare. As of now we are using VMWare (Vsphere 4.1) using local storage.
|
|
#
?
Jun 19, 2012 23:12
|
|
|
Additional details have emerged: - I'm not sure what model Reldata this is. It's one of the ones based on Supermicro chassis with vertically mounted drives. - It has come to light that apparently the particular appliance we have will not support the user load they are putting on it. I have no idea why they didn't figure this out BEFORE the migration. - I'm not sure if it's being used in iSCSI target mode or NAS mode but I speculate it is the latter. - The reason why I think that is ACLs aren't working properly. Even when permissions appear to be set correctly on a particular folder, it doesn't work the way it should. Specifically it seems to have issues with individual users, although groups tend to work ok. Essentially at this point it appears that it is in fact impossible for them to fix it because it won't even do what they're trying to make it do. They just figured this out today, and the migration happened on 6/15. The best part? We have no other options. We don't have any other hardware we can move it to and we can't roll back either because the person responsible is too proud to admit his mistake or because he did it in such a way that rolling back is now impossible. He actually tried to blame the helpdesk (my department) by saying that robocopy and the ACLs didn't work right because we left our computers on and had files open. Uh...that's not how file locking works. The most we would have had open is a folder window, not any actual files, and we certainly weren't locking anyone's user profile files. If I had something else to fall back on I would resign without notice tomorrow, because this is ridiculous. One person brought a $700 million company to a grinding halt.
|
|
#
?
Jun 20, 2012 01:15
|
|
|
Powdered Toast Man posted:He used robocopy. I have no idea what happened. Going from a windows server to SAMBA server emulating CIFS presents challenges for robocopy. He's a complete moron if he didn't test this first before doing the cutover. Common problems I've seen are: - Inappropriate permissions used by the account performing the copy, leading the ability to copy files but not to modify their permissions or attributes. I've had this happen with Robocopy before and the fix in that case was (strangely) that the account performing the copy needed to be a domain administrator. - Permissions are granted through local groups and those groups, and their members, are not created on the new device. For instance, NODE-C\Administrators sounds like the local admins group on the NAS so anyone who was a local admin on the old windows box should be on the NAS. - Improperly set robocopy flags that strip explicitly defined permissions and only leave inherited permissions. In this case your new files inherit permissions from the root of the share which only has NODE-C\Administrators. I've seen well paid consultants screw up these migrations by assuming automated copy tools do what you expect them to and not sanity checking the results.
|
|
#
?
Jun 20, 2012 01:23
|
|
|
somecallmetim posted:Anyone here have any experience with Dot Hill? We have a MSA2000 HP SAN which is just a rebranded Dot Hill SAN. It hasn't given us really any trouble in 3 years though we are only using it as the storage for our two node MSSQL cluster, nothing fancy.
|
|
#
?
Jun 20, 2012 02:36
|
|
|
somecallmetim posted:Anyone here have any experience with Dot Hill? We have about a 20k budget for this project and have looked at EMC's offering already (VNXe3300). No room in the budget for a second SAN right now, but we do have monies for a DR project later on this year. I might be able to cut out some of the software off of the quote, but I hear it is more expensive later on. Dot Hill... Now that's a name I haven't heard in a long time. Mostly because they typically OEM to folks like Sun and HP. The products are okay but won't have a lot of frills. That said you can probably get into NetApp or EMC the same price range. You'll want to make sure you consider the replication technologies available with each vendor if you're planning to do that. Also any recovery tools you might be using (such as VMware site recovery manager, which also offers host based replication.) I'm personally a huge advocate of SnapMirror from NetApp and RecoverPoint from EMC myself. I've worked with both products extensively for the last few years and have had nothing but glowing feedback and successes.
|
|
#
?
Jun 20, 2012 07:30
|
|
|
somecallmetim posted:Anyone here have any experience with Dot Hill? We have about a 20k budget for this project and have looked at EMC's offering already (VNXe3300). No room in the budget for a second SAN right now, but we do have monies for a DR project later on this year. I might be able to cut out some of the software off of the quote, but I hear it is more expensive later on. Any reason why you looked at the 3300 and not the 3100? I found the 3100 to be a lot cheaper for very little compromise.
|
|
#
?
Jun 20, 2012 08:02
|
|
|
NippleFloss posted:Going from a windows server to SAMBA server emulating CIFS presents challenges for robocopy. He's a complete moron if he didn't test this first before doing the cutover. This is why you always, always do a robocopy /L /LOG:filethatensuresidontlosemyjob.txt before you do the real copy. And for the love of god, you do things section by section, department by department, with time for testing in between.
|
|
#
?
Jun 20, 2012 08:55
|
|
|
Powdered Toast Man posted:[. One person brought a $700 million company to a grinding halt. If what you describe is true, once there is a light at the end of the tunnel, this guy needs to be let go.
|
|
#
?
Jun 20, 2012 09:36
|
|
|
spoon daddy posted:If what you describe is true, once there is a light at the end of the tunnel, this guy needs to be let go. I am amazed he has not been terminated and a consultant bought in to fix things already. Do you not have any change control procedures?
|
|
#
?
Jun 20, 2012 12:49
|
|
|
A wild envelope from EMC appears!! (Addressed to the wrong company but with the right C/O name on it) I wonder what's inside!   I've been made SAN admin for our new VNXe330 and VNX5300. So far I've not been impressed with the product at all. The VNXe3300 seems to be pretty glitchy. I haven't had a chance to play with the VNX5300 too much yet.
|
|
#
?
Jun 20, 2012 13:29
|
|
|
Goon Matchmaker posted:A wild envelope from EMC appears!! (Addressed to the wrong company but with the right C/O name on it) Yeah, awesome! Thanks for snail mailing me some poo poo you could have easily emailed. The largest storage and virtualization company in the world. I will say, Polycom does the same thing. Also, on the 5300... have fun with that. Been struggling with 2 of them over the past 6 months or so. I hear the VNXe line is supposed to be "easier" to implement / manage.
|
|
#
?
Jun 20, 2012 13:51
|
|
|
Actually he did (sort of) test it Friday night by having us check permissions on the copy and there were problems. He thought he had them fixed and went merrily on his way. It is of note that he said, and I quote, "This really scares me." He won't be fired. He's been at the company since it was formed in 2004 and he's only in his position as a result of nepotism.
|
|
#
?
Jun 20, 2012 14:39
|
|
|
For a $700 million company, two sysadmins seems a pretty small number.
|
|
#
?
Jun 20, 2012 15:16
|
|
|
paperchaseguy posted:For a $700 million company, two sysadmins seems a pretty small number. YOU ARE CORRECT SIR. They're trying to hire another. No luck so far. I wonder why?  I asked in a completely neutral way if the Reldata appliance we have could also be used as an iSCSI target, and wouldn't that take SMB/CIFS out of the equation? He responded by saying, well, it's a SAN with a built-in NAS head. He doesn't even know how the loving thing works. 
|
|
#
?
Jun 20, 2012 16:43
|
|
|
edit: doh, wrong NAS thread
evilmonkeh fucked around with this message at 17:35 on Jun 20, 2012 |
|
#
?
Jun 20, 2012 17:30
|
|
|
Powdered Toast Man posted:Additional details have emerged: I wouldn't say it was one guy who brought the company to a halt. Sure, he may have been the trigger, but it sounds like your company has a glaring lack of project and change management. A move like he did should've been validated, tested and rolled out in segments, with controls along the way to make sure poo poo worked. If that guy is going to remain employed, you need to distance yourself from that company, lest your career gets jeopardized due to stupidity beyond your control.
|
|
#
?
Jun 20, 2012 18:07
|
|
|
It sounds like a lot of other companies these days, honestly. Staff and budgets are cut to the point where you have no other choice but to fly by the seat of your pants. 2 sysadmins with 5,000 users in a company with 700m annually? Sounds like they are probably overworked like hell. First to go is the change management, then the documentation, then the testing, the day-to-day maintenance... It gets to the point where the only things you have time to do is put out fires and try to ram some improvements through, so you don't accrue so much technical debt that you'll never climb out of it. Then again, if he is the Director of IT then it is pretty much his priority #1 to make sure that the IT staff have the resources they need to do their job properly, and not half-assed. Of course, not a whole lot of IT Management realize this. Again, the guy is an idiot, but I can't be the only one who has experienced this type of thing first hand.
|
|
#
?
Jun 20, 2012 18:25
|
|
|
NippleFloss posted:StorageTek uses SUN Common Array Management, which is also pretty bad. CAM also includes a cli that is also pretty goofy, but is better than CAM. God, CAM is liquid dog poo poo; Thankfully, our last Sun array is rolling out the door in a few weeks. If the DS4000 used the same rebadged LSI/Engenio software as some of the other LSI resellers, it was pretty inoffensive. in a well actually fucked around with this message at 23:25 on Jun 25, 2012 |
|
#
?
Jun 20, 2012 19:12
|
|
|
theclaw posted:If the DS4000 used the same rebadged LSI/Engenio software as some of the other LSI resellers, it was pretty inoffensive.
|
|
#
?
Jun 20, 2012 19:31
|
|
|
Internet Explorer posted:It sounds like a lot of other companies these days, honestly. Staff and budgets are cut to the point where you have no other choice but to fly by the seat of your pants. 2 sysadmins with 5,000 users in a company with 700m annually? Sounds like they are probably overworked like hell. First to go is the change management, then the documentation, then the testing, the day-to-day maintenance... It gets to the point where the only things you have time to do is put out fires and try to ram some improvements through, so you don't accrue so much technical debt that you'll never climb out of it. You just described exactly how the IT department here runs. Another awesome example: last year, they needed to upgrade their Exchange infrastructure but didn't have the skills to do it. So they paid Dell something like $300,000 for hardware and consulting and Dell set up Exchange 2010 for them. There is absolutely no documentation on what they did. It only exists in the senior sysadmin's head.
|
|
#
?
Jun 20, 2012 20:06
|
|
|
That's pretty inexcusable for anywhere but a startup. And they could have (should have) paid Dell to give them documentation for a small amount extra. Obviously they're woefully understaffed on top of questionably competent. Where are the adults?
|
|
#
?
Jun 20, 2012 20:33
|
|
|
EMC VNC 5300 installation pain report: OH GOD IT BURNS :kraken: EMC shipped us the wrong disk configuration. 600GB drives in the reserved DPE slots instead of 300GB drives like we asked for as well as a extra 1TB drive that should have been a 600GB SAS drive. They also lied about 10GBe port aggregation. My boss is pissed. I'm confused as poo poo.
|
|
#
?
Jun 20, 2012 20:36
|
|
|
IT Vendor lied. Status: Shocked.
|
|
#
?
Jun 20, 2012 20:37
|
|
|
paperchaseguy posted:That's pretty inexcusable for anywhere but a startup. And they could have (should have) paid Dell to give them documentation for a small amount extra. COO just came over and asked what the flying gently caress is going on. He kinda got the brushoff. So. YOTJ, etc. Also, why is it I can't find any Reldata hardware for sale anywhere? And why is it so hard to find documentation? Is Starboard trying to pretend Reldata never existed?
|
|
#
?
Jun 20, 2012 21:15
|
|
|
skipdogg posted:IT Vendor lied. Status: Shocked. Powdered Toast Man posted:Also, why is it I can't find any Reldata hardware for sale anywhere? And why is it so hard to find documentation? Is Starboard trying to pretend Reldata never existed? evil_bunnY fucked around with this message at 21:26 on Jun 20, 2012 |
|
#
?
Jun 20, 2012 21:23
|
|
|
Powdered Toast Man posted:COO just came over and asked what the flying gently caress is going on. He kinda got the brushoff. So. This is part of your problem. Your company needs to spend some of that $700m a year on storage from an actual top tier storage vendor. When you go with a well known and widely used vendor you'll get much better support and have a broad community of users to go to with questions. Given the size and number of employees you're well above the SMB market that RelData was targeting. The saying "no one has ever gotten fired for buying IBM" is as true as ever (though you can easily replace IBM with any number of tier one storage providers).
|
|
#
?
Jun 20, 2012 21:32
|
|
|
NippleFloss posted:This is part of your problem. Your company needs to spend some of that $700m a year on storage from an actual top tier storage vendor. When you go with a well known and widely used vendor you'll get much better support and have a broad community of users to go to with questions. Given the size and number of employees you're well above the SMB market that RelData was targeting. The saying "no one has ever gotten fired for buying IBM" is as true as ever (though you can easily replace IBM with any number of tier one storage providers). It's no coincidence that IBM sells NetApp. 
|
|
#
?
Jun 20, 2012 21:35
|
|
|
Uh...yeah, about that. We actually have a Netapp FAS3420 with two shelves. It's being used for our "really" mission critical stuff, but originally that stuff was on Reldata hardware as well. Is the FAS3420 supposed to sound like a jet taking off all the time, by the way?
|
|
#
?
Jun 20, 2012 21:37
|
|
|
Powdered Toast Man posted:Uh...yeah, about that. We actually have a Netapp FAS3420 with two shelves. It's being used for our "really" mission critical stuff, but originally that stuff was on Reldata hardware as well. Depends on cooling but in general, they aren't quiet.
|
|
#
?
Jun 20, 2012 21:44
|
|
|
evil_bunnY posted:
This is where I'm at. I can't believe anyone would do something like this without an easily restorable verified backup and a rollback plan. If I didn't feel bad for PTM, I would call troll post.
|
|
#
?
Jun 20, 2012 21:47
|
|
|
skipdogg posted:This is where I'm at. I can't believe anyone would do something like this without an easily restorable verified backup and a rollback plan. If I didn't feel bad for PTM, I would call troll post. "Copy completed, data appears to be there, time to free up space that we need right now"!
|
|
#
?
Jun 20, 2012 22:04
|
|
|
Powdered Toast Man posted:COO just came over and asked what the flying gently caress is going on. He kinda got the brushoff. So. so... there are no adults? skipdogg posted:This is where I'm at. I can't believe anyone would do something like this without an easily restorable verified backup and a rollback plan. If I didn't feel bad for PTM, I would call troll post. Oh I believe it. It's far from the dumbest thing I've heard an ostensibly experienced IT professional do.
|
|
#
?
Jun 20, 2012 22:27
|
|
|
NippleFloss posted:I've seen enough stupid poo poo done by sysadmins or people who think they are sysadmins to know that this is totally a thing that could happen.
|
|
#
?
Jun 20, 2012 22:29
|
|
|
Thought we worked for either NetApp or a NetApp Var... Maybe that was someone else.
|
|
#
?
Jun 20, 2012 22:50
|
|
|
skipdogg posted:Thought we worked for either NetApp or a NetApp Var... Maybe that was someone else. I work for NetApp as an on-site professional services engineer with a DOD customer. Prior to that I worked for that customer as thier Storage and Backup lead. I've also worked as a consultant and as a sysadmin in the private sector. I can say that in my experience many admins have terrible organizational and planning skills. Thinks like change management, project planning, an failback plans are foreign concepts.
|
|
#
?
Jun 20, 2012 23:11
|
|
|
|
| # ? Apr 25, 2024 11:20 |
|
|
NippleFloss posted:I can say that in my experience many admins have terrible organizational and planning skills. Thinks like change management, project planning, an failback plans are foreign concepts. Which really, in my opinion, is not so much the fault of the admin and more the fault of management. It is their responsibility to make sure procedures are in place to enforce these types of things.
|
|
#
?
Jun 20, 2012 23:22
|
|