Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
notwithoutmyanus
Mar 17, 2009

Irritated Goat posted:

You're gonna have to work hard to get me to believe Fiserv isn't pants-on-head dumb. See: Datasafe

Oh do I have a FIS story for all of you.

I started at a bank 2 yrs ago, and I found out last year that for more than 2 years - we'd been billed around...$30k extra for the connection to FIS which was supposed to be a 10mb circuit. Instead, they were giving us 6.

When I found out about it, they denied it for about 3 months - literal radio silence to emails including me escalating further and further with them and including more and more management up the chain on our side due to lack of response, and then basically said "so, how do you want this? do you want free service for the next 12 months, or a check?" response from senior mgmt: check.

Typically :yotj: part? Of course I didn't see any prominent bonus or salary increase from literally getting the company $30k and am pretty much ready to look elsewhere.

notwithoutmyanus fucked around with this message at 11:36 on Feb 9, 2018

Adbot
ADBOT LOVES YOU

notwithoutmyanus
Mar 17, 2009
So, my generally shitheap financial company which was caught in the 70's is changing. We got a new CIO who was put in place to clean up a lot of poo poo. I even got the major purchases I've been asking for since I started this job. We've gotten people to stop being walking HR violations and that alone is amazing. Also, sometimes people don't silo everything off anymore, and sometimes the CIO is now putting a stop to silos. It's improving.

However, we have a basic problem here, which I wanted to ask the thread wisdom about, because I feel I'm missing something as is:

We have tons people who have a poo poo experience accessing files from their network share. Specifically it's a ton of users opening outlook archives across the share and giant excel files across the share. Most sites are hilariously bandwidth constrained, yet the CIFS share isn't showing any sort of performance warning as I think it averages it's metrics which it only gathers every 60 seconds, etc. At the same time, when I have an agent on some of these people's machines monitoring their quality of experience, it's showing enormous CIFS/SMB application latency to the share. All of these users are at site A, the share is at site B (HQ) and not very far at all geographically (10 miles away). Outlook freezes up accessing parts of the archive (across the share), and excel tends to freeze up with their giant excel files. Laptops all have 7200RPM shitters and not SSD because :downs: company. Also mostly windows 10 environment for the users. I see huge bandwidth spikes just clicking around the PST, possibly enough to support not enough bandwidth but other sides with a poo poo experience also have plenty of extra bandwidth. /endrant

In trying to identify the cause of the poo poo experience, am I missing something between a: outlook not supporting opening an archived PST in a network share and b: the exact same share being used for the pst's also being used for the giant excel files ? Storage/network share folks just sitting saying their poo poo is fine. I'm still wondering why it's identifying it as both CIFS and SMB, if that means something is hosed up in how it's transferring data?

notwithoutmyanus
Mar 17, 2009
Yeah, I wasn't sure why CIFS/SMB were somehow identified separately. Noise I guess.

Silo in this context is 2-3 teams buying poo poo that does the same thing - usually after having dedicated tools and shared access, or not telling other teams the poo poo they're doing (which impacts the other team), or people literally hiding the poo poo they're doing because they know they're doing it wrong.

notwithoutmyanus
Mar 17, 2009

AlternateAccount posted:

Wow, similar experience. Opening PSTs over a network share barely works when it works, and seems like it's a time bomb for Outlook completely making GBS threads itself and the PSTs becoming heavily corrupted after a few months. We've gotten most people migrated, but it was standard, trained practice to create a shitload of PSTs on a network share when I started. Ffs.

What agent did you use to do the SMB monitoring?

No one seems to believe me when I say hey, opening 20MB+ Excel files with tons of links over SMB from across the country is bad policy.

I used the solarwinds QOE agent out of sheer laziness because it's watching network response time separate from application response time from the perspective of the machine it's installed on. It's good for this, it's useless for HTTPS traffic.

notwithoutmyanus
Mar 17, 2009

Methylethylaldehyde posted:

Anything SMB is dogshit once your latency goes above about <1 ms. Once you're outside the local network, SMB goes from kinda ok, to dumpster fire tier in a hurry. Office products are especially bad at it because of how the user experience works, especially with complex cross-linked excel files. Excel will hang outright and just sorta sit there until the file downloads, then it'll open, then lock up resolving every single linked entry to another excel file or external data source. THEN once it's all done, will it load. On a 15MB Xlsx file, that can be upwards of 4 minutes over a 20 mbit VPN link. If the user clicks it a bunch, tries to open it a half dozen times, or force kills it and reopen it, it can corrupt the file hard.

Best solution I've found is DFS shares and a local copy of all files, with a decent replication policy in place.

Outlook and large PSTS over the network is a great way to spend a billion dollars trying to resolve an issue that's best fixed with a good backup policy, local PSTs files, and a ssd for the special snowflakes you just absolutely need 50 years of emails going back to the days of the SPARC mainframe they first sent ASCII nudes to eachother on.

Welp, this is what I expected. I suggested DFS and a backup solution for end users, got a blank stare. Gonna show this to the CIO and laugh my rear end off when I have to explain all the poo poo that lead up to it.

notwithoutmyanus
Mar 17, 2009
So I've been trying to rule out slow laptops from this SMB issue we're having. I've got monitoring going watching a few users. I see one has a massive disk queue spike after logging on, for around 45 minutes this morning. This has been ongoing for weeks.

I send out an email, trying to be proactive maybe an hour after the event, and I get a shrug from the impacted user.

Me: Sir, I saw a lot of slowness on your machine between 7:45AM and 8:45 AM this morning. Can you help me understand what you were doing so we can pin down the slowness? File access? Outlook?Network share?
Him: Well, I suppose it was a bunch of email downloading . :downs:

notwithoutmyanus
Mar 17, 2009
So my pretty bizarre workplace finally bought all the poo poo we need, some has come now, some is coming soon, all is well. :smith:
Of course I forgot that this means I'm like 5x busier as a team of one, and executive requests are flying in night and day.

notwithoutmyanus fucked around with this message at 10:22 on May 16, 2018

notwithoutmyanus
Mar 17, 2009

LethalGeek posted:

Somedays I feel the unholy IT trifecta of professors, doctors, and lawyers should be changed to professors, doctors, and paralegals.

Lawyers can be big babies sometimes but no one can be rude, cranky, and lovely like a paralegal IMO. They act like the place will fall apart without them and lol forever. Also they by far push back the hardest if you dare to tell them there are better ways to do something or a software change makes them have to do something different.

I remember seeing a job offer in my area specifically around my specialty plus a million other things (basically be the impossible everything specialist), for some big law firm. Network + storage + server admin + cloud + workstations + AD + virtualization + being considered L3 helpdesk. I kept getting a million recruiters about it and the position must have been open for half a year or more, easily. For humor, i looked on glassdoor and see what basically consisted of "the culture is immature/poo poo here". Memorable for the hilariously large wishlist as well as gently caress that place, clearly.

notwithoutmyanus
Mar 17, 2009
So, in tribute to the current title:

about 4 months and then again weeks ago I warned that our licenses were running out. New equipment has been installed and monitored, and now we're out of licenses. Clearly, we shouldn't pay our vendor until we have to. :cripes:

notwithoutmyanus
Mar 17, 2009
Meh, all our servers have a boring standard naming convention - our conference rooms, however, are named after sports and web browsers.

For once, I'd like to see conference rooms named after videogames or superheroes or even crypto would be amusing.

notwithoutmyanus
Mar 17, 2009

AlternateAccount posted:

Proofpoint + a giant [Ext] on the front of any outside emails seems to work pretty well for us. Also the way Proofpoint modifies(mangles the poo poo out of) URLs means that even if they realize a link is spurious after the fact, they can tell you YO, A USER CLICKED ON THIS BEFORE WE KNEW IT WAS BOGUS.

We have [THIS IS EXTERNAL SECURITY WARNING PLEASE BE CAREFUL] appended to every subject line. They're now so long we hit spam filters/junk constantly. Apparently nobody thought of that.
:cripes:

e: I need to :yotj: out of this poo poo, work life balance has been gone and recognition for busting my rear end has been "oh, sure we'll recognize you!" :bahgawd:

notwithoutmyanus
Mar 17, 2009

AlternateAccount posted:

I wish we could actually apply Reasonable Person Logic and not create entire-organization level security policy based on single outlier events that are highly unlikely to repeat.

Sure - if our infosec people were more technical than our helpdesk, maybe. Yet, I wouldn't even give them that much credit. I'd say some of the new ones even less than that.

notwithoutmyanus
Mar 17, 2009
So, for once I have a non poo poo boss! It's time to grow and :yotj: Storytime!

I'm actually happy about this, because this is the last conversation I had via email with my previous boss, who was thankfully deservedly canned. I ordered a $20 box of screws, and our receivables folks said they'd approve it and get my manager to approve it.

:downs: : "anus, what's this?" (forwards copy of approval request, CC's purchasing)
me: "it's an order for a box of screws."
:downs: "can you provide more details?"
me: "It's an order for a box of screws for network equipment"?
:downs: "can you please give me a better explanation?"
:sigh: "It's a box of screws for network equipment because sometimes things are unboxed and deployed, and we end up needing screws to mount it".
Meanwhile, purchasing tells me they'd never seen ex-boss do something like that before. :cripes:


Oh, and a more recent short one - verbatim, not involving a boss.
:frogdowns: "We want to poll the core switch" (for a bunch of data and we have a monitoring suite already)
Me: "you should probably get a change control for that"
:frogdowns: "What's a change control?"

Adbot
ADBOT LOVES YOU

notwithoutmyanus
Mar 17, 2009
:yotj: in progress, with giant pay bump and actual 9-5 flex hours (instead of 7-8 like I've been doing). I have never met as bad a company as my current one I'm exiting. Here's to finding out how wrong I am on the next one! :gbsmith:

  • 1
  • 2
  • 3
  • 4
  • 5