Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
Storysmith
Dec 31, 2006

Ursine Asylum posted:

I use that too! It fixes it in the sense that eventually it causes a more critical problem, which is also someone else's.

Sadly, sometimes creating a more critical problem is the only way to get things fixed sometimes, since that's when you can get the budget/time to spend on it. See: backups, RAID arrays, site security.

Or, to quote Dan Blumenthal, Broke gets fixed; crappy is forever. Sometimes things need a bit of helping to get them over that hump and into the realm of the fixable.

Adbot
ADBOT LOVES YOU

Storysmith
Dec 31, 2006

CitizenKain posted:

The joke was on me! Turns out the shithead in the call center never actually submitted a dispatch order, so I sat around all morning for nothing. I love life and everything in it.

I called their support, got someone who sounded like their soul hadn't left their body, they arranged a "high priority dispatch", but no time was selected so I just left. 3 hours later I get a call saying our VPN is up, so I guess while I was driving home and out of cell service someone showed up. gently caress it don't care.

CenturyLink support


Something a coworker made after the Savvis/CenturyLink TCAM exhaustion that basically left us invisible to half the internet for over a day with no communication:

Storysmith
Dec 31, 2006

slartibartfast posted:

Anybody here ever work for EPIC, the company that makes the health records software? I've heard sysadmins/helpdesk people bitching about their software in these threads, but that can be chalked up to a lot of different things. EPIC's got openings for my specialty, they pay on the high side of average, and I like WI beer, so I'm thinking about it.

Just curious if it's paradise with hookers and blow, just meh, or a complete clusterfuck.

Someone I know worked there in a non-sysadmin facility (whatever their term is for the person implementing new deployments and helping the client go live) and it paid well and was a good environment, as long as you don't mind working 70 hours a week. I can't speak for what it's like now, but this was 2007-2008.

They had a laundromat on site, which is one of those perks that makes you go "huh?"

So yeah, be cautious about work/life balance expectations if you go there.

Storysmith
Dec 31, 2006

captkirk posted:

Thing pissing me off a little:

My cell coverage in my apartment is sort of weak resulting in me missing an after hours call today. So my team lead told me to either get a new phone or use a Google Voice number to receive calls over wifi. Doing the calls over wifi requires me to stay around wifi though so I'm actually considering just spending $10/mo to have a Tracfone with minimal minutes just to receive after hours calls (I'm on call around 6 weeks of the year, so it ends up being like $15 an on call shift).

I'm not that pissed about it and I'm mostly pissed that there isn't a GSM carrier with reasonable coverage in my town.

If coverage is weak in your apartment and only your apartment, complaining to the carrier might get them to cough up a (what is it, micro? pico? femto?) cell that will fix the problem. Depending on your carrier and phone, wifi calling/uma/whatever else might be an option.

If coverage is weak everywhere switch providers to one that isn't terrible at the basic role of providing cellular phone service.

Storysmith
Dec 31, 2006

captkirk posted:

AT&T said I'm not eligible based on my area code. This is what I get for thinking "I'll support a provider who uses GSM!" Also, if I wanted to buy my own AT&T picocell it would be the cost of just buying a CDMA phone and switching to a CDMA carrier.

I'm assuming you're not on a phone that does wifi calling? I thought iOS did it natively now, unless that's just a t mobile thing.

Honestly, if they're going to push back on $10/mo on a line to actually reach you outside of business hours, I'm struggling to see how this is your problem to solve. "I'll respond promptly to any after-hours calls I actually receive."

Storysmith
Dec 31, 2006

Ursine Asylum posted:

manually taking ssns out of a database and manually putting them in google docs because they didn’t want to wait a week or two for a secure storage solution.

Backstory for this, please? What in the hell?

Storysmith
Dec 31, 2006

Gyshall posted:

eh, we have client execs who have been with the company 20+ years or so and like knowing they "have" the emails. I have no problem with dumping these into an online archive, disk space is cheap, etc. The Online Archive feature is basically what this is made for.

I don't know how people can feel this way with the possibility of lawsuits dragging every email they've sent into open court. If they're unaware, that's one thing, but I've told people about it and they still go "So?"

Storysmith
Dec 31, 2006


What the hell is oregano?

EDIT: this usenet posting says "it referred to an incident in which one of the original writers of BSD was arrested for crossing the canadian border with a bag full of oregano."
That's a bit of an obscure reference for a poster, no?

Storysmith
Dec 31, 2006

psydude posted:

If I say a "point to point link" what comes to mind? Maybe a DS3, or a T1, or another dedicated line that directly connects one router to another? Not an ethernet link?

Personally, I think of the giant drum-shaped antennas, and then site to site VPNs, but I'm willing to accept I'm weird for that.

Storysmith
Dec 31, 2006

Sickening posted:

What kind of business do you work for?

Clearly the kind that can't pay employees what they're worth, if these posts are to be believed.

Storysmith
Dec 31, 2006

bull3964 posted:

poo poo that pisses me off:

Local gas company just launched a new customer portal and not only is it balls slow and throws 500 errors half the time, I just noticed as I was setting up payment details that it's not even using SSL. Threw a Fiddler trace on it and plucked my credit card details right out of clear text.

The mind boggles. Needless to say I've switched back to paper billing.

Correct me if I'm wrong, but my understanding of Fiddler (in the one case where I used it) was that it provided a cert that you added to your cert store and it proceeded to MITM all the traffic you send and receive. You *should* see the clear text traffic in Fiddler, it's designed to MITM you. If you see it in wireshark or tcpdump without fiddler running, that's a thing to be worried about.

Storysmith
Dec 31, 2006

Bob Morales posted:

OMG WHAT SHOULD I DO

Just click ignore and quit calling me

ITS DOING ON EVERY COMPUTER ARE WE GETTING HACKED

You should probably figure out what the "some reason" is and work around it, because you're training these folks to get phished if you teach them "just click ignore and quit calling me when you see weird cert errors."

Storysmith
Dec 31, 2006

BaseballPCHiker posted:

I swear to god if we have been hacked I will try and make an impassioned stand for why we need to change or more likely try, get shot down, and start a job hunt. When I first came here we had ONE single domain admin account and password for every server, switch, etc. Also had a spread sheet with it all written out for every other single account you can think of. I finally managed to at the very least get us switched to a KeyPass setup with a shared database file. BUT they actually have a GPO (one of about 3) that makes everyone a local admin! Hooray!!! On top of that my boss has several times shot down the idea of having multiple service accounts because he doesnt want to have to remember them all or look them up.

This place is just weird. They wont do any of the above but we are about to pay out the rear end for ISE and 802.1x. I just dont get it.

Well that just makes sense. You gotta make sure you know which local admin cryptolockered the entire network.

Storysmith
Dec 31, 2006

nitrogen posted:

I do that poo poo and won't apologize for it. I am more likely to do it when working on issues, than just at regular times though.

Yeah, especially on a command line, gently caress your purity of essence. I will `cat thing | grep otherthing` because, statistically, I am likely to need to up-arrow and change otherthing, and I'd rather that be sitting at the end of my history.

If you're scripting, it's just like any other coding; do it in a way that makes the code's intentions clear to the people around you. If something would get you sneered at by dickbags on the internet but is what the other people on your team expect to see, it's a lot easier for everyone to maintain. For gently caress's sake, that article whines about using `ls *` in a script with "the `ls` is not very useful. It will just waste an extra process doing absolutely nothing." We're not paying by the PID here, people.

If I'm logged into a system, it's because something's gone wrong and I'm trying to fix it. I'm not code golfing, I'm trying to do my drat job. If the solution was an ideologically pure one-liner, we'd have automated it already.

CptJackLaser posted:

Luck was definitely part of it, I have no illusions about that. I know that my experience with entering the market was not normal. The cliche line "it's all about who you know" is how I got my job.

I became a sysadmin by spending my adolescence maintaining my own herd of linux machines instead of going to high school parties, and by knowing a dude who saw me floundering at a startup that was stiffing me on salary and said "hey, you could be a QA for my company. Come be a QA here." and then getting to know the other sysadmins and accepting the transfer.

It's not just who you know, it's also the ability to think on your feet once you get there and push the impostor syndrome down long enough to actually become the person they hired.

Storysmith
Dec 31, 2006


Impressive marbling. It's like a Kobe beeftop.

Storysmith
Dec 31, 2006

evol262 posted:

TL;DR

Do other people with a decade of experience feel like they "need" stuff to make them feel comfortable on default installs?

For certain values of 'default installs.'

My personal machine, where I do development and configuration management editing, has a crazily configured vim install that includes wonderful things like CtrlP and fugitive. I don't feel the need to distribute that on servers because, in my mind, if I'm logged into a server doing more than reading logs and tweaking config values, I'm at risk of creating another terrible special-snowflake machine that we can't easily rebuild and acquire the same configuration for via puppet.

The only thing I actually *do* miss when I'm shelled to a server, really, is zsh's fuzzy tab completion. I make typos like crazy, and being able to tab-complete my way from `/etc/init.d/aapche` to `/etc/init.d/apache2` or from /v/l/y<TAB> to /var/lib/yum is hella nice. But the fact that it would be installing another shell, with another security vulnerability list to track, and another dotfile to manage makes a strong case for just sucking it up and dealing with it.

We have a kickstart script we use to standardize (RHEL-based) server installs. The last thing it does is run puppet, taking the machine from RHEL's/CentOS's default install to our standard image. That standard image definitely has some things that are or were not in the default install: nmap, rsync, lsof, screen, tmux, an internet-sourced script that lets us know what processes are actually swapped out, the various PERC- and iDRAC-poking tools we base some of our monitoring around, etc.

We don't really distribute special snowflake configs for those utilities, though, with one exception: vim. We have a basic vimrc for root that sets nobackup and background=dark, because for the longest time, comments would be dark-blue-on-black and the first thing you'd do when editing any file was run ':set background=dark'.

That said, when it comes to these kinds of utilities, if someone came to me and said "hey, I need <thing> installed on <server>," I wouldn't outright deny them if they had a good reason. JQ, for instance, is now installed globally because someone asked for it on one project's machines, we put it there, and its effort reduction paid off. It's also how we got nmap installed: when you're dealing with systems with numerous firewalls between you and them, you sometimes need to know what ports can be seen from that server's perspective.

In other words, it's an odd balance for us, but it's at least centrally managed.

Storysmith
Dec 31, 2006


So, I need a new office chair for my home setup. How comfortable is that to sit in for ~8-12 hours?

Storysmith
Dec 31, 2006

flosofl posted:

Holy poo poo no.

Honestly, get a Herman Miller. I have an Aeron in my home office, and a Mirra at work.

I used to cheap out for the home office, and even with "comfortable" chairs I was burning through the cheap ones (sub $300) every couple years. I've had my current chair for over 6 and it's still fantastically comfortable and all the little levers and pneumatics still work perfectly. Chairs are like shoes, you don't want to go the budget route if you don't have to. Look at it as an investment in a productive work environment.

less than three posted:

We use the Steelcase Leap and they're fantastic. They're like $900-1k though. http://www.steelcase.com/eu-en/products/office-chairs/leap/

Argh, I was planning on getting an Eames Aluminum Group with casters but that Leap looks interesting as hell and just complicated matters. I think that was the chair my coworker got instead of an Aeron that disappeared the day he quit.

I sit in an Aeron when I'm at work; I'd rather sit in something else when I'm at home just for mental separation. I've also busted a couple of work's Aeron lumbar support things already.

Having gone through an outright absurd amount of chairs in my life, I know all about spending money right the first time; I just hadn't seen that weirdass task chair design before and was intrigued. $900 isn't exactly outrageous for a chair I'd be sitting in for 20-50 hours a week.

Storysmith
Dec 31, 2006

anthonypants posted:

Oh cool I found out why no one's ever run decent reports from our ticketing system:


gently caress you, Footprints.

How many tickets do you have that you're crossing the 15 minute threshold? Daaaamn.

Like, that's a terrible design, don't get me wrong, but browsers and web servers can be set to give you a hilariously large amount of leeway there.

Storysmith
Dec 31, 2006

Not pissing me off: Nginx.

I managed to get a web server hanging out between a group project's github.io page and the Public Internet with a valid external SSL cert in an afternoon, and part of that was waiting for the DNS validation change to propagate. Gets an A+ from ssllabs, nginx config is in a (local) git repo so I can see its history, and things Work.

Pissing me off: TLS and PKI in general.

The fact that I'm the only person in this group who's done this poo poo before recently enough that the knowledge isn't out-of-date and likely to lead to a misconfiguration or straight-up exploitation of the server. The incorrect howtos out there about openssl and nginx configurations, cargo-culted along by people because it sorta works-ish, the scars acquired in learning what to do, how to generate SSL certs with SANs, where to put them, all that jazz.

Egads, Let's Encrypt can't come soon enough.

I will never again shame someone who says "I'd have an SSL certificate for my site, but it's too hard to set up." There is a very real usability gap here.

Storysmith
Dec 31, 2006

evol262 posted:

Pissing you off: people who rely on howtos from idiots blogging instead of docs.

The nginx docs are super clear. Pretty much every CA has amazingly clear instructions for Windows and openssl. This poo poo isn't mod_authz_ldap or mod_authnz_ldap (or whatever horrible, barely documented bits apache uses now for ad auth now). There's no reason an admin who can tub two brain cells together can't get ssl working in an afternoon in 2015

Yeah. I'm aware of that page, which is part of why I was able to do it. Is everyone who has a blog setup? Do they even know how to tell if the VPS they're using is running nginx or Apache, let alone which of the 3 places config files could be for each?

And there's also this from nginx themselves that calls out the sheer number of lovely config howtos out there. Good thing that page you linked is the first Google result for "SSL nginx", so there's a chance nontechnical people will click it. Oh wait, it's third, behind ones from DigiCert and DO.

That page, incidentally, doesn't show you how to create and point at a different collection of Diffie-Hellman params, which is recommended these days due to weaknesses in the default set (see weakdh.org).

The thing that pissed me off wasn't "I was able to do this," it's that for people who don't have years of experience or aren't systems administrators in their daily life, this has to be the single most daunting black-box pray-it-works thing they'll ever try to do, break their website, and give up on.

Every "why Johnny can't encrypt" criticism is valid here in PKI land, even more so than PGP. If you don't think the usability of OpenSSL is a shitshow nightmare then congrats, you're the only person I've ever encountered who feels that way, including swaths of the crypto community.

Storysmith
Dec 31, 2006

From a few pages back, but...

Coredump posted:

The VM that runs our portal has gone belly up. Plus everyone on the server team is out today except me, the new guy. I love Fridays.

What symptoms are you seeing on your ovirt VMs? QEMU had(has) a bug that would cause disk I/O to go apeshit on a VM with a qcow2 disk, which would lead to it basically sitting there and blocking. If you viewed the console you'd see a ton of stuck task warnings, and logging in wouldn't work because it'd try to read /etc/passwd and write various files. The workaround was to use qemu-img to trigger a snapshot of the running VM, which would temporarily quiesce things but more importantly restore the state that was overwritten, and allow the VM to continue running. I can grab the sha of qemu that supposedly fixed it if this sounds like your problem; it happened infrequently enough on our machines (hilariously low I/O load) that the snapshot workaround seems sufficient.

Storysmith
Dec 31, 2006

Skandranon posted:

I'm sure he's been fired / hanged by the neck by now.

I also have trouble keeping track of what's at the core of the eight thousand virtualization solutions out there, so I might be misremembering and it might not actually be QEMU-backed.

Storysmith
Dec 31, 2006

pioneermax posted:

Oh this networked plasma cutter stopped working 2 and a half years ago when our IT supplier at the time made some "changes" can you fix it please

shoot me now

This is an Internet of things I can get behind. Please lock out/tag out, lest someone nmaps it at the wrong time and starts a cutting program.

Storysmith
Dec 31, 2006

Things pissing me off: I haven't been able to search in Outlook 2016 for Mac since August. Like, new things just aren't being indexed. I've done the weird MS suggestion of putting ~/Library/whatnot into OSX's Spotlight "privacy" tab and removing it, it clearly spends a lot of CPU time getting very warm trying to index things, and now I can't just not-see results for emails that arrived after August 17, but now can't see any search results at all.

I had Office 2011 installed alongside the Office 2016 preview, then got one of my company's first 2016 licenses to feel out how it is. I like Outlook 2016's whole "not running a background task that will constantly restart itself when you try to kill it to install updates" change from 2011, but I get way too drat much email without the ability to search through and filter it when I'm done, and I really like not having to use OWA.

Anyone else run into this before? Any suggestions before I blow this profile away and recreate it?

Storysmith
Dec 31, 2006

flosofl posted:

Try rebuilding the Spotlight index.

https://support.apple.com/en-us/HT201716

I did that for the relevant folder:

Storysmith posted:

I've done the weird MS suggestion of putting ~/Library/whatnot into OSX's Spotlight "privacy" tab and removing it, it clearly spends a lot of CPU time getting very warm trying to index things, and now I can't just not-see results for emails that arrived after August 17, but now can't see any search results at all.

Unless you want me to do it disk-wide? In which case, sure.

Storysmith
Dec 31, 2006

SIR FAT JONY IVES posted:

This was an official RHEL training. We scheduled it about six months before the class, which was in Dec 2014.

RHEL7 moved from sysvinit to systemd, right? Whatever your personal feelings about systemd, that would be a helluva reprogramming of over a decade of muscle memory performing even the most basic tasks to spring on someone. I'd definitely want a refund unless I already knew and could work with RHEL7.

Storysmith
Dec 31, 2006

Bob Morales posted:

I like how the link light for the port the cable is plugged into doesn't light up, but some other port that has nothing plugged into it lights up instead.



I still have a ton to learn about Cisco, but what is happening here? I know that some switches can have the port shut off via software, but that random link light is ...troubling.

Storysmith
Dec 31, 2006

poo poo not pissing me off:

Got into work a little before noon. Boss, DBA and I went out for lunch and got a bottle of Goose Island Bourbon County Stout each. I drank mine at my desk; boss decided to keep his for later and just drank Jameson on the rocks while we did the smallest amount of maintenance possible to keep basic services up.

I love this team.

nitrogen posted:

All you folks that work in data centers all day, I hope you're wearing hearing protection.

Filing this away under "things to do next time I visit our DC." Thanks for the tip.

Storysmith
Dec 31, 2006

ratbert90 posted:

poo poo that pissed me off today: Hyper-V and CentOS6.7

Why yes! Running yum update SHOULD destroy networking Microsoft! Thanks!

We tried this as an experiment once. Apparently if hyperV moves the VM off of the runner it's on and to another one, the VM's NIC's MAC address changes and (in a land where IPs are in any way tied to MAC, including through dhcp reservations) freaks networking the gently caress out. I'm genuinely interested in what the hell you ran into, because Terrible Virtualization Bugs is a hobby of mine.

Storysmith
Dec 31, 2006

Jeoh posted:

You can give the VM a static MAC address.

Yes; we just didn't realize that it would change in the first place. (I wasn't the one who created that VM.) Much like I didn't understand the two different ways Proxmox and Cloudstack use qcow2 for storage volumes until I was messing around trying to move a VM from one to the other, I learn best when faced with a real example.

Storysmith
Dec 31, 2006

namlosh posted:

Adding a host header entry for thing.org and https://www.thing.org on the correct website (I think they might be called mappings now actually) is how we used to host many sites on one ip and get them to show the correct website based on what the browser requested.

This won't work for ssl though... You have to have different ip's which is why one of my clients has 30 ip's on their single server

SNI exists, solves this very problem, and is supported on the client side by everything that isn't Windows XP or android 2.3, why not just use that? I can't imagine a decent hosting provider wouldn't support it, since it's literally designed to solve the problem of "hosting many websites on a single public IP."

You'll have a "default" SSL cert that gets shown to people who hit it via IP, and otherwise, the client negotiates what site's cert it's expecting via the https handshake.

Modern Apache and nginx do it, does IIS not?

(I've been building an nginx front end to an application server that works exactly as you described, only instead of 30 IPs, it's half a /24.)

Storysmith
Dec 31, 2006

stubblyhead posted:

What was wrong with the table?

Maybe it had only two legs?

Storysmith
Dec 31, 2006

Bob Morales posted:

Is there a posterboy for Azure like NetFlix for Amazon's service?

PagerDuty uses Azure's Fresno location, but that's less "poster boy" and more "the only company I've heard using Azure that isn't Microsoft or spun off from Microsoft".

Storysmith
Dec 31, 2006

Caconym posted:

our own users have db_owner

Sorry for your DBAs' livers.

Storysmith
Dec 31, 2006

Hurricane Electric's DNS is down. Not "the server is down and not returning results," as we have redundant providers and wouldn't be affected. No, we're in the hell or "returning empty results with NOERROR," so the majority of the Internet can't see us. I don't even know what to do against that.

Storysmith
Dec 31, 2006

Scaramouche posted:

Good god. Setting up OpenCart as a favour for a friend, post-install config keeps crapping out for some reason. Turns out that the randomly generated password (generated by opencart) can have ampersands in it, and the database login string (also made by opencart) doesn't escape it properly when building connection info:
https://github.com/opencart/opencart/issues/3185

So just to finish the install on this I had to go into two different php files and correct two very basic mistakes. Not filling me with confidence. Thank god I only have to install it, not actually setup/maintain it.

FYI the person running opencart is a straight up rear end in a top hat who antagonizes security researchers, so when people find things they tend to release it 0-day. There's a "community edition" of it that seems to be run by people who know not to bite the hands that feed them.

I'd advise not running opencart if your friend doesn't want to get hacked.

Storysmith
Dec 31, 2006

Scaramouche posted:

"We still want to keep our OpenCart site so can you do any improvements there first?"

(nota bene: this is not the opencart thing I mentioned in my previous post. Yes, there's two of them)

"No."

Just point to all the large companies using opencart for their ecommerce, like, uh

Exactly.

Storysmith
Dec 31, 2006

Wicaeed posted:

Currently pissing me off: The entire DevOps movement, and developers having a say in what the IT department uses for ANY solution.

Taking a quote from these guys website:


Basically they use a database to store poo poo, but it's custom so it's cool

Looking at their product, we'd basically have to take everything that Zabbix does out of the box and completely write something from the ground up to do what we want.

Sounds like the issue is your boss's boss springing this on you.

Without knowing the details of both stacks, you realize time series databases are basically the heart of anything that generates and consumes metric data, right? From the days of rrdtool to all the open source stuff underpinning the options you mention?

Developer buy-in on a platform is important. We had graphite and collectd providing box-level stats for almost a year before some of the devs tried pushing application metrics to it and standing up grafana for better dashboards, and the usefulness of graphite instantly tripled.

Contextualizing what the application is doing and the impact it has on the database is a lot easier when devs and ops people are using the same tool. And that's the fault of whoever is sending people chasing after ELK and Splunk simultaneously or two different monitoring/metrics platforms at once.

Adbot
ADBOT LOVES YOU

Storysmith
Dec 31, 2006

poo poo pissing me off: anyone run into a Windows CA refusing to read/process a CSR generated by openssl? Something changed somewhere along the line between when these certs were originally issued and now, when I have to renew them 3 years later, and it looks like the CA just thinks our CSRs are garbage. These aren't too complicated: sha256-signed requests for an internal fqdn and several subjectAltNames for them. ('wiki.company.local', 'wiki')
The only things that changed that I can think of is moving from sha1 to sha256, and migrating from one machine with the CA service to another. But we've gotten requests off of the new machine before, when created through the wizard or whatever.

I don't do Windows, and the Windows admin doesn't really do much Linux, so we're at a bit of an impasse.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply