Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
MJP
Jun 17, 2007

Are you looking at me Senpai?

Grimey Drawer

Docjowles posted:

Related, The Infinite Hows (or, the Dangers Of The Five Whys) by John Allspaw is a great (if very long) blog post on the topic of building a healthy culture around outages and mistakes.

Thank you for posting this. My last job had a boss who did just this and it wasn't just him - whenever something failed, or whenever a change went wrong, it was a witch hunt. Since the company still had pensions and there was a very real Warsaw Pact-ish environment between ops and engineering (Engineering being the USSR, Ops/Helpdesk and other internal IT groups being Poland, Lithuania, etc.) there was always an attempt at a blame game to pin it on some other gomer below you, or someone else, and to ensure that they got the scutwork of cleaning things up to prevent Rare Issue X from ever occurring.

It's good to know that other methodologies exist - we don't do this kind of blame-pinning where I am, but we have other organizational problems. Either way, promoting this kind of thinking - "how" vs. "why" - is a very good thing.

Adbot
ADBOT LOVES YOU

mayodreams
Jul 4, 2003


Hello darkness,
my old friend
Does anyone have any experience working for a National Laboratory / Department of Energy? I got a message from a recruiter about a job at Argonne and I've been really interested in working at National Lab for a long time.

Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.

mayodreams posted:

Does anyone have any experience working for a National Laboratory / Department of Energy? I got a message from a recruiter about a job at Argonne and I've been really interested in working at National Lab for a long time.
If it's anything like Brookhaven National Lab, it's a organizational trainwreck where you occasionally get to play with hardware an order of magnitude more powerful than anything you've ever seen in your life. If you're the kind of person who feeds on being surrounded by people of other disciplines who are way, way smarter than you, it might be your kind of place. Otherwise, it's a government IT job with all the frustrations of academia.

Gucci Loafers
May 20, 2006

Ask yourself, do you really want to talk to pair of really nice gaudy shoes?


Has anyone read about the technical-side of the Sony hack?

Was their IT Department terribly run and cut too many corners when they shouldn't have or was this something amazing like Stuxnet?

adorai
Nov 2, 2002

10/27/04 Never forget
Grimey Drawer

Tab8715 posted:

Was their IT Department terribly run and cut too many corners when they shouldn't have or was this something amazing like Stuxnet?
If I were to guess, they had little security once you were inside, and someone fell for a social engineering ploy.

Sickening
Jul 16, 2007

Black summer was the best summer.

Tab8715 posted:

Has anyone read about the technical-side of the Sony hack?

Was their IT Department terribly run and cut too many corners when they shouldn't have or was this something amazing like Stuxnet?

I did read something about text files with passwords so I assume it wasn't really mission impossible stuff.

MJP
Jun 17, 2007

Are you looking at me Senpai?

Grimey Drawer

adorai posted:

If I were to guess, they had little security once you were inside, and someone fell for a social engineering ploy.

What I want to know is where did North Korea get anyone with experience in disparate networks outside the country? Do they have people they trust enough to be doing IT for the government that could go into pentesting/actual penetration well enough to pull this off? Or did they somehow get China to loan them the technical corpus to pull it off?

Bhodi
Dec 9, 2007

Oh, it's just a cat.
Pillbug
NK has a tech arm of the government that is segregated from the local public and treated like kings. There's also the possibility of someone external finding access and selling a map and how-to to NK. There aren't any details either way as yet.

adorai
Nov 2, 2002

10/27/04 Never forget
Grimey Drawer

MJP posted:

What I want to know is where did North Korea get anyone with experience in disparate networks outside the country? Do they have people they trust enough to be doing IT for the government that could go into pentesting/actual penetration well enough to pull this off? Or did they somehow get China to loan them the technical corpus to pull it off?
I think you overestimate the difficulty of penetrating a corporate network. A script kiddie trojan embedded in a word document could be enough. Once inside, most corporate networks have plenty of weaknesses.

Griffon
May 14, 2003

Tab8715 posted:

Has anyone read about the technical-side of the Sony hack?

Was their IT Department terribly run and cut too many corners when they shouldn't have or was this something amazing like Stuxnet?

The CIO was quoted as saying that he wouldn't spend 10 million to protect 1 million.

http://mashable.com/2014/12/05/sony-hack-infosec-comments/

And they're saying the hack could cost $100 Million after the $171 Million loss on the playstation hack.

http://blogs.wsj.com/cio/2014/12/10/the-morning-download-sony-breach-could-cost-100-million/

I'm not sure the measure of the damage done to the relationships with Sony. Things like calling Angelina Jolie a spoiled brat now being public:
http://www.usmagazine.com/celebrity-news/news/sony-hack-key-events-from-leaked-emails-terror-threats-20141812

Griffon fucked around with this message at 21:37 on Dec 18, 2014

Paladine_PSoT
Jan 2, 2010

If you have a problem Yo, I'll solve it

I'm significantly more interested in what NK Hadn't yet released that caused Sony to cave.

Proteus Jones
Feb 28, 2013



psydude posted:

What. People will dig in and stay put in every job in every industry. I have no idea about application development because my experience with developers is limited to yelling at them for making their applications insecure as gently caress, but moving up rapidly in the networking world is insanely easy because there's such a huge demand for the skillset relative to the labor supply. In the end, it comes down to what you want to do: if you're interested in development, then obviously pursuing a development position makes sense. If you like networking, then pick up the CCNA book and start cranking. But get it out of your head that there's "little room for advancement" anywhere in this industry. It just takes you getting out of your comfort zone and being open to moving between companies and even geographic locations.

Can confirm there's an insane amount of career advancement in the network side.

Gucci Loafers
May 20, 2006

Ask yourself, do you really want to talk to pair of really nice gaudy shoes?


Sounds like it was more of combination of a massive government and a poorly run IT Department. Sounds like they'll have some job openings.

I did see some mentions of storing plain text passwords but all the Sony Leak discussion has been mostly removed from reddit.

angry armadillo
Jul 26, 2010
Am I the only one that's skeptical that it's the real NK making these threats

3 Action Economist
May 22, 2002

Educate. Agitate. Liberate.
Ugh.

I forgot about all the stupid "Come sell our bullshit insurance!" hits I'd get once I posted my resume.

angry armadillo posted:

Am I the only one that's skeptical that it's the real NK making these threats

Nope. That was my very first thought.

MagnumOpus
Dec 7, 2006

angry armadillo posted:

Am I the only one that's skeptical that it's the real NK making these threats

I'm skeptical about this attack being perpetrated by the NK gov't on account of it being quite a bit more than their usual saber rattling. That said, I am not skeptical of NK's ability to perpetrate complex cyberwarfare attacks. The NK gov't is not poor, and they spend a shitload of cash on a cyberwarfare program that includes putting the members up in luxury accomodations after they come back from 1-2 years of hands-on training with Chinese/Russian criminal hacker rings. Last year their attacks on SK businesses cost the SK economy $800M that we know of. The assymetric nature of cyber warfare makes it a great place to spend money, and they are actually doing it with great focus.

Proteus Jones
Feb 28, 2013



Colonial Air Force posted:

Ugh.

I forgot about all the stupid "Come sell our bullshit insurance!" hits I'd get once I posted my resume.


Yeah, but those knives basically sell themselves.

BurgerQuest
Mar 17, 2009

by Jeffrey of YOSPOS
My last day has arrived. Feels good. I will miss the work social group, tomorrow is going to be a very bad hangover.

Zero VGS
Aug 16, 2002
ASK ME ABOUT HOW HUMAN LIVES THAT MADE VIDEO GAME CONTROLLERS ARE WORTH MORE
Lipstick Apathy
This is awesome, live hacking attack map: http://map.ipviking.com/?_ga=1.98376799.153405815.1403529861

Right now you can see people hammering St. Louis, even their computers can't catch a break.

adorai
Nov 2, 2002

10/27/04 Never forget
Grimey Drawer
anyone doing cool stuff with Logstash? I am mostly interested in how you collect data in a meaningful way when it comes from a variety of disparate sources? Do you try to sort out everything with input rules and match statements, or do you just say gently caress it and run multiple collection servers?

QuiteEasilyDone
Jul 2, 2010

Won't you play with me?

Zero VGS posted:

This is awesome, live hacking attack map: http://map.ipviking.com/?_ga=1.98376799.153405815.1403529861

Right now you can see people hammering St. Louis, even their computers can't catch a break.

[Nuclear launch detected]

Kinda coasting right now, reaching out to Sysadmin positions on Indeed and seeing if anything sticks.

Methanar
Sep 26, 2013

by the sex ghost

MagnumOpus posted:

I'm skeptical about this attack being perpetrated by the NK gov't on account of it being quite a bit more than their usual saber rattling. That said, I am not skeptical of NK's ability to perpetrate complex cyberwarfare attacks. The NK gov't is not poor, and they spend a shitload of cash on a cyberwarfare program that includes putting the members up in luxury accomodations after they come back from 1-2 years of hands-on training with Chinese/Russian criminal hacker rings. Last year their attacks on SK businesses cost the SK economy $800M that we know of. The assymetric nature of cyber warfare makes it a great place to spend money, and they are actually doing it with great focus.

Why not just physically cut the cables leading to the country at that point?

Docjowles
Apr 9, 2009

Got a random LinkedIn message from a company my coworker recently left for. I love pretty much everything about my current job and employer. But said coworker gave me some info on his new compensation package and I'd be insane not to talk to them. It would be for a corporation at least 2 orders of magnitude larger than anywhere I've ever worked (startup lyfe :respek:) which makes me nervous. Current job offers extreme schedule flexibility which is worth a lot to me in the "having babies" phase of life. On the flip side my wife mostly stays at home so a major pay bump would really make a difference.

Surprise YOTJ in 2015? We shall see.

MagnumOpus
Dec 7, 2006

Methanar posted:

Why not just physically cut the cables leading to the country at that point?

NK situation is politically complex and volatile. That said, NATO just this past September released a new "enhanced" policy that specifically includes cyberwarfare within the definition of armed attacks that allow for Article 5 to be invoked. That's the one where NATO is required to go all-in with the attacked member state if war is declared. So like, who knows what will happen.

I assume that a lot of NK physical infrastructure routes through China though so they'd have to be in on it to completely shut out NK. Then there's still options on the black markets and with shady private companies that would be willing to give them access outside access via satellite or other means. NK can pretty much do any kind of shady poo poo they want because they are a rogue nation with a gun to the head of a western ally, and the US and China are stuck forever bargaining to keep them from pulling the trigger.

skooma512
Feb 8, 2012

You couldn't grok my race car, but you dug the roadside blur.

psydude posted:

What. People will dig in and stay put in every job in every industry. I have no idea about application development because my experience with developers is limited to yelling at them for making their applications insecure as gently caress, but moving up rapidly in the networking world is insanely easy because there's such a huge demand for the skillset relative to the labor supply. In the end, it comes down to what you want to do: if you're interested in development, then obviously pursuing a development position makes sense. If you like networking, then pick up the CCNA book and start cranking. But get it out of your head that there's "little room for advancement" anywhere in this industry. It just takes you getting out of your comfort zone and being open to moving between companies and even geographic locations.

Ah, I should have been more precise instead of rambling like I did. I was referring to network jobs within this particular account. Obviously I would do far better on the open market. However, I like where I'm at because the people are likable and the drama is minimal.

Tab8715 posted:

Huh?

Currently, you don't have the application expertise and it sounds like they're going to train you. If they do switch applications, what makes you think they'll lay you off as opposed to just re-training you onto something different?

Granted, lets say they do lay you off. If it's some specialized application it's not like you can't apply at another hospital that does use it.

They absolutely are going to train me, though my would be predecessor found the training and support to be lacking, especially at first.


After talking with my manager about it, I'm going to give it a shot. They made it very clear that I'm not expected to take the position simply because it's offered.

She also said the networking manager likes me and wouldn't mind having me on the team once things settle down :unsmith:. I'd obviously need to get educated before any of that happens, but it's nice to know I'm making a good impression overall.

Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.

adorai posted:

anyone doing cool stuff with Logstash? I am mostly interested in how you collect data in a meaningful way when it comes from a variety of disparate sources? Do you try to sort out everything with input rules and match statements, or do you just say gently caress it and run multiple collection servers?
It depends on how your data is coming in. If you're mostly dealing with bog-standard UDP syslog and trying to make it act like Splunk with rules and matchers, good luck. Generally, you'll have much better luck running the agents where you need them and making sure the data coming into Logstash is structured JSON to begin with.

On the other hand, if your applications have the ability to log structured JSON in the first place, you'll do much better. For this reason, Logstash seems to work a lot better with in-house apps than off-the-shelf stuff. For random unstructured logs from every device in your infrastructure, I'd shell out the money for Splunk if it was important.

Mr Shiny Pants
Nov 12, 2012

Che Delilas posted:

You've expressed some incredulity that this kind of stuff is a real job that you get paid relatively decent money for. I'm going to try and shed some light on this for you, because people like us (techs, nerds, smart people) tend to get down on themselves for not knowing enough and don't appreciate that they have real skills. The TLDR is that we have internalized a lot of little things that allow us to work with computers on a level that most people never get to.

The first major point is that we are computer literate, while the vast, vast majority of people are not. Computer literacy as I define it is basically the ability to figure out how to do something on a computer that you haven't done before. If we want to do something new, we use our heads, we think critically, we categorize, we rule things out. Perhaps you've seen this: http://xkcd.com/627/, which while simplified for the sake of humor, is a pretty damned good summary.

Most people never get there, they learn specific procedures by rote memorization; Creating a Word document. Printing a Word document. Logging off for the day. Checking their email. Running this month's payroll. If they want to learn anything new, they must have an illustrated step-by-step tutorial for the exact version of the thing they're using and God help them if a single screenshot is out of date.

The instant anything goes wrong, these people go into vapor lock. If they follow their normal steps (or if they think they did) and they don't get the same result they always get, they don't know what to do except try the same thing again. If there's an error message that pops up, they panic. Even if the message tells them exactly what to do, they are lost because it's something they've never done before. Googling the error message won't help them because their minds are not prepared to process the information (I'm generalizing here, many people have some ability to look beyond what they know, but a shockingly large number just do not). This segues nicely into my next point...

Googling it is a skill. It's hard for us to think of it that way because we do it all the time and we don't think about it. Googling the problem or error message sounds simple, but that's not the only thing you're doing. When you google an error message or problem, in your head you are taking dozens of little details about the problem space and sorting and prioritizing them. What's the program where the error is occurring? What is the user trying to do exactly? What operating system are they running? Can the problem be hardware related? If yes, what hardware? What model of that specific hardware are they using? Has this error happened before? Does it happen at regular intervals and/or specific times? What antivirus are they running? What web browser are they using?

You gather some of that information (depending on the kind of problem you're looking at) and you construct a google query. You're able to do that extremely rapidly because you have used computers a lot, and you're familiar with what information is likely to be relevant. It's very nearly automatic. There's a world of difference between "My computer is slow" and "Google Chrome 38 youtube video stutter and mouse latency" as search terms. Terrible example, but my point is that you have the ability to analyze, think critically, and quickly process and sort information about these devices and their problems, which are real skills that most people never acquire.

I just started reading this thread and I know it's from way back but I just want to say that this is wonderful post. Thanks!

Mr Shiny Pants
Nov 12, 2012

Misogynist posted:

It depends on how your data is coming in. If you're mostly dealing with bog-standard UDP syslog and trying to make it act like Splunk with rules and matchers, good luck. Generally, you'll have much better luck running the agents where you need them and making sure the data coming into Logstash is structured JSON to begin with.

On the other hand, if your applications have the ability to log structured JSON in the first place, you'll do much better. For this reason, Logstash seems to work a lot better with in-house apps than off-the-shelf stuff. For random unstructured logs from every device in your infrastructure, I'd shell out the money for Splunk if it was important.

Talking about Logstash: Anyone know how to get ElasticSearch to format the time right? It is always two hours off. I've tried transforming it with the Logstash config and if I configure it to output to the console it transforms it to the right time, stuffing it into ElasticSearch it does not transform the date to add the two hours.

I've searched around and it seems I am not the only one with this problem but so far a solution has not been forthcoming.

Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.

Mr Shiny Pants posted:

Talking about Logstash: Anyone know how to get ElasticSearch to format the time right? It is always two hours off. I've tried transforming it with the Logstash config and if I configure it to output to the console it transforms it to the right time, stuffing it into ElasticSearch it does not transform the date to add the two hours.

I've searched around and it seems I am not the only one with this problem but so far a solution has not been forthcoming.
I'm not really following, so I'll need a little more information: where is the data coming from, what is Logstash showing you, what is Elasticsearch showing you, how are you viewing what's coming back from Elasticsearch, and what are you expecting to see?

Mr Shiny Pants
Nov 12, 2012

Misogynist posted:

I'm not really following, so I'll need a little more information: where is the data coming from, what is Logstash showing you, what is Elasticsearch showing you, how are you viewing what's coming back from Elasticsearch, and what are you expecting to see?

Data is from the Windows eventlog. It comes in as a JSON string that gets handled by Logstash. In my configuration I have told logstash to not do anything to the data and stuff it right into Elastic Search. This results in the time from the events being two hours off from our local timezone.

So after some searching I've told logstash to transform the date by adding two hours on top of the one it gets. Effectively compensating for Eastic Search's time being off.

If run Logstash in this configuration and tell it to output to stdout and also ElasticSearch the output to stdout is correct but the date that goes into elastic search is not transformed.

This is from memory, the configuration is at work. I'll see if I can get them this afternoon. Thanks.

I checked the ES dates with Kibana and also with Curl. I might be misremembering some things, it's been awhile. The machine has the right timezone though.

Mr Shiny Pants fucked around with this message at 11:22 on Dec 19, 2014

Che Delilas
Nov 23, 2009
FREE TIBET WEED

Mr Shiny Pants posted:

I just started reading this thread and I know it's from way back but I just want to say that this is wonderful post. Thanks!

You're welcome. I enjoy this kind of analysis, and people in general have a lot of problems putting themselves in someone else's shoes and looking back at themselves objectively. In the case of IT people, we're so good wit computer that we lose our appreciation for how significant it is.

Dark Helmut
Jul 24, 2004

All growns up
That flowchart from xkcd is magical. I don't know how I've never seen it, so thanks!

CloFan
Nov 6, 2004

Zero VGS posted:

This is awesome, live hacking attack map: http://map.ipviking.com/?_ga=1.98376799.153405815.1403529861

Right now you can see people hammering St. Louis, even their computers can't catch a break.

I want to make this into a live wallpaper, this is neat looking

E: Ah hahaha if you switch tabs in Chrome, when you switch back it will 'catch up' with all the attacks and mostly just looks like a crazy deluge on St. Louis

E2: I think something broke : http://i.imgur.com/msZbVsT.jpg

CloFan fucked around with this message at 15:55 on Dec 19, 2014

Erwin
Feb 17, 2006

Misogynist posted:

It depends on how your data is coming in. If you're mostly dealing with bog-standard UDP syslog and trying to make it act like Splunk with rules and matchers, good luck. Generally, you'll have much better luck running the agents where you need them and making sure the data coming into Logstash is structured JSON to begin with.

On the other hand, if your applications have the ability to log structured JSON in the first place, you'll do much better. For this reason, Logstash seems to work a lot better with in-house apps than off-the-shelf stuff. For random unstructured logs from every device in your infrastructure, I'd shell out the money for Splunk if it was important.

I just set up Logstash and this is what I don't get. They created a product that can do the job of Splunk with some tweaking, but seem utterly baffled when people choose to use it that way (not literally, but the documentation is geared towards Lumberjack only and basically ignores syslog). I mean I get it, its better for custom stuff where you can create JSON before sending to Logstash, but syslog is a thing that exists already. The filters I've found online for Cisco ASA syslogs don't work (grok failures), but that could be Cisco's fault for not being consistent I guess.

I really don't know what my point is other than it's so close to being a Splunk replacement, but is determined to do its own slightly different thing.

Richard Noggin
Jun 6, 2005
Redneck By Default

QuiteEasilyDone posted:

[Nuclear launch detected]

Kinda coasting right now, reaching out to Sysadmin positions on Indeed and seeing if anything sticks.

How about a nice game of chess?

Inspector_666
Oct 7, 2003

benny with the good hair

Dark Helmut posted:

That flowchart from xkcd is magical. I don't know how I've never seen it, so thanks!

I hate xkcd, I think it's an unfunny comic for elitists.

That said, I printed out that flowchart, pinned it up next to my mother's computer and said I wasn't going to help her with any issues unless she had exhausted each step.

She hasn't had to call me since.

ElGroucho
Nov 1, 2005

We already - What about sticking our middle fingers up... That was insane
Fun Shoe

Inspector_666 posted:

I hate xkcd, I think it's an unfunny comic for elitists.

Good, I'm not alone

Docjowles
Apr 9, 2009

I was definitely in the camp that thought there was no chance North Korea was actually behind the Sony hacks. It just seemed like too convenient of a scapegoat, and too batshit crazy even for NK.

Welp.

The FBI posted:

As a result of our investigation, and in close collaboration with other U.S. government departments and agencies, the FBI now has enough information to conclude that the North Korean government is responsible for these actions.

Zero VGS
Aug 16, 2002
ASK ME ABOUT HOW HUMAN LIVES THAT MADE VIDEO GAME CONTROLLERS ARE WORTH MORE
Lipstick Apathy

ElGroucho posted:

Good, I'm not alone

Yes but http://goatkcd.com/sfw (still kinda :nws:) is the best webcomic ever

Adbot
ADBOT LOVES YOU

mayodreams
Jul 4, 2003


Hello darkness,
my old friend
We are having some email challenges with vendor and customer email getting caught in our filtering appliance. Some VP's in Canada start bitching and put in a request to white list a few vendors. Since my coworker's didn't recognize the domain name, we whitelisted rogers.com.

So now I have to remove it and tell the bitchy VP's that we cannot, in fact, whitelist 1/3 of Canada's home email address. :smithicide:

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply