Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
guppy
Sep 21, 2004

sting like a byob

orange sky posted:

What

How

Was it sinkholing the entire network?

If unchecked, looping up a switch will cause a broadcast storm that will eventually overwhelm the network. Spanning-Tree Protocol should catch and prevent the problem, although depending on what exactly is at the edge, there are cases where it won't.

Adbot
ADBOT LOVES YOU

Sepist
Dec 26, 2005

FUCK BITCHES, ROUTE PACKETS

Gravy Boat 2k
It still happens. 2 yeare ago I was working on a customer project to upgrade an aged HP campus with Cisco switches. Days before the upgrade someone looped through a VoIP switch and brought the whole network down. Their network admin had disabled spanning tree globally years before.

BaseballPCHiker
Jan 16, 2006

Year and a half ago we were onboarding a new client. Our help desk manager came on site because she wanted the customers to get to know her a bit better. She was a great manager, not a great tech.

So shes trying to be helpful with some computer moves, and somehow someway ends up in the back networking closet where she misidentifies a cable and plugs the new Cisco 2960x that was to be their new layer 3 switch into some janky non enterprise HP switch they already had and looped the network. That was at least a quick fix and the damage was more to our reputation than anything else.

bull3964
Nov 18, 2000

DO YOU HEAR THAT? THAT'S THE SOUND OF ME PATTING MYSELF ON THE BACK.


Many moons ago our small office didn't have managed switches so they didn't have spanning tree loop detection and one of our users somehow managed to bridge her physical network adapter in her notebook and the wifi card. That was a fun one to track down without any management tools on the switch level.

Defenestrategy
Oct 24, 2010

Is there a reason not to implement Spanning Tree? Seems like one of those things that just prevents bad poo poo and shouldn't add latency?

12 rats tied together
Sep 7, 2006

Most commonly is probably laziness, or maybe it was implemented poorly and caused an outage.

You might also be doing some kind of multipath routing which IIRC does not play nicely with spanning tree.

e: The single network engineer job I interviewed for, and got, the interview was about 99% spanning tree questions with a side of NAT and then when I started, my first project was to implement spanning tree because the previous 4 network guys did not want to.

Sickening
Jul 16, 2007

Black summer was the best summer.

Defenestrategy posted:

Is there a reason not to implement Spanning Tree? Seems like one of those things that just prevents bad poo poo and shouldn't add latency?

Unfounded network engineer pride.

klosterdev
Oct 10, 2006

Na na na na na na na na Batman!

PancakeTransmission posted:

Like other than emergencies... It's cheaper for me to grab a new cable than spend my time crimping. And we have contractors do it all anyway.
Whenever I have to run a cable in any room with a patch panel I usually cut the cable to length to avoid contributing to a gordian knot of a couple dozen cables looped back and forth a bunch of times where the cables are run. If you have a pre-made cable lying around you can also save time by cutting that cable to length and only crimping the cut end.

PancakeTransmission posted:

I've heard stories like this "we don't run Spanning Tree because our admins know how to avoid loops"... But they didn't account for those dastardly users

The most dangerous kind of user is one who thinks they know what they're doing because they did X in their home setup. Only thing worse than having to visit a site downed by an employee deciding to DIY it is having to go there again because they didn't learn their lesson the first time. They were lucky spanning tree was enabled on that site's core switch and they only downed their local work area.

BaseballPCHiker
Jan 16, 2006

Defenestrategy posted:

Is there a reason not to implement Spanning Tree? Seems like one of those things that just prevents bad poo poo and shouldn't add latency?

Nope.

Most common scenario I see is business buying cheap unmanaged network gear and outgrowing it quickly.

Sepist
Dec 26, 2005

FUCK BITCHES, ROUTE PACKETS

Gravy Boat 2k
People disable spanning-tree when they don't set the root priority right and suddenly a random gently caress-all switch connected at 100/Half is now responsible all L2 forwarding

Sepist
Dec 26, 2005

FUCK BITCHES, ROUTE PACKETS

Gravy Boat 2k
Jesus, we were paying AWS 12k/yr per FW instance for software bundles...meanwhile a year ago they bought 3 year licenses from a VAR to go BYOL. Converting the firewalls now saving the company 60k/yr....

GnarlyCharlie4u
Sep 23, 2007

I have an unhealthy obsession with motorcycles.

Proof

PancakeTransmission posted:

Like other than emergencies... It's cheaper for me to grab a new cable than spend my time crimping. And we have contractors do it all anyway.

I've heard stories like this "we don't run Spanning Tree because our admins know how to avoid loops"... But they didn't account for those dastardly users

Defenestrategy posted:

Is there a reason not to implement Spanning Tree? Seems like one of those things that just prevents bad poo poo and shouldn't add latency?

We did not have spanning tree configured properly.
So when someone plugged a Dell Powerconnect from like 1995 into the core Cisco switch it decided that it was going to become the root bridge and ALLLLLLLLLL the arp tables got hosed.
And continued to get hosed.
And got hosed some more.
The Cisco's just said "well okay I guess... good luck little buddy!" and the entire network poo poo the bed.
We thought there was a loop somewhere so we just started unplugging everything until I noticed that there was a random switch on my boss' desk with a goodwill $1.99 sticker on it, and I asked him where the gently caress it came from and how long it had been plugged in. He admitted it was plugged in not that long and didn't think to connect it to the MAJOR NETWORK OUTAGE that started moments after he plugged it in..

GnarlyCharlie4u fucked around with this message at 21:07 on Sep 13, 2019

Methanar
Sep 26, 2013

by the sex ghost

GnarlyCharlie4u posted:

We did not have spanning tree configured properly.
So when someone plugged a Dell Powerconnect from like 1995 into the core Cisco switch it decided that it was going to become the root bridge and ALLLLLLLLLL the arp tables got hosed.
And continued to get hosed.
And got hosed some more.
The Cisco's just said "well okay I guess... good luck little buddy!" and the entire network poo poo the bed.

the cool thing about stp is if you don't set a root bridge explicitly the protocol makes a determination by mac address with the smallest value, which is probably something really old.

GnarlyCharlie4u
Sep 23, 2007

I have an unhealthy obsession with motorcycles.

Proof

Methanar posted:

the cool thing about stp is if you don't set a root bridge explicitly the protocol makes a determination by mac address with the smallest value, which is probably something really old.

this is exactly what happened.
That Powerconnect was by far the oldest piece of network equipment in the entire building, and the Ciscos said "okay cool, you're the boss now"

e: I think it was a Dell Powerconnect 2324 so that would have been from around 2003

GnarlyCharlie4u fucked around with this message at 21:11 on Sep 13, 2019

mllaneza
Apr 28, 2007

Veteran, Bermuda Triangle Expeditionary Force, 1993-1952




12 rats tied together posted:

Most commonly is probably laziness, or maybe it was implemented poorly and caused an outage.

You might also be doing some kind of multipath routing which IIRC does not play nicely with spanning tree.

Or STP has memory leaks that will brick an sg330 until you power cycle it.

Woof Blitzer
Dec 29, 2012

[-]
Turn on BPDU guard fools.

nullfunction
Jan 24, 2005

Nap Ghost
Deploying software on Friday the 13th with a full moon tonight, how's everyone else finishing their week out?

SyNack Sassimov
May 4, 2006

Let the robot win.
            --Captain James T. Vader


Woof Blitzer posted:

Turn on BPDU guard fools.

100% this, I am constantly shocked by how many people don't have BPDU guard on. If a switch shouldn't plug into that port, it shouldn't be allowed to connect to that port.

(obviously I'm not talking about a network built of Dlinks I mean if you have proper switches).

Sepist
Dec 26, 2005

FUCK BITCHES, ROUTE PACKETS

Gravy Boat 2k

nullfunction posted:

Deploying software on Friday the 13th with a full moon tonight, how's everyone else finishing their week out?

Made a bunch of production BGP changes to 6 firewalls at 4:57 then immediately closed the laptop

Exit Strategy
Dec 10, 2010

by sebmojo

nullfunction posted:

Deploying software on Friday the 13th with a full moon tonight, how's everyone else finishing their week out?

Xcopy of a software engineer's drives because system won't boot and he's backup noncompliant and a release is coming Tuesday. Started it and walked out the door.

Bonzo
Mar 11, 2004

Just like Mama used to make it!
I'd love to get some opinions here.

An acquaintance of mine now works for my company but in another regional office. Their title (at least in Outlook) shows as Sr Account Exec but on Linkedin they've given themselves the title of Regional Account Director.

Is it strange that it bothers me? I mean, everyone lies on their resume but when you have the actual job, why lie to the public?

in a well actually
Jan 26, 2011

dude, you gotta end it on the rhyme

Bonzo posted:

I'd love to get some opinions here.

An acquaintance of mine now works for my company but in another regional office. Their title (at least in Outlook) shows as Sr Account Exec but on Linkedin they've given themselves the title of Regional Account Director.

Is it strange that it bothers me? I mean, everyone lies on their resume but when you have the actual job, why lie to the public?

Lol if your hr title matches your internal title matches your external title

Methanar
Sep 26, 2013

by the sex ghost

nullfunction posted:

Deploying software on Friday the 13th with a full moon tonight, how's everyone else finishing their week out?

Discovering that chef roles and runlists are entirely adhoc managed on 6 different chef clusters with zero source control for it at all and when I did a dumb knife role edit hack, that I only meant to solve the problem for 10 minutes while I figure out the real way its done here, I was actually doing it the way everyone else has ever done anything.

nullfunction
Jan 24, 2005

Nap Ghost

Methanar posted:

Discovering that chef roles and runlists are entirely adhoc managed on 6 different chef clusters with zero source control for it at all and when I did a dumb knife role edit hack, that I only meant to solve the problem for 10 minutes while I figure out the real way its done here, I was actually doing it the way everyone else has ever done anything.

See? You're already up to speed with your new job and have nothing to worry about.

monsterzero
May 12, 2002
-=TOPGUN=-
Boys who love airplanes :respek: Boys who love boys
Lipstick Apathy

PCjr sidecar posted:

Lol if your hr title matches your internal title matches your external title

Yeah, but it's only because management gave up and made everyone an analyst

Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.

Agrikk posted:

“Agrikk,

Can you please tell me the IP addresses that my Aurora cluster could be? I want to set up host DNS records for each up address so that I can ping them to determine which one is active so I can access it from my workloads if there is some kind of failure.”

Real email from a customer.


Edit: oh and the Aurora database was launched as multi-AZ into three /20 subnets.
I had a customer email me needing to know the physical address where their records are stored for some Medicare 855B form. They're stored on AWS. :shrug:

Woof Blitzer
Dec 29, 2012

[-]
Would be really great to not have to use 4 different chat programs to communicate to different teams.

CPColin
Sep 9, 2003

Big ol' smile.
Did everything finally stop letting people connect via various Pidgin plugins? I remember a time at an old job when people were split between, I think, Yahoo!, GChat, and either AIM or MSN. That was entertaining.

Internet Explorer
Jun 1, 2005





There has to be some theory or law that states matter-of-factly that the lower the TTL on DNS entries the longer lovely clients will keep it cached.

<insert it_was_dns.jpg>

mllaneza
Apr 28, 2007

Veteran, Bermuda Triangle Expeditionary Force, 1993-1952




nullfunction posted:

Deploying software on Friday the 13th with a full moon tonight, how's everyone else finishing their week out?

We did ! A few hundred Acronis clients (15% success), 600 SEP/Fireeye pushes (26%), and 55 machines with KACE got their first Microsoft security patches in years (65%) as a proof of concept. Nothing broke !

Before anyone asks, WSUS and SCCM are both outdated and horribly broken on the legacy domain. That's why I'm writing PowerShell code to do software deploys for a Fortune 100 company in 2019.

FISHMANPET
Mar 3, 2007

Sweet 'N Sour
Can't
Melt
Steel Beams
So I could have sworn we were talking about rundeck here but maybe not, but it reminded me of a problem I have.

There are a lot of customer requests I can easily script (reboot a machine, add memory to a vm, remove this person from a group) but I want a way to present this to customers. For a few process we're using Google Forms that plug into an Azure Automation runbook that does some magic to validate that someone is authorized to get what they've requested, but it's not very elegant.

I'm imagining some kind of front end where someone logs in and they see what they have access to and can just point and click their request and then it gets fired off into our automation.

I don't need an automation engine, I've got that, I want some kind of user facing portal to present it to customers. I don't even know what to call what I'm looking for to begin searching for it. I'm contemplating setting up a SharePoint site to see if it can do it, but I'm not even sure it's complex enough for what I want.

Docjowles
Apr 9, 2009

Yeah that is exactly what rundeck is for. The huge caution I would give is that it is extremely cumbersome/confusing to configure. It desperately needs someone who knows what the hell they’re doing to do a UI/UX overhaul. Also documentation. It’s a great irony that a tool to make Ops’ life easier is a huge pain in the dick to operate.

But if the people using the thing as opposed to configuring it are able to click buttons it should mostly be fine.

Methanar
Sep 26, 2013

by the sex ghost

FISHMANPET posted:

So I could have sworn we were talking about rundeck here but maybe not, but it reminded me of a problem I have.

There are a lot of customer requests I can easily script (reboot a machine, add memory to a vm, remove this person from a group) but I want a way to present this to customers. For a few process we're using Google Forms that plug into an Azure Automation runbook that does some magic to validate that someone is authorized to get what they've requested, but it's not very elegant.

I'm imagining some kind of front end where someone logs in and they see what they have access to and can just point and click their request and then it gets fired off into our automation.

I don't need an automation engine, I've got that, I want some kind of user facing portal to present it to customers. I don't even know what to call what I'm looking for to begin searching for it. I'm contemplating setting up a SharePoint site to see if it can do it, but I'm not even sure it's complex enough for what I want.

Give everyone full root to every system they need to make changes to.

Why are you IT people always getting in the way.

TheParadigm
Dec 10, 2009

FISHMANPET posted:


I don't need an automation engine, I've got that, I want some kind of user facing portal to present it to customers. I don't even know what to call what I'm looking for to begin searching for it. I'm contemplating setting up a SharePoint site to see if it can do it, but I'm not even sure it's complex enough for what I want.

Isn't that like, a 'service control panel?' back in the day, forums, etc always used to have a page with all the admin/special functions on it.

are you wondering what to call it, how to spin it, or both?

orange sky
May 7, 2007

What's your itsm platform? Doesn't it have some self service module?

Sickening
Jul 16, 2007

Black summer was the best summer.
I just had a network lead mention pots line in a design diagram for a totally new build out.

How upset should I be?

ChubbyThePhat
Dec 22, 2006

Who nico nico needs anyone else

Sickening posted:

I just had a network lead mention pots line in a design diagram for a totally new build out.

How upset should I be?

Giving massive benefit of the doubt, could this be for an alarm line or something?

Sickening
Jul 16, 2007

Black summer was the best summer.

ChubbyThePhat posted:

Giving massive benefit of the doubt, could this be for an alarm line or something?

Nope! They are Labeled (pots backup connections for emergency out of band management)

This is the DFW metro area. The land of every cell carrier know to mankind. Pots though.

Internet Explorer
Jun 1, 2005





We still have POTS lines for faxing. We're going to be moving in a few years and the plan was to move them....???

Adbot
ADBOT LOVES YOU

FISHMANPET
Mar 3, 2007

Sweet 'N Sour
Can't
Melt
Steel Beams

Docjowles posted:

Yeah that is exactly what rundeck is for. The huge caution I would give is that it is extremely cumbersome/confusing to configure. It desperately needs someone who knows what the hell they’re doing to do a UI/UX overhaul. Also documentation. It’s a great irony that a tool to make Ops’ life easier is a huge pain in the dick to operate.

But if the people using the thing as opposed to configuring it are able to click buttons it should mostly be fine.

From what I've been able to find rundeck would be very good to give to a T2 team to run automation a T3 team has written for them, but I'd want potentially non-technical users to be able to access it. Maybe that's what you mean by UI, but from what I've seen the UI would be a deal breaker.

TheParadigm posted:

Isn't that like, a 'service control panel?' back in the day, forums, etc always used to have a page with all the admin/special functions on it.

are you wondering what to call it, how to spin it, or both?

Mostly at this point what to call it, because maybe there's a name for it and there's a whole pile of solutions in that space, but I can't find them because I don't know what it's called.


orange sky posted:

What's your itsm platform? Doesn't it have some self service module?

Currently ServiceNow, but we can't get access to use that. We're in an RFP for a new ITSM product and so I don't know what the future product will be or its capabilities or my level of access to it. Access to the tool is a political problem that I may be able to solve on my own, but there's also a bit of chicken & egg problem where I don't know if I could do what I want in ServiceNow because I don't have access to try it, and I don't want to burn a bunch of energy to get access to it only to learn that it won't do what I want.

I also don't want to get too caught up in specific use cases and go down a rabbit hole of products just for that. For example, if I'm only talking about modifying VMs then probably most of what I want could be accomplished through some unholy combination of System Center Service Manager and System Center Virtual Machine Machine Manager. But I also have a use case of non-technical people modifying group membership, which could be done with Microsoft Identity Manager. But then I also have use cases of runbooks that run powershell to generate some kind of report on demand and send it to someone, and I'd like an easy way for someone to go in and click a button to generate that report.

FISHMANPET fucked around with this message at 15:13 on Sep 18, 2019

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply