Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
BaseballPCHiker
Jan 16, 2006

Anyone have any advice on how to get a provider (Comcast) to fix their ENS connection for a few sites? Im having a hell of a time trying to get them to do anything about it, besides saying their device is up and the connection is fine.

Basically about 3 days ago at 4am three of our sites went down all at the same time. I rush out to the sites, and get a few of them back up either over old dark fiber or cellular backups. Looking into it more it seems that EIGRP is failing, the l3 switches will receive hellos, form adjacencies, then fail to send out Hellos successfully and collapse, over and over.

At first I thought maybe it was a cisco bug or something, so I update code and still nothing. Next thing I did was to grab a new l3 switch, right out of the box, and configure it with the same setup to take out to the site. Had the exact same issue. So this time I plug it in to our Comcast provided Cienna device and take a Pcap to send to them. It basically just shows EIGRP failing over and over. Whats interesting is if I run a show CDP neighbors I only see like 3 out of our 20+ ENS sites.

Because this is an ENS point to multipoint setup I should be able to run a show CDP neighbors and see all of my other sites. I bring this up to Comcast and they shrug and go yeah thats weird but dont really have an answer for me.

The other interesting wrinkle is that we cut off service at two other ENS sites the day before all of this happened. When I brought this up to Comcast to see if they may have accidentally terminated service to the wrong locations I got literally nothing out of them.

At this point our sites have been down for 3 days and I'm running out of patience. Any advice on what I could do to help move this along? At this point I am 99% convinced its on Comcasts end, I just dont know how to help them along.

Adbot
ADBOT LOVES YOU

BaseballPCHiker
Jan 16, 2006

It gets better!

The Comcast tech I sent all of the info to just decided to close the ticket randomly apparently without notifying us!!! So I got to open another ticket for another one of the sites that are down and go through the same thing. Started spamming our account reps too to see if that gets us anywhere.

BaseballPCHiker
Jan 16, 2006

Our account rep is useless. And his sales engineer guy, I dont even know where to begin. Its like he devoted his life to AppleTalk or something and lives in some bizarro networking world because nothing ever seems to sink in with them.

We opened another ticket for our other down site, CC'd reps demanded to know why our other ticket was closed without notifying us and now my boss is asking about SLAs and service credits. We also started saying that one of these sites is a fire station and that this could effect public safety. We'll see what comes out of it.

Thankfully between dark fiber and cellular routers no site is hard down. First time I've ever had an outage with Comcast like this. They've been great and solid for the 5+ years we've had them until now.

BaseballPCHiker
Jan 16, 2006

Well its finally fixed.... one week later.

We had to open multiple tickets, call multiple techs over and over and over again but we FINALLY got someone who knew what they were doing or just got lucky. Apparently it was in their words "An issue with the line card on our core router and cost the traffic."

I am beyond pissed. Thankfully we had backup connections to keep these sites somewhat up but we were told for days the issue was on our end. I want to rip into our Comcast rep and tear him to shreds. Myself and my team has spent so much time on this proving it was Comcast. We better be getting a whole lot of service credits for this.

BaseballPCHiker
Jan 16, 2006

Sir, this is a Wendys.

I have no idea, probably need to disable UEFI secure boot or something dumb like that.

Actual Cisco related content. Anyone have any recommendations on YouTube channels, books, etc to get started with Network automation?

BaseballPCHiker
Jan 16, 2006

Ha do we work together?!?

Same thing happened to me a few weeks ago. Basically provider was down enough to impact service but not hard down so that failover switched over. We looked into doing some sort of weighted routing but just couldnt seem to get it right and its so rare that something like that happens that we just moved back to a manual process.

BaseballPCHiker
Jan 16, 2006

I also feel like networking is a bit safer of a career than traditional sysadmin work in the long term, though I also think the demise of sysadmin work in general is over exaggerated.

BaseballPCHiker
Jan 16, 2006

Just get hired at some shitshow dumpster fire of a MSP or something and deal with peoples ancient, crap, setups. Do that for a bit, get your CCNA, and then move on up.

I usually advocate against degrees specifically for IT. Certs are such a better bang for your buck value, and if you actually learn the material instead of just learning to pass a test I think you'll be in a much better position as a network engineer then if you went to school for some generic IT degree.

BaseballPCHiker
Jan 16, 2006

Famethrowa posted:

I'm mostly afraid of going help desk/MSP because of the entry level pay cut, but if thats inevitable...:smith:

Thats a tough situation and I'm not sure if there is a way around it, maybe some others here have made a similar mid career jump. All of the places I've worked as a network engineer wouldnt let someone new with just a CCNA do to much. You really need to get your feet wet somewhere to get started and get a good feel. That said, I've never worked for a BIG company or for a provider, they may be more inclined to just throw you to the wolves as a new person.

BaseballPCHiker
Jan 16, 2006

Thanks Ants posted:

Watchguard, you still admin them through some awful windows app.

Barracuda tried to sell me on their 'brand new' range of boxes and the demo started with a Windows app to manage them, nope nope nope.

As bad as Watchguard is I think I'd actually prefer it to Sonicwall.

Sonicwall just seems to get you 80% of the way there before it fucks you over. At least with Watchguard the config will bomb out early or refuse to load at the beginning.

BaseballPCHiker
Jan 16, 2006

I think Tufin will do something like that but its been a while since I've dealt with that software. Might want to look into it though.

BaseballPCHiker
Jan 16, 2006

The next exam refresh for Cisco should include correspondence with TAC, and looking up bug reports.

I swear 1/2 of my last networking job was either fixing bugs I encountered with Cisco devices, or updating IOS on devices to avoid bugs I hadnt encountered yet.

BaseballPCHiker
Jan 16, 2006

Just to pile on, Im just a Cisco guy, but for Cisco we'd have dedicated stacking cables in the back of the switches, not using SFP or copper ports for that.

Are you calling trunk links stack ports? Or is that how HP does stacking?

Also, when I was looking into something similar last year, Cisco had yet to come out with a fiber aggregation switch in their 9000 series of switches. Not sure if thats still the case or not.

BaseballPCHiker
Jan 16, 2006

Bob Morales posted:

FS actually has this in stock while the Cisco stuff is all backordered. We have been waiting like 2 months on a whole new Cisco wireless system.

Don't know about FS's support. They seem to have a decent rep on Reddit (lol). At these prices I could buy spares.

Edit: No coffee yet

Their support is OK. Like not the worst, but not the best. Dont expect Ubiquiti bad or Cisco (TAC back when they were good) level support. When I had to interact with them they appeared to be out of China, so I could really only get a quick response right away in the morning, otherwise it was next day.

BaseballPCHiker
Jan 16, 2006

Network Warrior was good last I looked like 5-6 years ago. Not sure how recently that has been updated or how relevant it still is.

BaseballPCHiker
Jan 16, 2006

You’d probably be better off just reading the config guide for the models you have honestly.

Also I’m pretty sure 2960s go end of support this year.

BaseballPCHiker
Jan 16, 2006

I knew it had to be close to eol. I replaced about 100 in 2020 with 9000 series catalysts.

Some days I miss being in networking. But then again I haven’t woke up to support a site with a fiber cut in over a year.

BaseballPCHiker
Jan 16, 2006

uhhhhahhhhohahhh posted:

Also gently caress ASAs. Can't even lab this properly because they aren't VRF aware and if I put BGP on them it'll start advertising routes between VRFs.

Isnt EOL for ASAs rapidly approaching? Like sometime in 2022?

Last I knew you could buy the new Firepower hardware and run ASA code on them still to buy you enough time to switch over to something besides a Cisco FW product.

BaseballPCHiker
Jan 16, 2006

GreenNight posted:

I have a stack of a dozen or so 2900 series and 3560-x series switches on the shelf.

Looks like 3560cgs are going for about $100 a pop! I have an old one thats just been sitting in my basement.... might be time to flip it.

BaseballPCHiker
Jan 16, 2006

Filthy Lucre posted:

Internal core here. The only internal traffic that would go through the firewall is traffic to/from the DMZ.

This is how I've always done it as well. You pay more money for L3 switches for a reason, let them handle routing as necessary.

BaseballPCHiker
Jan 16, 2006

I've just used Forescout personally. It worked well enough for me when I used it. I've sat through the dog and pony show for ISE, and did a bit with it in labs, but it was pretty pricey as I recall and my company never moved forward with it.

The newer/cooler NAC stuff seems to be more CASB focused, or do CASB with NAC as a throw in.

Adbot
ADBOT LOVES YOU

BaseballPCHiker
Jan 16, 2006

When you say manage, are you talking config pushes and backups? Or like a GUI to make changes? What exactly are you looking for?

Ansible seems to be the go to these days, or was at least last I looked into it.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply