Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
abigserve
Sep 13, 2009

this is a better avatar than what I had before
I seriously cannot stress this enough: do not consider asymmetric routing a feature of your design. Any edge design that includes asymmetric routing paths is broken. It is a road to ruin.

Adbot
ADBOT LOVES YOU

greatapoc
Apr 4, 2005

abigserve posted:

I seriously cannot stress this enough: do not consider asymmetric routing a feature of your design. Any edge design that includes asymmetric routing paths is broken. It is a road to ruin.

I’m not and I didn’t see the possibility for asymmetric routing if BGP was implemented correctly. I thought it would a reasonably simple failover scenario and in the event anything did go funky then the Palo Alto’s would be able to figure it out until I could correct the error.

abigserve
Sep 13, 2009

this is a better avatar than what I had before

greatapoc posted:

I’m not and I didn’t see the possibility for asymmetric routing if BGP was implemented correctly. I thought it would a reasonably simple failover scenario and in the event anything did go funky then the Palo Alto’s would be able to figure it out until I could correct the error.

In that example the "correct" topology would be to either have two independent firewalls (and take the hit that failover will force connection restarts ) or have a layer two network between the border routers and firewalls so regardless of which border router is actively routing traffic it transits the active firewall. Does that make sense?

greatapoc
Apr 4, 2005

abigserve posted:

In that example the "correct" topology would be to either have two independent firewalls (and take the hit that failover will force connection restarts ) or have a layer two network between the border routers and firewalls so regardless of which border router is actively routing traffic it transits the active firewall. Does that make sense?

Yep that makes sense, thanks. I guess I'm still just stuck on how to handle the /29 without using BGP. If I just get the provider to point another static out the other internet service it could still in theory want to send traffic out a failed link if their interface doesn't go down. On our side we could just track it with an IP SLA but I doubt they'll want to do something like that on their side.

ragzilla
Sep 9, 2005
don't ask me, i only work here


abigserve posted:

Any edge design that includes asymmetric routing paths is broken. It is a road to ruin.

Unless your edge is stateless by nature, in which case go hog wild.

abigserve
Sep 13, 2009

this is a better avatar than what I had before

greatapoc posted:

Yep that makes sense, thanks. I guess I'm still just stuck on how to handle the /29 without using BGP. If I just get the provider to point another static out the other internet service it could still in theory want to send traffic out a failed link if their interface doesn't go down. On our side we could just track it with an IP SLA but I doubt they'll want to do something like that on their side.

I would definitely use bgp.

Edit; also I would say there are few, if any, truly valid use cases for asymmetric routing. Non deterministic, sure, and there are always going to be asymmetric paths out there is the real world. But it should never factor into network design.

abigserve fucked around with this message at 02:45 on Oct 10, 2019

unknown
Nov 16, 2002
Ain't got no stinking title yet!


greatapoc posted:

Yep that makes sense, thanks. I guess I'm still just stuck on how to handle the /29 without using BGP. If I just get the provider to point another static out the other internet service it could still in theory want to send traffic out a failed link if their interface doesn't go down. On our side we could just track it with an IP SLA but I doubt they'll want to do something like that on their side.

I'd just recommend getting a second /29 for the other site.

Both sites work at all times, easily testable, just change a simple dns record which can be simply scripted.

greatapoc
Apr 4, 2005
I just set up a lab in EVE-NG with my current topology using a static route from the ISP side, OSPF internally and then went through the process of bringing up iBGP between the border, bringing the eBGP online to the ISP, advertising my route from both borders and then deleting the static route from the ISP router. Everything worked perfectly. I then setup the backup route with a higher MED and watched it change over so I'm pretty confident that's the best way to do it all as long as I can coordinate with the ISP to check they're receiving my routes and to delete the static. At least I know I can have everything running on my side in parallel.

This probably sounds pretty dull I've just never worked with BGP in production before.

Thanks Ants
May 21, 2004

#essereFerrari


I still can't see anything that you're running that requires failover of IP addresses.

greatapoc
Apr 4, 2005

Thanks Ants posted:

I still can't see anything that you're running that requires failover of IP addresses.

It requires devices that have public IP addresses to be reachable via a backup service that won’t have a static route pointed at it. If the primary goes down so do our addresses. Am I over complicating things and looking at it the wrong way?

greatapoc
Apr 4, 2005
Edit: double post

unknown
Nov 16, 2002
Ain't got no stinking title yet!


greatapoc posted:

I just set up a lab in EVE-NG with my current topology using a static route from the ISP side, OSPF internally and then went through the process of bringing up iBGP between the border, bringing the eBGP online to the ISP, advertising my route from both borders and then deleting the static route from the ISP router. Everything worked perfectly. I then setup the backup route with a higher MED and watched it change over so I'm pretty confident that's the best way to do it all as long as I can coordinate with the ISP to check they're receiving my routes and to delete the static. At least I know I can have everything running on my side in parallel.

This probably sounds pretty dull I've just never worked with BGP in production before.

You should assume that MEDs probably won't work - it's up to the ISP in question and many just drop whatever you set. So you'll probably have to set bgp communities on the advertised route which will then set the localpref on their side (if they even allow that!) - so get their published bgp community list.

Here's the other reason: When using BGP and multiple advertisements/sites, you're not guaranteed to have your traffic go the way you expect. I've seen multiple times when routing changes happen in the ISP and traffic flows change. Hot potato routing happens and it sends those packets via your backup path instead of your primary path, and now both paths are active. And even better is path based load balancing = packet 1 goes via pipe 1, packet 2 goes via pipe 2. Oh, and there's nothing you can do about it. Going to ask your ISP to turn off that new 100g peering connection they just set up? Yeah, no. Also, I mentioned asymmetric earlier - how are you handling your internal (outbound) routing, and what if there's a failure there and site A is going via site b firewall, but the firewall didn't go down... These are all things to consider.

Doing DNS based failover:

servicepublic.exampleX.com -cname-> serviceprivate.example.com -cname-> siteAaddress.example.com -A-> 192.0.2.2

servicepublic.exampleX.com -cname-> serviceprivate.example.com -cname-> siteBaddress.example.com -A-> 192.0.3.3

You can have multiple servicepublic cnames to a master serviceprivate name. That serviceprivate name is a just a cname to whatever site you want to use - that's 1 DNS record to change. Set the TTL to like 60 seconds, and bam, super fast failover.

Additional benefits: Now you have easy failover to the cloud too (Call it site C!). Business is taking off! You've now got a built in load balancing mechanism... create "siteLB.example" which can have 2 (or more) address records, or even sweeter, geo-ip capable (with the right dns server/service).

Don't tie yourself to an IP address.

Edit: Public example: $ dig www.amazon.com
;; ANSWER SECTION:
www.amazon.com. 253 IN CNAME www.cdn.amazon.com.
www.cdn.amazon.com. 60 IN CNAME d3ag4hukkh62yn.cloudfront.net.
d3ag4hukkh62yn.cloudfront.net. 15 IN A 13.225.199.69

Amazon can change their CDN provider easily, and the CDN can easily change the backend server I use at any time.

unknown fucked around with this message at 16:02 on Oct 10, 2019

greatapoc
Apr 4, 2005

unknown posted:

Don't tie yourself to an IP address.

Thanks a lot this is all really helpful. Certainly got a lot to think about.

unknown
Nov 16, 2002
Ain't got no stinking title yet!


unknown posted:

BGP is a sledgehammer

When BGP works (99% of the time), it's really nice.

When it breaks (aka doesn't do what you want/expect) is when find yourself on everyone's poo poo list for a long time because it broke due to some 3rd party that doesn't give a poo poo about you.

BaseballPCHiker
Jan 16, 2006

Tried to look up the EOL date on a 2960CX earlier today and couldn’t pull up the page. Gave up and forgot about, then my coworker had a hard time downloading something from the Cisco site. Forgot about that until we were coincidentally meeting with our Cisco rep this afternoon.

In his words: “Massive internal outage, heads are going to roll over this”.

tortilla_chip
Jun 13, 2007

k-partite
"We hosted it in San Jose, and they turned the power off"

abigserve
Sep 13, 2009

this is a better avatar than what I had before

greatapoc posted:

I just set up a lab in EVE-NG with my current topology using a static route from the ISP side, OSPF internally and then went through the process of bringing up iBGP between the border, bringing the eBGP online to the ISP, advertising my route from both borders and then deleting the static route from the ISP router. Everything worked perfectly. I then setup the backup route with a higher MED and watched it change over so I'm pretty confident that's the best way to do it all as long as I can coordinate with the ISP to check they're receiving my routes and to delete the static. At least I know I can have everything running on my side in parallel.

This probably sounds pretty dull I've just never worked with BGP in production before.

This is fine but you should use as-path prepending instead of MED as it will be better supported by most ISP's (check with them).

Your use case is like, the most basic implementation of BGP. The only time I think I wouldn't recommend it in your topology would be if you already had a GSLB implementation - I would just leverage that instead and use DNS as your failover mechanism.

greatapoc
Apr 4, 2005

abigserve posted:

This is fine but you should use as-path prepending instead of MED as it will be better supported by most ISP's (check with them).

I've just received their BGP routing policy handbook and it says they prefer MED but also support communities that will set the local preference on their side as well as AS prepending.

FatCow
Apr 22, 2002
I MAP THE FUCK OUT OF PEOPLE

ragzilla posted:

Unless your edge is stateless by nature, in which case go hog wild.

I go away from the thread for a few days and you beat me to this one.

I spent so much time beating "asymmetric routing is bad" out of this business.

Partycat
Oct 25, 2004

BaseballPCHiker posted:

Tried to look up the EOL date on a 2960CX earlier today and couldn’t pull up the page. Gave up and forgot about, then my coworker had a hard time downloading something from the Cisco site. Forgot about that until we were coincidentally meeting with our Cisco rep this afternoon.

In his words: “Massive internal outage, heads are going to roll over this”.

Yeah they missed a scheduled briefing with no explanation, can’t wait to find out who deleted what this time .

Kazinsal
Dec 13, 2011



Partycat posted:

Yeah they missed a scheduled briefing with no explanation, can’t wait to find out who deleted what this time .

Whatever happened it was bad enough that TAC couldn't open cases for a few hours. My team had a few customers left in some lovely positions because of that.

Leandros
Dec 14, 2008

tortilla_chip posted:

"We hosted it in San Jose, and they turned the power off"

There was some speculation among colleagues that this was indeed the case but it was quickly ruled out :v:

Leandros fucked around with this message at 21:48 on Apr 12, 2022

Leandros
Dec 14, 2008

E: please ignore. Connectivity issues while posting about connectivity issues, what a hoot!

Leandros fucked around with this message at 18:25 on Oct 11, 2019

Sprechensiesexy
Dec 26, 2010

by Jeffrey of YOSPOS
So, does anyone have any experience with Cisco Tetration ? My company wants me to evaluate it and I'm having a meeting with our account manager but some 3rd party feedback would be nice to.

Tetramin
Apr 1, 2006

I'ma buck you up.
Was troubleshooting an ASA that was replacing a sonicwall, the ipsec tunnel was up, and the FW could reach google DNS, but anything behind the firewall was unable to reach anything either on the other side of the tunnel, or public internet. I was beating my head against the wall because I've done dozens of these replacements and never had this much trouble.

Called TAC and the tech was pretty confused too, then he removed sfr fail-open from our policy, replacing it with sfr fail-open monitor. I guess I forgot to configure SFR because when I ran it, we got the EULA etc.

The tech told me 'oh yeah sfr fail-open will block all traffic if the module is down' and it was like 11pm so I said ok, its working so whatever. I checked the documentation this morning which confirmed my suspicion: 'The fail-open keyword sets the ASA to allow all traffic through, uninspected, if the module is unavailable.'.

It seems like this is doing the exact opposite of that? I have a bunch of ASA's that aren't using SFR with the setting turned on and no issues. Can anybody explain this? I'm waiting on a response from TAC, I kind of feel like he was just shooting in the dark and when it worked he mumbled about it being expected behavior.

Nuclearmonkee
Jun 10, 2009


I've had sfr fail-open let things through when the module was completely broken and non-functional. I thought that was the whole point of the command.

Does it just behave strangely if you have it in fail-open with the module installed but still awaiting setup? Never tried that honestly but it seems silly for it to work that way.

Tetramin
Apr 1, 2006

I'ma buck you up.

Nuclearmonkee posted:

I've had sfr fail-open let things through when the module was completely broken and non-functional. I thought that was the whole point of the command.

Does it just behave strangely if you have it in fail-open with the module installed but still awaiting setup? Never tried that honestly but it seems silly for it to work that way.

Yeah in my experience fail-open lets everything through, that does indeed seem to be the use purpose of it.

As to your question, yeah that's the exact state the device is in. SFR is installed and upgraded to the version we use, but I never went through the EULA and network configuration and it was blocking all traffic until we changed it to fail-open-monitor.

BaseballPCHiker
Jan 16, 2006

You know what I really hate ASDM on FirePowers. Actually I'm really starting to get sick of ASAs in general, seems like we hit a new bug every single week.

Anyway, not sure if this is a bug or something I broke but I'm guessing a bug.

Recently updated the certs on our ASA for our clientless SSL VPN connections. Just a simple cert update as ours was about to expire. Ever since users can sign in, authenticate using Duo 2-factor, and see WebVPN bookmarks. But if they click on them the page opens a new tab and just fails. As far as I can tell this is some odd Java issue but I really dont know, and I've spent the past few hours banging my head against the wall trying to figure this out. I've opened a TAC case and will hopefully hear back soon.

In the meantime has anyone run into this before? I've checked our backup configs just to make sure nothing else has changed and I dont see anything besides the cert upgrade.

EDIT: I found the problem. ASDM, Remote Access VPN, Clientless SSL VPN Access, Portal Bookmarks, select Bookmark group, select bookmark, edit, uncheck "Enable SmartTunnel"

Wasnt found int the config, the preview on the send command shows ASDM deleting some temp file off of disk0: and creating a new tempasdm#### file to throw on disk0.

BaseballPCHiker fucked around with this message at 20:12 on Oct 30, 2019

CrazyLittle
Sep 11, 2001





Clapping Larry
Narrator voice:
It was a dumb tunnel

volkadav
Jan 1, 2008

Guillotine / Gulag 2020
I didn't see a FAQ that touched on this in the head post, so: what's the easiest/least expensive way to get access to ios/rommon security updates as a hobbyist user? I picked up a c891fw off ebay for home use that seems perfectly functional but the images loaded are a bit out of date. The one hard quote I've found online for smartnet was ~$100/yr for this device from a third party, is that typical/reasonable? Is there a better option? (Slip :10bux: to friends with an active support account?)

:frogbon:

falz
Jan 29, 2005

01100110 01100001 01101100 01111010
Find the file names on cisco's website and then use Google skills and hope you find something similar.

wolrah
May 8, 2006
what?
I sometimes have luck googling the MD5/SHA hashes posted for various vendor restricted downloads, Cisco included. Side bonus is that also means you're ready to validate that the binary you just downloaded from MEGA or some random Czech open FTP is legit.

Thanks Ants
May 21, 2004

#essereFerrari


Yeah definitely verify the checksums of anything you get from an 'interesting' source

Tetramin
Apr 1, 2006

I'ma buck you up.
I PMd you if you can’t find it through googling. Idk if I can get it since I don’t have that device in my environment, but maybe Cisco portal access is enough..

Partycat
Oct 25, 2004

They’ve gotten somewhat more strict on that in the portal lately.

You can also go look for security advisories- sometimes they indicate they have released a free update to patch it, which you can get through TAC, but not major releases most of the time.

Kazinsal
Dec 13, 2011



Firepower Device Manager stopped deploying changes intermittently on our 2-month-old HA pair of 2110s about a week ago and completely stopped deploying changes last night, claiming a sync failure in the HA pair. The ASA dataplane, however, was running its HA just fine. Turns out the actual problem is that the FTD side's hard disk on the primary has taken a poo poo and partially went into a split brain state. The ASA portion didn't even flinch.

I hate firewalls. And especially loving Firepower.

BaseballPCHiker
Jan 16, 2006

We've got a pair of 2110s running right now in HA.

They'll probably be the last Cisco firewalls that ever run in this place. They've gotten so bad.

Palo Alto has a pretty great online demo/lab setup that we've been looking into. Probably the direction we'll go when it comes time to replace the 2110s.

https://portal.netdevgroup.com/learn/pan8-ce-pilot

Richard Noggin
Jun 6, 2005
Redneck By Default

BaseballPCHiker posted:

We've got a pair of 2110s running right now in HA.

They'll probably be the last Cisco firewalls that ever run in this place. They've gotten so bad.

Palo Alto has a pretty great online demo/lab setup that we've been looking into. Probably the direction we'll go when it comes time to replace the 2110s.

https://portal.netdevgroup.com/learn/pan8-ce-pilot

We made the switch from ASA with Firepower to PAN this year and don't regret it one bit. The only thing we still have (and plan on keeping) on the ASAs is AnyConnect and a couple IPSec tunnels.

Prescription Combs
Apr 20, 2005
   6

MF_James posted:

Cool, yeah I dropped the commands in and they worked but I wasn't sure if it would actually happen.

Then I realized I could set it to use TCP and just cap the traffic and see if it actually is sending logs messages.

I didn't get a lot of sleep last night OK!

Old as poo poo topic but unless you absolutely need to capture every connection to syslog, enable syslog permit host down or that ASA will block all traffic if that syslog server goes down.

E: when sending TCP syslog. Udp doesn't care obvs.

Prescription Combs fucked around with this message at 22:49 on Nov 22, 2019

Adbot
ADBOT LOVES YOU

MF_James
May 8, 2008
I CANNOT HANDLE BEING CALLED OUT ON MY DUMBASS OPINIONS ABOUT ANTI-VIRUS AND SECURITY. I REALLY LIKE TO THINK THAT I KNOW THINGS HERE

INSTEAD I AM GOING TO WHINE ABOUT IT IN OTHER THREADS SO MY OPINION CAN FEEL VALIDATED IN AN ECHO CHAMBER I LIKE

Client requirement is to capture all the logs for compliance (they are a bank). Used UDP in the end, but i might have swapped to TCP temporarily, I forget, been a while now.

I found that if you have 2 syslog servers configured, it won't send everything to both log servers (I haven't tested if it's because they are the same IP but different ports or if that will happen with 2 completely separate IPs), it will only send everything to the first server in teh list and then admin level events only to the second server.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply