Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!

quote:

Hello,

Do we support SMTP and POP3 protocols on our email server? I'd like to be able to send and receive mail occasionally through gmail.

Thanks,


Let's just tear out exchange, wipe our asses with the 250 cals, and do it your way.

Adbot
ADBOT LOVES YOU

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!

MrMoo posted:

Well OWA is utterly terrible at the best of times. Seemed to have broken image uploads for all OS X browsers for the last month or so, how they can mess up a simple HTML button is beyond me.



The version of safari that came with Yosemite also horribly broke Sharepoint.

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!
I'd like to extend a personal gently caress YOU to apple for killing PPTP in MacOS and making me reconfigure all 60-ish VPN routers I have.

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!

GnarlyCharlie4u posted:

heh We ran into some really weird mikrotik problems at my last job.
Like VPN traffic being transmitted out of order, ports randomly disabling themselves...

I remember some real good ones with the CCR and RB2011
You plug in a usb and the router shits itself.
It wouldn't boot unless something was trying to PoE - turned out to be a lovely power supply.
The IPSEC tunnel would get pissy and just drop all packets. You couldn't restore it until after you rebooted ALL of the endpoints.

A recent version of RouterOS would randomly and unfixably poo poo all your GRE tunnels.

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!
Those Mikrotik PPTP setups weren't my call :( I just have to change it now in the middle of a total shitstorm.


e: holy loving avatar

Farking Bastage fucked around with this message at 23:43 on Sep 16, 2016

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!

Kashuno posted:

We are finally going to go O365 next year :toot: I am probably more excited about migrating this stuff than I should be

https://portal.ondemand.quest.com/Auth/SignIn?product=quest.online.Migration&replyTo=https%3a%2f%2fmigration.ondemand.quest.com%2f


You'll thank me later.

Also there's a way to get a full exchange powershell to a 365 tenant(not just the azure user management side). I can build out an environment in minutes using scripts. I've recently migrated 3000 people to 365 and that Dell tool is amazing.

edit: appaarently {code} breaks tables

http://pastebin.com/i1Uhyeuv

I usually save that in my home directory as like "365" and invoke it with "./365"

Farking Bastage fucked around with this message at 23:44 on Sep 21, 2016

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!
See the pastebin. the forum software messed with that code in the code brackets. There's a cost ber mailbox with that tool but it will Migrate almost ANYTHING to 365. However, the limitation is non-exchange sources you will need to PST out any contacts, calendars, notes, or tasks.

Farking Bastage fucked around with this message at 23:48 on Sep 21, 2016

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!
We spend the better part of a year eliminating Centurylink as a last mile provider on all of our circuits in favor of Level3, now Centurylink has bought Level3 :negative:

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!

Vargatron posted:

It's 10/10 both ways. We need to implement QoS but getting this to work via AT&T is apparently difficult according to my networking guy.

Whaaaaaaaa

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!
Oh how I love new hardware day. HP has discontinued the venerable 3500-YL-48GPOE, so these are the replacement. Aruba 3810's, but with 40GBE stacking modules instead of 10.

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!

Thanks Ants posted:

I hope that piece of poo poo in the background is going in a dumpster

Oh I'm going to office space that loving thing.

In other news, client called in with "network" problems. Our clients have enough autonomy that they always end up with some dummy switch on the network that wrecks poo poo, but we can't tell them not to, just bill the crap out of them when they inevitably gently caress up.

STP Enabled : Yes
Force Version : MSTP-operation
IST Mapped VLANs : 1-4094
Switch MAC Address : 001b3f-dca280
Switch Priority : 32768
Max Age : 20
Max Hops : 20
Forward Delay : 15

Topology Change Count : 15,529
Time Since Last Change : 16 hours

:suicide:

e: that's not good...

code:
3500yl-48g-02# show mod

 Status and Counters - Module Information

  Chassis: 3500yl-48G J8693A         Serial Number:   
                              Allow V1 Modules:   Yes


                                                                       Core  Mod
  Slot  Module Description                     Serial Number  Status   Dump  Ver
  ----- -------------------------------------- -------------- -------- ----- ---
  A                                                           Failed   -


3500yl-48g-02#

Farking Bastage fucked around with this message at 16:05 on Nov 16, 2016

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!
I'm starting to regret going back to MSP right about now. Obviously, were terribly slow today, and the bosses called a meeting to not only suggest we go through customers setups looking for things to bill on, but also to say " We lose out on $30000 of revenue because of thanksgiving and christmas holidays, so keep that in mind.

Really?

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!
code:
Our technical support service has detected unauthorized access to your account.

Due to that, we had to block your account until you confirm your personal information.
To do so, please follow the instructions in the attachment.




Best Regards,
Iva Ruiz
Technical Support
and it has a .c attachment. They're not even trying anymore.

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!
We've had a problem with these SyncBak streaming boxes for a couple of weeks that's dirving us nuts. We queue them because they will rape all of your bandwidth if you let them. It turns out the default buffer on a PFIFO queue on a mikrotik is 50 packets. If it buffered more than that it was dropping the excess and loving up the video stream. Most other devices had some kind of error checking, but these apparently don't so if a packet got discarded it would gently caress up the video. Made new queues with bigger buffers and it seems to have fixed it.... I hope :ohdear:

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!
Poor Centurylink...



quote:

Update:
On December 27, 2018 at 02:40 GMT, CenturyLink identified a service impact in New Orleans, LA. The NOC is engaged and investigating in order to isolate the cause. Field Operations were engaged and dispatched for additional investigations. Tier IV Equipment Vendor Support was later engaged. During cooperative troubleshooting a device in San Antonio, TX was isolated from the network as it was seeming to broadcast traffic consuming capacity, which seemed to alleviate some impact. Investigations remained ongoing. Following the isolation of the San Antonio, TX device troubleshooting efforts focused on additional sites that teams were remotely unable to troubleshoot. Field Operations were dispatched to sites in Kansas City, MO, Atlanta, GA, New Orleans, LA and Chicago, IL. Tier IV Equipment Vendor Support continued to investigate the equipment logs to further assist with isolation. Once visibility was restored to the site in Kansas City, MO a filter was applied to the equipment to further alleviate the impact observed. All of the necessary troubleshooting teams in cooperation with Tier IV Equipment Vendor Support are working to restore remote visibility to the remaining sites at this time. We understand how important these services are to our clients and the issue has been escalated to the highest levels within CenturyLink Service Assurance Leadership.

A fiber multiplexer flaked out and broadcast stormed their entire network.

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!

Schadenboner posted:

Im teh Tier IV Support.

drat dude. That's gotta suck. Here's something I got from reddit when I posed the question of why in the gently caress was everything on the same broadcast domain:

quote:

Nah. I've heard rumors that this is a problem with their long haul DWDM network since there are a lot of transport circuits impacted in addition to Layer 3 services. My guess is the broadcast storm is on the management plane of that caused by lots of attempts to replicate with other nodes.

Edit: To expand a little bit. Most long haul fiber is lit with transport equipment like Ciena or Infinera. I believe CenturyLink is a Infinera network. They're big, so they're probably using something like the Infinera DTN-X XTC to light fiber at 1.2Tb/s and then sell subrate interfaces off of it. So you could buy 10G or 100G on a link between Denver and Chicago. Packets that go in Denver show up in Chicago. Since there's no direct fiber from Denver to Chicago, they map you to a channel on Denver to Kansas City and then a different channel from Kansas City to Chicago.

Everything is going SDN, even transport, so the definition of how to route traffic from Denver to Chicago is managed in software. My guess is the 'broadcast storm' is impacting the management layer of the transport network and preventing them from making changes to different nodes to restore services. That would explain the need to go touch each node to restore services.

Now when it comes to providing internet access, most ISPs will actually 'buy' transport from their long haul division. So there's a MPLS/L3 network layered on top the transport equipment to provide IP services. For example, CenturyLink will have a core router in Denver, connected to their Denver DWDM transport equipment at 400G, with a connection to Chicago where it connects at 400G to a core router in Chicago. All of the internet services you use come from that additional layer that uses circuits provisioned by the transport network. Sitting right next to that 400G path are other circuits for other carriers (Cogent, HE, Comcast, basically anyone that isn't operating their own national fiber network)

Usually the thing that fails is the MPLS/L3 layer, transport gear is extremely reliable outside of random fiber cuts and line card failures. When you hear about a network problem it's almost always the L3 layer. In this case, I'm guessing that the L3 layer is fine, but there's no or reduced connectivity between nodes due to the DWDM transport layer failure.

Luckily none of my sites in FL were affected on AS209 and AS2379

Farking Bastage fucked around with this message at 03:20 on Dec 28, 2018

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!

Lets Get Patchy posted:

Hell ya. We barely touched on powershell in my advanced sysadmin class last semester so I decided to get a few udemy classes and a book to study over the winter break. I really enjoy working with it so far.

It really comes in handy when you can script an O365 tenant for a migration. Before I went full network, I could fully build out a tenant down to the last note or contact in about 30 minutes.

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!
We got our pentest results back today. :stonk:

The vulnerabilities our team(network) found and reported to both the application and server groups in writing, 6 months ago, were the biggest things hit on the audit. Imagine that...

The main homepage had a unrestricted ASP relay.

Several users' passwords were dictionary guessed, several others compromised because their passwords were listed in public dumps and exploited against the office 365 portal.
A few had citrix application access, which was then exploited using an old method to get a CMD shell on the local Xenservers that hadn't been patched since God knows when. That, was in turn used to open powershell sessions with God rights on pretty much any server they wanted.

We have it in writing warning the other IT groups that this poo poo was not right, now they have it from from an outside source. At least they didn't find the AWS site that is still listening, despite our warnings, on port 80 and passing AD creds in the clear. :suicide:

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!
New Top of rack and IDF gear porn :D



Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!

PancakeTransmission posted:

Holy crap is that 3 whole boxes of 10g SFPs? That's gotta cost about the same as 3 individual Cisco brand transceivers!

We're locked into Cisco due to security accreditation policies and previous purchasing decisions :(

We still have 10+ year old switches in service. Guess that's what happens when you're not private sector :sigh:

We’re public sector with public safety and law enforcement. 4 cases of 10g to be exact. Two SX one LX and one ZX. All single mode. 18 grand for the the lot from fiberstore. That would have cost close to six figures if they were all extreme branded. My desk drawer has one of each of the branded ones just in case GTAC wants to give me poo poo.

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!
I don't know what it is about security system vendors, but gently caress the entire lot of them. Johnson Controls, Siemens, loving split pine tech, gently caress off. They were putting in new badge swipes for a building that we just spent about 80k completely re-cabling in cat6 and they pull this loving bullshit.



On top of all that, they put loving zip ties around my cable bundles instead of using the velcro. Who in their right loving mind thinks this is anything CLOSE to acceptable work. I'm about lose my poo poo.

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!

Thanks Ants posted:

I was convinced that Extreme had shitcanned Enterasys

They're in the process of it. They kept the policy bits, but they're EOL'ing pretty much everything we got now. I got a room full of x460 G2's and 690's to replace them with.

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!

Sickening posted:

I don't know whose cables are whose but I am the 12 ft patch cables connecting ports 12 inches away.

My vendor did that unfortunately. since we paid them to do that part, it had been decreed unsound for me to tear them all out and re-do.

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!

Nuclearmonkee posted:

Can't you tell the vendor "That's poo poo I'm not paying for it until you do it properly". Or did some goober sign off without looking at it or getting someone who knows what the gently caress to look at it.

I have stopped allowing them to do patch cables going forward and I do them all myself now. I get that modular switches are a real pain in the rear end to cable manage, but come the gently caress on. The linecards are 24 ports. The 48 port panels are broken off into 6 port sections. Just bundle them in sections indexed by length FFS. Anyhoo, those K10's are being EOL'ed to were going to stacked X460G2's and I'll be making them put 2u cable managers in between each panel from now on.

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!

skipdogg posted:

Oh yeah, things are nuts right now. Major Akamai issue right now. Affecting a lot of companies. Our customers can't login to our apps or website, this is just not good at all.


https://twitter.com/GossiTheDog/status/1418244159182413825?s=20

I wish that was all the problems I had today. Florida lightning just smoked almost all my network infrastructure at the main sewage treatment plant for the area. So far the tally is about 60k worth of burned up poo poo in a site that has extensive lightning mitigation measures.

5 Extreme X460g2-48P-10GE4 switches, each with two power supplies, a fan module, and a stacking module ($10,000 each)
13 CIsco 8811 and 8849 phones
2 WAP's
1 Fortinet FG101F firewall

so far.

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!
The guy who runs the SCADA side of the wastewater system just told me they lost around 150k worth of gear on their side. gently caress me.

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!

GreenNight posted:

Eh, hopefully your business has insurance.

Local government. Also getting fined by the EPA because when it hit the SCADA side, it shot the pumps up to max and put around 30000 extra gallons of wastewater through the sanitizers which diluted the chlorine saturation to below acceptable levels.

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!

ghostinmyshell posted:

What is the modern way of handing system notifications from linux servers like root sending an email that cronjob #2 failed? I thinking about some kind of splunk service but those notifications have to go somewhere to get indexed.

The way we used to do was send all of that to a public folder and review when necessary.

Wouldn't you stand up a syslog box and trigger that to notify via email when a predefined SNMP trap is received?

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!

22 Eargesplitten posted:

Well, after asking about complacency last week I've decided I'm looking for a new job. Company is refusing to even backfill positions because sales are down, like that's not going to result in customers leaving when our performance goes down because an already skeletal workforce is cut even farther. It will be years until I can get a promotion at this rate because the company is going to go into a death spiral until management that doesn't have their heads in their asses takes over.

You would do well to run like a motherfucker.

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!

MustardFacial posted:

code:
Dell EMC Networking OS 
Notice: this software is protected by copyright
Detecting hardware...panic:	 NMI ... calling panic

The operating system has halted.
:sigh:

Coming from a longtime network engineer, rip that poo poo out. When gear starts doing that, no matter how high end it is, it will just happen more and more frequently until it completely croaks or is replaced. I say this knowing full well that Dell switches are completely loving useless.

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!

MustardFacial posted:

Yeah, it's coming out I've already got a replacement coming from Dell.


Having not much experience with Dell switches I've always heard that their S-Series and above datacenter stuff was perfectly fine but their access layer N-Series was garbage. Having said that, this one in question is an S-series but all of our datacentre switches are Dell S-series and this is the only one to fail so far.

I will say though, in all of my years working on Cisco and Juniper gear, I've never seen a switch act the way this one does so I'm sort of inclined to agree with you, it's just hard to find concrete evidence on which Dell switches are poo poo and which are ok. Plus the cost savings on the Dell gear vs a Cisco Nexus or a Juniper EX means the higher ups are reticent on change.

I have about a dozen Enterasys S series blade chassis(24 SFP+per) that started doing very similar behavior in the last couple of years. GTAC would tell us to replace and rollup firmware, but it continued until we physically replaced them. Very similar behavior to yours and I suspect it was a bad block of memory and when the OSPF process would hit that space, the whole process poo poo itself. Not the thing you want happening when that thing's job is OSPF and is your holy grail of core devices because I don't have enough strands for a full VRRP/MLAG setup.

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!

abigserve posted:

Also in terms of hilarious switch software issues, my favourite was Enterasys 48port edge switches that, in a particular software version, would cause a broadcast storm when you enabled DHCP snooping

I had a similar problem with Cisco 8811 phones causing C5's to send out about 800 ipv6 multicast per second, per phone. That one was on cisco and one of their software patches. They like to use the term "caveat" because those arrogant fucks couldn't possibly admit that they may have actually hosed up. That was enough to crush a C5 and nearly tank a K10. I'm almost done replacing all my edge gear after nearly 2 years(300 switches). We were a full policy/NAC enterasys shop when Extreme bought Enterasys and that transition has been a bitch because of how the extreme asics can't hold all the policy rules we have. (enterasys did their own silicon and had extra memory on every port to accommodate the policy). All the C5's are now Extreme X460G2's, all the K10's are now stacked X460g2's, and the cores that were S series will soon be Extreme X690's running MLAG and VRRP(fully redundant paths if these grants go through).

Sometimes this business is fun when you get to build bad rear end poo poo.

Farking Bastage fucked around with this message at 19:30 on Jul 29, 2021

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!

LochNessMonster posted:

Give the support desk a heads up so they won’t be forwarding all users to you.

If they're like my support desk, it's automatically a network problem.

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!
We just got word to put the loving masks back on in the middle of a transition back to the office(currently 25%) because Florida sucks and the vaccinated are now responsible for protecting the unvaccinated apparently. I'm sorry but if they really want us to go back to office door closed mask on lockdown, then send us the gently caress back home until the boomers who keep electing swizzlefucks like Ron DeSantis do the only thing they have ever done for the greater good which is loving die.

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!

Biowarfare posted:

What kind of hot poo poo is going on down there?

"Florida’s DeSantis Signing Order To Block Mask Mandates"

I work for a municipal government. The county government in the same area mandated vaccines for all employees this week. Our leadership isn't that ballsy apparently.

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!

jaegerx posted:

What did y’all get for sysadmin day?

Another new person with PMP in their email signature that makes more than a network engineer.

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!

Coolnezzz posted:

All I received was a message in my close friends chat about it and then this post from you, so happy sysadmin day!

I was handed a surprise ISP speed upgrade this morning for a client and had to be on-site at 9. The tech from the ISP was already there when I arrived, they finally finished up after several rounds of testing with their NOC and said here's the fiber hand-off, can you test everything? BTW it's 2gig and here's where you connect your fiber to our 10g SFP+ transceiver. Our client didn't mention this would be a fiber hand-off, nor that it'd be 2gig, and prior to now it had been a 1gig copper setup. So as I was unprepared, I set things up while they were still there with a 1g copper transceiver from the ISP router to the edge firewall to get things up and running.

I thanked the tech and he went on his way, a few minutes later I'm getting yelled at by the client because I had no idea what I was doing and this had been planned for months and this bullshit happens every time I come to this site "instead of the good engineer who regularly comes here". I had only enabled half of the speed they were paying for and now felt like poo poo.

So I'm obviously upset as much as the client at this point, I call my account manager for this client and we start chatting and I conference in the client. I come to find out my coworker said he would take care of this entire setup "as long as you order these Sonicwall branded 10g SFP+ transceivers" and he will set everything up during his next visit, but as usual he documents nothing and tells no one about his plans, so the entire thing is in his head at this point.

Unfortunately my co-worker neglected to notice the 2x 10g interfaces on the Sonicwall were already used to link to the 2 core switches with Twinax cables when he told the client to buy the two new transceivers and he'd make it work. I tried to get the new ISP hand-off to link to these new transceivers in the 1gig slots to no avail, I even said to the ISP tech he could try to set them to 1g full duplex but decided half way through that sentence that this is a bad idea and it's not what the client is paying for. I also suggested LAGing them for a 2x 1g connection but also decided that was a bad idea in the end and decided on the 1g copper.

Anyway I drove home after writing some nasty emails and took a nap. So it all balances out, right?

I would be having that ISP's charred remains swept off my office floor.

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!
Where in the hell does Fortinet get the notion that I need to pay to ship my RMA back? Of all the ways to nickel and dime something. They're acting like they just went public for fucks sakes.

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!

Dick Trauma posted:

Dusting off Powershell because my boss has once again sent an email to the almost the entire company with an attachment that contains highly confidential material.

At this point you may as well save the exchange scripts and code them to be interactive.

George H.W. oval office posted:

Yea the O365 compliance powershell module to search and purge a spam email that got through is a life saver.

Back when I was doing a ton of mail migrations, I had the powershell exchange module down so cold, I could build out a full tenant in half a day, migrate every little thing in someone's outlook if non-exchange, and have them fully set up as soon as the transfer of existing data was finished.

Farking Bastage fucked around with this message at 14:53 on Aug 5, 2021

Adbot
ADBOT LOVES YOU

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!
Our CISO walked out when he was a middle manager under my current boss after cussing him out for being the micromanager he is about 5 years ago. Now he's CISO and is a rung above the ladder from that same boss. It's been entertaining to say the least.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply