Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Locked thread
BONGHITZ
Jan 1, 1970

at work?

Adbot
ADBOT LOVES YOU

Pittsburgh Fentanyl Cloud
Apr 7, 2003


Yes.

Iridium
Apr 4, 2002

Wretched Harp
i rebooted that one server with a big label on it that said "DO NOT REBOOT EVER"

i don't remember why. it turned out fine after a little while gently caress the police.

raspurtin
Apr 18, 2005

Remember when cisco switches defaulted to vtp server mode? Cratered the data center of a medium sized retail company with one once.

A Pinball Wizard
Mar 23, 2005

I know every trick, no freak's gonna beat my hands

College Slice
I was running a syntax to update a bunch of records in a customers dBASE system and fatfingered the filter so it started overwriting every record in the database

recovering was as easy as replacing the dbf file with a backup but people had been working even though I told them not to so they lost 20 people x 30 min of work in that database

A Pinball Wizard
Mar 23, 2005

I know every trick, no freak's gonna beat my hands

College Slice
i mean that's not counting "working for a company that uses dBASE in tyool 2016"

Satellit3
Oct 21, 2008

access 2010 was the last version to support dbf files. my fun tangential dbase fact


trap srpung etc

A Pinball Wizard
Mar 23, 2005

I know every trick, no freak's gonna beat my hands

College Slice

Satellit3 posted:

access 2010 was the last version to support dbf files. my fun tangential dbase fact


trap srpung etc

fun fact: this has literally impacted processes at my place of employment

Satellit3
Oct 21, 2008

lol. my it chumps were convinced to buy back some 2010 licenses or however it works

Moo Cowabunga
Jun 15, 2009

[Office Worker.




cool

Moo Cowabunga
Jun 15, 2009

[Office Worker.




I slaughtered the wrong sheep once. apparently it was a pet.

AtomD
May 3, 2009

Fun Shoe

Displeased Moo Cow posted:

I slaughtered the wrong sheep once. apparently it was a pet.
haha

meet my pet cat mittens, and my pet sheep muttons

Radio Paranoia
Jun 27, 2010

It is now safe to turn off your computer.

AtomD posted:

haha

meet my pet cat mittens, and my pet sheep muttons

Nice!

BooLoo
Oct 18, 2010

SLAM TIME
took production down across multiple countries for a few hours because i am bad at databases. YOLO.

indigi
Jul 20, 2004

how can we not talk about family
when family's all that we got?
as a teen at a catering company I put caff in the decaf dispenser and we got a bunch of complaints so we just started putting decaf in both and in the 4 following years I worked there never had a complaint

clone on the phone
Aug 5, 2003

I oversold an evening of primetime advertising by 30 seconds. Cost the station a cool mil. Didn't get fired, thumbs up.

The Leck
Feb 27, 2001

indigi posted:

as a teen at a catering company I put caff in the decaf dispenser and we got a bunch of complaints so we just started putting decaf in both and in the 4 following years I worked there never had a complaint
project mayhem proceeding nicely

indigi
Jul 20, 2004

how can we not talk about family
when family's all that we got?
these fuckin bourgeois coffee-snoggers don't enen tell the difference :smug:

Iridium
Apr 4, 2002

Wretched Harp

Satellit3 posted:

access 2010 was the last version to support dbf files. my fun tangential dbase fact

my company is a huge multibillion dollar enterprise that's still privately owned by a particular family. said family is mostly at arms length from it, but as a part of the corporate holding company we maintain certain aspects of their private poo poo. notably their charitable entity.

they get requests day in and day out from whatever loving cause feels like asking (ranging from multimillion dollar grants to certain eco charities cuz that's their cause du jour, down to some local theater group that wanted a few grand to build a new stage), and there's three old ladies up on the top floor that manage all the requests, store the data, get approvals, and then follow up to make sure it wasn't just the United Way guy buying another porsche again.

technically all of this is pretty private info, partly because it's uncouth to let the unwashed masses know what the family's causes are, but also because the app that does this stores poo poo like bank numbers for the finances behind all this poo poo.




so naturally it uses an Access database, in plaintext, stored on a shared drive that the three little old ladies had no way of understanding is actually open to the whole loving corporation. And still using Access 2010.

Radio Paranoia
Jun 27, 2010

It is now safe to turn off your computer.
left a security hole open that led to an enterprising russian dumping a bunch of our customer details on an obfuscated html page.

Powerful Two-Hander
Mar 10, 2004

Mods please change my name to "Tooter Skeleton" TIA.


forgot that SELECT INTO overwrites existing data and wiped a table on a prod db...but not an important one so who cares, just restored from backup and nobody noticed

pushed a front end change to a prod server without testing because we had no dev and the application poo poo itself, had to phone vendor support who were like "what do you mean you put it on prod?" but fixed it anyway....tbqh their own software was really at fault here as their own loader process broke even though it had not actually been changed

pretty minor tbh

Pittsburgh Fentanyl Cloud
Apr 7, 2003


Powerful Two-Hander posted:

forgot that SELECT INTO overwrites existing data and wiped a table on a prod db...but not an important one so who cares, just restored from backup and nobody noticed

pushed a front end change to a prod server without testing because we had no dev and the application poo poo itself, had to phone vendor support who were like "what do you mean you put it on prod?" but fixed it anyway....tbqh their own software was really at fault here as their own loader process broke even though it had not actually been changed

pretty minor tbh

Your work fuckup appears to be using MS SQL Server?!?

Captain Foo
May 11, 2004

we vibin'
we slidin'
we breathin'
we dyin'

i accidentally thought the storage array i migrated a bunch of VMs to was capable of handling the IO load but it was not and then i stranded a prod DB there for several hours because storage vMotion kept timing out

indigi
Jul 20, 2004

how can we not talk about family
when family's all that we got?
that doesn't sound like a fuckup at all. I absolve you of any guilt

Captain Foo
May 11, 2004

we vibin'
we slidin'
we breathin'
we dyin'

indigi posted:

that doesn't sound like a fuckup at all. I absolve you of any guilt

the entire thing was my fault and predicated on me not understanding the capabilities of the new array, that I recommended we buy

indigi
Jul 20, 2004

how can we not talk about family
when family's all that we got?
oh I thought it was a "well it should have been able to handle it but the hardware didn't perform optimally" type mistake

Radio Paranoia
Jun 27, 2010

It is now safe to turn off your computer.
watched helplessly as IT installed filezilla and teamviewer onto production machines :smith:

AtomD
May 3, 2009

Fun Shoe
i thin provisioned without monitoring datastore usage properly

a good chunk of the company's production servers were paused while i tried to delete things using vsphere's tremendously poo poo datastore browser

Captain Foo
May 11, 2004

we vibin'
we slidin'
we breathin'
we dyin'

indigi posted:

oh I thought it was a "well it should have been able to handle it but the hardware didn't perform optimally" type mistake

nah the hardware actually works really well for what we actually bought it for, not production vmware loads

AtomD posted:

i thin provisioned without monitoring datastore usage properly

a good chunk of the company's production servers were paused while i tried to delete things using vsphere's tremendously poo poo datastore browser

that thing is a Piece Of poo poo!

Robot Pride
Aug 2, 2010

by exmarx

Captain Foo posted:

that thing is a Piece Of poo poo!

Pendragon
Jun 18, 2003

HE'S WATCHING YOU
didn't deploy prototype code properly to my company's demo laptop. said prototype module didn't work properly for demonstration in front of prospect that was already thinking we were fuckups. didn't help that impression. in my defense, our method of deploying prototype code for demos was completely manual, and it was only a matter of time before this happened.

tasked with coming up with new UI framework to replace our late 90s ActiveX-powered abomination. chose Adobe Flex. lol.

boss asked for estimate on writing a new module in said new framework that was still under active development. I estimated two weeks. took three months due to the fact that the new UI framework didn't have all the nice features of the old one yet. learned an important lesson about estimating when dealing with new APIs.

thought I was on a development server. sudo shutdown -r now. I was on production. I color-coded my SSH windows after that.

made a change to the firewall to eliminate an unneeded rule. turns out the rule was what opened port 443 to our application. new company rule: no changes to firewall without manager approval and only after hours.

Satellit3
Oct 21, 2008

Pendragon posted:

thought I was on a development server. sudo shutdown -r now. I was on production. I color-coded my SSH windows after that.

lol

Captain Foo
May 11, 2004

we vibin'
we slidin'
we breathin'
we dyin'


that's a harsh one

Gazpacho
Jun 18, 2004

by Fluffdaddy
Slippery Tilde
took a job in amazon warehouse systems

lil bip
Mar 13, 2004

That ain't workin', that's the way you do it
I crashed a Wang mainframe on the second day of work because I touched the wrong cable on a krone frame.

Fuzzy Mammal
Aug 15, 2001

Lipstick Apathy
passed the buck on implementing encryption to someone else then got an email from sergey saying that dissidents were dying in oppressed countries because they couldn't communicate safely and it was basically my personal fault. that one sucked.

Pittsburgh Fentanyl Cloud
Apr 7, 2003


lil bip posted:

I crashed a Wang

heehee

Angela Merkle Tree
Jan 4, 2012

the definition of open: "mkdir android ; cd android ; repo init -u git://android.git.kernel.org/platform/manifest.git ; repo sync ; make"
College Slice
ran unit tests locally with the DATABASE_URL set to production
wondered why the test suite was taking so long to boot up (first step is to wipe/reload the db)

lost 2m rows of our client's data, taking the app down for 36 hours while we attempted restores from backups 3 times (the backups aren't actually reliable)

i dont actually consider it a fuckup though because this incident allowed us to move away from mongodb to a real database

Captain Foo
May 11, 2004

we vibin'
we slidin'
we breathin'
we dyin'

Angela Merkle Tree posted:

ran unit tests locally with the DATABASE_URL set to production
wondered why the test suite was taking so long to boot up (first step is to wipe/reload the db)

lost 2m rows of our client's data, taking the app down for 36 hours while we attempted restores from backups 3 times (the backups aren't actually reliable)

i dont actually consider it a fuckup though because this incident allowed us to move away from mongodb to a real database

Lmao, also your username is definitely not a fuckup, good job

Adbot
ADBOT LOVES YOU

Pittsburgh Fentanyl Cloud
Apr 7, 2003


Angela Merkle Tree posted:

ran unit tests locally with the DATABASE_URL set to production
wondered why the test suite was taking so long to boot up (first step is to wipe/reload the db)

lost 2m rows of our client's data, taking the app down for 36 hours while we attempted restores from backups 3 times (the backups aren't actually reliable)

i dont actually consider it a fuckup though because this incident allowed us to move away from mongodb to a real database

I"m boggled as to how people are able to nuke production poo poo or shut down production systems by setting incorrect vars or logging into the wrong system. What kind of mickey mouse operation lets you blow up production poo poo if you set the wrong DATABASE_URL?

  • Locked thread