|
Renegret posted:It's easy, just don't have bugs in your code. Reminds me of the time I did application support for a company that didn't have a testing team. "If the customer finds something that doesn't work they'll tell us anyway, so why bother".
|
#
?
Apr 11, 2018 16:18
|
|
|
|
| # ? Apr 29, 2024 00:17 |
|
|
We have no QA and rolled out an update to a customer in Australia (we're east coast USA), so their poo poo was broken overnight and now we can't work with them to troubleshoot because we come in when they go home  poo poo that works fine here doesn't work there so 
|
|
#
?
Apr 11, 2018 16:21
|
|
|
Which reminds me. I'm still waiting on root access for the production box. Thanks and gods bless.
|
|
#
?
Apr 11, 2018 16:24
|
|
|
suuma posted:We have no QA and rolled out an update to a customer in Australia (we're east coast USA), so their poo poo was broken overnight and now we can't work with them to troubleshoot because we come in when they go home Timezones.
|
|
#
?
Apr 11, 2018 16:25
|
|
|
Well, it's now issues with USB peripherals that worked previously and the server they're on is hosted somewhere off in Oceanic land so I don't think it's a time issue. vvv one of our techs was up til midnight trying to help them but we needed one of our software developers to actually fix the issue (dev has access to this hosted environment and tech support does not, yes its a bad system). suuma fucked around with this message at 16:33 on Apr 11, 2018 |
|
#
?
Apr 11, 2018 16:31
|
|
|
suuma posted:We have no QA and rolled out an update to a customer in Australia (we're east coast USA), so their poo poo was broken overnight and now we can't work with them to troubleshoot because we come in when they go home At an old job, we had people in Australia using the product but we closed at 8:00PM CST. Someone was on call basically for that 1 customer every night. We got a call maybe once every 6 months tops and got $50\week for it. 
|
|
#
?
Apr 11, 2018 16:31
|
|
|
"Everybody has a test system. Some people are lucky enough to have another system to run production in."
|
|
#
?
Apr 11, 2018 17:14
|
|
|
iospace posted:Which reminds me. Sounds like you need to talk to lowtax! Please do the needful!
|
|
#
?
Apr 11, 2018 23:09
|
|
|
suuma posted:We have no QA and rolled out an update to a customer in Australia (we're east coast USA), so their poo poo was broken overnight and now we can't work with them to troubleshoot because we come in when they go home do you work on the f-35?
|
|
#
?
Apr 11, 2018 23:20
|
|
|
iospace posted:Real IT pros and devs test in prod. We don’t have a test/training database for our ticketing system that we help the vendor develop. Everything gets ran straight in to production. I’m not the best at non-data pull SQL commands but I am the only one that gets to do them on this system. I accidentally brought down our whole reporting system a few weeks ago due to an incorrect parentheses. Worked fine when I used it for a small set of data but once it got opened up to the couple million entries in the database everything broke.
|
|
#
?
Apr 11, 2018 23:37
|
|
|
suuma posted:poo poo that works fine here doesn't work there so The disks spin the other way
|
|
#
?
Apr 11, 2018 23:39
|
|
|
Obsoletely Fabulous posted:We don’t have a test/training database for our ticketing system that we help the vendor develop. Everything gets ran straight in to production. I’m not the best at non-data pull SQL commands but I am the only one that gets to do them on this system. I accidentally brought down our whole reporting system a few weeks ago due to an incorrect parentheses. Worked fine when I used it for a small set of data but once it got opened up to the couple million entries in the database everything broke. That reminds me of a job interview I was on a few weeks ago, I talk to the IT director about their environment and their practices "Oh yeah we have pre-prod environment, dev environment, QA environment" etc etc. Then he brought in a DBA to ask me some questions and walked out of the room, at some point I asked him about how they do testing etc before rolling out software updates "Oh yeah that's kind of a shitshow, we test a lot in production, we don't really have a QA environment and none of the guys actually writing code know how to manipulate a DB properly so I'm constantly putting out fires when they push new code" ... one of these things is not like the other 
|
|
#
?
Apr 11, 2018 23:58
|
|
|
MF_James posted:That reminds me of a job interview I was on a few weeks ago, I talk to the IT director about their environment and their practices "Oh yeah we have pre-prod environment, dev environment, QA environment" etc etc. Then he brought in a DBA to ask me some questions and walked out of the room, at some point I asked him about how they do testing etc before rolling out software updates "Oh yeah that's kind of a shitshow, we test a lot in production, we don't really have a QA environment and none of the guys actually writing code know how to manipulate a DB properly so I'm constantly putting out fires when they push new code" ... one of these things is not like the other That's pretty shockingly honest, jesus. It must really be a dumpster fire if he warned you in the interview.
|
|
#
?
Apr 12, 2018 00:03
|
|
Cat Army
|
The Iron Rose posted:That's pretty shockingly honest, jesus. It must really be a dumpster fire if he warned you in the interview. That's how every small to medium business that develops does their poo poo, from what I've been exposed to. The best interview I had was a company that brought me in for my "scripting" knowledge and the fact that SQL was on my resume. The way they described it, they wanted me to help their Enterprise team develop a database driven configuration management platform where all clients would check in periodically, pull down any updated configurations, and report statistical data to the backend database. All configurations were XML templates stores in a CIFS share somewhere on the network. This was a Windows shop, and the interviewer wasn't familiar with Active Directory.
|
|
#
?
Apr 12, 2018 00:38
|
|
|
When I fill out a change request for something, there is a field in there that asks "Was this tested in the test environment?" I can't think of a time this will ever be anything but no. We can't get duplicate hardware to run in our DR because cost is an issue, how will we get hardware to run in test?
|
|
#
?
Apr 12, 2018 02:29
|
|
|
CitizenKain posted:We can't get duplicate hardware to run in our DR because cost is an issue Is this a one hand clapping kind of thing? How can you say you have a DR environment if there's no hardware? Is it just an empty room where the execs go to cry when they realize the money they're losing far outstrips the cost of a DR?
|
|
#
?
Apr 12, 2018 02:35
|
|
|
Our disaster recovery plan is the guy who doesn't work in the main office with the back up drives and tapes stored in his bedroom closet...
|
|
#
?
Apr 12, 2018 03:02
|
|
|
At a data center I worked out the DR plan was “dissolve the company” in contracts and people signed that poo poo.
|
|
#
?
Apr 12, 2018 03:15
|
|
|
The DR plan at my last place was "buy a bunch of poo poo from CDW and wait until it gets here assuming we have a facility at all" written in a Google Doc so pretty solid in the grand scheme I guess considering no one ever asked to actually see the DR plan itself, just if we had a DR plan at all.
|
|
#
?
Apr 12, 2018 04:04
|
|
|
Sheep posted:The DR plan at my last place was "buy a bunch of poo poo from CDW and wait until it gets here assuming we have a facility at all" written in a Google Doc so pretty solid in the grand scheme I guess considering no one ever asked to actually see the DR plan itself, just if we had a DR plan at all. I've referred to our DR plan as "Mostly prayer based." since we're probably FUBAR if things went really bad. They're working on one to have sites in other states support stuff, but that's gonna be a megaclusterfuck and I'm glad I don't have to manage it.
|
|
#
?
Apr 12, 2018 04:52
|
|
|
Last place I worked at had it designed pretty well. We had a development environment and a separate testing environment. Every 3 months. The dev and testing environment, would get copied from the production environment. This was to keep the dev and test environment up 2 date with production. It was really good.
|
|
#
?
Apr 12, 2018 07:21
|
|
|
Generally, non tech execs see what a proper DR plan costs and after finding out what it is for just scrap it (or severely neuter it) because "well we've never had that happen before  "And then they look great for saving the company a bunch of money. Of course the moment they realise that a DR plan needs to be enacted they are the first ones out the door swanning into a new job leaving everyone else to deal with the fallout. In the end, nothing is learned and the circle of life continues.
|
|
#
?
Apr 12, 2018 10:09
|
|
|
suuma posted:We have no QA and rolled out an update to a customer in Australia (we're east coast USA), so their poo poo was broken overnight and now we can't work with them to troubleshoot because we come in when they go home different system date formats? Ask me, an Australian, who has to run some servers with American date formats or certain software loving spaz's out when it rolls over to the 13 day of the month! Rudager fucked around with this message at 11:06 on Apr 12, 2018 |
|
#
?
Apr 12, 2018 11:04
|
|
|
Obsoletely Fabulous posted:We don’t have a test/training database for our ticketing system that we help the vendor develop. Everything gets ran straight in to production. I’m not the best at non-data pull SQL commands but I am the only one that gets to do them on this system. I accidentally brought down our whole reporting system a few weeks ago due to an incorrect parentheses. Worked fine when I used it for a small set of data but once it got opened up to the couple million entries in the database everything broke. Last week we deployed a set of code I worked on where I accidentally did a full table scan of a table that in our test data has like 1000 rows. In the prod database it has 340 million rows. When an account manager got on to test the fix, after we sent out the "code is deployed without issue!" email, loading the page I worked on would cause a web node to die from memory issues, but this was silent to the user and looked like just an unresponsive page. She then just kept hitting refresh. Oops!
|
|
#
?
Apr 12, 2018 12:24
|
|
|
Sefal posted:Last place I worked at had it designed pretty well. Ours is a nightly refresh of the databases from production, with WIP changes reapplied automatically to dev and test on an opt-in basis, so weird experiments and poo poo get cleaned out quickly. Life is good.
|
|
#
?
Apr 12, 2018 12:54
|
|
|
Rudager posted:different system date formats? It ended up being a bad build of our software. It's been updated and fixed. Yay no QA 
|
|
#
?
Apr 12, 2018 13:12
|
|
|
I once worked for a supplier of a non-essential website plugin for Sony UK. They at one point asked us what would happen to the service if London was nuked. Given we all lived and worked in London the internal response was "I don't think we'll give a poo poo".
|
|
#
?
Apr 12, 2018 14:13
|
|
|
Yeah it's like going through the DR plan about who comes into the office to get things running again after a war breaks out or something. Pretty sure I'll be more concerned with digging a bunker and stashing food away.
|
|
#
?
Apr 12, 2018 14:31
|
|
|
Our web app has a Contact Us page, where a user fills a form and it sends an email to our support@company.com address. An email came in:quote:Issue Report samsung on 5 pro smart phone motion button why
|
|
#
?
Apr 12, 2018 14:32
|
|
|
Sefal posted:Last place I worked at had it designed pretty well. Me: So, how often do we refresh our DEV environment? Boss: We don't. Me: Pardon? Boss: We can't do it because we test in our DEV environment. Me: I know, that's what it's for, but...the vendor doesn't take good care of DEV instances, and after each quarter they just start telling you to refresh the instance rather than fix anything. So we haven't refreshed since launch in Jan 17? Boss: Nope! I've been lobbying for months to just set a deadline and tell the developers "Hey, finish your testing by June 1 because we're nuking everything." Get shot down every time I bring it up. People are starting to test in production because DEV is falling apart. What could go wrong?
|
|
#
?
Apr 12, 2018 15:36
|
|
|
Thanks Ants posted:Yeah it's like going through the DR plan about who comes into the office to get things running again after a war breaks out or something. Pretty sure I'll be more concerned with digging a bunker and stashing food away. Yeah, I got grilled in an interview about DR/BC plans, went through 3 of our larger clients, 2 of which are based in hawaii, their DR/BC plans? Well, they have a cold/DR site on the islands. "But what if the islands get hit with a hurricane or some other natural disaster like a horrible volcano eruption?" "They are going to be too busy worrying about their families to care about work, also, the services they provide will be catastrophically down because they rely on physical infrastructure not being under magma"
|
|
#
?
Apr 12, 2018 17:09
|
|
|
MF_James posted:Yeah, I got grilled in an interview about DR/BC plans, went through 3 of our larger clients, 2 of which are based in hawaii, their DR/BC plans? Well, they have a cold/DR site on the islands. "But what if the islands get hit with a hurricane or some other natural disaster like a horrible volcano eruption?" "They are going to be too busy worrying about their families to care about work, also, the services they provide will be catastrophically down because they rely on physical infrastructure not being under magma" gently caress I hate that. I've dealt with asshats that keep escalating to more and more ridiculous situations until you're forced to say "I don't have an answer to how we'll deal with an earthquake swallowing our PROD location occurring simultaneously with a gas-main explosion at our DR site". Then the poo poo smugly says, "Well, I guess our DR/BC isn't comprehensive then, is it?"
|
|
#
?
Apr 12, 2018 17:23
|
|
|
Proteus Jones posted:gently caress I hate that. "By that point we will likely have ceased both daily operations as well as new hire onboarding, so yeah."
|
|
#
?
Apr 12, 2018 17:31
|
|
|
How much money do you have? I can make as many DR sites as you can afford, with the people to maintain them.
|
|
#
?
Apr 12, 2018 17:32
|
|
|
Well if, we are going to be all picky about it, you are supposed to have a DR site in a different geographical region: preferably on a different continent, ideally on a different tectonic plate. Justifiable if you are a major financial data processing site. Less justifiable if you're a download site for Bonzai Buddy
|
|
#
?
Apr 12, 2018 17:39
|
|
|
Just set up a mobile DR facility in a lifted off-road Mad Max RV so you can resume operations in the event of a global apocalypse and subsequent water war.
|
|
#
?
Apr 12, 2018 17:43
|
|
|
GreenNight posted:How much money do you have? I can make as many DR sites as you can afford, with the people to maintain them. How much would a DR site on the moon cost these days?
|
|
#
?
Apr 12, 2018 17:54
|
|
|
Truga posted:How much would a DR site on the moon cost these days? Pretty pricey, tapes don't like hot/cold, so you'd need a pretty robust box to store them in long term, with some pretty great AC and heat.
|
|
#
?
Apr 12, 2018 18:04
|
|
|
spog posted:Well if, we are going to be all picky about it, you are supposed to have a DR site in a different geographical region: preferably on a different continent, ideally on a different tectonic plate. A bank I worked for had to put their secondary datacenter in another state because Texas is all one power grid.
|
|
#
?
Apr 12, 2018 18:14
|
|
|
|
| # ? Apr 29, 2024 00:17 |
|
|
nullfunction posted:A bank I worked for had to put their secondary datacenter in another state because Texas is all one power grid. Texas makes a decent DR location for this same reason too.
|
|
#
?
Apr 12, 2018 18:16
|
|
