Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Locked thread
Ion Mage
Dec 21, 2006
I've been working with a small startup, which is aiming to create a phone app similar to uber. Currently, everything is still run on a single server. That is, both our website, as well as the server-side service that our app uses. The guy who originally set this server up has since left, well before I was onboard, so I never even met the guy. I have been hired to be the tech guy in his stead. Point is, I'm admin to a system which I did not set up myself, and for which I am the ONLY admin. Any technical questions I have need to be answered by myself alone. Even the guys making the app itself have little involvement with this server.

A few weeks ago, I had to install a new SSL certificate for our website, as the old one was due to expire. In order to do this, I ran into some technical issues and ultimately had to resort to running windows update to resolve them. There were over 200 updates to be installed, with the last update being over a year ago. This resolved the issue. This is a windows server 2012 R2 system.

A few days later, our website went down. Only the website. Our app's service was uninterrupted, and I was able to remote into the system as well. Once remoted in, I saw nothing wrong. We ultimately tried rebooting the system, and that worked. Subsequent searches of the event viewer and other logs turned up nothing to explain the outage.

This has recurred every few days. Once occurred at 2 AM, while I was asleep, and that let us observe that the problem self-resolved somehow. After about 45 minutes, the website went back up, with no human intervention.

There is no set time in which this happens. Its happened at various points in time, so I don't think its the side effect of any regularly scheduled thing.

The closest I can come up with is that server manager's logs had a sizable chain of ASP.net events throwing unhandled exceptions.

Also, as I mentioned earlier, there were a lot of updates that got installed, I suspect that one of them has thrown something off, but at 200 updates, pinpointing the right one is difficult, and thats assuming that is even the cause.
I doubt its the recently installed SSL certificate, if that was bad then people would be getting unsigned certificate warnings or somesuch, yes? I don't see why such a thing would outright bring down a web page

The site is running on IIS 8.5. The actual physical server is hosted by 1and1 hosting. The fact that the app's service was still working, as was the (interrelated) SQL service would indicate strongly that its not a hardware issue.

Ion Mage fucked around with this message at 04:33 on Dec 2, 2016

Adbot
ADBOT LOVES YOU

Alereon
Feb 6, 2004

Dehumanize yourself and face to Trumpshed
College Slice
Why are you even looking into this at all as opposed to turning up a new server you can support and maintain on a hosted cloud provider?

incoherent
Apr 24, 2004

01010100011010000111001
00110100101101100011011
000110010101110010

Ion Mage posted:

The actual physical server is hosted by 1and1 hosting.

Just. Stop.

Thanks Ants
May 21, 2004

#essereFerrari


Use a PaaS provider.

Ion Mage
Dec 21, 2006

Alereon posted:

Why are you even looking into this at all as opposed to turning up a new server you can support and maintain on a hosted cloud provider?

Actually, we plan to do something like that at a later date. I'm simply just curious to know what the problem is. And such a migration is still a ways off.

incoherent posted:

Just. Stop.

Sorry, please elaborate. It wasn't my decision, if thats what you mean.

JBark
Jun 27, 2000
Good passwords are a good idea.
This sounds kinda like when IIS spins down the worker process after the website hasn't been accessed in a while, then takes a while to spin it back up. If it's a really slow VM or running on really crappy hardware, maybe it's just taking a really long time to come back up and it looks like it is down? Usually in that case, you can just wait a couple minutes, then try the page load again and it will work.

I think there's an idle timeout in the advanced settings for the application pool that you can set to 0 to prevent this from happening. It's pretty useful for websites that are infrequently used, otherwise you find yourself waiting for it to come back up all the time.

incoherent
Apr 24, 2004

01010100011010000111001
00110100101101100011011
000110010101110010

Ion Mage posted:

Sorry, please elaborate. It wasn't my decision, if thats what you mean.

1and1 is a p. cruddy hosting service. If you're serious about hosting a scale sensitive app on one server\service on IIS you should bone up on managing iis by reading this book

https://www.manning.com/books/learn-windows-iis-in-a-month-of-lunches

Aunt Beth
Feb 24, 2006

Baby, you're ready!
Grimey Drawer
IIS will also auto-recycle app pools when a certain threshold is reached, memory usage or uptime. Might be running out of memory and unable to restart when certain processes are running.

GRINDCORE MEGGIDO
Feb 28, 1985


incoherent posted:

1and1 is a p. cruddy hosting service. If you're serious about hosting a scale sensitive app on one server\service on IIS you should bone up on managing iis by reading this book

https://www.manning.com/books/learn-windows-iis-in-a-month-of-lunches

I wonder which is worse - Heart Internet vs 1and1. The last company I worked for used Heart shared / vps / dedicated and it was as shittastic as you'd expect.

ohgodwhat
Aug 6, 2005

Dude I'll be honest I don't expect great things to come of your startup. Good luck.

GRINDCORE MEGGIDO
Feb 28, 1985


I was in a similar situation, working for a web dev company, when the programmer hosed off and left me with servers to administer.

I read a poo poo load of documentation on that hosting environment, Apache (Not IIS, or I'd be more specific), in particular, monitoring services and server resources, and becoming at least familiar with using logs, so that when the inevitable problem happens, you have more to research then "it's offline again at the same time". You'd be able to at least find out what's misbehaving.
I recommend doing that ASAP, because from personal experience, not being able to fix a fundamental problem like that when it's your job to be able to really sucks. (EFB by Ion Mage).

I mean... not to be a total dick, really, but so far, it sounds like you are just observing that it goes offline regularly.

GRINDCORE MEGGIDO fucked around with this message at 03:39 on Dec 26, 2016

Adbot
ADBOT LOVES YOU

jre
Sep 2, 2011

To the cloud ?



Hey op,

Is this a dedicated server or a vm ?
Does this server have a public ip or is there a load balancer in front of it ?

When you say the site is down, what do you actually mean by that ? Are you getting 500s or are requests to the server just timing out ?
If you look in the web server request logs, is there a gap in requests which corresponds with the downtime ?

What does the CPU usage per process, network usage, disk io per process , memory usage look like when it's normal. How does this change when the server is down ?

quote:

The guy who originally set this server up has since left, well before I was onboard, so I never even met the guy.
And you've changed all the passwords since the previous admin left, possibly not amicably ?

  • Locked thread