Search Amazon.com:
Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining us 94,000+ Goons, for the one-time price of $9.95! We charge money because it costs us $3,400 per month for bandwidth bills alone, and since we don't believe in shoving popup ads to our registered users, we try to make the money back through forum registrations.
  • Post
  • Reply
tronester
Aug 12, 2004
People hear what they want to hear.

I came across an issue with a client of ours yesterday.

They are set up with 4 2008 server boxes, one running as a database server for our software, and TS license manager, as well as TS session broker. The other three run as terminal servers.

What happened is that all of a sudden yesterday, all four servers somehow lost their trust tokens with the clients domain controller (a different server).

I had to fix the issue by disjoining all four servers from the domain, then rejoining them.

I have had this issue happen before on laptops that were not connected to the network for several weeks at a time, but never with servers.

What typically causes this?

The only abnormality I can think of is that the 4 servers were on their own Cisco switch/router, on a different subnet than any other equipment. Perhaps the Cisco device went down briefly causing the loss of sync?

Adbot
ADBOT LOVES YOU

BangersInMyKnickers
Nov 03, 2004

I have an oral fixation and it's not the sexy kind

Were the system event logs complaining about communication problems with the DC(s) before everything crapped out?

tronester
Aug 12, 2004
People hear what they want to hear.

BangersInMyKnickers posted:

Were the system event logs complaining about communication problems with the DC(s) before everything crapped out?

LOL, thats the part I forgot to mention. The client is in a very remote location states away, and consequently their internet access goes down often (even though they have a business grade T1).

I was unable to look at any of the logs, because I had to instruct one of their IT guys how to do all this over the phone.

He should be calling me back though later after the internet access comes back up, so I can go over the logs.

Nebulis01
Dec 30, 2003
Technical Support Ninny

BangersInMyKnickers posted:

Were the system event logs complaining about communication problems with the DC(s) before everything crapped out?

This would be my guess as well. Isn't the default Kerberos token valid for like 28 days though?

BangersInMyKnickers
Nov 03, 2004

I have an oral fixation and it's not the sexy kind

I can't remember the exact interval, but computer accounts in AD automatically update their password against the DC. There is a grace period where the old password becomes stale and needs to be refreshed and if you exceed that you can get exactly what you are describing where they just drop off the domain. Check those logs when you can, but if that is your situation then you may be better off dropping off a DC at that location and setting up a second domain site in AD and configuring the replication window between that site and your primary. That would leave something local to authenticate against in the (apparently likely) chance of network interruptions.

citywok
Sep 08, 2003
Born To Surf

We've got a site 7,000 miles away in the Philippines which has typhoons all the time taking the internet down, and we have considerably fewer issues than you do. it sounds like there has go to be a better connectivity solution for you out there somewhere than what you are using because it sounds like a pair of cups and a string.

CannedMeat
Apr 29, 2009


If your WAN link to the domain controller is that unreliable I'd suggest making one of your servers at the remote site a domain controller.

It's much easier to make sure AD replication is working from the one DC, than to check each server individually for errors in the log.

tronester
Aug 12, 2004
People hear what they want to hear.

Sorry, I wasn't too clear....

Their Domain controller is actually in the same room as the servers loaded with our software. The link speed between them is 100mbits. Which is why I am so confused on how this happened.

Like I said the switch or router that has our 4 servers on it is on a different subnet than the rest of their network, it has to go through a gateway (one hop) when I do a trace route from our servers to their domain controller. Thats about the only thing out of the ordinary. Unfortunately I am not sure how their network topology is set up.

TheRife
Jul 13, 2007

Philadelphia Sports: Classy and Sophisticated.



Its also possible someone accidentally deleted them out of active directory. If the domain is under their control and someone was "tinkering" with AD, this is most likely. That or something happened with the routing between those two subnets.

tronester
Aug 12, 2004
People hear what they want to hear.

I doubt anyone deleted the servers out of the active directory. They only have 3 IT people, and I don't know if they even know how to do that.

It probably does have something to do with routing between the subnets.

The device our 4 servers is plugged into has like 96 gigabit ports on it, they just never implemented it for the rest of their network, we plugged our stuff into it.

They have outside IT contractors that put all the Cisco networking equipment in. I think I will email them and see if they have any ideas.

LoKout
Apr 02, 2003

Professional Fetus Taster

Could be a silly question but is the local DC doing DNS also? AD requires DNS to do lookups for service providers (Kerberos and LDAP), so sometimes if the servers have trouble with DNS they'll fall off the domain. Also make sure that all the servers have that local DNS provider as their primary.

Kerberos tickets are renewed every 10 hours, though they don't expire for days and connectivity can be reestablished for about a month like previously mentioned. The only oddity you'll see without additional event logging enabled would be an inability for new users to log on to the boxes in question.

tronester
Aug 12, 2004
People hear what they want to hear.

LoKout posted:

Could be a silly question but is the local DC doing DNS also? AD requires DNS to do lookups for service providers (Kerberos and LDAP), so sometimes if the servers have trouble with DNS they'll fall off the domain. Also make sure that all the servers have that local DNS provider as their primary.

Kerberos tickets are renewed every 10 hours, though they don't expire for days and connectivity can be reestablished for about a month like previously mentioned. The only oddity you'll see without additional event logging enabled would be an inability for new users to log on to the boxes in question.

The local DC does DNS also, yes.



I checked the error logs, the only thing I could find was that the system lost contact with the domain controller, thus it could not apply group policy.

Rate Thread:
  • Post
  • Reply