Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
Spring Heeled Jack
Feb 25, 2007


Alright, I'm almost at my wits end with these guys. We've had issues with this vendor before and when we do, 99% of the time it's because of something stupid they've setup on their end.

We have a payment transaction that originates from a server on our network and submits it to them. For some currently unknown reason, transactions between 10:00am and about 10:15 have a very high likelyhood of failing. So the transaction gets all the way to their server, they take the payment, but they never relay back to us that they did, causing it to fail on our end even though they're already taken money. Not a good situation for the customer.

We have been unable to identify anything on our end running during this timeframe that would impact these transactions, and they haven't reported anything either. So we setup a call and run some packet caps on the app servers directly to correlate. Our next steps are to get gateway captures.

Our side:



Their side (they are 1hr behind us):




I'm not the best at reading packet caps but it looks like they are sending us the acknowledgement that they received the payment but its not making it to our server.


For reference, a capture of a completed transaction from our side:



Perhaps I'm reading the packet cap wrong but I would like if possible to get another set of eyes on this issue as a sanity check.

Adbot
ADBOT LOVES YOU

Cyks
Mar 17, 2008


Not the most experienced wireshark reader but it looks to me your side is closing the connection after waiting two minutes with no traffic and their end is finally getting around to sending their data a minute after the connection has already been terminated, for a transmission that should take all of 7 seconds from start to finish. Sounds like an issue on their end.

From personal experience, a service having issues at a specific time every day is from a server unable to process the number of requests it's receiving, either from legitimate traffic to a service or a administrative feature scheduled at a bad time.

Spring Heeled Jack
Feb 25, 2007


Cyks posted:

Not the most experienced wireshark reader but it looks to me your side is closing the connection after waiting two minutes with no traffic and their end is finally getting around to sending their data a minute after the connection has already been terminated, for a transmission that should take all of 7 seconds from start to finish. Sounds like an issue on their end.

From personal experience, a service having issues at a specific time every day is from a server unable to process the number of requests it's receiving, either from legitimate traffic to a service or a administrative feature scheduled at a bad time.

I should have clarified but yeah their firewall timeout is 120s so thatís why itís closing us off at that interval. Looking at it again I do see that theyíre trying to finish the transaction a minute after their firewall cuts us off. So it would appear that is the issue, or at least the indication of the issue on their end somewhere.

Weíve had tons of issues with this vendor in the past (rate limiting in their end, misc firewall fuckups, etc) and this is something that just started happening out of the blue about a month ago. They arenít admitting to anything yet so itís been my task to figure out the source of the issue. We canít identify any jobs or processes on our end that line up the the timeframe, so Iím 99.9% certain itís something theyíre doing.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply