Tuesday, August 16, 2016

Step-by-step client-server troubleshooting

Step-by-step client-server troubleshooting


As a software developer I want to describe one scenario repeating itself with clockwork persistence:
Somebody received something unexpected from the client working with a server and don't want know what to do with that.

Well then, take my hand and follow the path...

What's happened?

Usually in client server architecture the only part the victim has at hand is a client: usually it's a library or CLI that just proxies requests for service via some evil magic. So if client lib/CLI says you something you shouldn't believe it. Remember - it's a magic!

All you can say looking at some error message instead of expected nice lines of data or a status code, encouraging you with 'everything seems OK', is that 'client has something not matching positive result'.

Here comes first precaution: make sure you're using the right version of client! I'm serious: even if you've just installed a shiny new environment from a single distributive, the latter can be spoiled, and you can't bet your life it's perfect. So just do it.

For a lot of people it's really surprising that returned by the client error message may mean something, so read it!

At this point you should be able to make a decision: if it's the server malfunctioning or you've passed something wrong to the client from the beginning. The chances of latter are very high if the problem has occurred suddenly.

Enemy unknown

Well, here is the point where you usually believe you've done all right. It's always good to be sure you are not the one to blame, but it doesn't solve the problem.

The next level of this dungeon is the logging system, so make sure both client and server (especially server) log detailed messages. You may want to check some configuration files, get some documentation, find it obsolete, figure something out, and finally - has the detailed debug logging enabled.

ALARM: make sure you are about to analyze the right logs! It's a common case when logs are "somewhere else".

  1. Find traces of your problem in the logs of your client (if you are using CLI - just look to the screen)
  2. Mark the time of the incident!
  3. Find that time in server logs and see what happened there

Where are my logs?!

Haha! Have you forgotten about any proxies in between? ;)

There is nothing in the logs!

Really? Are you sure you skipped nothing? Well, call for a dev then...