Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

Correctly Stopping a Traveler Server

Daniel Nashed  24 October 2017 15:02:57
This is not new but I ran into this a couple of times on customer site. Specially on a Traveler HA server this becomes important.

Shutting down the Traveler servertask when the Domino server/service is stopped might lead to hang situations of the HTTP task.

The better way would be to shutdown the Traveler servertask first. But even that might lead to undesired results.


There is a special Traveler shutdown command "tell traveler shutdown" that can be used to let Traveler finish it's work and not accept any new requests before cleanly shutting down.


When you configured NTS_AUTOSTART_HTTP=true the Traveler task did start the HTTP task automatically but by default the HTTP task will not be shutdown automatically when you just shutdown the Traveler servertask.

If you configure NTS_AUTOSTOP_HTTP=true the Traveler servertask will automatically take care of shutting down the HTTP task in the right moment.


This is important for Traveler HA environments because the Traveler OSGi servlet will only connect to the local Traveler servertask which will than either process the request or forward it to another Traveler server (if another server holds the master monitor for the user).

So if you don't shutdown the HTTP task your load-balancer might still send requests to the Traveler server which reports back that the Traveler Server is not available ("IBM Traveler server is not available.").


That's an undesirable result for a load-balancer because usually the load-balance does not check for this status and might continue to send devices to that Traveler server.


It makes sense to configure the server to be either fully available or to not respond to HTTPS requests if the Traveler service is not available.

This is true if you shutdown the Traveler task or also during shutdown of the whole server.


My Domino start script for Linux allows to configure a pre-shutdown command which can be used to shutdown Traveler before you shutdown the whole server.


-- /etc/sysconfig/rc_domino_config --


# -- Command to execute before shutting down the Domino server --

DOMINO_PRE_SHUTDOWN_COMMAND="tell traveler shutdown"


# -- Delay before shutting down the Domino server after invoking the pre-shutdown command --

DOMINO_PRE_SHUTDOWN_DELAY=10


There is no simple solution for Windows because Domino is started as a Windows service.
But if you shutdown your server manually, you should shutdown the Traveler server and also use the notes.ini setting.


Sadly if you shutdown the whole server all processed will receive the quit command at the same time and will start to shutdown.
This means it could come to undesired timing issues when HTTP and Traveler do their shutdown at the same time.

A controlled Traveler server shutdown would be the cleaner solution.


Update to make it more clear:

Yes there is a big difference between using tell traveler quit and tell traveler shutdown!

If you quit a servertask the task has only limited time to continue to work with full resources available.
If a task or a server is in pending shutdown some resources are not available any more and cannot fully controlled terminate pending work.

Here is the decumentation for the shutdown command:

Shutdown [command]     - Stop accepting new work requests, allow current work to complete, and then Quit.  
If you specify a [command], then this command will be run after the server is idle but before it quits.

So it makes a lot of sense to use shutdown instead of quit.

------------


Another tip that might be still new for some customers. If your load-balancer or other monitoring software support authenticated (with user credentials), SSL enabled probes, there is a very nice feature in Traveler to allow an end to end check for your Traveler server availability.


HTTPS://traveler.acme.com/traveler?action=getStatus

Depending on your language settings you either get


a.) IBM Notes Traveler server is available.


b.) IBM Notes Traveler server is not available.

Or an error message for the end to end check for the user you specified. The user must be allowed to user Traveler and have a proper mailfile for this end to end test.

c.) The IBM Notes Traveler server cannot connect to your mail database mail/jdoe.nsf on server CN=domino.acme.com/OU=Srv/O=Acme.  Verify that your mail server mail database grants access to server CN=domino.acme.de/OU=Srv/O=Acme and is operational.  If this does not resolve the problem, your administrator may need to verify the network connection between the servers and that the IBM Notes Traveler server is allowed to access your mail server.


So I would recommend to
  • always use the "tell traveler shutdown" command first
  • enable HTTP Auto shutdown -> notes.ini NTS_AUTOSTOP_HTTP=true
  • use the options in my start script if you are running on Linux
  • have a look into the ?action=getStatus command if you didn't use that before
-- Daniel

Comments

1Jay Marme  24.10.2017 18:54:05  Correctly Stopping a Traveler Server

This is good information. I use tell traveler shutdown when patching but would also like a solution for OS restarts that are the result of Windows patching. Perhaps a program document to issue a tell traveler shutdown at the time the OS patches are scheduled to be applied?

2Craig Wiseman  25.10.2017 14:32:20  Correctly Stopping a Traveler Server

Thanks for this!

Is there a distinction I should be aware of between

"tell traveler q"

and

"tell traveler shutdown"?

I've always used "q" to shutdown traveler before http.

3Daniel Nashed  25.10.2017 14:42:13  Correctly Stopping a Traveler Server

Yes there is a big difference!

If you quit a servertask the task has only limited time to continue to work with full resources available.

If a task or a server is in pending shutdown some resources are not available any more and cannot fully controlled terminate pending work.

Here is the decumentation. Let me update the blog post to provide those details more clearly.

Shutdown [command] - Stop accepting new work requests, allow current work to complete, and then Quit.

If you specify a [command], then this command will be run after the server is idle but before it quits.

4Craig Wiseman  25.10.2017 15:02:39  Correctly Stopping a Traveler Server

Again, thanks!

I missed that distinction (obviously) and greatly appreciate the clarification.

5Thibaud Maes  25.10.2017 17:17:15  Correctly Stopping a Traveler Server

Thanks!

Didn't know the NTS_AUTOSTOP_HTTP setting!

I always instructed to perform a "tell http quit" before stopping the Traveler task and finally the server. This would then stop incoming requests from the load balancer but still allow the Traveler to handle requests load balanced from other Traveler servers until the stopping of that task.

greets!

6Daniel Nashed  26.10.2017 0:11:04  Correctly Stopping a Traveler Server

@Thibaud, no you should not stop HTTP before Traveler.

Traveler first needs to ensure requests are handled. Else the HTTP task might hang on shutdown.

On the other side you want HTTP to not take any new requests during shutdown.

The best way is really to enable the auto HTTP shutdown and to stop Traveler with shutdown before stopping the server.


  • [IBM Lotus Domino]
  • [Domino on Linux]
  • [Nash!Com]
  • [Daniel Nashed]