Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

 
alt

Daniel Nashed

 

Correctly Stopping a Traveler Server

Daniel Nashed  24 October 2017 13:02:57
This is not new but I ran into this a couple of times on customer site. Specially on a Traveler HA server this becomes important.

Shutting down the Traveler servertask when the Domino server/service is stopped might lead to hang situations of the HTTP task.

The better way would be to shutdown the Traveler servertask first. But even that might lead to undesired results.


There is a special Traveler shutdown command "tell traveler shutdown" that can be used to let Traveler finish it's work and not accept any new requests before cleanly shutting down.


When you configured NTS_AUTOSTART_HTTP=true the Traveler task did start the HTTP task automatically but by default the HTTP task will not be shutdown automatically when you just shutdown the Traveler servertask.

If you configure NTS_AUTOSTOP_HTTP=true the Traveler servertask will automatically take care of shutting down the HTTP task in the right moment.


This is important for Traveler HA environments because the Traveler OSGi servlet will only connect to the local Traveler servertask which will than either process the request or forward it to another Traveler server (if another server holds the master monitor for the user).

So if you don't shutdown the HTTP task your load-balancer might still send requests to the Traveler server which reports back that the Traveler Server is not available ("IBM Traveler server is not available.").


That's an undesirable result for a load-balancer because usually the load-balance does not check for this status and might continue to send devices to that Traveler server.


It makes sense to configure the server to be either fully available or to not respond to HTTPS requests if the Traveler service is not available.

This is true if you shutdown the Traveler task or also during shutdown of the whole server.


My Domino start script for Linux allows to configure a pre-shutdown command which can be used to shutdown Traveler before you shutdown the whole server.


-- /etc/sysconfig/rc_domino_config --


# -- Command to execute before shutting down the Domino server --

DOMINO_PRE_SHUTDOWN_COMMAND="tell traveler shutdown"


# -- Delay before shutting down the Domino server after invoking the pre-shutdown command --

DOMINO_PRE_SHUTDOWN_DELAY=10


There is no simple solution for Windows because Domino is started as a Windows service.
But if you shutdown your server manually, you should shutdown the Traveler server and also use the notes.ini setting.


Sadly if you shutdown the whole server all processed will receive the quit command at the same time and will start to shutdown.
This means it could come to undesired timing issues when HTTP and Traveler do their shutdown at the same time.

A controlled Traveler server shutdown would be the cleaner solution.


Update to make it more clear:

Yes there is a big difference between using tell traveler quit and tell traveler shutdown!

If you quit a servertask the task has only limited time to continue to work with full resources available.
If a task or a server is in pending shutdown some resources are not available any more and cannot fully controlled terminate pending work.

Here is the decumentation for the shutdown command:

Shutdown [command]     - Stop accepting new work requests, allow current work to complete, and then Quit.  
If you specify a [command], then this command will be run after the server is idle but before it quits.

So it makes a lot of sense to use shutdown instead of quit.

------------


Another tip that might be still new for some customers. If your load-balancer or other monitoring software support authenticated (with user credentials), SSL enabled probes, there is a very nice feature in Traveler to allow an end to end check for your Traveler server availability.


HTTPS://traveler.acme.com/traveler?action=getStatus

Depending on your language settings you either get


a.) IBM Notes Traveler server is available.


b.) IBM Notes Traveler server is not available.

Or an error message for the end to end check for the user you specified. The user must be allowed to user Traveler and have a proper mailfile for this end to end test.

c.) The IBM Notes Traveler server cannot connect to your mail database mail/jdoe.nsf on server CN=domino.acme.com/OU=Srv/O=Acme.  Verify that your mail server mail database grants access to server CN=domino.acme.de/OU=Srv/O=Acme and is operational.  If this does not resolve the problem, your administrator may need to verify the network connection between the servers and that the IBM Notes Traveler server is allowed to access your mail server.


So I would recommend to
  • always use the "tell traveler shutdown" command first
  • enable HTTP Auto shutdown -> notes.ini NTS_AUTOSTOP_HTTP=true
  • use the options in my start script if you are running on Linux
  • have a look into the ?action=getStatus command if you didn't use that before
-- Daniel

Comments

1Jay Marme  24.10.2017 16:54:05  Correctly Stopping a Traveler Server

This is good information. I use tell traveler shutdown when patching but would also like a solution for OS restarts that are the result of Windows patching. Perhaps a program document to issue a tell traveler shutdown at the time the OS patches are scheduled to be applied?

2Craig Wiseman  25.10.2017 12:32:20  Correctly Stopping a Traveler Server

Thanks for this!

Is there a distinction I should be aware of between

"tell traveler q"

and

"tell traveler shutdown"?

I've always used "q" to shutdown traveler before http.

3Daniel Nashed  25.10.2017 12:42:13  Correctly Stopping a Traveler Server

Yes there is a big difference!

If you quit a servertask the task has only limited time to continue to work with full resources available.

If a task or a server is in pending shutdown some resources are not available any more and cannot fully controlled terminate pending work.

Here is the decumentation. Let me update the blog post to provide those details more clearly.

Shutdown [command] - Stop accepting new work requests, allow current work to complete, and then Quit.

If you specify a [command], then this command will be run after the server is idle but before it quits.

4Craig Wiseman  25.10.2017 13:02:39  Correctly Stopping a Traveler Server

Again, thanks!

I missed that distinction (obviously) and greatly appreciate the clarification.

5Thibaud Maes  25.10.2017 15:17:15  Correctly Stopping a Traveler Server

Thanks!

Didn't know the NTS_AUTOSTOP_HTTP setting!

I always instructed to perform a "tell http quit" before stopping the Traveler task and finally the server. This would then stop incoming requests from the load balancer but still allow the Traveler to handle requests load balanced from other Traveler servers until the stopping of that task.

greets!

6Daniel Nashed  25.10.2017 22:11:04  Correctly Stopping a Traveler Server

@Thibaud, no you should not stop HTTP before Traveler.

Traveler first needs to ensure requests are handled. Else the HTTP task might hang on shutdown.

On the other side you want HTTP to not take any new requests during shutdown.

The best way is really to enable the auto HTTP shutdown and to stop Traveler with shutdown before stopping the server.

7Mahesh Dhuri  08.01.2018 16:19:11  Traveler: Other servers in the HADR pool are alive, but none are reachable.

Hello Daniel ,

There is frequently crash issue on Traveler server due to error Message PANIC: Invalid pool free chain therefore IBM has prepared HF to fix the same and needs to be install on FP9.

on 31 Dec 2017 we have installed FP9 and installed HF provided by IBM however after that we can see none of the Traveler servers in HA pool therefore we have un-installed FP9 along with HF however at the moment nothing is working for us , we have raised Severity 1 PMR with IBM 33552,300,624 however we are not getting and response from IBM as well from SWAT.

So far we have done below steps however no success.

- uninstalled FP9 as well HF on 2 Traveler servers out of 4.

- We have 4 Traveler servers however no server in HA pool

- We have re-added servers from HA rebooted all servers at OS level ,checked with Network and SQL team however nothing found any wrong.

We are now facing issue since long time and not getting proper response from IBM even raised Severity 1 PMR :(

your help would be highly appreciated.

kind regards,

Mahesh Dhuri

8MAHTAB QAISER  30.11.2021 23:38:00  Correctly Stopping a Traveler Server

Wondering if my environment serving applications also on http as well as traveler. Recommended steps at the end of this post is good for my environment too.

setup the notes.ini parameter NTS_AUTOSTOP_HTTP=true

a program document to shutdown the traveler server

a program document to restart the domino server.

Is that a right approach?

9Daniel Nashed  03.12.2021 14:26:06  Correctly Stopping a Traveler Server

@MAHTAB QAISER

The NTS settings are intended to start and stop the HTTP task with Traveler.

So you don't have to load the HTTP task separately and Traveler is in control of the HTTPs.

As a startup programm document you start Traveler.

But there is only a shutdown program document when you want to scheduled shutdown a task.

in that case you use server -c tell traveler shutdown.

But this does not help if you shutdown your Domino server. There is no shutdown server program document.

In case you are using Linux my start-script has a pre-shutdown command, which has a predefined, commented out shutdown option for Traveler...

-- Daniel

Links

    Archives


    • [HCL Domino]
    • [Domino on Linux]
    • [Nash!Com]
    • [Daniel Nashed]