Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

Seemless Cluster Failover

Daniel Nashed  26 February 2010 15:55:23


In Notes 7 you had to close a Notes database and reopen it to failover to a different server. The Notes Client did only use failover when opening a database with the right open options (failover options).
Notes 8 asks the user to switch to a different replica when the server goes down. This works in multiple areas including a view and also when you try to save a document and your server is down!

In many cases users have no idea what to do with the prompt message. There is a notes.ini setting HidePromptFailoverInc=1 which disables the failover prompt and automatically fails over without user interaction needed.

This does work for some but not all of the failover operations. For example if you try to save a document you are still prompted if you want to save the document to a different replica.

-- Daniel


LicenceKeyIssuesWithDomino

Daniel Nashed  5 February 2010 15:20:34

Domino 8 uses the IBM Licence Management routines (LUM) and ships with a full licence key in each release when you download it from the Passport or Partnerworld website. If you download it from the public demo locations it is timebombed.
I don't know why the full licensed code has licence keys. They are the same for each major release anyway.

They cause specially issues when you try to install multiple versions and mix different data directories and binary directories. Also when moving a data directory you can run into issues.
Those issues still occur in 8.5.1. Because the code on Windows still queries a registry key to query the location of the licence files.
The location is determined by Key: HKEY_LOCAL_MACHINE\SOFTWARE\Lotus\Domino\ Value: DataPath="c:\\Lotus\\Domino85\\Data"
This is true also for partitioned servers. In my installation this key was updated with the second partition but also the first partition used this location for license checking.

There are two files involved which are version depending Domino8.lic and nodelock. So those files from a 8.0 release will not work for 8.5.

So in cases where you see an error message like the following, you should check your versions and the registry values.

-- Daniel


An error occurred during license use management initialization.
Ensure that you are running Domino with a valid license file.
(OK)




ID Vault Support for iNotes Traveler BES

Daniel Nashed  27 January 2010 10:22:06

Since 8.5.1 Domino has support for external applications to leverage ID Vault to deploy Notes.IDs into the mailfile.
The current versions of Traveler, BES and iNotes do already support this functionality.
To get this working you need a special setting in the Security Policy / ID Vault section you have to enable "Allow Notes-based programs to use the Notes ID Vault" as shown below.




The second important part is that the application uses a new parameter in the C-API call to extract the Notes.ID.
This will hopefully be documented in the next C-API toolkit but has been mentioned in the security presentation (ID204) at Lotusphere last week.

STATUS LNPUBLIC SECExtractIdFileFromDB(
        DBHANDLE hDB,
        char *pProfileNoteName,
        DWORD ProfileNoteNameLength,
        char *pUserName,
        DWORD UserNameLength,
        char *pPassword,
        char *pPutIDFileHere,
        DWORD Reserved,
        void *pReserved);

To have this functionality enabled in your code you need to pass the full qualified, expanded name (CN=John Doe/O=Acme) to the pReserved parameter.
If the parameter is not passed and the ID is not already loaded into the profile you will receive an error.
In case the parameter if filled the code will automatically leverage ID Vault to download the Notes.ID from the Vault and attach it to the profile.

This is a very convenient way to deploy Notes.IDs for iNotes, Traveler and BES users with the current release.
If you are an ISV, leveraging this routine you should update your code to pass the pReserved Parameter to enable this functionality.

-- Daniel



Traveler 8.5.1.1 Encryption and Accepting Invitations on iPhone

Daniel Nashed  26 January 2010 15:46:16

Shortly before Lotusphere IBM shipped an update for the Lotus Traveler software.
Version 8.5.1.1 corrected a couple of issues and adds support for accepting invitations on the iPhone. Can also check with the native iPhone dialogs (calendar display) if the meeting fits in.

In addition to this IBM added support for reading encrypted mails for iPhones.
Because the iPhone itself does not support encypted mails IBM wrote a native iPhone helper app which you can download for free in the app store "IBM Lotus Notes Traveler Companion".
This app can be used to connect to the server to display encrypted messages.
You need to upload your Notes.ID to the mailfile (same way as iNotes or BES) and the application prompts you for the ID password to decypt the message and display it in the app.

The app needs to be configured (server name, user name, password) and you have to specify some settings on the Traveler server.

Both features need Traveler 8.5.1.1 and also need some configuration changes in NTSConfig.xml.
The features are both disabled by default and you have to check the readme for details (http://www.ibm.com/support/docview.wss?uid=swg24019529&aid=2).

Thanks for those two new great featues!
Both have been requested by many customers


-- Daniel

performance impact with private on first use views/folders

Daniel Nashed  22 December 2009 20:44:30

The following issue exists for a very long time. But it took until last year until it has been discovered. We have found it at two customers with performance problems using annotated client_clock data and it took a while to convince IBM support hat this can have a bigger performance impact for customer -- specially when using WAN connections.

Meanwhile the problem has been described in a technote (http://www.ibm.com/support/docview.wss?uid=swg21322578) / SPR# RSTN7K2EM4 with information from our PMR but I want to give some additional background we figured out.

-- Daniel


Technical background:

Notes/Domino uses an internal cache for design elements to avoid the overhead of searching design elements every time. The design cache is leveraged by the API like NIFFindDesignNote. This works locally and specially also for remote transactions. It's a single transaction invoked on client side to request the NoteID of the design element.

The design cache is based on a kind of hash table using the names of the design elements. If you use private on first use views/folders at some point (depending on the length of the names of the design elements and some other details) the design cache overflows (due to internal structure issues) and is completely disabled. This happens with about 40 entries as described in the technote.

Once this happened the Notes Client will try to open the design collection (a kind of special view) and reads thru the design collection step by step requesting buffer by buffer with data from the design collection. When you are lucky the design element is in the first part of the collection but most of the time it is in the middle or end of the collection. The more design elements the database has the more reads the client has to perform. And specially when a design element cannot be found always the full design collection is read.

Here is a real world example from a customer application how the transactions look when the design cache works and when it is broken.

With design cache

(3-507 [331]) FINDDESIGN_NOTES: 0 ms. [34+16=50] 

Without design cache
 
(3-299 [309]) OPEN_COLLECTION(REPC1256E62:004B8651-NTFFFF0020,0000,0000): 0 ms. [42+34=76] 
(4-299 [310]) READ_ENTRIES(REPC1256E62:004B8651-NTFFFF0020): 0ms. [76+65148=65224] 
(5-299 [311]) READ_ENTRIES(REPC1256E62:004B8651-NTFFFF0020): 16 ms. [76+65388=65464] 
(6-299 [312]) READ_ENTRIES(REPC1256E62:004B8651-NTFFFF0020): 0ms. [76+63918=63994] 
(7-299 [313]) READ_ENTRIES(REPC1256E62:004B8651-NTFFFF0020): 0ms. [76+65506=65582] 
(8-299 [314]) READ_ENTRIES(REPC1256E62:004B8651-NTFFFF0020): 0ms. [76+65498=65574] 
(9-299 [315]) READ_ENTRIES(REPC1256E62:004B8651-NTFFFF0020): 15 ms. [76+64920=64996] 
(10-299 [316]) READ_ENTRIES(REPC1256E62:004B8651-NTFFFF0020): 0 ms. [76+63688=63764] 
(11-299 [317]) READ_ENTRIES(REPC1256E62:004B8651-NTFFFF0020): 0 ms. [76+64522=64598] 
(12-299 [318]) READ_ENTRIES(REPC1256E62:004B8651-NTFFFF0020): 0 ms. [76+65402=65478] 
(13-299 [319]) READ_ENTRIES(REPC1256E62:004B8651-NTFFFF0020): 0 ms. [76+65478=65554] 
(14-299 [320]) READ_ENTRIES(REPC1256E62:004B8651-NTFFFF0020): 16 ms. [76+65296=65372] 
(15-299 [321]) READ_ENTRIES(REPC1256E62:004B8651-NTFFFF0020): 0 ms. [76+65238=65314] 
(16-299 [322]) READ_ENTRIES(REPC1256E62:004B8651-NTFFFF0020): 0 ms. [76+65386=65462] 
(17-299 [323]) READ_ENTRIES(REPC1256E62:004B8651-NTFFFF0020): 0 ms. [76+9480=9556] 
(18-299 [324]) CLOSE_COLLECTION(REPC1256E62:004B8651-NTFFFF0020): 0 ms. [12+0=12] 

You see that instead of one transaction with 50 bytes of data transferred.
In the error case we have 16 transactions with 14 transactions receiving up to 64k of data.
In my case the test has been performed on a local machine without any network delay.
But in normal customer cases in a WAN we have maybe 100-200 ms or more for each of those transactions.

If this happs for each requested design element (depending how it is queried).


How to find if a database has lost the design cache

You can see it by checking the client_clock data and I wrote a little tool to scan databases if the design cache is still in place.
If you want to test individual databases I can send you the lite version of this tool.
I wrote another tool that scans whole servers and also allows to delete private on first use view/folder entries.


How to recover

Once the design cache is gone it is quite difficult in most case to get it enabled again. First of all you have to get rid of the private on first use folder/view instances and than compact the database. In most cases this might re-enable the design cache but in many cases you have to create a new instance of the database.


How to avoid the problem

Currently the only solution is to avoid private on first use views/folders because for larger applications you always have more than 40 instances of a private on first use view/folder.

There is one work-around to avoid the client transactions for searching the design collection using a hotfix.
The hotfix ignores that the DB lost it's design cache and let the server do the search in the design collection without sending all the entries to the client.
But this is clearly just a temporary work-around which can also have some side-effects (the hotfix is currently still in test).

What will be done in future

There is no simple fix for this issue. The full solution to avoid that the design cache breaks needs a change which cannot be done in the current code stream - according to the current information we got from IBM support.


I hope this post helps customers and other Business Partners with their applications.

Feel free to contact me offline to get a copy of the lite version of the analysis tool.

back from holidays have a great x-mas time

Daniel Nashed  22 December 2009 09:36:05

After two great weeks in Safaga/Egypt I am back home. We have been very lucky to arrive before some of the airports closed because of snow ... And from 25 C to -10 C was quite a change ...
This was my first trip shortly before x-mas time and we really enjoyed the two weeks in the sun.
After finalizing and submitting two Lotusphere sessions just in time after two weeks away from keyboard I can start preparing for x-mas ...

I wish you and your families a great x-mas time and all the best for 2010!!!

-- Daniel



One of the pools next to the beach



Entrance of the hotel already prepared for x-mas... the main hall even had a huge x-mas tree ...

    Deadlock on Serverstartup with certain Extension Managers in Domino 8.5.1

    Daniel Nashed  26 November 2009 01:02:45

    There has been some confusion about this issue which has been posted in Ulrich's blog (http://www.eknori.de/2009-11-13/disable_log_dbgroup/)

    Multiple ISVs including myself contacted Ulrich offline about the info he got from another ISV and it turned out that this issue only affects some very specific Extension-Managers.

    After checking with support we finally got clarification published in the following TN (http://www.ibm.com/support/docview.wss?&uid=swg21410986).
    The problem occurred due to optimization some very specific optimization in low level Domino code.

    I have been in touch with a coulple of ISVs who ran into this issue to check what actually caused this issue.
    The problem seem mainly to occur when Extension-Managers use NSF or other C-API calls in the init phase of the DLL (MainEntryPoint).
    The C-API documentation states that no C-API calls are allowed in this routine but many applications use calls in this area.

    What is interesting is that other business partners suffer from the issue when another partner's Extension Manager is installed.

    The suggested Notes.ini entry mentioned in the TN (DISABLE_LOG_DBGROUP=1) is a work-around which causes the old not optimized behavior until the application is changed.
    IBM is working on another fix for 8.5.2 which hopefully solves this issue.

    But this clearly is one of the good examples why we should do only what the C-API states as supported (features vs. behaviors).

    On the other side this shows that IBM still tries to help customers and ISVs to get their applications working.

    Let me know what you think about this ...

    -- Daniel

    PMR ESR/SR Lookup Widget

    Daniel Nashed  13 November 2009 10:35:53

    Last week I was looking for some good examples to demo for a customer. When I opened a new PMR I noticed that with the new system there is an URL which contains the PMR number.
    The PMR example turned out to be quite complex and a good example. Implementing a lookup for SPRs, TNs in fixlist and Kbase was the logical next step and more easy afterwards.
    I am now using those lookups for a week and it is quite convenient.

    This blog post is a quick introduction how to use LiveText and own recognizers in the Notes 8.x standard client.
    It is not a complete introduction into LiveText and does not start at intro level. This blog post is only to explain details of the text recognizers and how to map parameters and demo the PMR example.
    You first have to enable Widgets and LiveText in general and have to be familiar with Widgets in general.

    A PMR number consists of 3 parts Number, Branch and CountryCode

    If you look into an example PMR you see the 3 parts. Sometimes there is a "#" between PMR, sometimes it is separated by comma, sometimes with blanks.
    But in general the format is very similar and looks like this.

    PMR # 66129 SGC 724

    The new IBM SR tool allows URLs like this which contain those three parts and allows a direct jump into the PMR.

    https://www.ibm.com/support/servicerequest/problemDescriptionSelect.action?userType=0&sourceAppl=XSR&srNumber=66129&branch=SGC&retainCountryCode=724


    What we need to implement the Widgets are the following components.

    - a Recognizer for parsing the PMR number format
    - a Content Type the Recognizer can feed
    - a Widget invoking the URL passing the parameters defined in the new Content Type
    - a New Action which binds Content Type and Widget together


    Let's start with the recognizer. First of all we need a name for the recognizer and a regular expression matching the PMR number.
    I tried to match all formats got thru the SR system per mail and what normally used when typing a PMR.
    It always starts wth PMR can have a ":" or "#" or just a blank in between. Has a 5 digit number. Than a 3 digit alpha/number string and a 3 digit number.
    Can be separated by comma or blank and there might be more blanks in between.

    The Recognizer uses regular expressions which are a lot of fun and are around in Unix for a long time.

    Here is the current regular expression I am using to parse a PMR number.

    \bPMR[#: ]*?([0-9]{5})[ ,]*?([A-Z0-9]{3})[ ,]*?([0-9]{3})\b


    The important part in this regular expression is that parts of the string which should be used as separated parameters are put together in round brackets.
    I marked the three parameters in the same colors I used for my sample PMR above.

    Once you have defined those three parameters the next step is to map them to a content type.
    Because there is no build-in content type for PMR numbers you have to create a new one (see below) with the three parameters and name them accordingly.

    After creating the content type you can start mapping the parameters.
    The Grouping Number is used to specify the parameter. Grouping number 0 stands for the complete text that matched the regular expression.

    Group Number 1 stands for the expression in the first round bracket (red).
    Group Number 2 stands for the next bracket and so on ...

    You can simply add more Content Properties and map them with the corresponding grouping.

    When this is done you have defined a new recognizer and mapped it to a content type.
    The next step is to create a Widget which can consume our new content type.


    Configuration of new Recognizer



    Configuration of new Content Type




    Now we can build our Widget by invoking the Widget Wizzard. You select a WebSite Widget and just paste in the URL above.
    It would make sense to replace the PMR number with one of your active PMR numbers because the Wizard connects to the website and asks for authentication.
    The authentication is cached and you only have to enter it when the session expires.
    I tried to get the automated authentication working but I did not get the IBM login form working.
    I was able to set the username and password but the page was not submitted. Did anyone get it working with the IBM account?




    Because the parameters are passed via URL the request type is a HTTP GET.






    In the next step you might want to change the name of the Widget.
    The wire action is defined in a later step.




    Passing multiple parameters for the PMR can be configured in the Advanced tab.
    In this example all parameters are required but not all of them will be filled using the PMR recognizer.
    Therefore they are set to required and have a default value.

    The other Parameters will be mapped using the wire option when configuring a LiveText action.
    Those names will match the corresponding names from the PMR Content Type.




    After finalizing the Widget (I left out some of the screens) we have defined the PMR Widget.

    The only missing part is the LiveText Wire between the Recognizer, Content-Type and the Widget.

    When configuring the new action for the Widget (right click on the Widget) you have to select the right Content Type and choose the right parameters using the Advanced tab.




    Because of the three parameters we need the advanced tab.




    This was the last step to wire all the components together.
    Now you should be able to recognize content in incoming mail and other documents.


    Below is an example of what types of PMR formats are automatically recognized with this regular expression.
    If you have other formats that do not work yet let me know and I will update the regular expression if possible.

    You see also the other LiveText I build for TNs and SPRs. The actions I am currently using is a FT search in Kbase and Fixlist database.
    I am still playing around with Widgets my first look into this... I hope you find my example helpful.


    -- Daniel


    Enabling Storage Optimization Features in New Databases

    Daniel Nashed  9 November 2009 17:38:56

    When you have enabled storage optimization features like LZ1, data+design compression and DAOS, you want new databases and replicas also to have the same storage optimization options turned on when the database instance is created.
    I have tested with ND 8.5.1 different ways to create new databases and most settings are preserved even when creating a new mail-file from template or when a new archive is created from the mail database.

    The changes are propagated thru the flags in the icon note which are also copied when you create an archive or a new database from template.
    So if you want new databases automatically have the right storage optimization features ensure that the right settings are enabled on the template already.
    This is great news and helps to ensure that the settings are automatically set for new databases.
    In the migrations phase you still have to enable those settings manually on each instance of the database (because the flags are only propagated when a new instance is created).

    While testing I ran into one issue which is currently a bigger road block for us and worked before in 8.5.
    When the archive is created the DAOS status of an archive database is not preserved when the compact -a creates a new archive.
    All other settings are correctly preserved only the DAOS flag is missing.

    I have opened a new PMR and got a "SPR # SHEZ7XGL5H2 DAOS flag is not preserved when creating a server based archive via compact -a"

    According to support I am the first customer reporting this regression. We need this functionality in a larger migration where we push around 2 TB of data from a single mail-server to an archive server and due to storage constrains we need the NLOs directly in the DAOS store which is impossible for us to enable in time during the archive operation thru this issue.

    If this functionality is important for you or your customers, you should open a PMR to add weight to this SPR to get it fixed in the next FP.

    -- Daniel



    Using mobile optimzed websites in Sidebar

    Daniel Nashed  2 November 2009 21:46:49

    Today I looked into some nice demos what to show customers about Widgets and had an little idea ...
    There are many services I am using on my phone. Some are a native app but others are an optimized website which in some cases works really great in the sidebar. Here is a nice example.

    -- Daniel



    here is another example for the start with a much smaller sidebar (it's optimized to float).