Daniel Nashed 30 January 2016 14:47:59There is a new IF1 for Domino 9.0.1 that includes two fixes we have waited for in the TLS area specially when communicating with STARTTLS and web-services as posted before on my blog. SPR #KLYHA57S37 - Disable TLS Session Resumption on outbound connections by default This fix addresses and issue for outgoing STARTLS sessions on SMTP. See some more details in my other blog post --> http://blog.nashcom.de/nashcomblog.nsf/dx/tls-1.2-connection-issues-with-protection.outlook.com.htm SPR #MKENA4SQ7R - Domino TLS 1.2 Client Hello does not offer a Signature Algorithm extension causing some handshakes to fail The second issue is a problem with a missing security algorithm extension that causes connection issues which happened in many customer environments -- and it looks like this happened depending on the certificate used in some cases. And also what the remote server supported. The fix implements the missing extensions and improves compatibility. SPR #KLYHA5YRVP - Recommended security fix for IBM Domino (technote 1974958) The Domino SLOTH vulnerability is about collision attack with the MD5 hash function that is used in the TLS handshake. The fix addresses this issue. Here are the main details from the TN describing the SPR. CVEID: CVE-2015-7575 DESCRIPTION: The TLS protocol could allow weaker than expected security caused by a collision attack when using the MD5 hash function for signing a ServerKeyExchange message during a TLS handshake. An attacker could exploit this vulnerability using man-in-the-middle techniques to impersonate a TLS server and obtain credentials. See more details here -> http://www.ibm.com/support/docview.wss?uid=swg21974958 SPR #DKENA32JMP - Add support for Extended Master Secret (RFC 7627) to TLS 1.2 This is a quite new RFC which has been implemented by Microsoft and Google for their browsers recently. Both sides need to support this extension! Domino does now support this extension which eliminates a risk of a man-in- the-middle attack in some situations described in the RFC below. The Transport Layer Security (TLS) master secret is not
cryptographically bound to important session parameters such as the
server certificate. Consequently, it is possible for an active
attacker to set up two sessions, one with a client and another with a
server, such that the master secrets on the two sessions are the
same. Thereafter, any mechanism that relies on the master secret for
authentication, including session resumption, becomes vulnerable to a
man-in-the-middle attack, where the attacker can simply forward
messages back and forth between the client and server. This
specification defines a TLS extension that contextually binds the
master secret to a log of the full handshake that computes it, thus
preventing such attacks. https://www.ietf.org/mail-archive/web/ietf-announce/current/msg14570.html
Daniel Nashed 30 January 2016 14:17:19If you are attending IBM ConnectED in Orlando and you are interested in Linux you should attend the Linuxfest Session. Thanks to Bill Malchisky we made it again into the agenda! I am looking forward to this session and will bring the brand new Start Script Version 3.1.0 with many enhancements. Here is a copy of Bills' original post. Looking forward to this session. -- Daniel Linuxfest VII Gets a Slot at IBM Connect 2016 Bill Malchisky January 28 2016 02:00:00 AM Linuxfest VII - The Penguin Awakens
After many months of planning and working with the events team, we are pleased to announce that Linuxfest is back for our seventh year. This is the only session at IBM Connect dedicated exclusively to Linux and IBM software. As we moved back to the last day lunch break, be certain to bring your box lunch and join us for an informative session Linux and IBM. New this year, we are in Event Connect and Session Preview Tools.
Date: Wednesday, 3 February
Time: 11:45 - 12:45 PM
Place: Orange G
Session ID: TI-1118
Audience: Admins, Developers, Architects
Speakers: Bill Malchisky, Wes Morgan, and Daniel Nashed
Ask questions and get informative answers from three passionate leading IBM on Linux SMEs.
Whether you've already deployed IBM technology on Linux or are "just interested," join us for the seventh installment of what has become an IBM Connect tradition. In this open discussion of the latest on Linux from IBM, you'll hear Business Partners, IBMers and IBM Champions talk about the most recent developments around our favorite operating system, share tips and tricks, and open the floor to your questions, successes and commentary as well. This is not a roadmap/strategy session; instead, it's a chance for you to learn what's out there for Linux, pick up technical know-how to ease your deployments, and connect with other IBM customers using Linux.
Daniel Nashed 16 January 2016 15:23:21Traveler 18.104.22.168 is the first update shipped this year. It comes with a number of fixes. See details here --> http://www.ibm.com/support/docview.wss?uid=swg21700212#9019 And it solves an important issue for Traveler HA Servers. There is a technote describing the issue in detail and you should have a look into the new command introduced in this version as soon you have updated your servers. The following TN #1974741 "Two scenarios where multiple accounts for users could be created on an IBM Traveler server HA pool" explains the new command and problem situation that might occur. Have a look into the TN if you are running Traveler HA --> http://www.ibm.com/support/docview.wss?uid=swg21974741 There are two new "features" introduced with Traveler 22.214.171.124 - Calendar Ghosting (which was added in 126.96.36.199) is enabled by default for IBM Verse clients starting in release 188.8.131.52 - And the new "DbAccountsCheck" which can be used to diag and fix the problem described in the TN mentioned above.
Daniel Nashed 7 January 2016 11:57:08Two of my customers had issues connecting to the Microsoft hosted environment over TLS 1.2 once we got the session resumption working (see previous blog posts). My environment had the same configuration and could connect just fine. It looks like the servers are behaving different with different certificates. That's the only difference we saw in configuration. After a couple of tests and working with IBM support we got a hotfix that we successfully tested yesterday. I know of 3 customers who solved their connection issues that way. The error you see in the logs is the following: TLS/SSL connection 184.108.40.206(64892) -> 220.127.116.11(25) failed with client certificates NOT supported by server signature algorithms SMTPClient: SSL handshake error: 1C7Ah Router: No messages transferred to ACME.COM (host acme.mail.protection.outlook.COM) via SMTP: SSL IO error. Remote session no longer responding. SPR # MKENA4SQ7R Domino TLS 1.2 Client Hello does not offer a Signature Algorithm extension causing some handshakes to fail This is one of the SPRs planned for the next IF. There are other open issues that should be also fixed as well like the outgoing session resumption issues. Short description what happens. TLS 1.2 defines an extension to the Client Hello (signature algorithms) and this is officially required for TLS1.2 in contrast to earlier TLS versions. Some servers implement the RFC quite strict and that could cause connection issues over TLS 1.2 The fix ensures that the signature algorithms are send which includes all the currently supported algorithms: 06 01 - SHA512/RSA 05 01 - SHA384/RSA 04 01 - SHA256/RSA 03 01 - SHA224/RSA 02 01 - SHA1/RSA 01 01 - MD5/RSA"
Daniel Nashed 15 December 2015 21:18:43 We have been running into some issues and I got multiple customers reporting that outgoing STARTTLS did not work in some cases specially for some German provides like web.de and gmx.net. The error you see when enabling debugging is SSLEncodeClientHello> We offered SSL/TLS version TLS1.0 (0x0301) FindCipherSpec> Cipher spec DHE_RSA_WITH_AES_256_CBC_SHA256 (107) is not supported with TLS1.0 It turned out that session resumtion in combination with the new introduced TLS 1.2 causes some interoperability issues. The outgoing session does use TLS 1.0 instead of TLS 1.2 in some cases because of session resumption. Session resumption is specially important for incoming HTTPS connections. But it is also used for outbount connections. When TLS 1.0 is used instead of TLS 1.2 your server might chose a cipher that is not supported in combination with TLS 1.2 and the connection will fail with an error message like this TLS/SSL connection 192.168.1.1(39040) -> 192.168.1.2(25) failed with server chose unsupported cipher spec 0x006B The current work-around is to disable resumable sessions with the following notes.ini parameter SSL_RESUMABLE_SESSIONS=1 You should be aware that this causes some performance impact for incoming connections like HTTPS. IBM is working on a solution. Stay tuned for more details. -- Daniel
Daniel Nashed 5 December 2015 16:19:05 I have been helping a customer who had issues with Backup Exec for Domino. They got issues with their backups. The error message pointed to issues with their tapes. But it turned out it had to do with the DAOS integration which is not fully working with Domino 9.0.1 The error they got pointed to issues with the back media: Final Error Code: e00084ca HEX (0xe00084ca HEX) or a00084cd HEX (0xa00084cd HEX) Final Error Description: The data being read from the media is inconsistent. Final Error Category: Backup Media Errors But it turned out that the backup had issues with querying the DAOS files for databases After a long discussion with support we turned off the daosmgr commands which are used during the backup of a database (registry settings). But you cannot separate DAOS NLO backup and NSF backup. There is no way to completely separate NSF and DAOS backup. It was a surprise that Backup Exec has so "deep" DAOS integration but I think they did not implement it the right way. Running a daosmgr LISTNLOs for each database during backup does not sound right to me. There is a setting described here -> https://www.veritas.com/support/en_US/article.TECH136835 to disable querying information for the backup and the customer is now doing that operation manually at restore.
They set HKEY_LOCAL_MACHINE\SOFTWARE\Symantec\Backup Exec For Windows\Backup Exec\Engine\Domino\DAOS listnlo mode = 3 and the issues have stopped. Speaking with during the long going support incident Symantec they finally said that Domino 9.0.1 is not supported and they are declaring end of life for their product and even the customer has valid maintenance they are not getting any updates. So they have been working on it for a while and had no support for 9.0.1 and finally decided to declare the product end of life instead of fixing issues and supporting 9.0.1!!! They closed the support incident and the customer asked for a formal statement from product support in writing. If you are using Backup Exec it sounds like it is time to look for a different backup solution! I saw multiple customer reports with similar issues and they are all not getting any help. If someone else is using Backup Exec you should be aware of the situation. It is not only the current problem but also the way they did not care about the customer problem and that they have vaild support contract and they are not supporting their product on a current Domino release. Now that Symantec and Veritas are separate companies we have to see what is going to happen (https://www.symantec.com/page.jsp?id=separation-strategy) . My customer has valid maintenance for their product but Veritas does now own the product. Not sure if the developers are still on board on either side. But in any case they decided to not support it any more.
Daniel Nashed 4 December 2015 16:14:52This week Domino 9.0.1 FP5 has been released.
The client fixpack seems to have issues. I have seen a Support Flash alert and a couple of customers/partners contacted me with problems.
On the client side I would wait until those problems have been resolved.
But on the server side you should look into implementing FP5 soon.
I have deployed it on my production server and I have now also incoming and outgoing "STARTTLS" enabled with additional logging via my SpamGeek application.
In addition to a couple of security fixes the new version also has some detail fixes in the TLS area which will help to get better logging and compatibility with other environments which is specially important for STARTTLS.
I am currently still having SSLV2 HELLO enabled on my server and I keep monitoring the logs.
And I have noticed some strange behaviour which I already have reported to IBM.
With 9.0.1 FP4 IF2 IBM changed the default cipher list and you did not need to set SSLCipherSpec in most cases because they did a great job enabling only all secure ciphers by default and putting the other ciphers on the weak list.
In addition to security fixes there is also an new JVM patch included in FP5.
The current JVM is 1.6 SR16 FP15 and there is a separate technote available with details what is fixed.
Some of the fixes in Domino and also in the JVM provide better protection against "Logjam security vulnerability".
There are also a couple of fixes to address memory leaks -- some are in the security area.
So I would recommend considering the update soon at least on server side for external servers.
It works well for me and there is additional logging that can be helpful. Update 10.12.2015: We still see some outgoing TLS connection problems for STARTTLS. I first thought this could be fixed by setting the SSLCipherSpec explicitly.
But it did turn out that it does not fix it. Still troubleshooting what is going wrong. One customer has a PMR open and I am also tracing...
Daniel Nashed 30 November 2015 14:58:43 Most backup solutions are still not really flexible when it comes to restore operations. I am currently involved into some backup projects and build a tool that can be used on top of a Domino aware backup solution. Some software can disable replication when restoring a NSF file. Other applications can change the replica when restoring a database. But I have not seen application that can do both at the same time. And there are also other operations that could make sense. I would wish backup vendors would support more options. What backup vendor do you use beside TDP and EMC Networker? What are your experiences with backup and specially with restore? Which other restore operations would you wish to have? Here is what I would expect from a restore operation and what I currently added to my add-on tool that can be used after restoring a database Disable Replication Disabling replication is needed in many restore operations when you restore into a different location. If replication is still enabled, deletion stubs might replicate back to the temporary restored database. In most of the cases the restore is needed because of deleted documents. Assign a new Replica-ID In many cases it makes sense to assign a new replica-id to avoid conflicts with the existing database. In that case the database cannot replicate and also operations that find the database by replica-id will not find this temporary restored database. We have seen applications which locate the database by replica and would possibly find the restored database. In that case a restore with a different replica-id makes sense. Also when you assigning a new replica the icon on the desktop is never stacked over the current database. That can be helpful when you want to provide a link to a database to the user. Disable all agents If you restore a database you usually don't want that agents are executed in the temporary restored database. There is a database option that no background agends are allowed in the database, which presents agents to run. In most cases this makes a lot of sense to avoid conflicts with the existing database Mark a databases out of service Marking a databases out of service in a cluster helps to avoid access to this database for normal users. Admins can still access the database. This can be helpful for the restored database but more often this is important for the existing database which you might want to troubleshoot. Take a database off-line / Bring a database on-line Taking a database online of offline is part of the backup API and can be helpful to force a database to be online after a restore operation. Or it can be useful to avoid access to a database before deleting the database. Best is to combine both operations. Take a database off-line before deleting it and having the program wait a certain time. This provides you with the best chance to delete a database Delete a database Sometimes you have to delete a database from command-line. While the server is running it is not a good idea at all to physically delete the database on OS level! You have to use the Domino API to delete the database in a save way and have Domino aware of the delete. It is completely unsupported to delete a database on OS-level on all platforms when the server is running! Change Database Title Change a database title or add a prefix can be very helpful if you want to send a link to the user with this new database to indicate it's a restore of his mailfile. Rename a database There are different use-cases where you want to rename a database. You cannot rename a database on OS level while the server is running. This can only be done from Domino server side using the API. Additional Ideas that might be interesting I have a couple of additional ideas that could be interesting. One of the most often requested features I could think of would be to copy folders including documents to the current database. -- Create a folder from the current inbox design of the target database -- Check if documents already exist in the database and only add the document to the folder if the document exists already --> check by UNID -- Copy documents from the restore database to the target database -- Different options to find a database. The best would be probably to find target database by mailfile owner if not specified manually What do you think? I am wondering that those type of operations are not already implemented in backup applications. And I would be interested in your feedback. -- Daniel nshrestore: Syntax: dbname.nsf [Options] -v Verbose Logging -d Disable Replication -n Assign New Replica-ID -a Disable Agents -o Mark Out Of Service -i Mark in Service -b Bring DB Online -f Take DB Offline -x Delete DB -w Wait time for take DB Offline (Default: 30) -t Change DB Title -p Add Prefix to DB Title -r Rename DB
Daniel Nashed 21 November 2015 09:34:21I should have blogged about this earlier. It was in my 2013 IBM Connected presentation but beside the TN and my presentation there is not much information.
If you are using Domino clustering on Win2008 or higher you should really disable the port Stealth mode!
This week I ran into a customer crash situation with repeated crashs which took a while to fix.
The failover on their Win2012 R2 servers was painful slow.
In Win2008 Microsoft introduced a feature called the Port Stealth mode.
This new "security feature" is enabled by default and is independent from the Windows Firewall.
If Domino does not listen any more for NRPC port 1352 Windows will discard all TCP IP packets for new and also existing connections.
That means the Notes client still thinks that the server is there and tries again to send TCP packages until the TCP timeout is reached.
The client is hanging for 30 up to 60 seconds until the failover occurs because Windows does not reject the packages from the client.
Once you disabled the Stealth mode via registry values, the client failover is again almost immediate.
You should also enable silent cluster failover in the desktop policy to avoid any prompts and the failover is almost seamless in most of the cases.
And in current Domino releases the client will also fail back to the home-mail-server later on.
To disable the port Stealth mode you have to set the registry values mentioned in the technote and we had to restart Windows to ensure the settings have effect.
The changes only take effect when your restart Windows! We have multiple customers reporting it even for Windows 2012 R2.
IBM Technote --> https://www.ibm.com/support/docview.wss?uid=swg21498755
The IBM TN is referencing the following Microsoft Technote --> http://msdn.microsoft.com/en-us/library/ff720058%28v=prot.10%29.aspx
Daniel Nashed 18 November 2015 17:32:57Last call! In case you did not know yet.
There is a new type of event organized by DNUG next Tuesday.
I am very interested to see how the feedback to this new event type is.
The event is free for DNUG members and in case you are not a member there is a small fee.
Also the way to get enroll is different. The DNUG board to make it easier and tries different ways to organise the event.
I am looking forward to the event and I hope to see many of you next week!
The sessions are all in German but since my blog is English I am still writing this blog entry in English.
See the agenda below and there are more details in the event document.
|Zeit ||Thema ||Speaker |
|9:00 – 9:15 Uhr ||Begrüßung || |
|9:15 – 10:15 Uhr ||Client-Strategie: Welcher Client in welcher Umgebung ||Christian Henseler |
|10:30 – 11:15 Uhr ||Client-Strategie: IMSMO – Outlook 2013 als Frontend für Domino |
|Manfred Lenz (IBM) |
|11:30 – 12:30 Uhr ||Client-Strategie: Calendaring – Koexistenzen, Interoperabilitäten und Troubleshooting ||Anett Hammerschmidt (AHT Consulting)|
Manfred Lenz (IBM)
|12:30 – 13:30 Uhr ||Mittagspause || |
|13:30 – 14:15 Uhr ||Client-Server-Lizenzen: Endlich Durchblick bei IBM Lizenzen für IBM Notes Domino, Connections und Sametime ||Michael Deery|
|14:30 – 15:30 Uhr ||Server-Security: Domino Security – Best Practices ||Daniel Nashed (Nash!Com) |
|15:30 – 16:00 Uhr ||Kaffeepause || |
|16:00 – 17:00 Uhr ||Client-Server-Ausblick: IBM Verse und ein Ausblick auf die Dinge die bei IBM noch in der Pipeline sind ||Olaf Börner (BCC Unternehmensberatung GmbH) |
|17:00 – 17:15 Uhr ||Ende || |
Die Veranstaltung ist für DNUG Mitglieder kostenlos. Für Nicht-Mitglieder wird ein Unkostenbeitrag von € 90,- netto erhoben.