Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

    Notes Domino Platform - What is Notes/Domino and what is special about it?

    Daniel Nashed  10 January 2019 22:26:42

    In the last couple of month I did workshops with customers to check where they are with their Notes and Domino environment and what they are going to do in the future.

    My part was to explain what Notes/Domino really is and why this is so different than other application platforms.


    Based on my workshop and the presentation I put together a summary. If you want a copy of my German workshop slides, drop me a note.

    The following gives you a high level overview of Notes/Domino which describes the basic architecture and some of the most important core features.

    It is not a complete list of functionality nor describes all the details but it gives you an idea about the concept behind it.


    There is an interesting new Forrester study "The Total Economic Impact™ of IBM Domino" which provides some additional background.

    https://www.ibm.com/common/ssi/cgi-bin/ssialias?htmlfid=01022901USEN

    My summary is more a technical summary that gives you an idea about what is great about the platform.
    And this is not including the new exiting new development options introduced in Notes/Domino 10 which we should look into separately.


    -- Daniel



    Introduction


    When Notes was first released most of the core functionality was already in place and the core architecture never had to change because it was from ground up designed in a way that makes it very flexible, robust and secure.


    Looking into the first memorandum written by Mitch Kapor you can already see the most important aspects and the core idea behind Notes -->
    https://app.box.com/s/lv7lm64d1gbvkwyzxdct9sqzavot2mh6).

    Later there was a very interesting book published by Lotus Oktober 2000 which is still available. It describes the architecture and core components in a way, which has never been published before or after.

    The foreword already describes the core idea behind the product in a few words.


    „In 1989 Lotus Notes® made the world a little smaller.
    Since its first release, Notes™ has enabled teams of people to work together, even when separated by great distances or when individual contributions are made at different times.
    .. Notes continues to evolve innovative, new capabilities that support its initial core concept: Collaboration made possible through shared, secure, online databases."

    (Source :
    http://www.lotus.com/ldd/doc/uafiles.nsf/docs/inside-notes).

    Today we are used to work in shared teams in different countries and timezones but when Notes was first designed there wasn't a wide spread network that everyone could use.

    I recall the times where our servers had modems and later ISDN cards to communicate. Times changed but the concept behind it is still modern and very powerful.

    And when it comes to an integrated, rapid application development platform there is stil no comparable product on the market.



    What is IBM Notes


    When we look into the basic concepts behind Notes/Domino we see a  very flexible, document oriented. none-relational databases system (today you would name it "NoSQL").


    It is a Rapid Application Development & Deployment platform with integrated development and run-time environment.


    Lotus integrated email/calendar functionality into the base environment on same infrastructure and APIs

    Thus extendibility, customization of functionality and design is part of the core product.


    Notes and Domino has support for @Formula, Lotus Script/Com, Java, Javascript and C.

    This includes integration options like own servertasks or even Extension Managers, client extension points

    All of this has been designed into the product since day one.


    The application is document centric with flexible fields and field types per document (including Richtext and MIME).

    So you can have any number of fields of any type in any document. There is no fixed structure that limits you to a certain structure.


    It is not the best choice for large number of often changing records/documents that have more relational structure.

    But you can integrate via ODBC, JDBC and today with flexible Web-Services/REST services.


    Notes/Domino separates UI/Design/Code and Data which is also still a modern approach.

    Leveraging simple but very flexible and powerful architecture: e.g. "forms, views, folders, agents" you can write and extent applications flexible and in rapid application development style.


    The NSF storage (Notes database) holds the design and data at the same time. Both use separated but very similar "note-types" in the database.
    This makes easy to replicate among servers and clients. And this also helps to maintain the design leveraging "templates".


    Notes/Domino supports XML since Version 5 and allows to export/import documents and also design elements!


    The own client/server communication protocol "NRPC" allows transparent client/server access which allows you to work with the same database either on-line on a server or on a local instance "replica" of the database.

    For that reason Notes/Domino has build-in replication for Notes databases between clients and servers with full local support for databases.


    At the time Notes was first releases there wasn't any central directory.

    So Notes/Domino invented an integrated directory containing information for users, groups, servers and other information.

    Domino was one of the first products to support external access and sync via LDAP. And there are also other interfaces to access directory data.

    Today integration with central directories like Active Directory is important. There are multiple options to integrate and sync and also for Single Sign On (e.g. SAML or Kerberos).


    Out of the box you can open, edit and save back attachments with external applications like Office and other applications in any document.
    In addition the Notes client offers optional MAPI support for Office applications for mail integration for local applications.


    One feature that has been taken for granted by many users and developers is the integrated, performance optimized full text index for server and also local databases!

    You can use it in any database and you can use it on Lotus Script. Java and also C-API level!


    The tight integration with email and the private public key infrastructure allows to build flexible work-flow applications where you can digitally sign documents.
    Tight integration means that all functionality is available in one application in contrast to other applications that need separate products to implement comparable functionality -- if other products support the same functionality at all.

    In many cases you end up having multiple other application platforms depending on the solution you choose.



    Development Platform


    Since Version 8/8.5 Notes integrates with the Eclipse development and run-time environment.

    This offered new integration points and application options while keeping all existing functionality from the C/C++ application platform.

    At the time it was introduced this was a good idea. But using Eclipse was quite challenging specially from performance point of view.

    Today we see a move away from this integration and we will probably see other client options soon.


    The development platform (Admin/Design Client) has low footprint on normal desktop with local or server based applications.


    One simple but very powerful part of the application platform is the formula engine which can be used in forms, views and folders to calculate information that is displayed or stored in the application.

    This is is comparable to the @formula uses in spread sheet applications (no wonder looking into the roots of the inventors working on Lotus 1-2-3, Lotus Symphony and earlier products).


    This is on one side a very simple but also a powerful and very effective way to process and present data.



    Full Upward Compatibility


    From the very beginning Notes was design to be upward compatible!

    So all functionality remains supported in a newer version without much modification needed.


    A new version might come with a new On Disk Structure (ODS) which brings new database back-end functionality.

    Clients and servers encapsulate the ODS details for the local NSF so even earlier clients can access servers with later versions.


    Once you updated the client and server code you can simply update the ODS with standard tools/maintenance compact servertasks.

    Updates -- even major releases -- are simple without data migration or export/import!


    Custom applications usually work without or with little modification on new major release.

    Standard unmodified applications are "just" updated via updated templates by the standard design servertask.


    The client UI is consistent cross versions, even when new functionality is added.


    But on the other side this means UI/design enhancements for your existing applications and application maintenance is often neglected because everything just works.

    In fact many applications worked for years without any change. That's why nobody touched them even for UI modernization.



    JVM Support


    It took quite a while until IBM decided to implement JVM 1.8 in Domino and they implemented just in time before Oracle stopped supporting JVM 1.6.


    But  we now have JVM 1.8 support since 9.0.1 FP8 (runtime) 9.0.1 FP10 (compile time)

    An IBM JVM Team provides JVM based on Oracle JVM which is updated in every FP to the latest version available.



    Notes/Domino Security


    Flexible databases access with multiple levels is integrated part of the database design since day one.


    - Manager/Designer, Editor/Author, Reader, Depositor with additional roles

    - Access on document-level with simple but effective mechanisms like Reader/Author fields with names, groups and roles

    - Encryption on field/document level and local database level

    - Execution Control List (ECL) for detailed controlled for signed application code!
    - Integrated Public Key Infrastructure (PKI) document/email encryption based on RSA technology

    - Native S/MIME integration (public certs need deployment process, most customers use gateway encryptions)


    - Certificated based authentication via Notes.ID

    - Access via Certificate and optional password checking

    - Since Domino 8.5 simplified Notes.ID management via "ID-Vault"


    - HTTP via user/password with native SSO (LTPA token, also used in other IBM products)

    - Optional Authentication via X.509 Certificate

    - Optional Authentication with SAML 2.0



    Current Encryption Standards


    - HTTP/ SMTP, LDAP, IMAP, POP3 use current encryption like

    - TLS 1.2 with ECDHE Ciphers (since 9.0.1 FP4) and SHA256 certificates
    - SSL-Labs „A“ Rating


    - Notes NRPC Port Encytion (9.0.1 FP8)

    - Encryption using AES-128 CBC /  AES-128 GCM / AES-256 GCM / RC2-128

    - Ticket Algorithm HMAC-SHA 1 - HMAC-SHA 512 (Default: HMAC-SHA 256)


    - Notes data encryption AES 128 or AES 256



    HTTP/HTTPS Support


    Any database can be presented in web. Web representation with current technologies for example via XPages "Responsive-Design" with responsive design, infinite scolling etc


    - Hybrid applications with Web/Notes Client/mobile access


    - New modern web interface for mail.

    - Verse on Prem (VOP) does currently use a simple add-on installer and leverages the existing infrastructure


    - REST Interfaces for read/write with specific rest services for mail, calendar etc

    - Strategic use of REST with Open Standard documentation with new functionality in the latest Feature Packs



    Domino 64bit Support


    Domino supports native 64bit on all platforms.

    64bit has benefits specially in virtualized environments because it allows better optimization if all components leverage 64bit architecture.

    Larger memory support is also helpful for special applications servers like Traveler, specific HTTP work-loads or the IBM Enterprise Integrator.


    Domino as an application usually does not need more than 4 GB RAM for a normal server configuration.
    IBM put a lof of energy into I/O optimization. There fore not more RAM is needed by the application.

    You would still can benefit on larger memory configurations where the OS will leverage the file-system cache (specially in virtualized environments).

    But the existing code is performance and resource optimized and does not need more RAM on it's own like other applications do.



    64 GB Database Limit


    64 GB remains the physical limit of a Notes/Domino database

    But there are many options to optimize databases to either

    a.) reduce the storage needed
    b.) move some data outside the physical NSF file


    Document Design/Compression → Reduces Design and Document data (everything that is not Index or attachment) up to 50%


    DAOS (Domino Attachment and Object Services) = External Storage for "attachment" data (database objects are move to the DAOS storage)
    Leveraging DAOS officially supports 1 TB per NSF database. Usually for mail-databases attachments would cover 70% of the mail-data.


    NIFNSF = Put view/folder index to a separate file (.ndx) → Maximum size of the .ndx file: 1 TB!


    So when leveraging those technologies the physical limit of the NSF does not matter any more on server level.

    In addition a single NSF file would only hold data for a single mail user or a single application. So the 64 GB is per Notes NSF.


    On client side the 64 GB limit still applies because the optimization is not available in Notes 9.0.1.

    But on the other side more than 64 GB of data in a single application on client side might not be the best idea.

    Domino offers integrated archive functionality which would help to reduce data on the archive mail database or other application databases.


    For Notes/Domino 10 the maximum physical database size is increased to 256 GB.


    Database Optimized Storage


    Leveraging Database Compression, DAOS and NIFNSF also helps your to optimized performance and helps you to reduce storage and backup space dramatically!


    With Document/Design Compression you save additional storage space.

    It is transparent to the application with up to 50 % of data and design reduction just by enabling the feature via standard compact


    DAOS


    Domino Attachment and Object Services (DAOS) moves attachment data to a central store with one file per attachment (configurable threshold usually 256 KB generating NLO files with the attachment) where it is stored deduplicated.


    In addition to deduplication and NSF data reduction you can store data on a NetApp or other technologies which allow deduplication on storage container level to further reduce storage needs.

    DAOS also allows to reduce backup costs leveraging online, incremental backup. The NLO is only stored and backuped once per server!


    NIFNSF


    A quite new feature introduced in Domino 9.0.1 Feature Pack 8 allows to store the index in a separate file for performance reasons and also to further reduce backup costs.

    The separate file on disk allows optimized locking and the NIF index does not need to be backuped.



    Domino Clustering


    Application level Active/Active Clustering with separate, independent server instances with separate storage

    No SAN mirroring or special storage environment required.


    The Notes Client is "cluster aware" with automatic fail-over and fail-back


    A cluster replicator servertask is used for almost real-time replication.


    Downtime is minimized downtime because maintenance is performed on each server separately and Domino clusters are operating completely independent!

    A single databases can also be taken off-line. The key here is that Domino as a "service" is hight-available and the individual servers can be down for planned maintenance.



    Domino Transaction Logging/Backup


    Transaction Logging is fully integrated into Domino NSF infrastructure since Domino 5.0


    Ensures database consistence and performance for each NSF file.

    Fast restart without fixup after failure (similar to journal file-system on Linux).


    Translog functionality has been derived from DB2 functionality.


    Dedicated backup API interface used by 3rd parties for on-line backup without downtime

    Full backup/Incremental backups with Point in Time recovery per NSF file



    Notes Domino Diagnostic & Fault Recovery


    Outstanding, fully integrated diagnostic functionality for crash, hand, memory leaks etc.

    NSD (Notes System Diagnostic) provides detailed diagnostic information for IBM to pinpoint and fix issues.


    Fault-Recovery provides fast restart after server crash

    Automated diagnostic collection (ADC) after restart collects diagnostic data from client or server.



    Mobile Mail/PIM Access


    „Mobile First“ is one of the important IBM strategic goals.


    IBM Traveler as a separate, easy to install solution leverages the standard Domino server stack.


    Mobile access via iOS, Android, BB10 for Mail, Calendar, Contacts (and Todo)

    Native Verse Client (App) for iOS and Android via SyncML

    iOS Mail App Support via Active Sync

    BB10 Support via Active Sync



    My Conclusion


    Today mail is commodity. Notes and other products support similar features compared to other platforms.

    Replacing Notes as a mail-platform alone does not provide much benefit and has migration risks and costs.
    Specially coexistence during the migration or with existing applications can be tricky.


    The key benefit for Notes/Domino as a platform is unmatched rapid application development.

    There is no easy way to migrate applications and if you end up replacing just part of the applications you have to support and pay for both platforms for a longer time period.


    Alone the full integration of application with mail, calendar and contacts and workflow functionality is difficult to replace by one single product.

    Customers migrating most of the times have to move applications to multiple platforms and write new applications.


    Notes/Domino is stable, integrated, performing, high available platform.


    It has good TCO as an application platform compared to other platforms. There is no single platform providing all functionality used by customers today.


    In addition today offers Domino REST APIs to be more flexible and used in a micro-services landscape and can be even better integrated with other enterprise applications.

    Comments

    1Andy Brunner  11.01.2019 9:00:54  Notes Domino Platform - What is Notes/Domino and what is special about it?

    Very good article. Thanks Daniel!

    2Paul Withers  11.01.2019 14:46:11  Notes Domino Platform - What is Notes/Domino and what is special about it?

    A wonderfully comprehensive list of features. As you mention, the good foundations are why Domino has endured. I'm sure many customers have forgotten so much of this because they just take it for granted. And Domino V10 has added more, with increased limits, gRPC, IAM, New Relic reporting, not to mention more robust clustering monitoring and maintenance.

    3Uwe Brahm  11.01.2019 15:11:23  Notes Domino Platform - What is Notes/Domino and what is special about it?

    Thanks Daniel!

    This is a very nice piece that helps us even internally. Many people are not even

    technically aware of the features already

    built in.

    4Daniel Nashed  11.01.2019 18:38:42  Notes Domino Platform - What is Notes/Domino and what is special about it?

    Thanks for your feedback Paul!

    Yes I left out the new features and want to present them separately.

    Actually there are good IBM presentations about most of the new stuff from IBM!

    Let me add just one detail about the "New Relic reporting". It's very customizable!

    You can customize it with notes.ini settings and the only thing you need to deploy it are those notes.ini settings!

    No no coding, no additional software just notes.ini settings. You point it to the target application of your choice and the server will publish the stats.

    I have a blog post about it with a sample database with an agent that acts as a consumer of the stats to play with the stats.

    -- Daniel

    5Wayne Scarano  11.01.2019 19:27:03  Notes Domino Platform - What is Notes/Domino and what is special about it?

    Thanks! Great document.

    I would mention in the security section about 2FA (something you have and something you know) with Notes IDs. MFA is all the rage now but in the 90s we had to fight with firms who didn't want to 'manage the Notes ID" because it was a hassle. Translation - security is a hassle but everyone agrees MFA is the standard. In the same vein I would mention how it was ahead of it's time.

    Is MongoDB the first NoSQL? No, the first document database was Notes and because it WASN'T relational it was dismissed because it wasn't mainstream. Also, with the best replication engine ever invented it allowed the concept of not requiring a master/secondary (primary/secondary). Any copy was legitimate and useful for disaster recovery and lower latency (regional/local copies).

    While it was 15 years ahead of it's time it was also 15 years behind the times in terms of user interface (rich text needed a rewrite), new features (on par with Outlook), performance (separating indexes), and completing the task of making it web and cloud native.

    Because Notes abstracted us from a lot of tedious tasks you have to do on your own, it allowed for faster development. This is the same I'm seeing with Serverless technology (Cloud 2.0) and eventually 3.0. The cloud becomes the platform to build upon with abstracted services that are fully managed.

    6Clemens Hagendorf  14.01.2019 14:34:14  Notes Domino Platform - What is Notes/Domino and what is special about it?

    A very good overview. Thank you very much Daniel !

    In this context, I come to a statement that I once snapped: Any migration away from Domino automatically causes coexistence with Domino.

    #dominoforever

    Archives


    • [IBM Lotus Domino]
    • [Domino on Linux]
    • [Nash!Com]
    • [Daniel Nashed]