Streamlining passwords and achieving SSO for users on Windows platforms

 Abstract

This article provides in-depth configuration settings for leveraging Active Directory to authenticate users, allowing elimination of Domino Internet passwords for users on Windows platform. This configuration is useful for Web only users, as well as for users who also access Domino with Notes. These settings can be used in combination with the 8.5.1 Notes shared login feature that eliminates the user's Notes password.

Single sign-on (SSO) can mean many things, but in general the goal is to reduce the number of password prompts that users must respond to. From an administrative standpoint, the goal also includes reducing the number of passwords required for users to remember, since fewer password problems will result in fewer help desk calls and lower administrative cost. In IBM Lotus Notes and Domino products, we have a variety of features which can be used together to reduce administration cost. Customers on Microsoft® Windows® platforms can take advantage of SSO features including Notes shared login, and Windows single sign-on for Web clients. For your users who are both Notes users as well as Domino Web users, you can achieve SSO and reduce administrative cost by eliminating both Notes passwords and Domino Internet passwords. For any authentication scenarios requiring password verification, you can choose to rely on Microsoft Windows Active Directory passwords already in place for all Windows users.

Eliminating Notes passwords vs. synchronizing passwords

When using Notes on Windows, the Notes shared login feature in release 8.5.1 allows users to start Notes without having to provide a Notes password. Users only need to log in to Windows using their Windows password. In this scenario, the important password for the user to remember and manage is the Windows password.

Many customers want their users to deal with the Windows password only, and historically may have deployed the old Notes 'single logon' feature that synchronized the Windows password with the Notes password, while optionally also synchronizing the Domino Internet password (if configured in the user's security policy). While it previously made sense to try to keep the three passwords (Windows, Notes, and Domino Internet password) in synch, the 8.5.1 Notes shared login feature effectively eliminates the Notes password so that there isn't a Notes password to keep synchronized with a Windows password. This is good news! Now you don't need password synchronization, which often is an administrative headache!

If your goal is to streamline the number of passwords, we recommend the Notes shared login feature that eliminates Notes passwords. Additionally you can eliminate Domino Internet passwords, so that there is no further need to synchronize any password with Windows. You can set up Web users to be authenticated directly against the Windows password managed in Microsoft Active Directory, which is described below.

This article does not cover the Notes shared login configuration itself, or Notes ID vault which can conveniently be used in conjunction to manage id synchronization and id backup. If you are deploying Notes shared login to eliminate Notes passwords, see instructions here: Using Notes shared login to eliminate Notes password prompts at the IBM Lotus Notes and Domino Information Center.

Eliminating Domino Internet passwords

The Domino 8.5.1 release included Windows single sign-on for Web clients. This feature allows a Web user to access Domino resources without providing an Internet password. The underlying technology does not use the Windows password per se, but rather leverages the user's Windows operating system login (i.e. Kerberos security) credentials. The result is that a logged in Windows user is not challenged for a password when browsing to Domino on the Web. Windows single sign-on for Web clients is targeted to your Domino server on the Windows platform. Where Windows single sign-on is operational, the Domino Internet password is unused.

Independent of whether you deploy Windows single sign-on for Web clients (or any other Domino Multi-server session authentication SSO feature), it is important to note that passwords are still needed in some Web scenarios. The Windows single sign-on for Web clients feature eliminates the need for a Domino password only for intranet access scenarios. In order to leverage the user's Windows Kerberos security credentials, the Windows single sign-on feature requires that the user's Windows machine can directly interface with the Windows domain controller. Windows single sign-on cannot be used in Internet scenarios (e.g. user login to Domino across a firewall), and obviously not in cases where a Domino server does not offer the Windows single sign-on feature, therefore in some scenarios a Web user needs to supply a password. If you prefer the user to supply the Windows password rather than a Domino Internet password, you can set up Web users to be authenticated directly against the Windows password managed in Active Directory (see below).

This article does not cover all instructions to configure Web SSO for Domino, or the Windows single sign-on for Web clients feature. See instructions here in the Notes/Domino Information Center: Setting up Windows single sign-on for Web clients

Leveraging Active Directory to authenticate users

If you prefer a Web user to supply the Windows password rather than a Domino Internet password, you can set up Web users to be authenticated directly against the Windows password managed in Microsoft Active Directory. This configuration requires setting up Directory Assistance to Active Directory, as well as an appropriate supporting directory configuration.

The configuration ensures that a user's directory record is found in Active Directory in order that the Active Directory password information can be used to authenticate the user. Also the configuration must ensure that the user's Active Directory record is associated with any corresponding Person record found for the user in Domino Directory. While the user can be authenticated against the Active Directory password, the association between Active Directory and Domino Directory allows the user's name to be mapped to the Notes name found on Domino database ACLs (Access Control Lists). To enable the user's successful authorization to access Domino resources, the Domino server must recognize the user according to the Notes name contained on the Domino ACL.

Usually the Domino server would be configured for Multi-server session authentication SSO, but SSO is not strictly required.

Follow the steps below to manage Web user authentication in Active Directory and eliminate Domino Internet passwords. This configuration requires you to add users' Notes distinguished names to Active Directory user accounts.

Step 1
The Domino server must be configured to use a directory assistance database. In the directory assistance database, create an LDAP directory assistance document to use to connect to the Active Directory server. The following table describes some of the most important fields to configure in the LDAP directory assistance document.

Tab	Field	Value	Comment
Basics	Make this domain available to	Notes Clients and Internet Authentication/Authorization	Required LDAP Clients is optional
Basics	Group Authorization	Yes or No	Select Yes if you want to use Active Directory groups in database ACLs.
Naming Contexts (Rules)	Trusted for Credentials	Yes	--
LDAP	Attribute to be used as Notes Distinguished Name		Attribute in Active Directory that stores users' Notes distinguished names. A directory administrator may need to extend the Active Directory schema to add an attribute for this name if there is no existing attribute that already contains the Notes distinguished name. Alternatively it may be feasible to use the altSecurityIdentities attribute, if not already in use for another purpose. A directory synchronization tool such as IBM Tivoli Directory Integrator can be used to populate the attribute with the Notes names. The value stored in the attribute must adhere to valid distinguished name syntax. In Active Directory use LDAP comma (,) separators in the Notes names rather than the Notes forward slash (/) separators; for example: cn=Betty Zechman,ou=Marketing,o=Renovations rather than cn=Betty Zechman/ou=Marketing/o=Renovations Used to link this Active Directory record to a Notes distinguished name for determining user access to Domino resources.
LDAP	Type of search filter to use	Active Directory	--

• If Multi-server session authentication (SSO) is deployed for Web access to your Domino server, you will also need to consider the following item in the LDAP directory assistance document:

Tab	Field	Value	Comment
Basics	Attribute to be used as name in an SSO token	$DN	Required only if there is an IBM SSO server authenticating users against Active Directory so that users' LTPA tokens contain their Active Directory names. Requires "Map names in LTPA token" to be enabled in the Web SSO Configuration document.

• For use within the intranet, if Windows Single Sign-on for Web clients is being deployed as a Multi-server session authentication (SSO) option on your Domino web server, the following items in the LDAP directory assistance document should also be configured:

Tab	Field	Value	Comment
Basics - SSO configuration	Windows single sign-on for Web clients	Enabled	Enables efficient name lookups based on users' Active Directory logon (Kerberos) names. In combination with "Attribute to be used as Notes Distinguished Name", allows the user's Kerberos identity to be associated with the Domino name.
Basics - SSO configuration	Kerberos realm	Active Directory domain	Specify in upper case characters, for example, AD.ACME.COM.

Step 2
If a user has a Person document in the Domino Directory, make the following edits to the Person document to set up for authenticating the user for Internet access using the Active Directory password. Person documents are optional for Web users who are not LotusiNotes users.

Tab	Field	Value	Comment
Basics	Internet Address	Value of the mail attribute in the user's Active Directory account	Used to link Web user Person document to the Active Directory user account.
Basics	Internet Password (HTTPPassword)	None	Remove the password to use user's Active Directory passwords for Internet access that requires user password verification. When password removed, you should set directory access to prevent users from adding a Domino Internet password themselves. When password removed, Domino verifies user passwords in Active Directory in situations when Windows single sign-on is not available.

Step 3
If a user has a Domino Person document but you have removed the Domino Internet password, disable the following Internet password settings in users' effective Security Settings policy document:

Tab	Field	Value	Comment
Password Management Basics	Allow Users to Change Internet Password over HTTP	No	The default behavior is Yes. If there is no Security Settings policy document specified for users, create one in order to change the default behavior.
Password Management Basics	Update Internet Password When Notes client Password Changes	No	--
Password Management Basics	Enforce Password Expiration	Disabled (or Notes Only)	The Notes Only setting is enforceable for Windows users only if you are not deploying Notes shared login, which eliminates Notes passwords.

Step 4
Specify the following setting in the Server documents of participating Domino servers:

Tab	Field	Value	Comment
Security - Internet Access	Internet authentication	Fewer name variations with higher security	--

Step 5
If Multi-server session authentication (SSO) is deployed for Web access, additional configuration is needed in the Domino Directory Web SSO Configuration document. If the SSO servers are authenticating users against Active Directory, specify the following setting in the Web SSO Configuration document:

Tab	Field	Value	Comment
Basics - Token Configuration	Map names in LTPA tokens	Enabled	Used to map Active Directory distinguished names in SSO LTPA tokens to Notes distinguished names for determining user access to Domino resources. Used to ensure functioning SSO at servers that authenticate the user against Active Directory.

Quick guide to setting up SSL using Domino as the Certificate Authority

Question

What steps can you follow to set up SSL on a Lotus® Domino® server using Domino as the Certificate Authority? Is there a quick guide other than the detailed steps in the Lotus Domino Administrator help?

Answer

Below are the steps to configure SSL on a Lotus Domino server using Domino as the Certificate Authority (CA). This method takes advantage of the Domino CA's keyring form that automatically creates the server key ring and the server certificate request, signs it with the CA certificate, then installs the CA certificate and the signed server certificate into the server key ring. For detailed steps and information, refer to the Domino Administrator Help section "Using a Domino 5 certificate authority."
Note: This method of configuring SSL does not use the server task called the CA Process. For information on that method of configuring SSL, which was introduced in Domino 6, refer to "Quick guide to securing a Domino server with SSL using the CA process" (#1193730).

Steps to create the files:
1. Create the Domino Certificate Authority database from the Domino R5 Certificate Authority (cca50.ntf) template if needed.

2. Open the Certificate Authority database (from the Domino Administrator, click Files, and open the Domino Certificate Authority application). Select the first option "1. Create Certificate Authority Key Ring & Certificate."

Picture of Setup options:

3. Fill in the required fields on the form to create the key ring file for your Certificate Authority. Enter a unique name for the Common Name field of the form to identify the Certificate Authority.

Picture of Key Ring form


4. Click the button "Create Certificate Authority Key Ring." This creates a key ring file (by default named CAkey.kyr) for the Domino Certificate Authority.

5. From the main menu, select "2. Configure Certificate Authority Profile" to set a profile for the Certificate Authority. Most of the information on this page does not need to change, and the fields that are blank are optional and can be left blank if desired. Be sure that the caKey.kyr (or the name entered on for the Key Ring File Name in Step 3) is listed at the top. The last entry, "Default validity period", has a default value of 2 years. The maximum for this is 10, but the validity period defined must result in an end date earlier than the validity date of the Certificate Authority's key ring itself. Verify the information on the page is correct, then click Save and Close.

6. From the main menu, select the third option, "3. Create Server Key Ring & Certificate." You also start from this option when you want to make an additional server's key ring.

7. Fill in the required fields on the Create CA Server Key Ring form. Be sure to enter the fully qualified host name in the "Common Name" field. This host name should be identical to the name Web users will be entering to access the server.

The CA Certificate Label field should be "CAKeyPair" if you initially used "1. Create Certificate Authority Key Ring & Certificate."

(Note: You use this form to create any subsequent key ring files for other servers that need SSL enabled.) Picture of the Server Key Ring form


8. Once the form is complete, select "Create Server Key Ring." Domino then creates the server key ring (by default keyfile.kyr), creates the server certificate request, signs it with the CA certificate, then installs the CA certificate and the signed server certificate into the server key ring.

9. Copy the key ring files (by default named keyfile.kyr and keyfile.sth) to the Data directory of the Domino server on which you wish to enable SSL. You can find these files in the data directory of the Notes client that you used to create them.

Steps to configure SSL on the server:
1. Verify that the key ring files created previously are in the Data directory of the Domino server.
2. Open the Server document for this server. Go to the Ports -> Internet Ports tab.
3. If necessary, change the entry in the SSL key file name field to reflect the name of the server key ring file.
4. Make sure that SSL port status is set to enabled. Optionally, to force SSL to be used for all connections, change "TCP/IP Port Status" to "Redirect to SSL."
5. Save and close the Server document.
6. Restart the HTTP task at the server console.

Steps for Web browser (optional):
Because you are using a private Certificate Authority, in some cases when a Web browser user access the server using HTTPS, the user might see the following warning message:

"The security certificate was issued by a company you have not chosen to trust..."

Picture of complete warning message


To resolve this warning, you have to trust the Certificate Authority. Once you have trusted the Certificate Authority into your browser, the security check is cleared and any subsequent access does not result in a security alert.

1. To trust the Domino Certificate Authority, open the Certificate Authority database in your Web browser.
2. Select "Accept This Authority In Your Browser" and follow the on-screen steps to install the trusted root into your browser.

Or you can execute steps in the Web browser to accept the certificate. Steps for commonly used Web browsers are provide below.

For Microsoft Internet Explorer 6:
1. Click "View Certificate"
2. Click "Install Certificate"
3. Follow the on-screen prompts

For Microsoft Internet Explorer 7:
The warning message that appears is "There is a problem with this website's security certificate."
1. In the warning message, click "Continue to this website"
2. In the next window, click the Certificate Error next to the address bar to launch a pop-up dialog in which you click "View Certificates"
3. Click on the "Certification Path" tab, and select the top-most entry, which should have red X. Then click "View Certificate"
4. Click "Install Certificate"
5. Follow the on-screen prompts in the installation wizard

For Firefox:
1. Click "Accept this certificate permanently"
2. Click Ok

ibm.com

Lotus Domino SSL Certificates Installation

SSL Certificate Installation in Lotus Domino

Note: Installing SSL Certificates on Lotus Domino Web Server requires that the Certificate files be merged into the same Key Ring that was used to generate the CSR. If you do not have this Key Ring then you will need to have a new certificate issued with a new CSR from a new Key Ring. The certificates must be installed to the Key Ring in the correct order (see example of order below):

TrustedRoot.crt
DigiCertCA2.crt (if included in your zip file)
DigiCertCA.crt
your_domain_name.crt

1. Open Domino Server Certificate Administration (CERTSRV.NSF). This is in the System Databases in the administration panel of Notes.
   
   
2. Choose "Install Trusted Root Certificate into Key Ring". Enter the file name of the key ring that was made when you created your CSR, then install the Trusted Root Certificate (TrustedRoot.crt). You may get a message that the root certificate is already installed as a trusted root. If you receive this message continue to step 3.
   
   
3. Again choose "Install Trusted Root Certificate into Key Ring". Enter the file name of the key ring, then install your Intermediate Certificate(s). If your zip file includes DigiCertCA2.crt, install that file first, and then install repeat this process and install DigiCertCA.crt next to this same location. Otherwise, you only need to install the DigiCertCA.crt file here.
   
   
4. This time choose "Install Certificate into Key Ring". Enter the file name of the Key Ring, then install your Primary Certificate (your_domain_name.crt) using the "Merge Certificate into Key Ring" button.
   
   Your SSL Certificate now installed to your Key Ring and it is ready for use on your Lotus Domino Server.
   
   http://www.digicert.com

 

DAOS Backup and Restore

Table of Contents

• 1 Backup considerations for DAOS
• 2 Order matters
• 3 DAOS index files
• 4 Transaction logging
• 5 Command examples
• 6 DAOS enable and disable considerations
• 7 Space and time savings
• 8 Restoring DAOS objects
• 9 Restoring documents or NSF files with DAOS attachments
• 10 Dealing with damaged files and clusters
• 11 Options for restoration
• 12 Offline archival

This article describes how to refine your backup and restore procedures when using the popular Domino Attachment and Object Service (DAOS) feature, also known as "attachment consolidation."

Backup considerations for DAOS

In a standard Notes® database (NSF), the attachments are stored inside of the NSF file itself, and the database is self-contained. In order to back up a standard Notes database, only the NSF file itself needs to be backed up. After you introduce DAOS, the NSFs that participate in DAOS contain only references to the NLO files where the attachment content is stored. As a result, backing up the NSF alone is no longer enough. The NLO data needs to be backed up as well.

When performing standard NSF backups, there are two main approaches. The individual NSF being backed up can be taken offline (or the entire server can be stopped) so that no changes occur to the NSF over the duration of the backup operation. The other method allows the NSF and server to remain active, but requires using a backup utility program that interacts with the Domino® backup/restore API. Using a utility ensures that a consistent copy of the NSF contents is recorded, despite any changes that occur to the NSF over the duration of the backup operation. If you are using a backup utility that does not use the backup/restore API, you must stop the Domino server and all Domino-related applications for the duration of the backup.

None of the processing for NSF backups needs to change for DAOS. The change needed to accommodate DAOS is simply a procedural addition: in addition to backing up the NSF data, you must also back up the NLO data.

Backing up the NLO files in the DAOS repository can be done either while the Domino server is down, or when it is up and running. The backup does not require the use of any Domino API-based utilities. Once NLO files are written initially, Domino never modifies their contents, so the backup mechanism does not have to work around file-write activity. NLO files can be backed up as any other generic file on the file system. Only the NLO files that are complete and not in the process of being written to or renamed need to be backed up. Any that are busy can be skipped until the next backup. Most backup applications will automatically skip files that they can not read because of other activity.

Order matters

If you shut down the Domino server during the backup process, the NSF and NLO files can be backed up in any order. If you must keep the Domino server up and running during the backup process, it is important to back up all the NSF data before backing up the NLO files. The reason has to do with the addition of references to new NLO files in an active system, described in this section.

When you back up an NSF that participates in DAOS, there are some number of NLO references contained in that NSF at the time of the backup. Since there is some duration to the backup operation for all NSFs, the number of references to NLO files may be increasing over that duration in a system that is operating during the backup process. If there were (for example) 10,000 NLO files referenced collectively by all the NSFs at the beginning of the NSF backup process, there could be 10,100 by the time the last NSF is backed up.

Likewise, the backup of the NLO data has a duration as well, so while there might have been 10,100 NLO files at the beginning of the NLO backup process, there could be 10,200 by the time the last NLO is backed up.

In this scenario, the backed up version of the NSFs could reference at most only 10,100 NLO files. Because the NLO backup was done after the NSF backup process, the NLO backup included at least that many, but may have as many as 10,200 NLO files. Worst case, there are more NLO files backed up than strictly necessary to satisfy the NSF references. Since all accesses to the NLO files are done through the NSFs, and the NSFs were done first, all of the referenced NLO files are guaranteed to exist in the set of NLO files that were backed up. If there is an error accessing an NLO file in order to back it up because it's in use, that can safely be ignored. If the file is being written, the activity must have occurred after the NSF was backed up; therefore, this NLO file does not need to be in the corresponding set of NLO files, and will be backed up as part of the next cycle.

The deferred deletion interval should be set to a period longer than your chosen backup cycle. In this way, the NLOs are not pruned (physically deleted) prior to the next backup. Instead, the actual deletion is deferred until they've aged accordingly.

If you were to have a shorter or nonexistent deletion interval—the feature can be disabled by setting it to zero in the DAOS tab of the server document—it opens a window of time during which a deleted attachment is non-recoverable, as the NLO file has been physically deleted before the backup has occurred.
Avoid pruning NLO files from the repository (by issuing a prune command at the Domino console) before they have had a chance to be backed up; you will prevent them from being recoverable. When an attachment is deleted, and the associated NLO file's reference count goes to zero, it becomes a candidate for deletion. The deferred deletion interval determines when the deletion actually occurs. If the deferred deletion interval is set (as recommended) to be longer than the backup cycle, all NLOs will be in existence for at least one backup cycle, and therefore any NLO can be recovered later.

After the initial full backup of the NLO files in the DAOS repository, you can perform incremental backups, which save only the data that has changed since the last backup. NLO files are ideal candidates for incremental backup because there are no changes to them after their initial creation.

One NLO file is created for each unique attachment, so it is possible to have a very large number of NLO files in large deployments. The maximum number of files per numbered DAOS subdirectory is 40,000, and there can be 1000 subdirectories, for a maximum total of 40 million NLO files. Check with your backup utility specifications to see if there is a limit on the total number of files it will manage, and monitor the growth of the DAOS directory file population accordingly.

DAOS index files

The daos.cfg and daoscat.nsf files should not be backed up. (Note that this is a change from earlier recommendations) These two index files can be re-created from the DAOS repository and the NSFs participating in DAOS if necessary. If these files become corrupted, they can be safely deleted while Domino is not running. They will be created on startup automatically.

The daos.cfg file helps manage the files in the DAOS repository. The NLO files are stored in subdirectories (0001, 0002, and so forth) underneath the base DAOS directory. For several reasons (including performance), DAOS limits the number of NLO files in each subdirectory. The daos.cfg file keeps track of how many files are currently in each subdirectory so that DAOS puts new files in subdirectories where the count of files is below that limit. As NLOs are deleted, the corresponding file count is decremented, allowing backfilling of older subdirectories. The daos.cfg file is expendable, and will be re-created at Domino startup time if it is missing.

The daoscat.nsf file contains two indexes. One is a list of all NSFs that are holding NLO references (DAOS ID Table, or DIT.) The second is a list of all NLOs that exist, and the DAOS repository subdirectory they exist in (DAOS Object Index, or DOI). There are no externally visible parts to this NSF, and there are no privileges that apply to change that. The DIT is modified when an NSF acquires its first NLO reference. The DOI is modified when a new (unique) NLO is created. The daoscat.nsf file is expendable, and will be re-created at Domino startup time if it is missing. Since a full resync can take a significant amount of time, only empty indexes are created by this process at startup. A resync operation should be done as soon as it is convenient, however.

In some cases it could be necessary to fully reboot the server until the daoscat.nsf and the daos.cfg are re-created.

A DAOS resync operation (“tell daosmgr resync”) fully re-populates these two indexes from scratch. You can also run the command “[n]daosmgr resync”if you want to perform a resync operation with the Domino server shut down.

Transaction logging

Because all NSFs that participate in DAOS have to also participate in transaction logging, the contents of all their attachments will be included in the log. Any NLO files that are created as a result of activity to the NSF will be re-created if the log is replayed.

Command examples

Using the Tivoli Storage Manager (TSM), the command to back up the DAOS repository would be:

dsmc incremental c:\lotus\domino\data\daos

where the path specified is the full one to the DAOS repository.

Since the NLO files are being backed up incrementally, the initial backup will be quite large, but subsequent ones will be much smaller. The total footprint of the DAOS directory will be written out during the first backup.

DAOS enable and disable considerations

Once a Domino server has DAOS enabled, and NSFs are selected to participate in DAOS, their attachments are stored in the DAOS repository. If DAOS is subsequently disabled, the attachments that were in DAOS remain in DAOS until they are re-integrated into the NSF. Any DAOS references that remain in the NSF will continue to be serviced by DAOS, even if it is disabled. An NSF that contains DAOS references is not self-contained, and must continue to be treated as an active DAOS participant as long as it has DAOS references. To re-integrate the DAOS attachments into an NSF and remove the DAOS references, you can process the NSF with the “compact -c -daos off” command. Once that is done, the NSF will be self-contained again, and can be treated as a normal NSF.

Furthermore, to ensure that the DAOS enablement change takes effect completely, the Domino server as well as all processes that use the Domino API (compact, resync, backup, etc) are stopped. This allows the API to terminate completely, so the status change can be picked up at the next startup.

Space and time savings

The disk footprint savings with DAOS continues into the backup processing as well. The NLO files represent the static data that used to reside in the NSF, and was backed up every cycle even though it hadn't changed. In a typical mail environment, a large reduction in the NSF footprint plus a very small amount of NLO data means the reduction in the NSF footprint translates almost directly into a reduction in the backup footprint. Not only is the duplicate data being eliminated, the mailfile data is being separated into static and dynamic components. By applying an incremental backup regimen to the static (NLO) data, only the NLO files created since the last backup cycle need to be processed. That represents typically a very small amount of data compared with the entire set of NLO files.

Using DAOS enables backup software solutions to optimally backup the NLO files. This is because once an NLO is written to disk, it never changes. Therefore, the file need only be backed up once in its lifetime. Based on the example shown in the DAOS Estimator document [link], the space saving per full backup would be 38.8 GB, roughly equal to the number of shared NLO's times the average NLO size.

In the incremental backup case, duplicate NLO's will not be backup up again. Thus, the space savings from DAOS is directly proportional to the numbe r of duplicate NLO's seen in the environment, and the backup time savings is the product of the space saved and the backup throughput.

Restoring DAOS objects

A well-preserved DAOS repository makes for fast and easy restoration. And, while not really a backup mechanism, the default deferred deletion interval allows for accidentally deleted attachments to be saved from physical deletion up to 30 days after a mishap. Simply pull the document out of the trash folder, if soft deletion is enabled. If it's too late for that, restore from backup, then resynchronize the DAOS catalog using the server console command "tell daosmgr resync force" -- DAOS will once again recognize that the NLO has references.

Although an NLO will survive for the period of time specified by the deferred deletion interval, if soft deletion is disabled or a backup of the referencing document has not been made, there is no way to get at the contents of the NLO, especially if encryption is enabled (the default).

Restoring documents or NSF files with DAOS attachments

To restore either a full NSF or a single document, the process starts off the same. You must first restore the database and then the missing NLOs. To do this using Tivoli Data Protection for Domino, you would issue the command:

domdsmc restore -into

To determine which missing NLO files to bring back from the Domino server console, run

tell daosmgr listnlo -o missing.txt MISSING restoreddatabasename.nsf

The resulting missing.txt file is then fed into the restore command With the Tivoli Storage Manager (TSM), the command would be

dsmc restore -filelist=missing.txt -inact 

If you are restoring the entire NSF, you are done. Note that any restoration operation will put the DAOS catalog into the Needs Resync state, so a resync operation should be performed as soon as convenient.

If you need only one document, you can now copy and paste it to its intended destination.
For a complete recovery after a catastrophic failure, the NSF and NLO files can be restored, followed by replaying the archived transaction logs. This will result in the most up-to-date recovery situation.

Dealing with damaged files and clusters

If an NSF is damaged, and you have clustered servers or replicas of the NSF on another server, you have several options.

1. Replicate each entire NSF – New replicas can be created from the existing NSFs on the clustermate(s). Each new replica should be marked as DAOS enabled. As the replication occurs, the associated attachments will be saved to DAOS.
2. Copy NSFs, replicate missing attachments – All of the necessary NSFs are copied from the clustermate(s) to the server being repaired. This will create a copy of the document data without attachment data. Fixup -j -D is then run, deleting all documents that contain DAOS references to NLO files that do not exist. Subsequent replication will re-create those documents along with the associated attachments, which will be stored in DAOS.
3. Copy NSFs, copy/restore NLOs - All of the necessary NSFs are copied from the clustermate(s) to the server being restored. The command 'tell daosmgr listnlo missing somefile.nsf' is then issued for each individual NSF to generate a list of the NLO files that do not exist in the DAOS repository. Those NLO files are then restored from backup, or copied from the clustermate(s). (Note that copying the NLO files from another Domino server will work only if DAOS encryption is turned off. DAOS encryption is on by default, and uses the server key to do the encryption; therefore enc rypted NLO files are not portable between Domino servers.)
4. Copy full NSFs and re-extract – If you have a replica on another server that is not DAOS-enabled, the NSF can be copied to the server being restored. The attachments will be inline in that copy of the NSF, and 'compact -c -daos on' can be issued to extract the inline attachments out to DAOS.
5. Reintegrate NSFs and re-extract – If you have a replica on another server that is DAOS-enabled, but encryption prevents using the NLO files directly, you can run a 'compact -c -daos off' on the other server to re-integrate the attachments into the NSF. Once that is done, the NSF can be copied to the server being restored, and you can use 'compact -c -daos on' to extract the attachments to DAOS again.

Options for restoration

The need for restoring NLO files depends partly on the deferred deletion interval. If the restore is happening from a snapshot that's within the interval (for example, the interval is 30 days, and the NSF is being restored from last week's backup) it's not possible for any of the NLOs to have been deleted yet, so there shouldn't be a need to restore any NLOs. If the NSF being restored is older than the interval (for example, the interval is 30 days, but the NSF being restored is from 3 months ago), it's possible that some of the NLOs have been deleted, and would need to be restored.

Some of this also depends on what the reason for the restore is. If it's a catastrophic failure, you should restore the NSF(s) and run 'tell daosmgr listnlo missing filename.nsf' to get a list of all of the NLOs needed to make it whole again. That list should then be fed into the restore utility to restore those NLOs as well.

If it's just a matter of getting a single document with attachments back, you don't really need everything to be made whole to access just that one set of attachments. In the spirit of “Work smarter not harder,” you can restore the NSF, and then attempt to access the desired document (and attachments, if any) and finally deal with any missing NLOs that are mentioned during that operation. If there aren't any attachments on the document, there's no other work to be done. If there are attachments, the NLOs may still be there, so it's worth trying to access them before doing anything else. If any are missing, you'll get a console message that mentions the name of the NLO, at which point you can restore only what you need.

Offline archival

For offline archival purposes, it is recommended that the attachments be re-integrated into the NSF using a 'compact -c -daos off' operation prior to archiving. That eliminates the need to archive all the individual NLO files referred to by the NSF also. For example, if an employee was leaving the company, and their mail account was being closed and archived, this approach would be appropriate.

lotus.com

DAOS Best Practices

When it comes to configuring the Domino Attachment and Object Service (DAOS), you may be asking yourself - and us in turn - what's the right way to set it up? For example, is there an optimum “Minimum Size” setting? Should the repository go under the data directory or on its own drive? What's the best “Deferred Deletion Interval” in relation to my backup and restore schedule? This guide, and the documents it references, attempts to answer these questions of individual, site-specific configuration in general terms with guidelines for adapting and modifying them based on measurements made against your particular environment.

Where To Locate Your DAOS Base Path Repository

By default the DAOS repository resides under the server's data directory and defines a single container as indicated by the “DAOS Base Path” setting in the DAOS tab of the server document. So, on Windows for example, if you use the default, “DAOS”, and your data directory is C:\Lotus\Domino\Data, the full path to the repository would be C:\Lotus\Domino\Data\DAOS. For Domino 8.5, only this one container may be specified.

However, this default location, chosen simply for being well-known, may not be the most efficacious. Some things to consider:

1. What is the total capacity of all file attachments? With only one container in Domino 8.5, flexibility may be important when choosing the best DAOS base path. You'll want to be sure you have significant storage capacity or the ability to reconfigure a logical drive as space needs increase. Use the Domino Attachment and Object Service Estimator to plan for your storage requirements. 2. What I/O costs do I expect to incur? DAOS base path I/O is significantly less than that of Domino's Data directory. In benchmark tests, DAOS repository I/O was 94% less than that of the server's Data directory. Lower performing storage (a NAS device, for example) can be used here. 3. Can I use lower cost or external storage devices? In many cases, you might find attachments are infrequently accessed -- for example, when they're part of old email messages collecting the proverbial dust in one's inbox. In these environments, locating DAOS on lower cost storage (tier 3) devices may be indicated. On the other hand, if full text indexing, agents, or other applications make heavy use of the consolidated attachments, “lower cost” storage may cost you in performance. Note: Externalizing the DAOS repository in this manner does not mean you can map multiple Domino servers to the same container. This is an unsupported configuration as of this publication and could very well lead to data loss due to encryption with the server's key. NLO files cannot be shared across Domino servers. Note: Modifying the location of the DAOS repository at a later time is allowed and requires that you first change the “DAOS Base Path” field on the DAOS tab in the server document, stop the Domino server and then relocate the existing subdirectory structure with its NLO files to the new location. On server restart, the modification will take place seamlessly. 4. Why is it recommend to not locate the DAOS Base Path under Domino's Data directory? Many of Domino's tasks, including fixup, compact and the admin client, scan Domino's Data directory and will also scan the NLO files if the files are located under that sub-directory. The scanning of NLO files for tasks other than DAOSMGR will add additional unneeded overhead to the processing done by Domino's tasks.

Optimum Minimum Size For Participation

By default, the minimum size setting for an attachment to make use of DAOS is 4096 bytes. While we recommend using 64000 as the lowest value you should use here (1048576 on iSeries), there are a number of things to consider when determining the best DAOS minimum size setting for your system.

1. Do not set the minimum size lower than the default setting. Due to attachment file overhead, setting the minimum size to anything lower than the default size would actually be less efficient than storing the attachment in the NSF file. 2. Set a minimum size that is a multiple of your file system's disk block size. By choosing a minimum size that is a multiple of the disk block size, you optimize disk usage. To ascertain the disk block size for your file system, on a Windows NTFS, use “fsutil fsinfo ntfsinfo ” and take note of the “Bytes Per Cluster”. This is the disk block size. On Solaris, you could use df -g and take note of the block size. On AIX you need be super user to do determine block size, then use lsfs -q and look for block size. On Linux you also need to be super user to find the block size, then use df -k to determine the device name of your filesystem and the uses dumpe2fs | grep 'Block size' to determine the block size. 3. Take note of possible limitations on number of files. The smaller you make the setting, the more attachments will qualify for DAOS consolidation. The larger you make the setting, the fewer will qualify. In Domino 8.5, the DAOS repository allows for one container with up to 1,000 subcontainers, each with a maximum of 40,000 NLO files. Thus the storage capacity of DAOS is limited to 40 million distinct objects. This is a significant number of files, so if you expect to come anywhere close to approaching it, you should check the limits on your backup and restore solution, as some applications and file systems have limitations on maximum number of files. Refer to your operating system and/or backup application guidelines. 4. The current recommendation for the lowest value you should use on IBM iSeries is 1048576 (1M) to avoid overwhelming the filesystem and backup utilities.

To get an idea of how many files various settings would generate, you can run them through the Domino Attachment and Object Service Estimator.

The ultimate goal with this setting is to minimize the number of files in your DAOS repository and maximize the amount of disk space saved.

Deferred Deletion Interval

DAOS automatically deletes NLO files that are no longer being referenced by any databases. This deletion of NLO files is known as “pruning” and occurs at the specified “Deferred Deletion Interval.”

Establishing a useful “Deferred Deletion Interval” for your server involves a few considerations, primary among them your backup and restore schedule. You want to ensure that NLO files which are no longer needed remain in existence at least as long as your backup cycle. In this way, they will not be deleted before the next backup.

A secondary consideration is the size of attachments typically stored in the repository for your server. If they are usually quite large, you may want to have them cleaned up as quickly as possible after there are no longer any references to them.

If your deferred deletion interval is set too high, NLO files which are no longer needed will continue to take up valuable space in the repository. If your deferred deletion interval is set too low, you could be deleting NLO files that have not yet been backed up, thus making it difficult, and in some cases not even feasible, to restore them. It is important to find a balance that satisfies both your backup schedule and the system integrity of a neatly pruned environment.

Pruning

Pruning can also be manually triggered to override the automatic deferred deletion interval. The administrator can issue the console command “tell daosmgr prune x” to forcibly delete unreferenced NLOs that are x days old. This will recover the disk space still being used by unreferenced NLO files immediately rather than waiting for the automatic deferred deletion interval to do so. When performing this action, you must consider your backup cycles. As with setting the deferred deletion interval too low, pruning too soon could delete NLO files that have not yet been backed up.

When A Notes Database Should Use DAOS

There are several good reasons to select an NSF file for participation in DAOS consolidation:

• It contains or is likely to have multiple copies of the same attachments.
  Even a single NSF can benefit from DAOS consolidation.
• It resides on a server where the same attachments appear across multiple NSF files.
  If others are also referencing attachments present in your database, why not share?
• It contains very large attachments.
  In this case, it may not matter how many other NSF files hold the attachments in question. If they're large enough, the simple step of storing them outside the NSF can make common operations against that database much faster.

While DAOS can always benefit your data, DAOS has less benefit under the follow conditions:

• Databases have lots of small attachments.
  Attachment consolidation is less efficient in this scenario due to disk blocking. You can, however, eliminate this issue by adjusting the minimum size setting upward.
• There is little or no attachment duplication across databases.
  Backup would still benefit due to extracting static data, but you would have little disk space reduction.
• Databases contain few or no attachments.
  In 8.5, DAOS stores only file attachment data.
• Databases need to be quickly portable.
  Because DAOS object files (NLOs) cannot be shared across Domino servers, it is more difficult to move DAOS-enabled Notes databases from server to server.

It's not necessary for all databases on a server to leverage DAOS, but for those that do, the savings in both space and time, as for example, in much accelerated compact operations, are significant.

Mailbox

Although DAOS will work in any configuration, it operates most efficiently when it is enabled on both the mail.box files and individual mail files. Enabling transaction logging and DAOS on mail.box will enable the Router to optimize the delivery of DAOS based attachments. This can result in significant I/O savings for the case where the same attachment is sent to multiple recipients on the same Domino server.

I/O Activity for mail delivery of email with an attachment

DAOS Enabled		Document Writes	Attachment Read(s)	Attachment Write(s)	Comments
MAIL.BOX	Mail files
No	No	1 + N	1 + N	1 + N
Yes	No	1 + N	1 + N	1 + N
No	Yes	1 + N	1 + N	1 + N
Yes	Yes	1 + N	1	1	Maximum reduction for both I/O & CPU

* N is the number of recipients

Note: If you have multiple mail.box files, you must enable transaction logging and DAOS on all of them to leverage DAOS object copy optimization, which streamlines delivery of attachments.

Since Domino creates new mailboxes as needed, you should also set these properties on the mail.box template. If you choose not to enable DAOS or transaction logging on the mail box, DAOS will still be used by any DAOS-enabled mail files. Using DAOS on mail.box(es) only affects the optimized routing (delivery) of attachments.

When an incoming document is received at mail.box, it is stored there until it is delivered to the individual mail file(s) of the recipient(s). Several results are possible:

• If DAOS is not enabled anywhere, the document will be stored in mail.box, and the attachment will be stored inline. As the document is delivered to the recipient mail file(s), the document and the contents of the attachment are read from mail.box, and the document is written to the mail file with the attachment inline. Total I/O cost to deliver to N recipients: 1 + N doc writes, 1 + N attachment reads, 1 + N attachment writes.
• If DAOS is enabled on both mail.box and destination mail file(s), any attachments in that document will be extracted and converted to NLO files as it is being written to mail.box. The document and DAOS ticket are written to the destination mail file(s). IMPORTANT: In the case where both mail.box and the mail file are DAOS-enabled, the contents of the attachment will not be written again as the document is delivered, only a reference to the existing NLO file will be copied. Total I/O cost to deliver to N recipients: 1 + N doc writes, 1 attachment read, 1 attachment write.
  
  If DAOS is enabled on mail.box but not on the destination mail file(s), any attachments in that document will be extracted and converted to NLO files as it is being written to mail.box. Since the destination mail file(s) is/are not using DAOS, the attachment must be stored inline, and the contents of the attachment will be read out of mail.box (which has it stored in DAOS) in order to do that. Total I/O cost to deliver to N recipients: 1 + N doc writes, 1+N attachment read, 1+N attachment writes.
• If DAOS is not enabled on mail.box, but is for the destination mail file(s), the attachment will be stored inline in mail.box. As the document is delivered to the recipient mail file(s), the contents of the attachment are read out of the mail.box document, and a temporary NLO file is created for each destination mail file so that a checksum can be calculated. If an NLO file with the same checksum already exists, the temporary file is deleted. In the case of N recipients, this process will be repeated N times, even though only one NLO file will remain at the end of the process. Total I/O cost to deliver to N recipients: 1 + N doc writes, 1+N attachment read, 1+N attachment writes. IMPORTANT: In this case, although the end result (a single NLO file per unique attachment) is the same, the I/O cost is significantly increased over the case where mail.box is enabled for DAOS.

Mail Journaling

For 8.5, it is recommended that you not enable DAOS on the mail journal (mailjrn.nsf).

Encryption

By default, DAOS employs encryption to safeguard its repository. This setting is separate from encryption settings that apply to an NSF or document. The encryption is done with the server key, so the resulting NLO files can be read only on a server that uses that same key. This may be a consideration for backup or redundant server setup.

The performance hit for DAOS encryption is negligible; testing showed a 5% CPU increase with no change in I/O versus unencrypted data. However, if your organization has reason to disable it, we've provided the server notes.ini setting DAOS_ENCRYPT_NLO, which can be set to zero to affect that change. To determine the current status of encryption, use the ”sh stat daos” command from the server console.

There are some storage area network devices that have the ability to deduplicate files. If DAOS_ENCRYPT_NLO=0 is not specified, deduplicating will not occur across Domino servers. The encryption will make even duplicate files unqualified for deduplicating because they will encrypted with each server's key.

Compression

While DAOS is compatible with compression, there are a couple of points to remember:

1. It's possible for an attachment to disqualify itself from DAOS consolidation by compressing to a size smaller than the Minimum Size setting. 2. The same attachment, undergoing different compression types, LZ1 versus Huffman versus no compression, will be seen by DAOS as different objects and will, therefore, be shared only with others of like type.

Resynchronization of the Catalog

In order to ensure that NLO files are not physically deleted when there are still ticket holders referencing them, if there is any reason to question the accuracy of a reference count, DAOS puts itself in a safe mode whereby no deletes are allowed to proceed. This state is signaled to the administrator via the Domino Domain Monitoring systems and is reported as a “NEEDS RESYNC” state from the “tell daosmgr status catalog” server console command.

To perform this catalog resynchronization manually, type “tell daosmgr resync” from the server console.

Note: The duration of a resync can be significant and depends on the number of DAOS-enabled databases, the number of NLO files in your environment, and your system configuration. It can take several hours to complete causing its execution to overlap into normal business hours. Although Domino and DAOS are functional while a resync is in progress, there may be a degradation in performance while it is running.

Prior to 8.5.2, if you find it necessary to halt the resync operation, you can interrupt it by issuing “tell daosmgr quit” from the server's console. However, for continued operation of DAOS, immediately restart the DAOSMGR task using the console command “load daosmgr.” When it is convenient to continue the resync operation, issue “tell daosmgr resync” again from the server's console and resync processing will continue where it left off.

8.5.2 introduces many improvements to the performance of the resync operation. Resync will complete sooner in this release and improvements to keep the catalog in sync have been made. Resync can now be scheduled to work with in a time window by setting DAOS_RESYNC_START_TIME and DAOS_RESYNC_STOP_TIME. Setting a time window will eliminate the need to stop and start daosmgr when resyncing the catalog.

Typically, the only downside to being in "Needs Resync" state is that DAOS suspends pruning operations. Unless you are experiencing errors, it is usually better to wait to do a resync until the next available maintenance window.

Antivirus

Consider DAOS interaction with antivirus scans. It's critical the DAOS base path and .NLO file extension have the same anti-virus policy as the Domino data directory and .NSF file extension. If the two file types have different policies, consolidating attachments from an existing NSF using “compact -daos on -c” can result in NLO files being quarantined as they're extracted from the NSF. A user who then opens a quarantined attachment would see a “Missing NLO” error message.

A Domino add-in antivirus program is recommended so that the attachments are scanned as they pass through the server. You should not scan the NLO files directly. (Similarly, the transaction log files should be excluded from scans.)

The screenshot below shows how you'd configure Symantec Antivirus to exclude the .NLO file extension. From the Configure folder on the left, select File System Auto-Protect. Click the Exclusions button, then the Extensions button. Type “NLO” without the quotes and click Add.

Recommended versions of Notes Client and Domino Server

For the Notes Client anything before 8.5.1 or post 8.5.1 FP3 (including this version) should be used. For the Domino Server, the minimum version should be 8.5.1 FP3. For more information see the Technote 1446397 "Attachment corruption related to DAOS and Domino 8.5.1 ".

Ideally the Domino Server should be at 8.5.2 to gain both operational and performance improvements made to DAOS catalog synchronization as well as other DAOS improvements.

Worse practices

1. Too small a minimum participation size - getting too "greedy"
Limit the participation size to what will resulting in reasonable disk space amount with the least number of nlo files. Let's review the following output from the "Domino Attachment and Object Service Estimator":

DAOS Minimum Size versus number of NLO's and Disk Space:

0.0 KB will result in 2226347 .nlo files using 185.5 GB
64.0 KB will result in 1092894 .nlo files using 175.7 GB
128.0 KB will result in 708403 .nlo files using 163.6 GB
256.0 KB will result in 422087 .nlo files using 145.9 GB
512.0 KB will result in 219833 .nlo files using 120.2 GB
1.0 MB will result in 93628 .nlo files using 87.8 GB
2.0 MB will result in 36576 .nlo files using 56.6 GB
3.0 MB will result in 17499 .nlo files using 38.0 GB
4.0 MB will result in 9717 .nlo files using 26.3 GB
8.0 MB will result in 1576 .nlo files using 6.5 GB

While the theoretical maximum (first line) would generate approximately 2.2M files using 185 GB this would not be the ideal participation size for two important reasons.

The first reason is the benefit of DAOS can be realized without having to maintain 2.2M NLO files. A more reasonable participation size between 128k-256K would result in the number of NLO files in the range of 422,087 to 708,403. Taking an average of the two participation sizes, a value of 192K would result in approximately 500,000 NLO files with a disk space size of about 150 GB. The net result would be an 80 % total size yield with a little less than 1/4 of the theoretical maximum NLO files to maintain in your backup/restore procedures.

The second reason is that reducing the minimum participation size is more easily done then increasing the minimum participation size.

2. Deleting the daoscat.nsf and/or daos.cfg to "fix" problems.
The daoscat.nsf and the daos.cfg are vital files for DAOS. daoscat.nsf keeps data about the location of NLO files with reference counts. daos.cfg maintains data about the DAOS configuration. If one or both of these files are deleted, DAOS will rebuild the files; a processes that will take hours to complete. While the files are being recreated the location of individual NLO files and the reference count will be in transition which would result in intermittent access to attachments. Removal of corrupted daoscat.nsf should only be done when all other options for getting the file back have failed.

3. Long deferred deletion intervals set on a hub server.
When mail routing or replication hub servers are DAOS enabled, NLO files will be created and marked for deletion. Until Prune is run as scheduled by the "Deferred Deletion Interval", the files will exist in the DAOS Repository.

4. Failing to monitor the DAOS catalog state.
DDM events are generated for DAOS state changes including the state of the DAOS catalog. It is important to keep the DAOS catalog in a synchronized state to make sure prune can run and to keep access to the DAOS manager functioning correctly. Make sure to monitor the state of the DAOS catalog either with via DDM events or directly via the server console with "show daosmgr status catalog" and make sure the catalogState reports "SYNCHRONIZED".

5. No Backup/Restore procedures.
It is worth repeating the phrase timing is everything regarding backup and restore. It is important to time the backup of nsf and nlo files to be more frequent than the "Deferred Deletion Interval" because prune will remove the NLO files. Another important point is to not only have a well planned Backup/Restore procedure implemented, but to validate the restore procedure regularly to ensure that data can be restored before there is an emergency. For more information refer to the DAOS Backup and Restore

Transaction logging and DAOS

System Database File	Should DAOS be enabled on this database?	Should Archive transaction logging be enabled on this Database?	Should Circular transaction logging be enabled on this Database?	Comments
NAMES.NSF	No	Yes	Yes
LOG.NSF	No	No	Yes	If this is not transaction logged consider deleting it after a crash as it could impact startup time. It may require a fixup and this could take an extremely long time if log.nsf is very large.
ADMIN4.NSF	No	Yes	Yes
MAIL.BOX	Yes	Yes	Yes	If you want to use DAOS and not transaction log mailbox bodies you can use the following in 8.51. NSF_DONT_LOG_MAILBOX_BODY=1 RM_NO_LOG_LARGE_OBJECTS=1 RM_NO_LOG_OBJECTS_IN_MAILBOX=1
DBDIRMAN.NSF	No	No	No
CLUBUSY.NSF	No	No	Yes
DDM.NSF	No	No	Yes
STATREP.NSF	No	No	Yes
STATMAIL.NSF	No	No	Yes
CLDBDIR.NSF	No	No	Yes
WEBADMIN.NSF	No	No	Yes
CLDBDIR.NSF	No	No	Yes
BUSTIME.NSF	No	No	Yes
EVENTS4.NSF	No	No	Yes
STATLOG.NSF	No	No	Yes
REPORTS.NSF	No	No	Yes
MTSTORE.NSF	No	No	Yes
ACTIVITY.NSF	No	No	Yes

Domino Tuning Tips (all platforms)

The following tips will help Administrators to optimize their Notes/Domino environment. It is important for Administrators to understand that any configuration change should be closely monitored for its effects. Make single changes at a time to ensure adverse effects are easy to spot and changes can be backed-out.

View Index Updates

It is possible to tune the Domino update task to take advantage of the additional CPU resource of multi-core hardware. There are a several NOTES.INI parameters that you may use:

Update Task (View Index Updates)

If the system’s resources allow it, additional Update tasks may be started on the server, by adding the parameter, Updaters=[number], to the NOTES.INI. As a rule of thumb, do not exceed a maximum of one Update task per CPU. For example, to enable two Update tasks, apply this parameter:

• Updaters=2

Full Text Index Updates

A separate thread can be created solely for full text updates. This separate thread will only be responsible for updating full text indexes, so view updates will continue to occur even if a large full text index is being rebuilt. Consider applying this NOTES.INI parameter if many full text indexes are present on a Domino server and sufficient CPU resource remain:

• UPDATE_FULLTEXT_THREAD=1

Maintaining view indexes

The NOTES.INI parameter DEFAULT_INDEX_LIFETIME_DAYS=[number of days] allows administrators to set a default lifetime for database view indexes if none was selected by the database designer in the Design View Attributes dialog box.

The default value for this parameter is 45. Setting this to a lower value can have two benefits:

• Save Disk Space
• Reduce amount of work Update task must perform

Creating view indexes is disk intensive and requires CPU and memory resource, so it is important for Administrators and designers to strike a balance. It is not recommended to set this value to lower than 14 (two weeks).

Disable Transaction Logging For Certain Databases

Turning transaction logging may impact your enabled DAOS features relative to mail. In most cases, disabling Transaction Logging is not recommended because you lose the benefits of fast server restart. Disabling transaction logging on a database will cause Fixup to run on that database, creating the potential for slow restart. Furthermore, you will not be able to perform incremental backups using the transaction logs for that database.
For some databases, however, it might not be necessary to enable transaction logging. This is because it generates additional transaction log traffic on the disk and causes the transaction log drive to fill up quicker. Examples of databases that are suitable for disabling of transaction logs are:

• Mail boxes (mail.box) - Do not disable Transactional Logging on Mail.boxes if you are running DAOS.
• Server Log (log.nsf)
• Monitoring Results (statrep.nsf)
• Statistics Mail-in (statmail.nsf)
• Web Administration (webadmin.nsf)
• Schedule (clubusy.nsf or busytime.nsf)
• Cluster Database Directory (cldbdir.nsf)

Table 1 highlights NOTES.INI parameters to force new instances of the certain system databases created on server startup to have transaction logging disabled. They do not disable transaction logging on existing databases, or if they are created manually.

NOTES.INI Parameter	System Database
MailBoxDisableTXNLogging=1	Mail.box (See note below)
Log_DisableTXNLogging=1	Log.nsf
Schedule_DisableTXNLogging=1	Clubusy.nsf or busytime.nsf

Table 1: NOTES.INI parameters to disable database transaction logging.
Note: Do not disable Transactional Logging on Mail.boxes if you are running DAOS.

Replication

If you have more than one Domino server in your environment then you will need to set up replication. The following tips should help optimize the time and resource requirements to replication data around your environment.

Multiple Replicator Tasks

The database replicator (replica) task on a server is responsible for handling scheduled replication requests, as configured in server connection documents. The Cluster Replicator (clrepl) task is event driven and responds to changes in a database. These changes are then replicated to other cluster members.

Each replicator task only replicates with one server at a time, therefore the first replication must complete before the next one can start. The number of concurrent replicator tasks running can be set with the following NOTES.INI parameter:

• Replicators=[number]
• 
  

In environments where more than two servers participate in the cluster, additional clrepl tasks can be enabled. The number of concurrent clrepl tasks running can be set with NOTES.INI parameter:

• Cluster_Replicators=[number]

Administrators should monitor Replica.Cluster Domino statistics. For instance, the WorkQueueDepth Domino statistic indicates the number of changes waiting to be replicated to cluster members. If it is continually increasing, enable additional clrepl tasks.
The following statistics give indication on the current, average and peak Cluster work queue depth:

• Replica.Cluster.WorkQueueDepth
• Replica.Cluster.WorkQueueDepth.Avg
• Replica.Cluster.WorkQueueDepth.Max
• 
  

Replication Triangulation

When a client or server replicates with a remote server, it keeps a log of the name of the remote server and the time and date in the Replication History. Domino uses the replication history to determine which documents to scan for changes during the next replication. The purpose of Replication Triangulation is to make each server aware of every other server which maintains a replica of the same database, and which has had a successful replication.

In a large environment (hundreds of servers in a domain), the number of replication history events to maintain can cause a significant performance impact to the Domino server. Maintaining replication triangulation history for databases which exist on hundreds or thousands of servers is too expensive. This can manifest itself in the form of increased CPU activity for the replica task.

You can disable replication triangulation with the following server NOTES.INI parameters (available in Domino 7.0 and later)

• NSF_REPLHIST_NO_TRI=1
• REPL_NO_WS_TRI_HIST=1
• REPL_NO_REMOTE_TRI_HIST=1
• 
  

For local replicas, use Notes Client NOTES.INI parameters:

• NSF_REPLHIST_NO_TRI=1 [This will prevent existing triangulated entries from being read]
• REPL_NO_WS_TRI_HIST=1 [This will prevent new triangulated entries from being written]

Note: After setting the NOTES.INI parameters, the replication history must be purged from each replica of the databases affected.


Further information is available in IBM technote 1270104.

Internal Caches

In Domino, caches are used to store frequent lookups in memory, to prevent the data being continually read from disk. Since the mechanical hard-disk is commonly the slowest component of the server, caches help improve performance and make a more responsive user experience. The following sections provide Administrators with tips to ensure cache sizes are properly set.

NLCache

NLCache is the name lookup cache. The default is 16MB before Domino 8.5.2. Beginning in Domino 8.5.2, the default is 64MB. It can be increased as needed up to 4GB.

To determine if you need to increase the NLCache, use the show stat database or show NLcache console command. For example:
Database.NAMELookupCacheCacheSize = 16,447,205 (current size of cache)
Database.NAMELookupCacheLookups = 1,879,903
Database.NAMELookupCacheMaxSize = 16,777,216 (default maximum size)
Database.NAMELookupCacheMisses = 1,362,746

A relatively high number of misses compared to lookups indicate that you should increase the maximum allowed cache size. Administrators should increase the size of the NLCache in increments, so try doubling the default to begin with, if necessary. Modify using the ini parameter:

• NLCache_Size= For example, set 67108864, which sets NLCache_Size to 64MB.

Group Cache

When the server needs to lookup the members of a group (For example, in the event of an authentication request) it first checks the Group Cache. It will store results in the Group Cache to optimize performance. Groups are invalidated during updates or when cache is full. The default size is 4MB and it can be increased to 15MB.

If the cache needs to rebuild frequently because not enough data can be cached, this can slow down group lookups. Therefore, frequent or very large group updates can slow down server.

Verify GroupCache statistics with “Show stat net”
NET.GroupCache.Hits = 155
NET.GroupCache.Misses = 10
NET.GroupCache.NumEntries = 9
NET.GroupCache.Size = 65,406
NET.GroupCache.Used = 2,716

NET.GroupCache.Misses indicates the number of times a group was not found in the cache and so had to be read from disk. Administrators should increase the size of the Group Cache in increments until the number of misses reduces. Modify size with the NOTES.INI parameter:

• Group_Cache_Size= For example set 15360 , which sets Group_Cache_Size to 15MB.

Unified Buffer Manager (UBM)

The UBM (formerly referred to as the NSF Buffer Pool) is used to increase performance by caching disk I/O requests to all databases on a server. The UBM is typically the largest contribution of shared memory usage on the server. In earlier releases of Domino, the maximum size of the UMB was automatically calculated as 3/8th's of the physical RAM in a server. In a server with many gigabytes of RAM, this could result in an unacceptably high UMB upper-limit. Consider a server with 4GB RAM installed:
Max UMB Size = (4 x 3/8) = 1.5GB.
If the UMB were to grow to its maximum size of 1.5GB, there is a risk that shared memory may exceed process limits and result in a server instability.
Domino 8 now enforces a maximum UBM size of 512 MB default on Windows32, Linux, AIX, Solaris, i5/OS and z/OS platforms. On 64-bit Domino, the maximum UBM size is 1024 MB. Therefore, there is no need to take action to properly size the UBM in Domino 8. For additional details on this new behavior, see IBM technote #1268988.
For recommendations on setting NSF_BUFFER_POOL_SIZE_MB, read IBM technote 1286171.

NSF_DbCache_MaxEntries

The NSF_DbCache_MaxEntries determines the number of databases that a server can hold in its database cache at one time. If your server has sufficient memory, you can improve the performance of the server by increasing the number of databases that Lotus Domino can cache in memory at one time. The default value is 25 or the NSF_Buffer_Pool_Size divided by 300 KB, whichever value is greater.

You should monitor the Database.DbCache.Hits statistic on your server. This indicates the number of times a database open request was satisfied by finding the database in cache. A high value indicates database cache is working effectively. If the ratio of Database.DbCache.Hits to InitialDbOpen is low, you might consider increasing NSF_DbCache_Maxentries.

For detail information on how to set the number of databases cached simultaneously in Lotus Domino, read IBM technote 1279893.

NSF Monitor Pool

The Monitor Pool caches event monitors which include server and client mail rules. If set too low the Monitor Pool can become full. Side effects include causing new calendar notices not to display in the Calendar Mini View.

The default pool size is 40 MB. The maximum value for this pool is 256/512 MB.

To see the current values review stats
Database.MonitorPool.Event.Used = 52309
Database.MonitorPool.Monitors.Used = 1184
Database.MonitorPool.Size = 41943040

Check the LOG.NSF for the error “Insufficient memory – NSF monitor pool is full” to know if you should increase this pool for the Domino server. Many customers find a value between 80 and 100 MB to work best. Increase this value if you are approaching the maximum by using the NOTES.INI setting:

• NSF_MONITOR_POOL_SIZE_MB=

Multiple Mail Boxes

Additional mailboxes are required on a server only when the amount of mail traffic prevents the router from being able to effectively access the mail box files and access conflicts are encountered. Access conflicts occur when other threads or processes lock the mailbox, while another entry tries to access the mailbox and is denied.

The typical number of mail boxes on a mail server ranges from one to four, depending on the volume of mail. IBM technote 1148438 provides Administrators with the procedure to determine whether additional mailboxes are required on busy servers. If you feel you are in need of another mailbox. Change the server configuration to 2 mail boxes.

Tips for Server Based Mail Rules

Keep the number to the absolute minimum, based on business requirements. Rules require processing by the mail router each time it delivers mail. Therefore the fewer rules, the fewer server resources are required.
Place rules most likely to be triggered at the top of list and use the “Stop processing further mail rules” option.
Searching the memo body is CPU intensive and will cause and increase in the CPU used by the SMTP task

Use the following NOTES.INI parameter to cap the number of mail rules per user's mailfile or server mail.box:

• MailMaxFilters=<1 to 100>

This IBM Case study provides Administrators with an indication to the impact mail rules have on a server.

Tuning User Sessions

There are two NOTES.INI parameters that define how many client user sessions a server can have concurrently open and also how long a session remains open. Keeping sessions open requires memory on the server, whereas opening a new session consumes a small amount of CPU resource. Therefore a balance must be achieved.

• Server_MaxSessions=[number] (Maximum number of concurrent open sessions)
• Server_Session_Timeout=[number minutes] (maximum duration of idle activity before the Domino server terminates the session)

A timeout value of 30-45 minutes is recommended for most customers. For more information read IBM technote 1089879.

Domino Configuration Tuner (DCT)

The Domino Configuration Tuner (DCT) provides easy-to-use self-service configuration analysis for more robust installations and better performance of Domino servers. Make sure you are up-to-date with suggested changes from Lotus Domino by periodically updating DCT. For more information, read "Domino Configuration Tuner (DCT) provides easy-to-use self-service configuration" - IBM technote 4019358.

Optimizing server performance (Top 10 ways to improve your server performance sidebar)

Top 10 ways to improve your server performance (sidebar)
By analyzing a variety of NotesBench reports, published over the last two years by NotesBench Consortium members, we came up with a list of the top 10 ways you can improve the performance of your server. The list shows you how to improve your server capacity and response time.

1. Make sure your server memory matches the number of users you want to support. Most NotesBench vendors use 300K-400K per active user. They also set their NSF_BUFFER_POOL_SIZE to the maximum for their memory configuration. This setting isn't necessary, because the Domino server initially obtains a quarter of available memory and grows only if necessary (depending on the load). You should use published physical memory configurations as a ceiling for memory configuration decisions.
2. Distribute I/O among separate devices. For example, you can put the OS kernel on one drive, the page file on another, the Domino executable on a third, and finally the Domino data files on a fourth drive. In some cases, NotesBench vendors point their log.nsf file to a location different from the default data directory (using the log= setting in the server's NOTES.INI file).
3. I/O subsystem improvements. For example you can:
   • Move from EISA-based systems (such as, controllers) to PCI-based systems
   • Exchange EISA/PCI boards in favor of PCI-only boards (this way, lower speed EISA devices won't decrease the I/O throughput)
   • Use striping to improve performance
   • Use multiple I/O controllers to distribute logical volumes (and use file pointers to databases across separate controllers). Make sure you have the latest BIOS for your I/O subsystem. This is an inexpensive way to remove a likely throughput bottleneck.
4. Use faster disk drives. You can improve disk drive speeds from 5,400 rpm to 7,200 rpm. For most Windows NT systems, NotesBench vendors use 2GB disk drives. For Solaris and IBM Netfinity systems, the drives were larger: 4GB. For AS/400, the drives were even larger: 8GB.
5. Increase the stripe size. NotesBench vendors use a stripe size of 8K (Digital's systems) or 16K (IBM Netfinity reports). (The IBM Netfinity report provides additional information on I/O settings such as w IOQ Depth, Outbound Posting, PCI Line Prefetch, and Address Bit Permitting.)
6. Use faster CPUs. NotesBench vendors have moved beyond the Pentium, Sparc, and PowerPC processors, which were in the 100-200Mhz range, to higher speed processors. However, they consistently use P6-based systems over the Pentium II systems for high-end Domino server loads. The size of your Level 2 cache should match your expected user loads and the response time you want. Vendors have moved from 256K to 512K, 1MB to 2MB Level 2 cache systems, especially on their greater than two-CPU configurations.
7. Improve your network. NotesBench vendors have:
   • Moved from 10Mbps cards and networks to 100Mbps configurations
   • Used multiple LAN segments (one for each partition) to isolate network traffic, at the high-end user loads
8. Change your network protocol to IP. Vendors were initially (two years ago) using NetBIOS and SPX internally, but have unanimously moved to IP for their performance publishing efforts.
9. Upgrade to a newer release of Domino. NotesBench vendors have moved from Domino Release 4.5a SMP version to Domino Release 4.52B SMP version for higher capacity results. The first Domino Release 4.6a result (AS/400) on a RAID5 configuration indicates a reliable configuration can still provide competitive response time with a properly designed I/O architecture.
10. Use Domino partitioned servers. NotesBench vendors have increased scaling of active user loads and leveraged their more powerful configurations (faster clock cycles, fiber-connected I/O subsystems, OS kernel to CPU binding, and multiple I/O controllers) by using partitioned servers.

How we came up with these recommendations
To understand how we came up with our top 10 list, we will take you through the performance analysis of Number 2 in the list -- to distribute I/O among separate devices. Initially, many vendors placed the kernel, page, and Domino executables on one volume and the Domino data files on another. However, both volumes were on the same controller. Lately, the NotesBench reports show improvements in performance when the volumes are separated across multiple controllers, and individual volumes are separated across disks. What this means is that we found that vendors put the OS kernel on one drive, page file on another, Domino executable on a third, and finally the Domino data files on a fourth drive. In some cases, they pointed their log.nsf file to a location different from the default data directory (using the log= setting in the server's NOTES.INI file). Vendors who distributed the I/O over several disk drives had better server performance overall, and could support a higher capacity of users. For example, in a NotesBench report published in May of 1996, Digital Equipment Corporation set up a server with the following specifications:

• CPUs: four 133Mhz CPUs
• Memory: 512MB
• Domino: Release 4.1

They placed the operating system and the Domino executable on drive C:\, the page file on drive D:\, and the Notes\data directory on drive E:\. They could support a maximum capacity of 1,500 users with this configuration.
In a NotesBench report published in September of 1997, IBM Corporation set up a server with the following specifications:

• CPUs: three 200MHz1Intel Pentium Pro processors
• Memory: 1GB2
• Domino: Release 4.51

They placed the operating system on drive C:\, the page file on drive C:\, the Notes\data directory on drive E:\, and the Domino executable on drive E:\. They supported a Mail-only workload of 3,500 active mail users. In a four-processor configuration, they supported a MailDB workload of 2,900 active users. These examples led us to the conclusion that distributing I/O over several disk drives had better server performance overall, and could support a higher capacity of users. We went through many other NotesBench reports to collect the data shown in our top 10 list. You can visit the NotesBench Web site yourself to view published data and test results. Visiting the site may help you to come up with other ways to improve your server's performance.

ibm.com