DAOS Backup and Restore

Table of Contents

• 1 Backup considerations for DAOS
• 2 Order matters
• 3 DAOS index files
• 4 Transaction logging
• 5 Command examples
• 6 DAOS enable and disable considerations
• 7 Space and time savings
• 8 Restoring DAOS objects
• 9 Restoring documents or NSF files with DAOS attachments
• 10 Dealing with damaged files and clusters
• 11 Options for restoration
• 12 Offline archival

This article describes how to refine your backup and restore procedures when using the popular Domino Attachment and Object Service (DAOS) feature, also known as "attachment consolidation."

Backup considerations for DAOS

In a standard Notes® database (NSF), the attachments are stored inside of the NSF file itself, and the database is self-contained. In order to back up a standard Notes database, only the NSF file itself needs to be backed up. After you introduce DAOS, the NSFs that participate in DAOS contain only references to the NLO files where the attachment content is stored. As a result, backing up the NSF alone is no longer enough. The NLO data needs to be backed up as well.

When performing standard NSF backups, there are two main approaches. The individual NSF being backed up can be taken offline (or the entire server can be stopped) so that no changes occur to the NSF over the duration of the backup operation. The other method allows the NSF and server to remain active, but requires using a backup utility program that interacts with the Domino® backup/restore API. Using a utility ensures that a consistent copy of the NSF contents is recorded, despite any changes that occur to the NSF over the duration of the backup operation. If you are using a backup utility that does not use the backup/restore API, you must stop the Domino server and all Domino-related applications for the duration of the backup.

None of the processing for NSF backups needs to change for DAOS. The change needed to accommodate DAOS is simply a procedural addition: in addition to backing up the NSF data, you must also back up the NLO data.

Backing up the NLO files in the DAOS repository can be done either while the Domino server is down, or when it is up and running. The backup does not require the use of any Domino API-based utilities. Once NLO files are written initially, Domino never modifies their contents, so the backup mechanism does not have to work around file-write activity. NLO files can be backed up as any other generic file on the file system. Only the NLO files that are complete and not in the process of being written to or renamed need to be backed up. Any that are busy can be skipped until the next backup. Most backup applications will automatically skip files that they can not read because of other activity.

Order matters

If you shut down the Domino server during the backup process, the NSF and NLO files can be backed up in any order. If you must keep the Domino server up and running during the backup process, it is important to back up all the NSF data before backing up the NLO files. The reason has to do with the addition of references to new NLO files in an active system, described in this section.

When you back up an NSF that participates in DAOS, there are some number of NLO references contained in that NSF at the time of the backup. Since there is some duration to the backup operation for all NSFs, the number of references to NLO files may be increasing over that duration in a system that is operating during the backup process. If there were (for example) 10,000 NLO files referenced collectively by all the NSFs at the beginning of the NSF backup process, there could be 10,100 by the time the last NSF is backed up.

Likewise, the backup of the NLO data has a duration as well, so while there might have been 10,100 NLO files at the beginning of the NLO backup process, there could be 10,200 by the time the last NLO is backed up.

In this scenario, the backed up version of the NSFs could reference at most only 10,100 NLO files. Because the NLO backup was done after the NSF backup process, the NLO backup included at least that many, but may have as many as 10,200 NLO files. Worst case, there are more NLO files backed up than strictly necessary to satisfy the NSF references. Since all accesses to the NLO files are done through the NSFs, and the NSFs were done first, all of the referenced NLO files are guaranteed to exist in the set of NLO files that were backed up. If there is an error accessing an NLO file in order to back it up because it's in use, that can safely be ignored. If the file is being written, the activity must have occurred after the NSF was backed up; therefore, this NLO file does not need to be in the corresponding set of NLO files, and will be backed up as part of the next cycle.

The deferred deletion interval should be set to a period longer than your chosen backup cycle. In this way, the NLOs are not pruned (physically deleted) prior to the next backup. Instead, the actual deletion is deferred until they've aged accordingly.

If you were to have a shorter or nonexistent deletion interval—the feature can be disabled by setting it to zero in the DAOS tab of the server document—it opens a window of time during which a deleted attachment is non-recoverable, as the NLO file has been physically deleted before the backup has occurred.
Avoid pruning NLO files from the repository (by issuing a prune command at the Domino console) before they have had a chance to be backed up; you will prevent them from being recoverable. When an attachment is deleted, and the associated NLO file's reference count goes to zero, it becomes a candidate for deletion. The deferred deletion interval determines when the deletion actually occurs. If the deferred deletion interval is set (as recommended) to be longer than the backup cycle, all NLOs will be in existence for at least one backup cycle, and therefore any NLO can be recovered later.

After the initial full backup of the NLO files in the DAOS repository, you can perform incremental backups, which save only the data that has changed since the last backup. NLO files are ideal candidates for incremental backup because there are no changes to them after their initial creation.

One NLO file is created for each unique attachment, so it is possible to have a very large number of NLO files in large deployments. The maximum number of files per numbered DAOS subdirectory is 40,000, and there can be 1000 subdirectories, for a maximum total of 40 million NLO files. Check with your backup utility specifications to see if there is a limit on the total number of files it will manage, and monitor the growth of the DAOS directory file population accordingly.

DAOS index files

The daos.cfg and daoscat.nsf files should not be backed up. (Note that this is a change from earlier recommendations) These two index files can be re-created from the DAOS repository and the NSFs participating in DAOS if necessary. If these files become corrupted, they can be safely deleted while Domino is not running. They will be created on startup automatically.

The daos.cfg file helps manage the files in the DAOS repository. The NLO files are stored in subdirectories (0001, 0002, and so forth) underneath the base DAOS directory. For several reasons (including performance), DAOS limits the number of NLO files in each subdirectory. The daos.cfg file keeps track of how many files are currently in each subdirectory so that DAOS puts new files in subdirectories where the count of files is below that limit. As NLOs are deleted, the corresponding file count is decremented, allowing backfilling of older subdirectories. The daos.cfg file is expendable, and will be re-created at Domino startup time if it is missing.

The daoscat.nsf file contains two indexes. One is a list of all NSFs that are holding NLO references (DAOS ID Table, or DIT.) The second is a list of all NLOs that exist, and the DAOS repository subdirectory they exist in (DAOS Object Index, or DOI). There are no externally visible parts to this NSF, and there are no privileges that apply to change that. The DIT is modified when an NSF acquires its first NLO reference. The DOI is modified when a new (unique) NLO is created. The daoscat.nsf file is expendable, and will be re-created at Domino startup time if it is missing. Since a full resync can take a significant amount of time, only empty indexes are created by this process at startup. A resync operation should be done as soon as it is convenient, however.

In some cases it could be necessary to fully reboot the server until the daoscat.nsf and the daos.cfg are re-created.

A DAOS resync operation (“tell daosmgr resync”) fully re-populates these two indexes from scratch. You can also run the command “[n]daosmgr resync”if you want to perform a resync operation with the Domino server shut down.

Transaction logging

Because all NSFs that participate in DAOS have to also participate in transaction logging, the contents of all their attachments will be included in the log. Any NLO files that are created as a result of activity to the NSF will be re-created if the log is replayed.

Command examples

Using the Tivoli Storage Manager (TSM), the command to back up the DAOS repository would be:

dsmc incremental c:\lotus\domino\data\daos

where the path specified is the full one to the DAOS repository.

Since the NLO files are being backed up incrementally, the initial backup will be quite large, but subsequent ones will be much smaller. The total footprint of the DAOS directory will be written out during the first backup.

DAOS enable and disable considerations

Once a Domino server has DAOS enabled, and NSFs are selected to participate in DAOS, their attachments are stored in the DAOS repository. If DAOS is subsequently disabled, the attachments that were in DAOS remain in DAOS until they are re-integrated into the NSF. Any DAOS references that remain in the NSF will continue to be serviced by DAOS, even if it is disabled. An NSF that contains DAOS references is not self-contained, and must continue to be treated as an active DAOS participant as long as it has DAOS references. To re-integrate the DAOS attachments into an NSF and remove the DAOS references, you can process the NSF with the “compact -c -daos off” command. Once that is done, the NSF will be self-contained again, and can be treated as a normal NSF.

Furthermore, to ensure that the DAOS enablement change takes effect completely, the Domino server as well as all processes that use the Domino API (compact, resync, backup, etc) are stopped. This allows the API to terminate completely, so the status change can be picked up at the next startup.

Space and time savings

The disk footprint savings with DAOS continues into the backup processing as well. The NLO files represent the static data that used to reside in the NSF, and was backed up every cycle even though it hadn't changed. In a typical mail environment, a large reduction in the NSF footprint plus a very small amount of NLO data means the reduction in the NSF footprint translates almost directly into a reduction in the backup footprint. Not only is the duplicate data being eliminated, the mailfile data is being separated into static and dynamic components. By applying an incremental backup regimen to the static (NLO) data, only the NLO files created since the last backup cycle need to be processed. That represents typically a very small amount of data compared with the entire set of NLO files.

Using DAOS enables backup software solutions to optimally backup the NLO files. This is because once an NLO is written to disk, it never changes. Therefore, the file need only be backed up once in its lifetime. Based on the example shown in the DAOS Estimator document [link], the space saving per full backup would be 38.8 GB, roughly equal to the number of shared NLO's times the average NLO size.

In the incremental backup case, duplicate NLO's will not be backup up again. Thus, the space savings from DAOS is directly proportional to the numbe r of duplicate NLO's seen in the environment, and the backup time savings is the product of the space saved and the backup throughput.

Restoring DAOS objects

A well-preserved DAOS repository makes for fast and easy restoration. And, while not really a backup mechanism, the default deferred deletion interval allows for accidentally deleted attachments to be saved from physical deletion up to 30 days after a mishap. Simply pull the document out of the trash folder, if soft deletion is enabled. If it's too late for that, restore from backup, then resynchronize the DAOS catalog using the server console command "tell daosmgr resync force" -- DAOS will once again recognize that the NLO has references.

Although an NLO will survive for the period of time specified by the deferred deletion interval, if soft deletion is disabled or a backup of the referencing document has not been made, there is no way to get at the contents of the NLO, especially if encryption is enabled (the default).

Restoring documents or NSF files with DAOS attachments

To restore either a full NSF or a single document, the process starts off the same. You must first restore the database and then the missing NLOs. To do this using Tivoli Data Protection for Domino, you would issue the command:

domdsmc restore -into

To determine which missing NLO files to bring back from the Domino server console, run

tell daosmgr listnlo -o missing.txt MISSING restoreddatabasename.nsf

The resulting missing.txt file is then fed into the restore command With the Tivoli Storage Manager (TSM), the command would be

dsmc restore -filelist=missing.txt -inact 

If you are restoring the entire NSF, you are done. Note that any restoration operation will put the DAOS catalog into the Needs Resync state, so a resync operation should be performed as soon as convenient.

If you need only one document, you can now copy and paste it to its intended destination.
For a complete recovery after a catastrophic failure, the NSF and NLO files can be restored, followed by replaying the archived transaction logs. This will result in the most up-to-date recovery situation.

Dealing with damaged files and clusters

If an NSF is damaged, and you have clustered servers or replicas of the NSF on another server, you have several options.

1. Replicate each entire NSF – New replicas can be created from the existing NSFs on the clustermate(s). Each new replica should be marked as DAOS enabled. As the replication occurs, the associated attachments will be saved to DAOS.
2. Copy NSFs, replicate missing attachments – All of the necessary NSFs are copied from the clustermate(s) to the server being repaired. This will create a copy of the document data without attachment data. Fixup -j -D is then run, deleting all documents that contain DAOS references to NLO files that do not exist. Subsequent replication will re-create those documents along with the associated attachments, which will be stored in DAOS.
3. Copy NSFs, copy/restore NLOs - All of the necessary NSFs are copied from the clustermate(s) to the server being restored. The command 'tell daosmgr listnlo missing somefile.nsf' is then issued for each individual NSF to generate a list of the NLO files that do not exist in the DAOS repository. Those NLO files are then restored from backup, or copied from the clustermate(s). (Note that copying the NLO files from another Domino server will work only if DAOS encryption is turned off. DAOS encryption is on by default, and uses the server key to do the encryption; therefore enc rypted NLO files are not portable between Domino servers.)
4. Copy full NSFs and re-extract – If you have a replica on another server that is not DAOS-enabled, the NSF can be copied to the server being restored. The attachments will be inline in that copy of the NSF, and 'compact -c -daos on' can be issued to extract the inline attachments out to DAOS.
5. Reintegrate NSFs and re-extract – If you have a replica on another server that is DAOS-enabled, but encryption prevents using the NLO files directly, you can run a 'compact -c -daos off' on the other server to re-integrate the attachments into the NSF. Once that is done, the NSF can be copied to the server being restored, and you can use 'compact -c -daos on' to extract the attachments to DAOS again.

Options for restoration

The need for restoring NLO files depends partly on the deferred deletion interval. If the restore is happening from a snapshot that's within the interval (for example, the interval is 30 days, and the NSF is being restored from last week's backup) it's not possible for any of the NLOs to have been deleted yet, so there shouldn't be a need to restore any NLOs. If the NSF being restored is older than the interval (for example, the interval is 30 days, but the NSF being restored is from 3 months ago), it's possible that some of the NLOs have been deleted, and would need to be restored.

Some of this also depends on what the reason for the restore is. If it's a catastrophic failure, you should restore the NSF(s) and run 'tell daosmgr listnlo missing filename.nsf' to get a list of all of the NLOs needed to make it whole again. That list should then be fed into the restore utility to restore those NLOs as well.

If it's just a matter of getting a single document with attachments back, you don't really need everything to be made whole to access just that one set of attachments. In the spirit of “Work smarter not harder,” you can restore the NSF, and then attempt to access the desired document (and attachments, if any) and finally deal with any missing NLOs that are mentioned during that operation. If there aren't any attachments on the document, there's no other work to be done. If there are attachments, the NLOs may still be there, so it's worth trying to access them before doing anything else. If any are missing, you'll get a console message that mentions the name of the NLO, at which point you can restore only what you need.

Offline archival

For offline archival purposes, it is recommended that the attachments be re-integrated into the NSF using a 'compact -c -daos off' operation prior to archiving. That eliminates the need to archive all the individual NLO files referred to by the NSF also. For example, if an employee was leaving the company, and their mail account was being closed and archived, this approach would be appropriate.

lotus.com

DAOS Best Practices

When it comes to configuring the Domino Attachment and Object Service (DAOS), you may be asking yourself - and us in turn - what's the right way to set it up? For example, is there an optimum “Minimum Size” setting? Should the repository go under the data directory or on its own drive? What's the best “Deferred Deletion Interval” in relation to my backup and restore schedule? This guide, and the documents it references, attempts to answer these questions of individual, site-specific configuration in general terms with guidelines for adapting and modifying them based on measurements made against your particular environment.

Where To Locate Your DAOS Base Path Repository

By default the DAOS repository resides under the server's data directory and defines a single container as indicated by the “DAOS Base Path” setting in the DAOS tab of the server document. So, on Windows for example, if you use the default, “DAOS”, and your data directory is C:\Lotus\Domino\Data, the full path to the repository would be C:\Lotus\Domino\Data\DAOS. For Domino 8.5, only this one container may be specified.

However, this default location, chosen simply for being well-known, may not be the most efficacious. Some things to consider:

1. What is the total capacity of all file attachments? With only one container in Domino 8.5, flexibility may be important when choosing the best DAOS base path. You'll want to be sure you have significant storage capacity or the ability to reconfigure a logical drive as space needs increase. Use the Domino Attachment and Object Service Estimator to plan for your storage requirements. 2. What I/O costs do I expect to incur? DAOS base path I/O is significantly less than that of Domino's Data directory. In benchmark tests, DAOS repository I/O was 94% less than that of the server's Data directory. Lower performing storage (a NAS device, for example) can be used here. 3. Can I use lower cost or external storage devices? In many cases, you might find attachments are infrequently accessed -- for example, when they're part of old email messages collecting the proverbial dust in one's inbox. In these environments, locating DAOS on lower cost storage (tier 3) devices may be indicated. On the other hand, if full text indexing, agents, or other applications make heavy use of the consolidated attachments, “lower cost” storage may cost you in performance. Note: Externalizing the DAOS repository in this manner does not mean you can map multiple Domino servers to the same container. This is an unsupported configuration as of this publication and could very well lead to data loss due to encryption with the server's key. NLO files cannot be shared across Domino servers. Note: Modifying the location of the DAOS repository at a later time is allowed and requires that you first change the “DAOS Base Path” field on the DAOS tab in the server document, stop the Domino server and then relocate the existing subdirectory structure with its NLO files to the new location. On server restart, the modification will take place seamlessly. 4. Why is it recommend to not locate the DAOS Base Path under Domino's Data directory? Many of Domino's tasks, including fixup, compact and the admin client, scan Domino's Data directory and will also scan the NLO files if the files are located under that sub-directory. The scanning of NLO files for tasks other than DAOSMGR will add additional unneeded overhead to the processing done by Domino's tasks.

Optimum Minimum Size For Participation

By default, the minimum size setting for an attachment to make use of DAOS is 4096 bytes. While we recommend using 64000 as the lowest value you should use here (1048576 on iSeries), there are a number of things to consider when determining the best DAOS minimum size setting for your system.

1. Do not set the minimum size lower than the default setting. Due to attachment file overhead, setting the minimum size to anything lower than the default size would actually be less efficient than storing the attachment in the NSF file. 2. Set a minimum size that is a multiple of your file system's disk block size. By choosing a minimum size that is a multiple of the disk block size, you optimize disk usage. To ascertain the disk block size for your file system, on a Windows NTFS, use “fsutil fsinfo ntfsinfo ” and take note of the “Bytes Per Cluster”. This is the disk block size. On Solaris, you could use df -g and take note of the block size. On AIX you need be super user to do determine block size, then use lsfs -q and look for block size. On Linux you also need to be super user to find the block size, then use df -k to determine the device name of your filesystem and the uses dumpe2fs | grep 'Block size' to determine the block size. 3. Take note of possible limitations on number of files. The smaller you make the setting, the more attachments will qualify for DAOS consolidation. The larger you make the setting, the fewer will qualify. In Domino 8.5, the DAOS repository allows for one container with up to 1,000 subcontainers, each with a maximum of 40,000 NLO files. Thus the storage capacity of DAOS is limited to 40 million distinct objects. This is a significant number of files, so if you expect to come anywhere close to approaching it, you should check the limits on your backup and restore solution, as some applications and file systems have limitations on maximum number of files. Refer to your operating system and/or backup application guidelines. 4. The current recommendation for the lowest value you should use on IBM iSeries is 1048576 (1M) to avoid overwhelming the filesystem and backup utilities.

To get an idea of how many files various settings would generate, you can run them through the Domino Attachment and Object Service Estimator.

The ultimate goal with this setting is to minimize the number of files in your DAOS repository and maximize the amount of disk space saved.

Deferred Deletion Interval

DAOS automatically deletes NLO files that are no longer being referenced by any databases. This deletion of NLO files is known as “pruning” and occurs at the specified “Deferred Deletion Interval.”

Establishing a useful “Deferred Deletion Interval” for your server involves a few considerations, primary among them your backup and restore schedule. You want to ensure that NLO files which are no longer needed remain in existence at least as long as your backup cycle. In this way, they will not be deleted before the next backup.

A secondary consideration is the size of attachments typically stored in the repository for your server. If they are usually quite large, you may want to have them cleaned up as quickly as possible after there are no longer any references to them.

If your deferred deletion interval is set too high, NLO files which are no longer needed will continue to take up valuable space in the repository. If your deferred deletion interval is set too low, you could be deleting NLO files that have not yet been backed up, thus making it difficult, and in some cases not even feasible, to restore them. It is important to find a balance that satisfies both your backup schedule and the system integrity of a neatly pruned environment.

Pruning

Pruning can also be manually triggered to override the automatic deferred deletion interval. The administrator can issue the console command “tell daosmgr prune x” to forcibly delete unreferenced NLOs that are x days old. This will recover the disk space still being used by unreferenced NLO files immediately rather than waiting for the automatic deferred deletion interval to do so. When performing this action, you must consider your backup cycles. As with setting the deferred deletion interval too low, pruning too soon could delete NLO files that have not yet been backed up.

When A Notes Database Should Use DAOS

There are several good reasons to select an NSF file for participation in DAOS consolidation:

• It contains or is likely to have multiple copies of the same attachments.
  Even a single NSF can benefit from DAOS consolidation.
• It resides on a server where the same attachments appear across multiple NSF files.
  If others are also referencing attachments present in your database, why not share?
• It contains very large attachments.
  In this case, it may not matter how many other NSF files hold the attachments in question. If they're large enough, the simple step of storing them outside the NSF can make common operations against that database much faster.

While DAOS can always benefit your data, DAOS has less benefit under the follow conditions:

• Databases have lots of small attachments.
  Attachment consolidation is less efficient in this scenario due to disk blocking. You can, however, eliminate this issue by adjusting the minimum size setting upward.
• There is little or no attachment duplication across databases.
  Backup would still benefit due to extracting static data, but you would have little disk space reduction.
• Databases contain few or no attachments.
  In 8.5, DAOS stores only file attachment data.
• Databases need to be quickly portable.
  Because DAOS object files (NLOs) cannot be shared across Domino servers, it is more difficult to move DAOS-enabled Notes databases from server to server.

It's not necessary for all databases on a server to leverage DAOS, but for those that do, the savings in both space and time, as for example, in much accelerated compact operations, are significant.

Mailbox

Although DAOS will work in any configuration, it operates most efficiently when it is enabled on both the mail.box files and individual mail files. Enabling transaction logging and DAOS on mail.box will enable the Router to optimize the delivery of DAOS based attachments. This can result in significant I/O savings for the case where the same attachment is sent to multiple recipients on the same Domino server.

I/O Activity for mail delivery of email with an attachment

DAOS Enabled		Document Writes	Attachment Read(s)	Attachment Write(s)	Comments
MAIL.BOX	Mail files
No	No	1 + N	1 + N	1 + N
Yes	No	1 + N	1 + N	1 + N
No	Yes	1 + N	1 + N	1 + N
Yes	Yes	1 + N	1	1	Maximum reduction for both I/O & CPU

* N is the number of recipients

Note: If you have multiple mail.box files, you must enable transaction logging and DAOS on all of them to leverage DAOS object copy optimization, which streamlines delivery of attachments.

Since Domino creates new mailboxes as needed, you should also set these properties on the mail.box template. If you choose not to enable DAOS or transaction logging on the mail box, DAOS will still be used by any DAOS-enabled mail files. Using DAOS on mail.box(es) only affects the optimized routing (delivery) of attachments.

When an incoming document is received at mail.box, it is stored there until it is delivered to the individual mail file(s) of the recipient(s). Several results are possible:

• If DAOS is not enabled anywhere, the document will be stored in mail.box, and the attachment will be stored inline. As the document is delivered to the recipient mail file(s), the document and the contents of the attachment are read from mail.box, and the document is written to the mail file with the attachment inline. Total I/O cost to deliver to N recipients: 1 + N doc writes, 1 + N attachment reads, 1 + N attachment writes.
• If DAOS is enabled on both mail.box and destination mail file(s), any attachments in that document will be extracted and converted to NLO files as it is being written to mail.box. The document and DAOS ticket are written to the destination mail file(s). IMPORTANT: In the case where both mail.box and the mail file are DAOS-enabled, the contents of the attachment will not be written again as the document is delivered, only a reference to the existing NLO file will be copied. Total I/O cost to deliver to N recipients: 1 + N doc writes, 1 attachment read, 1 attachment write.
  
  If DAOS is enabled on mail.box but not on the destination mail file(s), any attachments in that document will be extracted and converted to NLO files as it is being written to mail.box. Since the destination mail file(s) is/are not using DAOS, the attachment must be stored inline, and the contents of the attachment will be read out of mail.box (which has it stored in DAOS) in order to do that. Total I/O cost to deliver to N recipients: 1 + N doc writes, 1+N attachment read, 1+N attachment writes.
• If DAOS is not enabled on mail.box, but is for the destination mail file(s), the attachment will be stored inline in mail.box. As the document is delivered to the recipient mail file(s), the contents of the attachment are read out of the mail.box document, and a temporary NLO file is created for each destination mail file so that a checksum can be calculated. If an NLO file with the same checksum already exists, the temporary file is deleted. In the case of N recipients, this process will be repeated N times, even though only one NLO file will remain at the end of the process. Total I/O cost to deliver to N recipients: 1 + N doc writes, 1+N attachment read, 1+N attachment writes. IMPORTANT: In this case, although the end result (a single NLO file per unique attachment) is the same, the I/O cost is significantly increased over the case where mail.box is enabled for DAOS.

Mail Journaling

For 8.5, it is recommended that you not enable DAOS on the mail journal (mailjrn.nsf).

Encryption

By default, DAOS employs encryption to safeguard its repository. This setting is separate from encryption settings that apply to an NSF or document. The encryption is done with the server key, so the resulting NLO files can be read only on a server that uses that same key. This may be a consideration for backup or redundant server setup.

The performance hit for DAOS encryption is negligible; testing showed a 5% CPU increase with no change in I/O versus unencrypted data. However, if your organization has reason to disable it, we've provided the server notes.ini setting DAOS_ENCRYPT_NLO, which can be set to zero to affect that change. To determine the current status of encryption, use the ”sh stat daos” command from the server console.

There are some storage area network devices that have the ability to deduplicate files. If DAOS_ENCRYPT_NLO=0 is not specified, deduplicating will not occur across Domino servers. The encryption will make even duplicate files unqualified for deduplicating because they will encrypted with each server's key.

Compression

While DAOS is compatible with compression, there are a couple of points to remember:

1. It's possible for an attachment to disqualify itself from DAOS consolidation by compressing to a size smaller than the Minimum Size setting. 2. The same attachment, undergoing different compression types, LZ1 versus Huffman versus no compression, will be seen by DAOS as different objects and will, therefore, be shared only with others of like type.

Resynchronization of the Catalog

In order to ensure that NLO files are not physically deleted when there are still ticket holders referencing them, if there is any reason to question the accuracy of a reference count, DAOS puts itself in a safe mode whereby no deletes are allowed to proceed. This state is signaled to the administrator via the Domino Domain Monitoring systems and is reported as a “NEEDS RESYNC” state from the “tell daosmgr status catalog” server console command.

To perform this catalog resynchronization manually, type “tell daosmgr resync” from the server console.

Note: The duration of a resync can be significant and depends on the number of DAOS-enabled databases, the number of NLO files in your environment, and your system configuration. It can take several hours to complete causing its execution to overlap into normal business hours. Although Domino and DAOS are functional while a resync is in progress, there may be a degradation in performance while it is running.

Prior to 8.5.2, if you find it necessary to halt the resync operation, you can interrupt it by issuing “tell daosmgr quit” from the server's console. However, for continued operation of DAOS, immediately restart the DAOSMGR task using the console command “load daosmgr.” When it is convenient to continue the resync operation, issue “tell daosmgr resync” again from the server's console and resync processing will continue where it left off.

8.5.2 introduces many improvements to the performance of the resync operation. Resync will complete sooner in this release and improvements to keep the catalog in sync have been made. Resync can now be scheduled to work with in a time window by setting DAOS_RESYNC_START_TIME and DAOS_RESYNC_STOP_TIME. Setting a time window will eliminate the need to stop and start daosmgr when resyncing the catalog.

Typically, the only downside to being in "Needs Resync" state is that DAOS suspends pruning operations. Unless you are experiencing errors, it is usually better to wait to do a resync until the next available maintenance window.

Antivirus

Consider DAOS interaction with antivirus scans. It's critical the DAOS base path and .NLO file extension have the same anti-virus policy as the Domino data directory and .NSF file extension. If the two file types have different policies, consolidating attachments from an existing NSF using “compact -daos on -c” can result in NLO files being quarantined as they're extracted from the NSF. A user who then opens a quarantined attachment would see a “Missing NLO” error message.

A Domino add-in antivirus program is recommended so that the attachments are scanned as they pass through the server. You should not scan the NLO files directly. (Similarly, the transaction log files should be excluded from scans.)

The screenshot below shows how you'd configure Symantec Antivirus to exclude the .NLO file extension. From the Configure folder on the left, select File System Auto-Protect. Click the Exclusions button, then the Extensions button. Type “NLO” without the quotes and click Add.

Recommended versions of Notes Client and Domino Server

For the Notes Client anything before 8.5.1 or post 8.5.1 FP3 (including this version) should be used. For the Domino Server, the minimum version should be 8.5.1 FP3. For more information see the Technote 1446397 "Attachment corruption related to DAOS and Domino 8.5.1 ".

Ideally the Domino Server should be at 8.5.2 to gain both operational and performance improvements made to DAOS catalog synchronization as well as other DAOS improvements.

Worse practices

1. Too small a minimum participation size - getting too "greedy"
Limit the participation size to what will resulting in reasonable disk space amount with the least number of nlo files. Let's review the following output from the "Domino Attachment and Object Service Estimator":

DAOS Minimum Size versus number of NLO's and Disk Space:

0.0 KB will result in 2226347 .nlo files using 185.5 GB
64.0 KB will result in 1092894 .nlo files using 175.7 GB
128.0 KB will result in 708403 .nlo files using 163.6 GB
256.0 KB will result in 422087 .nlo files using 145.9 GB
512.0 KB will result in 219833 .nlo files using 120.2 GB
1.0 MB will result in 93628 .nlo files using 87.8 GB
2.0 MB will result in 36576 .nlo files using 56.6 GB
3.0 MB will result in 17499 .nlo files using 38.0 GB
4.0 MB will result in 9717 .nlo files using 26.3 GB
8.0 MB will result in 1576 .nlo files using 6.5 GB

While the theoretical maximum (first line) would generate approximately 2.2M files using 185 GB this would not be the ideal participation size for two important reasons.

The first reason is the benefit of DAOS can be realized without having to maintain 2.2M NLO files. A more reasonable participation size between 128k-256K would result in the number of NLO files in the range of 422,087 to 708,403. Taking an average of the two participation sizes, a value of 192K would result in approximately 500,000 NLO files with a disk space size of about 150 GB. The net result would be an 80 % total size yield with a little less than 1/4 of the theoretical maximum NLO files to maintain in your backup/restore procedures.

The second reason is that reducing the minimum participation size is more easily done then increasing the minimum participation size.

2. Deleting the daoscat.nsf and/or daos.cfg to "fix" problems.
The daoscat.nsf and the daos.cfg are vital files for DAOS. daoscat.nsf keeps data about the location of NLO files with reference counts. daos.cfg maintains data about the DAOS configuration. If one or both of these files are deleted, DAOS will rebuild the files; a processes that will take hours to complete. While the files are being recreated the location of individual NLO files and the reference count will be in transition which would result in intermittent access to attachments. Removal of corrupted daoscat.nsf should only be done when all other options for getting the file back have failed.

3. Long deferred deletion intervals set on a hub server.
When mail routing or replication hub servers are DAOS enabled, NLO files will be created and marked for deletion. Until Prune is run as scheduled by the "Deferred Deletion Interval", the files will exist in the DAOS Repository.

4. Failing to monitor the DAOS catalog state.
DDM events are generated for DAOS state changes including the state of the DAOS catalog. It is important to keep the DAOS catalog in a synchronized state to make sure prune can run and to keep access to the DAOS manager functioning correctly. Make sure to monitor the state of the DAOS catalog either with via DDM events or directly via the server console with "show daosmgr status catalog" and make sure the catalogState reports "SYNCHRONIZED".

5. No Backup/Restore procedures.
It is worth repeating the phrase timing is everything regarding backup and restore. It is important to time the backup of nsf and nlo files to be more frequent than the "Deferred Deletion Interval" because prune will remove the NLO files. Another important point is to not only have a well planned Backup/Restore procedure implemented, but to validate the restore procedure regularly to ensure that data can be restored before there is an emergency. For more information refer to the DAOS Backup and Restore

Transaction logging and DAOS

System Database File	Should DAOS be enabled on this database?	Should Archive transaction logging be enabled on this Database?	Should Circular transaction logging be enabled on this Database?	Comments
NAMES.NSF	No	Yes	Yes
LOG.NSF	No	No	Yes	If this is not transaction logged consider deleting it after a crash as it could impact startup time. It may require a fixup and this could take an extremely long time if log.nsf is very large.
ADMIN4.NSF	No	Yes	Yes
MAIL.BOX	Yes	Yes	Yes	If you want to use DAOS and not transaction log mailbox bodies you can use the following in 8.51. NSF_DONT_LOG_MAILBOX_BODY=1 RM_NO_LOG_LARGE_OBJECTS=1 RM_NO_LOG_OBJECTS_IN_MAILBOX=1
DBDIRMAN.NSF	No	No	No
CLUBUSY.NSF	No	No	Yes
DDM.NSF	No	No	Yes
STATREP.NSF	No	No	Yes
STATMAIL.NSF	No	No	Yes
CLDBDIR.NSF	No	No	Yes
WEBADMIN.NSF	No	No	Yes
CLDBDIR.NSF	No	No	Yes
BUSTIME.NSF	No	No	Yes
EVENTS4.NSF	No	No	Yes
STATLOG.NSF	No	No	Yes
REPORTS.NSF	No	No	Yes
MTSTORE.NSF	No	No	Yes
ACTIVITY.NSF	No	No	Yes

Domino Tuning Tips (all platforms)

The following tips will help Administrators to optimize their Notes/Domino environment. It is important for Administrators to understand that any configuration change should be closely monitored for its effects. Make single changes at a time to ensure adverse effects are easy to spot and changes can be backed-out.

View Index Updates

It is possible to tune the Domino update task to take advantage of the additional CPU resource of multi-core hardware. There are a several NOTES.INI parameters that you may use:

Update Task (View Index Updates)

If the system’s resources allow it, additional Update tasks may be started on the server, by adding the parameter, Updaters=[number], to the NOTES.INI. As a rule of thumb, do not exceed a maximum of one Update task per CPU. For example, to enable two Update tasks, apply this parameter:

• Updaters=2

Full Text Index Updates

A separate thread can be created solely for full text updates. This separate thread will only be responsible for updating full text indexes, so view updates will continue to occur even if a large full text index is being rebuilt. Consider applying this NOTES.INI parameter if many full text indexes are present on a Domino server and sufficient CPU resource remain:

• UPDATE_FULLTEXT_THREAD=1

Maintaining view indexes

The NOTES.INI parameter DEFAULT_INDEX_LIFETIME_DAYS=[number of days] allows administrators to set a default lifetime for database view indexes if none was selected by the database designer in the Design View Attributes dialog box.

The default value for this parameter is 45. Setting this to a lower value can have two benefits:

• Save Disk Space
• Reduce amount of work Update task must perform

Creating view indexes is disk intensive and requires CPU and memory resource, so it is important for Administrators and designers to strike a balance. It is not recommended to set this value to lower than 14 (two weeks).

Disable Transaction Logging For Certain Databases

Turning transaction logging may impact your enabled DAOS features relative to mail. In most cases, disabling Transaction Logging is not recommended because you lose the benefits of fast server restart. Disabling transaction logging on a database will cause Fixup to run on that database, creating the potential for slow restart. Furthermore, you will not be able to perform incremental backups using the transaction logs for that database.
For some databases, however, it might not be necessary to enable transaction logging. This is because it generates additional transaction log traffic on the disk and causes the transaction log drive to fill up quicker. Examples of databases that are suitable for disabling of transaction logs are:

• Mail boxes (mail.box) - Do not disable Transactional Logging on Mail.boxes if you are running DAOS.
• Server Log (log.nsf)
• Monitoring Results (statrep.nsf)
• Statistics Mail-in (statmail.nsf)
• Web Administration (webadmin.nsf)
• Schedule (clubusy.nsf or busytime.nsf)
• Cluster Database Directory (cldbdir.nsf)

Table 1 highlights NOTES.INI parameters to force new instances of the certain system databases created on server startup to have transaction logging disabled. They do not disable transaction logging on existing databases, or if they are created manually.

NOTES.INI Parameter	System Database
MailBoxDisableTXNLogging=1	Mail.box (See note below)
Log_DisableTXNLogging=1	Log.nsf
Schedule_DisableTXNLogging=1	Clubusy.nsf or busytime.nsf

Table 1: NOTES.INI parameters to disable database transaction logging.
Note: Do not disable Transactional Logging on Mail.boxes if you are running DAOS.

Replication

If you have more than one Domino server in your environment then you will need to set up replication. The following tips should help optimize the time and resource requirements to replication data around your environment.

Multiple Replicator Tasks

The database replicator (replica) task on a server is responsible for handling scheduled replication requests, as configured in server connection documents. The Cluster Replicator (clrepl) task is event driven and responds to changes in a database. These changes are then replicated to other cluster members.

Each replicator task only replicates with one server at a time, therefore the first replication must complete before the next one can start. The number of concurrent replicator tasks running can be set with the following NOTES.INI parameter:

• Replicators=[number]
• 
  

In environments where more than two servers participate in the cluster, additional clrepl tasks can be enabled. The number of concurrent clrepl tasks running can be set with NOTES.INI parameter:

• Cluster_Replicators=[number]

Administrators should monitor Replica.Cluster Domino statistics. For instance, the WorkQueueDepth Domino statistic indicates the number of changes waiting to be replicated to cluster members. If it is continually increasing, enable additional clrepl tasks.
The following statistics give indication on the current, average and peak Cluster work queue depth:

• Replica.Cluster.WorkQueueDepth
• Replica.Cluster.WorkQueueDepth.Avg
• Replica.Cluster.WorkQueueDepth.Max
• 
  

Replication Triangulation

When a client or server replicates with a remote server, it keeps a log of the name of the remote server and the time and date in the Replication History. Domino uses the replication history to determine which documents to scan for changes during the next replication. The purpose of Replication Triangulation is to make each server aware of every other server which maintains a replica of the same database, and which has had a successful replication.

In a large environment (hundreds of servers in a domain), the number of replication history events to maintain can cause a significant performance impact to the Domino server. Maintaining replication triangulation history for databases which exist on hundreds or thousands of servers is too expensive. This can manifest itself in the form of increased CPU activity for the replica task.

You can disable replication triangulation with the following server NOTES.INI parameters (available in Domino 7.0 and later)

• NSF_REPLHIST_NO_TRI=1
• REPL_NO_WS_TRI_HIST=1
• REPL_NO_REMOTE_TRI_HIST=1
• 
  

For local replicas, use Notes Client NOTES.INI parameters:

• NSF_REPLHIST_NO_TRI=1 [This will prevent existing triangulated entries from being read]
• REPL_NO_WS_TRI_HIST=1 [This will prevent new triangulated entries from being written]

Note: After setting the NOTES.INI parameters, the replication history must be purged from each replica of the databases affected.


Further information is available in IBM technote 1270104.

Internal Caches

In Domino, caches are used to store frequent lookups in memory, to prevent the data being continually read from disk. Since the mechanical hard-disk is commonly the slowest component of the server, caches help improve performance and make a more responsive user experience. The following sections provide Administrators with tips to ensure cache sizes are properly set.

NLCache

NLCache is the name lookup cache. The default is 16MB before Domino 8.5.2. Beginning in Domino 8.5.2, the default is 64MB. It can be increased as needed up to 4GB.

To determine if you need to increase the NLCache, use the show stat database or show NLcache console command. For example:
Database.NAMELookupCacheCacheSize = 16,447,205 (current size of cache)
Database.NAMELookupCacheLookups = 1,879,903
Database.NAMELookupCacheMaxSize = 16,777,216 (default maximum size)
Database.NAMELookupCacheMisses = 1,362,746

A relatively high number of misses compared to lookups indicate that you should increase the maximum allowed cache size. Administrators should increase the size of the NLCache in increments, so try doubling the default to begin with, if necessary. Modify using the ini parameter:

• NLCache_Size= For example, set 67108864, which sets NLCache_Size to 64MB.

Group Cache

When the server needs to lookup the members of a group (For example, in the event of an authentication request) it first checks the Group Cache. It will store results in the Group Cache to optimize performance. Groups are invalidated during updates or when cache is full. The default size is 4MB and it can be increased to 15MB.

If the cache needs to rebuild frequently because not enough data can be cached, this can slow down group lookups. Therefore, frequent or very large group updates can slow down server.

Verify GroupCache statistics with “Show stat net”
NET.GroupCache.Hits = 155
NET.GroupCache.Misses = 10
NET.GroupCache.NumEntries = 9
NET.GroupCache.Size = 65,406
NET.GroupCache.Used = 2,716

NET.GroupCache.Misses indicates the number of times a group was not found in the cache and so had to be read from disk. Administrators should increase the size of the Group Cache in increments until the number of misses reduces. Modify size with the NOTES.INI parameter:

• Group_Cache_Size= For example set 15360 , which sets Group_Cache_Size to 15MB.

Unified Buffer Manager (UBM)

The UBM (formerly referred to as the NSF Buffer Pool) is used to increase performance by caching disk I/O requests to all databases on a server. The UBM is typically the largest contribution of shared memory usage on the server. In earlier releases of Domino, the maximum size of the UMB was automatically calculated as 3/8th's of the physical RAM in a server. In a server with many gigabytes of RAM, this could result in an unacceptably high UMB upper-limit. Consider a server with 4GB RAM installed:
Max UMB Size = (4 x 3/8) = 1.5GB.
If the UMB were to grow to its maximum size of 1.5GB, there is a risk that shared memory may exceed process limits and result in a server instability.
Domino 8 now enforces a maximum UBM size of 512 MB default on Windows32, Linux, AIX, Solaris, i5/OS and z/OS platforms. On 64-bit Domino, the maximum UBM size is 1024 MB. Therefore, there is no need to take action to properly size the UBM in Domino 8. For additional details on this new behavior, see IBM technote #1268988.
For recommendations on setting NSF_BUFFER_POOL_SIZE_MB, read IBM technote 1286171.

NSF_DbCache_MaxEntries

The NSF_DbCache_MaxEntries determines the number of databases that a server can hold in its database cache at one time. If your server has sufficient memory, you can improve the performance of the server by increasing the number of databases that Lotus Domino can cache in memory at one time. The default value is 25 or the NSF_Buffer_Pool_Size divided by 300 KB, whichever value is greater.

You should monitor the Database.DbCache.Hits statistic on your server. This indicates the number of times a database open request was satisfied by finding the database in cache. A high value indicates database cache is working effectively. If the ratio of Database.DbCache.Hits to InitialDbOpen is low, you might consider increasing NSF_DbCache_Maxentries.

For detail information on how to set the number of databases cached simultaneously in Lotus Domino, read IBM technote 1279893.

NSF Monitor Pool

The Monitor Pool caches event monitors which include server and client mail rules. If set too low the Monitor Pool can become full. Side effects include causing new calendar notices not to display in the Calendar Mini View.

The default pool size is 40 MB. The maximum value for this pool is 256/512 MB.

To see the current values review stats
Database.MonitorPool.Event.Used = 52309
Database.MonitorPool.Monitors.Used = 1184
Database.MonitorPool.Size = 41943040

Check the LOG.NSF for the error “Insufficient memory – NSF monitor pool is full” to know if you should increase this pool for the Domino server. Many customers find a value between 80 and 100 MB to work best. Increase this value if you are approaching the maximum by using the NOTES.INI setting:

• NSF_MONITOR_POOL_SIZE_MB=

Multiple Mail Boxes

Additional mailboxes are required on a server only when the amount of mail traffic prevents the router from being able to effectively access the mail box files and access conflicts are encountered. Access conflicts occur when other threads or processes lock the mailbox, while another entry tries to access the mailbox and is denied.

The typical number of mail boxes on a mail server ranges from one to four, depending on the volume of mail. IBM technote 1148438 provides Administrators with the procedure to determine whether additional mailboxes are required on busy servers. If you feel you are in need of another mailbox. Change the server configuration to 2 mail boxes.

Tips for Server Based Mail Rules

Keep the number to the absolute minimum, based on business requirements. Rules require processing by the mail router each time it delivers mail. Therefore the fewer rules, the fewer server resources are required.
Place rules most likely to be triggered at the top of list and use the “Stop processing further mail rules” option.
Searching the memo body is CPU intensive and will cause and increase in the CPU used by the SMTP task

Use the following NOTES.INI parameter to cap the number of mail rules per user's mailfile or server mail.box:

• MailMaxFilters=<1 to 100>

This IBM Case study provides Administrators with an indication to the impact mail rules have on a server.

Tuning User Sessions

There are two NOTES.INI parameters that define how many client user sessions a server can have concurrently open and also how long a session remains open. Keeping sessions open requires memory on the server, whereas opening a new session consumes a small amount of CPU resource. Therefore a balance must be achieved.

• Server_MaxSessions=[number] (Maximum number of concurrent open sessions)
• Server_Session_Timeout=[number minutes] (maximum duration of idle activity before the Domino server terminates the session)

A timeout value of 30-45 minutes is recommended for most customers. For more information read IBM technote 1089879.

Domino Configuration Tuner (DCT)

The Domino Configuration Tuner (DCT) provides easy-to-use self-service configuration analysis for more robust installations and better performance of Domino servers. Make sure you are up-to-date with suggested changes from Lotus Domino by periodically updating DCT. For more information, read "Domino Configuration Tuner (DCT) provides easy-to-use self-service configuration" - IBM technote 4019358.

Optimizing server performance (Top 10 ways to improve your server performance sidebar)

Top 10 ways to improve your server performance (sidebar)
By analyzing a variety of NotesBench reports, published over the last two years by NotesBench Consortium members, we came up with a list of the top 10 ways you can improve the performance of your server. The list shows you how to improve your server capacity and response time.

1. Make sure your server memory matches the number of users you want to support. Most NotesBench vendors use 300K-400K per active user. They also set their NSF_BUFFER_POOL_SIZE to the maximum for their memory configuration. This setting isn't necessary, because the Domino server initially obtains a quarter of available memory and grows only if necessary (depending on the load). You should use published physical memory configurations as a ceiling for memory configuration decisions.
2. Distribute I/O among separate devices. For example, you can put the OS kernel on one drive, the page file on another, the Domino executable on a third, and finally the Domino data files on a fourth drive. In some cases, NotesBench vendors point their log.nsf file to a location different from the default data directory (using the log= setting in the server's NOTES.INI file).
3. I/O subsystem improvements. For example you can:
   • Move from EISA-based systems (such as, controllers) to PCI-based systems
   • Exchange EISA/PCI boards in favor of PCI-only boards (this way, lower speed EISA devices won't decrease the I/O throughput)
   • Use striping to improve performance
   • Use multiple I/O controllers to distribute logical volumes (and use file pointers to databases across separate controllers). Make sure you have the latest BIOS for your I/O subsystem. This is an inexpensive way to remove a likely throughput bottleneck.
4. Use faster disk drives. You can improve disk drive speeds from 5,400 rpm to 7,200 rpm. For most Windows NT systems, NotesBench vendors use 2GB disk drives. For Solaris and IBM Netfinity systems, the drives were larger: 4GB. For AS/400, the drives were even larger: 8GB.
5. Increase the stripe size. NotesBench vendors use a stripe size of 8K (Digital's systems) or 16K (IBM Netfinity reports). (The IBM Netfinity report provides additional information on I/O settings such as w IOQ Depth, Outbound Posting, PCI Line Prefetch, and Address Bit Permitting.)
6. Use faster CPUs. NotesBench vendors have moved beyond the Pentium, Sparc, and PowerPC processors, which were in the 100-200Mhz range, to higher speed processors. However, they consistently use P6-based systems over the Pentium II systems for high-end Domino server loads. The size of your Level 2 cache should match your expected user loads and the response time you want. Vendors have moved from 256K to 512K, 1MB to 2MB Level 2 cache systems, especially on their greater than two-CPU configurations.
7. Improve your network. NotesBench vendors have:
   • Moved from 10Mbps cards and networks to 100Mbps configurations
   • Used multiple LAN segments (one for each partition) to isolate network traffic, at the high-end user loads
8. Change your network protocol to IP. Vendors were initially (two years ago) using NetBIOS and SPX internally, but have unanimously moved to IP for their performance publishing efforts.
9. Upgrade to a newer release of Domino. NotesBench vendors have moved from Domino Release 4.5a SMP version to Domino Release 4.52B SMP version for higher capacity results. The first Domino Release 4.6a result (AS/400) on a RAID5 configuration indicates a reliable configuration can still provide competitive response time with a properly designed I/O architecture.
10. Use Domino partitioned servers. NotesBench vendors have increased scaling of active user loads and leveraged their more powerful configurations (faster clock cycles, fiber-connected I/O subsystems, OS kernel to CPU binding, and multiple I/O controllers) by using partitioned servers.

How we came up with these recommendations
To understand how we came up with our top 10 list, we will take you through the performance analysis of Number 2 in the list -- to distribute I/O among separate devices. Initially, many vendors placed the kernel, page, and Domino executables on one volume and the Domino data files on another. However, both volumes were on the same controller. Lately, the NotesBench reports show improvements in performance when the volumes are separated across multiple controllers, and individual volumes are separated across disks. What this means is that we found that vendors put the OS kernel on one drive, page file on another, Domino executable on a third, and finally the Domino data files on a fourth drive. In some cases, they pointed their log.nsf file to a location different from the default data directory (using the log= setting in the server's NOTES.INI file). Vendors who distributed the I/O over several disk drives had better server performance overall, and could support a higher capacity of users. For example, in a NotesBench report published in May of 1996, Digital Equipment Corporation set up a server with the following specifications:

• CPUs: four 133Mhz CPUs
• Memory: 512MB
• Domino: Release 4.1

They placed the operating system and the Domino executable on drive C:\, the page file on drive D:\, and the Notes\data directory on drive E:\. They could support a maximum capacity of 1,500 users with this configuration.
In a NotesBench report published in September of 1997, IBM Corporation set up a server with the following specifications:

• CPUs: three 200MHz1Intel Pentium Pro processors
• Memory: 1GB2
• Domino: Release 4.51

They placed the operating system on drive C:\, the page file on drive C:\, the Notes\data directory on drive E:\, and the Domino executable on drive E:\. They supported a Mail-only workload of 3,500 active mail users. In a four-processor configuration, they supported a MailDB workload of 2,900 active users. These examples led us to the conclusion that distributing I/O over several disk drives had better server performance overall, and could support a higher capacity of users. We went through many other NotesBench reports to collect the data shown in our top 10 list. You can visit the NotesBench Web site yourself to view published data and test results. Visiting the site may help you to come up with other ways to improve your server's performance.

ibm.com

Using the Message Recall feature in IBM Lotus Notes and Domino V8

The ability to recall mail messages is one of the most requested features for IBM Lotus Notes and Domino V8. Enabled on the server and client by default, this new feature lets you recall mail messages that were sent in error. This article explains how Message Recall works, discusses how it is configured and controlled, and covers the finer points of planning and deploying the feature.
How Message Recall works: The basics
Message Recall is simple to use from a user's point of view. You simply open the Sent view of your Lotus Notes V8 mail file, highlight the mail you want to recall, and click the Recall Message button in the Action bar (see figure 1).

Figure 1. Action bar showing Recall Message button on the right

A dialog box then displays that shows the original recipients, any of which you can deselect or choose to get a response (see figure 2).

Figure 2. Recall Message dialog box

After you click OK, the recall request is acknowledged with a dialog box. If everything is set up properly, the messages are removed from the recipients’ mail. You receive a report telling you which messages were recalled (and whether they had been read) and whether any messages could not be recalled and why. Now let's investigate what’s behind this simple process.

Back to top

Configuring Message Recall
Lotus Notes and Domino V8 ship with Message Recall enabled. Whether you build a new system from scratch or upgrade servers and users, this feature is ready to use. If you have a large system that will be upgraded over a period of time, you may want to disable the feature at first, to allow time to plan for user support and training.
The Server Configuration document is used to configure Message Recall. This central location lets you easily allow or disallow it for all servers. Mail Policy documents can further refine the settings, as can individual user preferences.
If you do not have any Server Configuration documents, Message Recall is still turned on by default, and the default settings, listed here, apply:

• Message Recall: Enabled
• Allow recall of messages with unread status: Unread Only
• Do not allow recall of messages older than: 14 days

If your deployment plan calls for the feature to be rolled out later, you must create a Server Configuration document to turn it off for now.
To disable Message Recall, simply open the Lotus Domino V8 Server Configuration document and change the Message Recall setting from Enabled, as seen in Figure 3, to Disabled.

Figure 3. Message Recall tab in the Server Configuration document

Back to top

Requirements for Message Recall
The requirements for Message Recall are:

• A Lotus Domino V8 or later server containing the mail to be recalled.
• A mail file based on a Lotus Notes V8 or later mail template with which to recall the message.
• A copy of the mail message in the sender’s mail file. (This is usually found in the Sent view, but the Message Recall button is also found in the All Documents view.)
• Permission given at the server to perform Message Recall (and optionally at the policy and the recipient’s mail-file level, where the feature can be turned off).

Message Recall not only works with these minimal requirements but also works even if the recalling user is on a server that is not Lotus Domino V8. All that is required is a Lotus Notes V8 mail template containing the button to recall the message and that the message being called resides on a Domino V8 server. Moreover, interim servers between the recaller and the recipient can be any version because the request at that point is simply an email message.

Back to top

Limitations of Message Recall
Message Recall works only on mail that is routed over NRPC. This means that neither mail routed to the Internet nor internal mail routed over SMTP can be successfully recalled.
If a Mail Policy exists that limits the use of this feature for a user, this policy applies and can limit the feature’s functionality. If there is no policy, or if the policy allows a user to change the recall setting, each user can choose to disable the feature in his or her Mail Preferences.
Only mail that resides on a Lotus Domino V8 or later server can be successfully recalled. The Lotus Domino V8 router does the work, so a recall request sent to a Lotus Domino V7 server does not succeed, and the recalling user receives a report stating that the server does not support Message Recall.
A copy of the message must be saved in the sender’s mail file to recall it successfully. If the sender did not save the message when it was sent, Message Recall cannot be used. Because signatures are checked (for security purposes), mail must be recalled from the mail file from which it was sent, by the original sender. A delegatee can recall a message from another user’s mail file, but only if the delegatee sent it. This means that a Lotus Notes Administrator or an administrative assistant is not able to recall a message sent by another user, unless that person has access to the user’s ID and password.
If a message has been forwarded by a recipient, the forwarded message is not recalled by the original sender’s recall request because the UNID differs from that of the original. In other words, Message Recall does not chase down forwarded copies of an email. Any forwarded copies must be recalled by the person who forwarded them.
Message Recall is not available with IBM Lotus Domino Web Access. Mail sent to a Lotus Domino Web Access user can be recalled if it is on a Lotus Domino V8 server and the recalling user has a Lotus Notes V8 mail template to use to initiate the recall.
Message Recall works only for email, not for Calendar and To Do items. Users see a pop-up box stating "This message type cannot be recalled" if they attempt to recall a Calendar or To Do item. These items can be cancelled or changed using the Reschedule or Cancel options that exist in all versions of Lotus Notes calendars.

Back to top

How the feature works: The details
When the original sender clicks the Recall Message button from the Sent (or All Documents) view, the Lotus Notes client creates a recall request for the highlighted message. This request is mailed to each recipient or group that the user chooses. (Note in figure 2 that the addresses of Internet users are displayed, and a recall request can be mailed, but Message Recall does not succeed for these users.)

Tip

If you plan to run a mixed-version system for a while, consider putting a mail rule on your pre-Lotus Domino V8 SMTP servers to reject these messages. Otherwise, recall reports are sent to the Internet, where they may cause confusion for the recipient. Such a mail rule can be set to not accept messages with a form containing "recall." You can then disable this rule when the SMTP server is upgraded to Lotus Domino V8.

The document UNID is used to identify the message in the recipient’s mail file. If the server is capable of performing a recall, the message is located and deleted. The router removes the message completely and leaves only a deletion stub, allowing the message to be removed from replica copies as well. Even if the recipient has soft deletions enabled, the message is never left in the Trash folder. Because the UNID of the document is used to locate the message, any copies that have been moved to folders are also removed.
The recalling user receives a report with the results of the recall attempt. If the message was successfully deleted, the report contains this information, noting whether or not the message was read. If the recall attempt fails, the report states the reason it failed. The recalling user has two very similar notices in his or her mail file, one in the Sent folder and one in the Inbox, after the message is recalled. Both are needed to properly inform the user which messages were recalled, so users need to be told to retain both notices until they are satisfied with the results of the recall. Both notices can then be deleted.
The recipient of the original message is not notified that a recall has occurred. The message is simply no longer in the recipient's mail file.
If you attempt to recall a message that was sent using SMTP, the Lotus Domino V8 router sends you a Non-Delivery Report stating “Message Recall Requests cannot be routed via SMTP.”

Back to top

Controlling Message Recall
Although the Message Recall feature is enabled, disabled, and configured at the server level, it can be refined and controlled at other points, providing companies the flexibility to use the feature the way they want.
Controls start at the Server Configuration document, in which the feature is enabled or disabled; selections are available to allow the recall of read, unread, or both; and time limits are available ranging from weeks down to minutes.
Mail Policy control is available for Lotus Notes Administrators. The Message Recall options are as follows and are shown in figure 4:

• User is allowed to recall sent messages: yes or no.
• Other users are allowed to recall messages they sent to this user: yes or no.
• Allow recall of messages with unread status: Unread only or Both read and unread.
• Do not allow recall of messages older than: specify number of weeks, days, hours, or minutes.

Figure 4. Mail Policy settings in the Server Configuration document

The Mail Policy can be applied to a subset of users who may be under legal obligation to keep all messages, or perhaps applied to limit who is allowed to recall messages at all. The Mail Policy overrides the settings in the Server Configuration document. For instance, if the server allows unread mail to be recalled, but the Mail Policy applied to a user allows unread and read mail to be recalled, the Mail Policy applies, and both unread mail and read mail can be recalled. This provides very specific control for individual users.
Unless Mail Recall is disabled at the server or via a policy, users can also control Message Recall. Under User Preferences - Basics, they can select or deselect “Allow others to recall mail sent to me.” If the user deselects this option, the setting causes a notice to be sent back to the recalling user stating that the message cannot be recalled.

Back to top

Message Recall and legal compliance
Message Recall is a new concept to many organizations that use Lotus Notes and Domino. Questions have arisen over whether this new feature conflicts with legal requirements, specifically many of the newer laws concerning the retention of email.
Solutions that are designed for compliance purposes usually require that all mail be journaled as it is sent, before it arrives at the recipient’s file. Message Recall does not affect this type of solution. In fact, not only is the original message journaled, the recall request is journaled also.
If the original message is ever needed for compliance purposes, it is in the mail journal and in the off-site storage that is part of your compliance solution. As added assurance, Message Recall requests themselves cannot be recalled. Thus, a robust mail compliance solution should have no problem dealing with Message Recall, and many companies already have custom-built or other mail recall solutions that are fully compliant.

Back to top

Understanding the finer points
Because Lotus Notes mail can be read offline, with the user disconnected from a network, the read and unread status of a local replica of a mail file can be different from that of a server-based copy until replication takes place. Thus, a recall request sent to the server may report that a message was unread when, in fact, the user read it offline.
Mobile devices can present a challenge to users wishing to recall all copies of a message. Unless a handheld device is configured to process deletions from the server-based mail, messages are not removed from the handheld device. The owner of the handheld device controls this setting. Also, depending on the vendor, Message Recall may not be possible from the device itself.
Mail sent to users with ambiguous names is not recalled; the recall message cannot determine which recipient was the one selected by the sender originally.
Mail can be recalled from groups successfully, but if the group membership changes between the time the mail was sent and recalled, recall notices may not be sent to original recipients who have since been removed from the group, and recall notices may be sent to users who never got the original message. Also, you may recall messages from groups that include Internet addresses, but only those users in the group with valid Lotus Notes addresses receive a recall notice. The other requests result in failure notices.
Finally, messages can always be printed and forwarded or even have screenshots made of them. Message Recall is not effective in these cases. Message Recall is not guaranteed to get rid of any and all traces of a message; rather, it is intended to give users the ability to recover from errors made in sending messages.

Back to top

Conclusion
Lotus Notes and Domino V8 Message Recall is yet another tool for users, and one that may become quite popular. When or if you choose to deploy the feature, consider offering training to your users so they are able to use it effectively.

www.ibm.com

Move a Lotus Domino server to a new certifier without a reinstall

Charles TANABAL
Suppose that there is a company name change and you must to move your Lotus Domino server to a new certifier, without changing any server settings. Learn step-by-step how to move your Lotus Domino server without having to reinstall it or change any settings using the Server1/ExistingCertifier will be moved to Server1/NewCertifier example.

1. Create a new certifier (/Newcertifier).
2. Be sure to cross-certify in both directions (/Newcertifier-> /ExistingCertifier and /ExistingCertifier->/Newcertifier).
3. Create a new server registration with the new certifier (Server1/NewCertifier).
4. In your "all servers documents" view, duplicate the Server1/ExistingCertifier server document.

Related resources from SearchDomino.com: Securely connect Lotus Domino servers on different domainsLotus Domino Server performance-tuning pointers Top 10 Lotus Notes Domino administration tips of 2007

5. Edit the duplicate document and change the server name to "Server1Temp/NewCertifier."
6. Edit the server document for Server1/Newcertifer (the newly registered document) and copy the value of the field "Certified Public Key" to your clipboard, under the Administration tab.
7. Next, paste in your new server document by replacing the

Show Me More

• More on Lotus Notes Domino Performance
• Get help from the community
• Powered by ITKnowledgeExchange.com
• "Certified Public Key" value in the Server1Temp/NewCertifier server document.
• Delete the Server1/NewCertifier server document.
• Edit the document for Server1Temp/NewCertifier and change the name to Server1/Newcertifier. Save and close out.
• Duplicate all connexion documents where Server1/ExistingCertifer is the source or destination server.
• Next, change Server1/Existingcertifier to Server1/NewCertifier in all duplicated documents.
• With your admin client, add Server1/NewCertifier in the access control list (ACL) of all Lotus Notes databases marked with 'Manager' access. Specify it as "Administration Server," using the advanced tab for MailFiles and other Lotus Notes databases, if needed.
• Replicate names to all Lotus Domino servers.
• Use a code-generated button to send an email to all Lotus Notes users with Server1 as their home server, so that they can change the Mailserver Field in their location document. Below is the LotusScript code to create this button. Note: Clicking this button will change the servername information in all location and connections documents of the local address book. Domino administrators should adapt this code to meet their own specifications.
  

  Sub Click(Source As Button) 
  Dim session As New NotesSession 
  Dim db As NotesDatabase 
  Dim view As NotesView 
  Dim doc As NotesDocument 
  Dim nextdoc As NotesDocument 
  Dim destLocation As Variant 
  Dim privnab As New NotesDatabase
  ("","names.nsf") 
          
  If Not (privnab.isopen) Then 
                  privnab.Open "","" 
  End If 
  Set view = db.GetView("($Connections)") 
  Set doc = view.GetFirstDocument 
  While Not doc Is Nothing 
  destServer = doc.GetItemValue("Destination") 
  Set nextdoc=view.GetNextDocument(doc) 
  'THE BELOW LINE MUST BE MODIFIED 
  TO SPECIFY THE RELATIVE SERVER NAME: 
  If destServer(0) = 
  "CN=serverName/O=ExistingCertifier" 
  Then 
  'If you want to remove the document include this line: 
  'Call doc.Remove(True) 
  'If you want to update the Destination
   field include the below instead: 
  Call doc.replaceitemvalue
   ("Destination","CN=servername/O=NewCertifier") 
  'If you want to update the Destination 
  Server Address entry include the below: 
  'Call doc.replaceitemvalue 
  ("OptionalNetworkAddress",
  "<IP address>") 
  'Include the line below if you've 
  included either of the ReplaceItemValue calls: 
  Call doc.save (True,True) 
  End If 
  Set doc=nextdoc 
  Wend 
          
  Set view = db.GetView("($Locations)") 
  Set doc = view.GetFirstDocument 
  While Not doc Is Nothing 
  destServer = Lcase
  (doc.GetItemValue("MailServer")) 
  Set nextdoc=view.GetNextDocument(doc) 
  'THE BELOW LINE MUST BE 
  MODIFIED TO SPECIFY THE 
  RELATIVE SERVER NAME: 
  If destServer(0) = 
  "cn=servername/o=existingcertifier" 
  Then 
  'If you want to remove the document
   include this line: 
  'Call doc.Remove(True) 
  'If you want to update the Destination 
  field include the below instead: 
  Call doc.replaceitemvalue ("Destination",
  "CN=Servername/O=NewCertifier") 
  'If you want to update the Destination 
  Server Address entry include the below: 
  'Call doc.replaceitemvalue 
  ("OptionalNetworkAddress","<IP address>") 
  'Include the line below if you've included 
  either of the ReplaceItemValue calls: 
  Call doc.save (True,True) 
  End If 
  Set doc=nextdoc 
  Wend 
  End Sub
  

• Replace the "mailserver" field with "CN=Server1/O=NewCertifier" for all users specifying server1 as Homeserver. Stop the Lotus Domino server.
• Replicate names to all Lotus Domino servers.
• Stop Server1.
• Rename the existing Server ID (Server.id) file under the Domino Directory to ExistingCertifier as the extension (Server.ExistingCertifier). Note: Place the ID file that was created during the registration of Server1/NewCertifer and change the name to Server.id to ensure that it corresponds with what is in your notes.ini file.
• Copy the notes.ini file to Notes.ini.Existingcertifier.
• Edit your notes.ini file, find Server1/ExistingCertifier and replace it with Server1/newCertifier, and then save and close.
• Restart your Lotus Domino server.

Securely connect Lotus Domino servers on different domains

Jim MC
One of the many challenges Lotus Notes Domino administrators face is the increased workload that results from having a distributed setup of their IT organization. This is especially true if they are working on multiple domains. When exchanging information across more than one domain, security is a particular concern. This reader-submitted tip explains how to safely and securely connect Lotus Domino servers that are located on different domains through a process called cross-certification.

---

In the case of Domino servers being located in different domains, administrators can cross-certify the servers to communicate, connect, and exchange information with each other. Cross-certifying allows Lotus Notes users in one domain access to data in another domain -- while simultaneously maintaining security at its highest levels.
Here are the steps you should follow to carry out the cross-certification process:

1. Create a "safe copy" of an existing user ID file and open your Lotus Notes client.

   Related resources from SearchDomino.com: Expert Advice: Cannot connect to new domainProtect

• More on Lotus Notes Domino Access, Permissions and Authentication
• Get help from the community
• Powered by ITKnowledgeExchange.com

1. Lotus Notes from malicious code with the Domino ECL Lotus Notes Domino Access, Permissions and Authentication Reference Center

2. From the File menu, locate the User Security option. The location of this menu option will vary, depending on your installed version of Lotus Notes.
3. Select the Your Certificates tab and also the Export Notes ID (Safe Copy) tab from the Other Actions dropdown list. When prompted, click Save to create the SAFE.ID file. This will create a safe copy of the ID file for a Lotus Notes user in the first domain.
4. Transfer the created file to the destination Lotus Domino server (i.e. the Domino server located in second domain).
5. Copy the file to diskette, shared directory folder, CD-ROM, or otherwise, transfer the file to the Lotus Domino server.
6. Launch the Domino Administrator client.
7. Select the File -> Open Server menu options to connect to the Domino server.
8. From the main navigation window, select the Configuration tab.
9. Now click Certification and Cross-Certify from the right-most side. If the options are not displayed, click on the Tools button to expand the list configuration options.
10. From the Choose a Certifier dialog window, choose the Certifier ID button and select the CERT.ID file associated with the Lotus Domino server. This is a special ID file that was automatically created when the Domino server was installed. A copy of the file will probably be stored on the Domino server. Select the file and click OK to continue.
11. When prompted, specify the password associated with the server CERT.ID file and click OK again. You must know this password to continue with the process.
12. You will now be prompted to select the safe copy of the ID file. This will enable all Lotus Notes users in the first domain to access the Lotus Domino server in the second domain. Click OK after the SAFE.ID file has been selected.
13. Click the Cross Certify button to generate the cross-certificate for the destination Domino Server Directory. Note that the first time you connect to the destination Domino server you will be prompted to create a digital certification for the destination server. This is a one-time event so just click on the "Yes" button when that message is displayed.