Search This Blog

Saturday, November 19, 2011

DAOS Best Practices

When it comes to configuring the Domino Attachment and Object Service (DAOS), you may be asking yourself - and us in turn - what's the right way to set it up? For example, is there an optimum “Minimum Size” setting? Should the repository go under the data directory or on its own drive? What's the best “Deferred Deletion Interval” in relation to my backup and restore schedule? This guide, and the documents it references, attempts to answer these questions of individual, site-specific configuration in general terms with guidelines for adapting and modifying them based on measurements made against your particular environment.


Where To Locate Your DAOS Base Path Repository



By default the DAOS repository resides under the server's data directory and defines a single container as indicated by the “DAOS Base Path” setting in the DAOS tab of the server document. So, on Windows for example, if you use the default, “DAOS”, and your data directory is C:\Lotus\Domino\Data, the full path to the repository would be C:\Lotus\Domino\Data\DAOS. For Domino 8.5, only this one container may be specified.

However, this default location, chosen simply for being well-known, may not be the most efficacious. Some things to consider:
    1. What is the total capacity of all file attachments? With only one container in Domino 8.5, flexibility may be important when choosing the best DAOS base path. You'll want to be sure you have significant storage capacity or the ability to reconfigure a logical drive as space needs increase. Use the Domino Attachment and Object Service Estimator to plan for your storage requirements. 2. What I/O costs do I expect to incur? DAOS base path I/O is significantly less than that of Domino's Data directory. In benchmark tests, DAOS repository I/O was 94% less than that of the server's Data directory. Lower performing storage (a NAS device, for example) can be used here. 3. Can I use lower cost or external storage devices? In many cases, you might find attachments are infrequently accessed -- for example, when they're part of old email messages collecting the proverbial dust in one's inbox. In these environments, locating DAOS on lower cost storage (tier 3) devices may be indicated. On the other hand, if full text indexing, agents, or other applications make heavy use of the consolidated attachments, “lower cost” storage may cost you in performance. Note: Externalizing the DAOS repository in this manner does not mean you can map multiple Domino servers to the same container. This is an unsupported configuration as of this publication and could very well lead to data loss due to encryption with the server's key. NLO files cannot be shared across Domino servers. Note: Modifying the location of the DAOS repository at a later time is allowed and requires that you first change the “DAOS Base Path” field on the DAOS tab in the server document, stop the Domino server and then relocate the existing subdirectory structure with its NLO files to the new location. On server restart, the modification will take place seamlessly. 4. Why is it recommend to not locate the DAOS Base Path under Domino's Data directory? Many of Domino's tasks, including fixup, compact and the admin client, scan Domino's Data directory and will also scan the NLO files if the files are located under that sub-directory. The scanning of NLO files for tasks other than DAOSMGR will add additional unneeded overhead to the processing done by Domino's tasks.

Optimum Minimum Size For Participation



By default, the minimum size setting for an attachment to make use of DAOS is 4096 bytes. While we recommend using 64000 as the lowest value you should use here (1048576 on iSeries), there are a number of things to consider when determining the best DAOS minimum size setting for your system.
    1. Do not set the minimum size lower than the default setting. Due to attachment file overhead, setting the minimum size to anything lower than the default size would actually be less efficient than storing the attachment in the NSF file. 2. Set a minimum size that is a multiple of your file system's disk block size. By choosing a minimum size that is a multiple of the disk block size, you optimize disk usage. To ascertain the disk block size for your file system, on a Windows NTFS, use “fsutil fsinfo ntfsinfo ” and take note of the “Bytes Per Cluster”. This is the disk block size. On Solaris, you could use df -g and take note of the block size. On AIX you need be super user to do determine block size, then use lsfs -q and look for block size. On Linux you also need to be super user to find the block size, then use df -k to determine the device name of your filesystem and the uses dumpe2fs | grep 'Block size' to determine the block size. 3. Take note of possible limitations on number of files. The smaller you make the setting, the more attachments will qualify for DAOS consolidation. The larger you make the setting, the fewer will qualify. In Domino 8.5, the DAOS repository allows for one container with up to 1,000 subcontainers, each with a maximum of 40,000 NLO files. Thus the storage capacity of DAOS is limited to 40 million distinct objects. This is a significant number of files, so if you expect to come anywhere close to approaching it, you should check the limits on your backup and restore solution, as some applications and file systems have limitations on maximum number of files. Refer to your operating system and/or backup application guidelines. 4. The current recommendation for the lowest value you should use on IBM iSeries is 1048576 (1M) to avoid overwhelming the filesystem and backup utilities.

To get an idea of how many files various settings would generate, you can run them through the Domino Attachment and Object Service Estimator.

The ultimate goal with this setting is to minimize the number of files in your DAOS repository and maximize the amount of disk space saved.

Deferred Deletion Interval



DAOS automatically deletes NLO files that are no longer being referenced by any databases. This deletion of NLO files is known as “pruning” and occurs at the specified “Deferred Deletion Interval.”

Establishing a useful “Deferred Deletion Interval” for your server involves a few considerations, primary among them your backup and restore schedule. You want to ensure that NLO files which are no longer needed remain in existence at least as long as your backup cycle. In this way, they will not be deleted before the next backup.

A secondary consideration is the size of attachments typically stored in the repository for your server. If they are usually quite large, you may want to have them cleaned up as quickly as possible after there are no longer any references to them.

If your deferred deletion interval is set too high, NLO files which are no longer needed will continue to take up valuable space in the repository. If your deferred deletion interval is set too low, you could be deleting NLO files that have not yet been backed up, thus making it difficult, and in some cases not even feasible, to restore them. It is important to find a balance that satisfies both your backup schedule and the system integrity of a neatly pruned environment.

Pruning



Pruning can also be manually triggered to override the automatic deferred deletion interval. The administrator can issue the console command “tell daosmgr prune x” to forcibly delete unreferenced NLOs that are x days old. This will recover the disk space still being used by unreferenced NLO files immediately rather than waiting for the automatic deferred deletion interval to do so. When performing this action, you must consider your backup cycles. As with setting the deferred deletion interval too low, pruning too soon could delete NLO files that have not yet been backed up.

When A Notes Database Should Use DAOS



There are several good reasons to select an NSF file for participation in DAOS consolidation:
  • It contains or is likely to have multiple copies of the same attachments.
    Even a single NSF can benefit from DAOS consolidation.
  • It resides on a server where the same attachments appear across multiple NSF files.
    If others are also referencing attachments present in your database, why not share?
  • It contains very large attachments.
    In this case, it may not matter how many other NSF files hold the attachments in question. If they're large enough, the simple step of storing them outside the NSF can make common operations against that database much faster.

While DAOS can always benefit your data, DAOS has less benefit under the follow conditions:
  • Databases have lots of small attachments.
    Attachment consolidation is less efficient in this scenario due to disk blocking. You can, however, eliminate this issue by adjusting the minimum size setting upward.
  • There is little or no attachment duplication across databases.
    Backup would still benefit due to extracting static data, but you would have little disk space reduction.
  • Databases contain few or no attachments.
    In 8.5, DAOS stores only file attachment data.
  • Databases need to be quickly portable.
    Because DAOS object files (NLOs) cannot be shared across Domino servers, it is more difficult to move DAOS-enabled Notes databases from server to server.

It's not necessary for all databases on a server to leverage DAOS, but for those that do, the savings in both space and time, as for example, in much accelerated compact operations, are significant.

Mailbox



Although DAOS will work in any configuration, it operates most efficiently when it is enabled on both the mail.box files and individual mail files. Enabling transaction logging and DAOS on mail.box will enable the Router to optimize the delivery of DAOS based attachments. This can result in significant I/O savings for the case where the same attachment is sent to multiple recipients on the same Domino server.

I/O Activity for mail delivery of email with an attachment
DAOS Enabled
Document WritesAttachment Read(s)Attachment Write(s)Comments
MAIL.BOX
    Mail files




No
No
1 + N
1 + N
1 + N

Yes
No
1 + N
1 + N
1 + N

No
Yes
1 + N
1 + N
1 + N

Yes
Yes
1 + N
1
1
    Maximum reduction for both I/O & CPU
* N is the number of recipients

Note:
If you have multiple mail.box files, you must enable transaction logging and DAOS on all of them to leverage DAOS object copy optimization, which streamlines delivery of attachments.

Since Domino creates new mailboxes as needed, you should also set these properties on the mail.box template. If you choose not to enable DAOS or transaction logging on the mail box, DAOS will still be used by any DAOS-enabled mail files. Using DAOS on mail.box(es) only affects the optimized routing (delivery) of attachments.

When an incoming document is received at mail.box, it is stored there until it is delivered to the individual mail file(s) of the recipient(s). Several results are possible:
  • If DAOS is not enabled anywhere, the document will be stored in mail.box, and the attachment will be stored inline. As the document is delivered to the recipient mail file(s), the document and the contents of the attachment are read from mail.box, and the document is written to the mail file with the attachment inline. Total I/O cost to deliver to N recipients: 1 + N doc writes, 1 + N attachment reads, 1 + N attachment writes.
  • If DAOS is enabled on both mail.box and destination mail file(s), any attachments in that document will be extracted and converted to NLO files as it is being written to mail.box. The document and DAOS ticket are written to the destination mail file(s). IMPORTANT: In the case where both mail.box and the mail file are DAOS-enabled, the contents of the attachment will not be written again as the document is delivered, only a reference to the existing NLO file will be copied. Total I/O cost to deliver to N recipients: 1 + N doc writes, 1 attachment read, 1 attachment write.

    If DAOS is enabled on mail.box but not on the destination mail file(s), any attachments in that document will be extracted and converted to NLO files as it is being written to mail.box. Since the destination mail file(s) is/are not using DAOS, the attachment must be stored inline, and the contents of the attachment will be read out of mail.box (which has it stored in DAOS) in order to do that. Total I/O cost to deliver to N recipients: 1 + N doc writes, 1+N attachment read, 1+N attachment writes.
  • If DAOS is not enabled on mail.box, but is for the destination mail file(s), the attachment will be stored inline in mail.box. As the document is delivered to the recipient mail file(s), the contents of the attachment are read out of the mail.box document, and a temporary NLO file is created for each destination mail file so that a checksum can be calculated. If an NLO file with the same checksum already exists, the temporary file is deleted. In the case of N recipients, this process will be repeated N times, even though only one NLO file will remain at the end of the process. Total I/O cost to deliver to N recipients: 1 + N doc writes, 1+N attachment read, 1+N attachment writes. IMPORTANT: In this case, although the end result (a single NLO file per unique attachment) is the same, the I/O cost is significantly increased over the case where mail.box is enabled for DAOS.

Mail Journaling



For 8.5, it is recommended that you not enable DAOS on the mail journal (mailjrn.nsf).

Encryption



By default, DAOS employs encryption to safeguard its repository. This setting is separate from encryption settings that apply to an NSF or document. The encryption is done with the server key, so the resulting NLO files can be read only on a server that uses that same key. This may be a consideration for backup or redundant server setup.

The performance hit for DAOS encryption is negligible; testing showed a 5% CPU increase with no change in I/O versus unencrypted data. However, if your organization has reason to disable it, we've provided the server notes.ini setting DAOS_ENCRYPT_NLO, which can be set to zero to affect that change. To determine the current status of encryption, use the ”sh stat daos” command from the server console.

There are some storage area network devices that have the ability to deduplicate files. If DAOS_ENCRYPT_NLO=0 is not specified, deduplicating will not occur across Domino servers. The encryption will make even duplicate files unqualified for deduplicating because they will encrypted with each server's key.

Compression



While DAOS is compatible with compression, there are a couple of points to remember:
    1. It's possible for an attachment to disqualify itself from DAOS consolidation by compressing to a size smaller than the Minimum Size setting. 2. The same attachment, undergoing different compression types, LZ1 versus Huffman versus no compression, will be seen by DAOS as different objects and will, therefore, be shared only with others of like type.

Resynchronization of the Catalog



In order to ensure that NLO files are not physically deleted when there are still ticket holders referencing them, if there is any reason to question the accuracy of a reference count, DAOS puts itself in a safe mode whereby no deletes are allowed to proceed. This state is signaled to the administrator via the Domino Domain Monitoring systems and is reported as a “NEEDS RESYNC” state from the “tell daosmgr status catalog” server console command.

To perform this catalog resynchronization manually, type “tell daosmgr resync” from the server console.

Note: The duration of a resync can be significant and depends on the number of DAOS-enabled databases, the number of NLO files in your environment, and your system configuration. It can take several hours to complete causing its execution to overlap into normal business hours. Although Domino and DAOS are functional while a resync is in progress, there may be a degradation in performance while it is running.

Prior to 8.5.2, if you find it necessary to halt the resync operation, you can interrupt it by issuing “tell daosmgr quit” from the server's console. However, for continued operation of DAOS, immediately restart the DAOSMGR task using the console command “load daosmgr.” When it is convenient to continue the resync operation, issue “tell daosmgr resync” again from the server's console and resync processing will continue where it left off.

8.5.2 introduces many improvements to the performance of the resync operation. Resync will complete sooner in this release and improvements to keep the catalog in sync have been made. Resync can now be scheduled to work with in a time window by setting DAOS_RESYNC_START_TIME and DAOS_RESYNC_STOP_TIME. Setting a time window will eliminate the need to stop and start daosmgr when resyncing the catalog.

Typically, the only downside to being in "Needs Resync" state is that DAOS suspends pruning operations. Unless you are experiencing errors, it is usually better to wait to do a resync until the next available maintenance window.

Antivirus



Consider DAOS interaction with antivirus scans. It's critical the DAOS base path and .NLO file extension have the same anti-virus policy as the Domino data directory and .NSF file extension. If the two file types have different policies, consolidating attachments from an existing NSF using “compact -daos on -c” can result in NLO files being quarantined as they're extracted from the NSF. A user who then opens a quarantined attachment would see a “Missing NLO” error message.

A Domino add-in antivirus program is recommended so that the attachments are scanned as they pass through the server. You should not scan the NLO files directly. (Similarly, the transaction log files should be excluded from scans.)

The screenshot below shows how you'd configure Symantec Antivirus to exclude the .NLO file extension. From the Configure folder on the left, select File System Auto-Protect. Click the Exclusions button, then the Extensions button. Type “NLO” without the quotes and click Add.



Recommended versions of Notes Client and Domino Server



For the Notes Client anything before 8.5.1 or post 8.5.1 FP3 (including this version) should be used. For the Domino Server, the minimum version should be 8.5.1 FP3. For more information see the Technote 1446397 "Attachment corruption related to DAOS and Domino 8.5.1 ".

Ideally the Domino Server should be at 8.5.2 to gain both operational and performance improvements made to DAOS catalog synchronization as well as other DAOS improvements.

Worse practices



1. Too small a minimum participation size - getting too "greedy"
Limit the participation size to what will resulting in reasonable disk space amount with the least number of nlo files. Let's review the following output from the "Domino Attachment and Object Service Estimator":

DAOS Minimum Size versus number of NLO's and Disk Space:

0.0 KB will result in 2226347 .nlo files using 185.5 GB
64.0 KB will result in 1092894 .nlo files using 175.7 GB
128.0 KB will result in 708403 .nlo files using 163.6 GB
256.0 KB will result in 422087 .nlo files using 145.9 GB
512.0 KB will result in 219833 .nlo files using 120.2 GB
1.0 MB will result in 93628 .nlo files using 87.8 GB
2.0 MB will result in 36576 .nlo files using 56.6 GB
3.0 MB will result in 17499 .nlo files using 38.0 GB
4.0 MB will result in 9717 .nlo files using 26.3 GB
8.0 MB will result in 1576 .nlo files using 6.5 GB

While the theoretical maximum (first line) would generate approximately 2.2M files using 185 GB this would not be the ideal participation size for two important reasons.

The first reason is the benefit of DAOS can be realized without having to maintain 2.2M NLO files. A more reasonable participation size between 128k-256K would result in the number of NLO files in the range of 422,087 to 708,403. Taking an average of the two participation sizes, a value of 192K would result in approximately 500,000 NLO files with a disk space size of about 150 GB. The net result would be an 80 % total size yield with a little less than 1/4 of the theoretical maximum NLO files to maintain in your backup/restore procedures.

The second reason is that reducing the minimum participation size is more easily done then increasing the minimum participation size.

2. Deleting the daoscat.nsf and/or daos.cfg to "fix" problems.
The daoscat.nsf and the daos.cfg are vital files for DAOS. daoscat.nsf keeps data about the location of NLO files with reference counts. daos.cfg maintains data about the DAOS configuration. If one or both of these files are deleted, DAOS will rebuild the files; a processes that will take hours to complete. While the files are being recreated the location of individual NLO files and the reference count will be in transition which would result in intermittent access to attachments. Removal of corrupted daoscat.nsf should only be done when all other options for getting the file back have failed.

3. Long deferred deletion intervals set on a hub server.
When mail routing or replication hub servers are DAOS enabled, NLO files will be created and marked for deletion. Until Prune is run as scheduled by the "Deferred Deletion Interval", the files will exist in the DAOS Repository.

4. Failing to monitor the DAOS catalog state.
DDM events are generated for DAOS state changes including the state of the DAOS catalog. It is important to keep the DAOS catalog in a synchronized state to make sure prune can run and to keep access to the DAOS manager functioning correctly. Make sure to monitor the state of the DAOS catalog either with via DDM events or directly via the server console with "show daosmgr status catalog" and make sure the catalogState reports "SYNCHRONIZED".

5. No Backup/Restore procedures.
It is worth repeating the phrase timing is everything regarding backup and restore. It is important to time the backup of nsf and nlo files to be more frequent than the "Deferred Deletion Interval" because prune will remove the NLO files. Another important point is to not only have a well planned Backup/Restore procedure implemented, but to validate the restore procedure regularly to ensure that data can be restored before there is an emergency. For more information refer to the DAOS Backup and Restore

Transaction logging and DAOS


    System Database File
    Should DAOS be enabled on this database?
    Should Archive transaction logging be enabled on this Database?
    Should Circular transaction logging be enabled on this Database?
    Comments
    NAMES.NSF
No
Yes
Yes

    LOG.NSF
No
No
Yes
    If this is not transaction logged consider deleting it after a crash as it could impact startup time. It may require a fixup and this could take an extremely long time if log.nsf is very large.
    ADMIN4.NSF
No
Yes
Yes

    MAIL.BOX
Yes
Yes
Yes
    If you want to use DAOS and not transaction log mailbox bodies you can use the following in 8.51. NSF_DONT_LOG_MAILBOX_BODY=1 RM_NO_LOG_LARGE_OBJECTS=1 RM_NO_LOG_OBJECTS_IN_MAILBOX=1 
    DBDIRMAN.NSF
No
No
No

    CLUBUSY.NSF
No
No
Yes

    DDM.NSF
No
No
Yes

    STATREP.NSF
No
No
Yes

    STATMAIL.NSF
No
No
Yes

    CLDBDIR.NSF
No
No
Yes

    WEBADMIN.NSF
No
No
Yes

    CLDBDIR.NSF
No
No
Yes

    BUSTIME.NSF
No
No
Yes

    EVENTS4.NSF
No
No
Yes

    STATLOG.NSF
No
No
Yes

    REPORTS.NSF
No
No
Yes

    MTSTORE.NSF
No
No
Yes

    ACTIVITY.NSF
No
No
Yes

Friday, November 11, 2011

Domino Tuning Tips (all platforms)

The following tips will help Administrators to optimize their Notes/Domino environment. It is important for Administrators to understand that any configuration change should be closely monitored for its effects. Make single changes at a time to ensure adverse effects are easy to spot and changes can be backed-out.


View Index Updates


It is possible to tune the Domino update task to take advantage of the additional CPU resource of multi-core hardware. There are a several NOTES.INI parameters that you may use:

Update Task (View Index Updates)


If the system’s resources allow it, additional Update tasks may be started on the server, by adding the parameter, Updaters=[number], to the NOTES.INI. As a rule of thumb, do not exceed a maximum of one Update task per CPU. For example, to enable two Update tasks, apply this parameter:
  • Updaters=2

Full Text Index Updates


A separate thread can be created solely for full text updates. This separate thread will only be responsible for updating full text indexes, so view updates will continue to occur even if a large full text index is being rebuilt. Consider applying this NOTES.INI parameter if many full text indexes are present on a Domino server and sufficient CPU resource remain:
  • UPDATE_FULLTEXT_THREAD=1

Maintaining view indexes


The NOTES.INI parameter DEFAULT_INDEX_LIFETIME_DAYS=[number of days] allows administrators to set a default lifetime for database view indexes if none was selected by the database designer in the Design View Attributes dialog box.

The default value for this parameter is 45. Setting this to a lower value can have two benefits:
  • Save Disk Space
  • Reduce amount of work Update task must perform

Creating view indexes is disk intensive and requires CPU and memory resource, so it is important for Administrators and designers to strike a balance. It is not recommended to set this value to lower than 14 (two weeks).

Disable Transaction Logging For Certain Databases

Turning transaction logging may impact your enabled DAOS features relative to mail. In most cases, disabling Transaction Logging is not recommended because you lose the benefits of fast server restart. Disabling transaction logging on a database will cause Fixup to run on that database, creating the potential for slow restart. Furthermore, you will not be able to perform incremental backups using the transaction logs for that database.
For some databases, however, it might not be necessary to enable transaction logging. This is because it generates additional transaction log traffic on the disk and causes the transaction log drive to fill up quicker. Examples of databases that are suitable for disabling of transaction logs are:
  • Mail boxes (mail.box) - Do not disable Transactional Logging on Mail.boxes if you are running DAOS.
  • Server Log (log.nsf)
  • Monitoring Results (statrep.nsf)
  • Statistics Mail-in (statmail.nsf)
  • Web Administration (webadmin.nsf)
  • Schedule (clubusy.nsf or busytime.nsf)
  • Cluster Database Directory (cldbdir.nsf)

Table 1 highlights NOTES.INI parameters to force new instances of the certain system databases created on server startup to have transaction logging disabled. They do not disable transaction logging on existing databases, or if they are created manually.

    NOTES.INI Parameter
    System Database
    MailBoxDisableTXNLogging=1
    Mail.box (See note below)
    Log_DisableTXNLogging=1
    Log.nsf
    Schedule_DisableTXNLogging=1
    Clubusy.nsf or busytime.nsf
Table 1: NOTES.INI parameters to disable database transaction logging.
Note: Do not disable Transactional Logging on Mail.boxes if you are running DAOS.

Replication


If you have more than one Domino server in your environment then you will need to set up replication. The following tips should help optimize the time and resource requirements to replication data around your environment.

Multiple Replicator Tasks


The database replicator (replica) task on a server is responsible for handling scheduled replication requests, as configured in server connection documents. The Cluster Replicator (clrepl) task is event driven and responds to changes in a database. These changes are then replicated to other cluster members.

Each replicator task only replicates with one server at a time, therefore the first replication must complete before the next one can start. The number of concurrent replicator tasks running can be set with the following NOTES.INI parameter:
  • Replicators=[number]

In environments where more than two servers participate in the cluster, additional clrepl tasks can be enabled. The number of concurrent clrepl tasks running can be set with NOTES.INI parameter:
  • Cluster_Replicators=[number]

Administrators should monitor Replica.Cluster Domino statistics. For instance, the WorkQueueDepth Domino statistic indicates the number of changes waiting to be replicated to cluster members. If it is continually increasing, enable additional clrepl tasks.
The following statistics give indication on the current, average and peak Cluster work queue depth:
  • Replica.Cluster.WorkQueueDepth
  • Replica.Cluster.WorkQueueDepth.Avg
  • Replica.Cluster.WorkQueueDepth.Max

Replication Triangulation


When a client or server replicates with a remote server, it keeps a log of the name of the remote server and the time and date in the Replication History. Domino uses the replication history to determine which documents to scan for changes during the next replication. The purpose of Replication Triangulation is to make each server aware of every other server which maintains a replica of the same database, and which has had a successful replication.

In a large environment (hundreds of servers in a domain), the number of replication history events to maintain can cause a significant performance impact to the Domino server. Maintaining replication triangulation history for databases which exist on hundreds or thousands of servers is too expensive. This can manifest itself in the form of increased CPU activity for the replica task.

You can disable replication triangulation with the following server NOTES.INI parameters (available in Domino 7.0 and later)
  • NSF_REPLHIST_NO_TRI=1
  • REPL_NO_WS_TRI_HIST=1
  • REPL_NO_REMOTE_TRI_HIST=1

For local replicas, use Notes Client NOTES.INI parameters:
  • NSF_REPLHIST_NO_TRI=1 [This will prevent existing triangulated entries from being read]
  • REPL_NO_WS_TRI_HIST=1 [This will prevent new triangulated entries from being written]
Note: After setting the NOTES.INI parameters, the replication history must be purged from each replica of the databases affected.


Further information is available in IBM technote 1270104.

Internal Caches


In Domino, caches are used to store frequent lookups in memory, to prevent the data being continually read from disk. Since the mechanical hard-disk is commonly the slowest component of the server, caches help improve performance and make a more responsive user experience. The following sections provide Administrators with tips to ensure cache sizes are properly set.

NLCache


NLCache is the name lookup cache. The default is 16MB before Domino 8.5.2. Beginning in Domino 8.5.2, the default is 64MB. It can be increased as needed up to 4GB.

To determine if you need to increase the NLCache, use the show stat database or show NLcache console command. For example:
Database.NAMELookupCacheCacheSize = 16,447,205 (current size of cache)
Database.NAMELookupCacheLookups = 1,879,903
Database.NAMELookupCacheMaxSize = 16,777,216 (default maximum size)
Database.NAMELookupCacheMisses = 1,362,746

A relatively high number of misses compared to lookups indicate that you should increase the maximum allowed cache size. Administrators should increase the size of the NLCache in increments, so try doubling the default to begin with, if necessary. Modify using the ini parameter:
  • NLCache_Size= For example, set 67108864, which sets NLCache_Size to 64MB.

Group Cache


When the server needs to lookup the members of a group (For example, in the event of an authentication request) it first checks the Group Cache. It will store results in the Group Cache to optimize performance. Groups are invalidated during updates or when cache is full. The default size is 4MB and it can be increased to 15MB.

If the cache needs to rebuild frequently because not enough data can be cached, this can slow down group lookups. Therefore, frequent or very large group updates can slow down server.

Verify GroupCache statistics with “Show stat net”
NET.GroupCache.Hits = 155
NET.GroupCache.Misses = 10
NET.GroupCache.NumEntries = 9
NET.GroupCache.Size = 65,406
NET.GroupCache.Used = 2,716

NET.GroupCache.Misses indicates the number of times a group was not found in the cache and so had to be read from disk. Administrators should increase the size of the Group Cache in increments until the number of misses reduces. Modify size with the NOTES.INI parameter:
  • Group_Cache_Size= For example set 15360 , which sets Group_Cache_Size to 15MB.

Unified Buffer Manager (UBM)


The UBM (formerly referred to as the NSF Buffer Pool) is used to increase performance by caching disk I/O requests to all databases on a server. The UBM is typically the largest contribution of shared memory usage on the server. In earlier releases of Domino, the maximum size of the UMB was automatically calculated as 3/8th's of the physical RAM in a server. In a server with many gigabytes of RAM, this could result in an unacceptably high UMB upper-limit. Consider a server with 4GB RAM installed:
Max UMB Size = (4 x 3/8) = 1.5GB.
If the UMB were to grow to its maximum size of 1.5GB, there is a risk that shared memory may exceed process limits and result in a server instability.
Domino 8 now enforces a maximum UBM size of 512 MB default on Windows32, Linux, AIX, Solaris, i5/OS and z/OS platforms. On 64-bit Domino, the maximum UBM size is 1024 MB. Therefore, there is no need to take action to properly size the UBM in Domino 8. For additional details on this new behavior, see IBM technote #1268988.
For recommendations on setting NSF_BUFFER_POOL_SIZE_MB, read IBM technote 1286171.


NSF_DbCache_MaxEntries


The NSF_DbCache_MaxEntries determines the number of databases that a server can hold in its database cache at one time. If your server has sufficient memory, you can improve the performance of the server by increasing the number of databases that Lotus Domino can cache in memory at one time. The default value is 25 or the NSF_Buffer_Pool_Size divided by 300 KB, whichever value is greater.

You should monitor the Database.DbCache.Hits statistic on your server. This indicates the number of times a database open request was satisfied by finding the database in cache. A high value indicates database cache is working effectively. If the ratio of Database.DbCache.Hits to InitialDbOpen is low, you might consider increasing NSF_DbCache_Maxentries.

For detail information on how to set the number of databases cached simultaneously in Lotus Domino, read IBM technote 1279893.

NSF Monitor Pool


The Monitor Pool caches event monitors which include server and client mail rules. If set too low the Monitor Pool can become full. Side effects include causing new calendar notices not to display in the Calendar Mini View.

The default pool size is 40 MB. The maximum value for this pool is 256/512 MB.

To see the current values review stats
Database.MonitorPool.Event.Used = 52309
Database.MonitorPool.Monitors.Used = 1184
Database.MonitorPool.Size = 41943040

Check the LOG.NSF for the error “Insufficient memory – NSF monitor pool is full” to know if you should increase this pool for the Domino server. Many customers find a value between 80 and 100 MB to work best. Increase this value if you are approaching the maximum by using the NOTES.INI setting:
  • NSF_MONITOR_POOL_SIZE_MB=

Multiple Mail Boxes


Additional mailboxes are required on a server only when the amount of mail traffic prevents the router from being able to effectively access the mail box files and access conflicts are encountered. Access conflicts occur when other threads or processes lock the mailbox, while another entry tries to access the mailbox and is denied.

The typical number of mail boxes on a mail server ranges from one to four, depending on the volume of mail. IBM technote 1148438 provides Administrators with the procedure to determine whether additional mailboxes are required on busy servers. If you feel you are in need of another mailbox. Change the server configuration to 2 mail boxes.

Tips for Server Based Mail Rules


Keep the number to the absolute minimum, based on business requirements. Rules require processing by the mail router each time it delivers mail. Therefore the fewer rules, the fewer server resources are required.
Place rules most likely to be triggered at the top of list and use the “Stop processing further mail rules” option.
Searching the memo body is CPU intensive and will cause and increase in the CPU used by the SMTP task

Use the following NOTES.INI parameter to cap the number of mail rules per user's mailfile or server mail.box:
  • MailMaxFilters=<1 to 100>

This IBM Case study provides Administrators with an indication to the impact mail rules have on a server.

Tuning User Sessions


There are two NOTES.INI parameters that define how many client user sessions a server can have concurrently open and also how long a session remains open. Keeping sessions open requires memory on the server, whereas opening a new session consumes a small amount of CPU resource. Therefore a balance must be achieved.
  • Server_MaxSessions=[number] (Maximum number of concurrent open sessions)
  • Server_Session_Timeout=[number minutes] (maximum duration of idle activity before the Domino server terminates the session)

A timeout value of 30-45 minutes is recommended for most customers. For more information read IBM technote 1089879.

Domino Configuration Tuner (DCT)


The Domino Configuration Tuner (DCT) provides easy-to-use self-service configuration analysis for more robust installations and better performance of Domino servers. Make sure you are up-to-date with suggested changes from Lotus Domino by periodically updating DCT. For more information, read "Domino Configuration Tuner (DCT) provides easy-to-use self-service configuration" - IBM technote 4019358.

Tuesday, November 8, 2011

Optimizing server performance (Top 10 ways to improve your server performance sidebar)

Top 10 ways to improve your server performance (sidebar)
By analyzing a variety of NotesBench reports, published over the last two years by NotesBench Consortium members, we came up with a list of the top 10 ways you can improve the performance of your server. The list shows you how to improve your server capacity and response time.
  1. Make sure your server memory matches the number of users you want to support. Most NotesBench vendors use 300K-400K per active user. They also set their NSF_BUFFER_POOL_SIZE to the maximum for their memory configuration. This setting isn't necessary, because the Domino server initially obtains a quarter of available memory and grows only if necessary (depending on the load). You should use published physical memory configurations as a ceiling for memory configuration decisions.
  2. Distribute I/O among separate devices. For example, you can put the OS kernel on one drive, the page file on another, the Domino executable on a third, and finally the Domino data files on a fourth drive. In some cases, NotesBench vendors point their log.nsf file to a location different from the default data directory (using the log= setting in the server's NOTES.INI file).
  3. I/O subsystem improvements. For example you can:
    • Move from EISA-based systems (such as, controllers) to PCI-based systems
    • Exchange EISA/PCI boards in favor of PCI-only boards (this way, lower speed EISA devices won't decrease the I/O throughput)
    • Use striping to improve performance
    • Use multiple I/O controllers to distribute logical volumes (and use file pointers to databases across separate controllers). Make sure you have the latest BIOS for your I/O subsystem. This is an inexpensive way to remove a likely throughput bottleneck.
  4. Use faster disk drives. You can improve disk drive speeds from 5,400 rpm to 7,200 rpm. For most Windows NT systems, NotesBench vendors use 2GB disk drives. For Solaris and IBM Netfinity systems, the drives were larger: 4GB. For AS/400, the drives were even larger: 8GB.
  5. Increase the stripe size. NotesBench vendors use a stripe size of 8K (Digital's systems) or 16K (IBM Netfinity reports). (The IBM Netfinity report provides additional information on I/O settings such as w IOQ Depth, Outbound Posting, PCI Line Prefetch, and Address Bit Permitting.)
  6. Use faster CPUs. NotesBench vendors have moved beyond the Pentium, Sparc, and PowerPC processors, which were in the 100-200Mhz range, to higher speed processors. However, they consistently use P6-based systems over the Pentium II systems for high-end Domino server loads. The size of your Level 2 cache should match your expected user loads and the response time you want. Vendors have moved from 256K to 512K, 1MB to 2MB Level 2 cache systems, especially on their greater than two-CPU configurations.
  7. Improve your network. NotesBench vendors have:
    • Moved from 10Mbps cards and networks to 100Mbps configurations
    • Used multiple LAN segments (one for each partition) to isolate network traffic, at the high-end user loads
  8. Change your network protocol to IP. Vendors were initially (two years ago) using NetBIOS and SPX internally, but have unanimously moved to IP for their performance publishing efforts.
  9. Upgrade to a newer release of Domino. NotesBench vendors have moved from Domino Release 4.5a SMP version to Domino Release 4.52B SMP version for higher capacity results. The first Domino Release 4.6a result (AS/400) on a RAID5 configuration indicates a reliable configuration can still provide competitive response time with a properly designed I/O architecture.
  10. Use Domino partitioned servers. NotesBench vendors have increased scaling of active user loads and leveraged their more powerful configurations (faster clock cycles, fiber-connected I/O subsystems, OS kernel to CPU binding, and multiple I/O controllers) by using partitioned servers.
How we came up with these recommendations
To understand how we came up with our top 10 list, we will take you through the performance analysis of Number 2 in the list -- to distribute I/O among separate devices. Initially, many vendors placed the kernel, page, and Domino executables on one volume and the Domino data files on another. However, both volumes were on the same controller. Lately, the NotesBench reports show improvements in performance when the volumes are separated across multiple controllers, and individual volumes are separated across disks. What this means is that we found that vendors put the OS kernel on one drive, page file on another, Domino executable on a third, and finally the Domino data files on a fourth drive. In some cases, they pointed their log.nsf file to a location different from the default data directory (using the log= setting in the server's NOTES.INI file). Vendors who distributed the I/O over several disk drives had better server performance overall, and could support a higher capacity of users. For example, in a NotesBench report published in May of 1996, Digital Equipment Corporation set up a server with the following specifications:
  • CPUs: four 133Mhz CPUs
  • Memory: 512MB
  • Domino: Release 4.1
They placed the operating system and the Domino executable on drive C:\, the page file on drive D:\, and the Notes\data directory on drive E:\. They could support a maximum capacity of 1,500 users with this configuration.
In a NotesBench report published in September of 1997, IBM Corporation set up a server with the following specifications:
  • CPUs: three 200MHz1Intel Pentium Pro processors
  • Memory: 1GB2
  • Domino: Release 4.51
They placed the operating system on drive C:\, the page file on drive C:\, the Notes\data directory on drive E:\, and the Domino executable on drive E:\. They supported a Mail-only workload of 3,500 active mail users. In a four-processor configuration, they supported a MailDB workload of 2,900 active users. These examples led us to the conclusion that distributing I/O over several disk drives had better server performance overall, and could support a higher capacity of users. We went through many other NotesBench reports to collect the data shown in our top 10 list. You can visit the NotesBench Web site yourself to view published data and test results. Visiting the site may help you to come up with other ways to improve your server's performance.
ibm.com

Monday, November 7, 2011

Using the Message Recall feature in IBM Lotus Notes and Domino V8

The ability to recall mail messages is one of the most requested features for IBM Lotus Notes and Domino V8. Enabled on the server and client by default, this new feature lets you recall mail messages that were sent in error. This article explains how Message Recall works, discusses how it is configured and controlled, and covers the finer points of planning and deploying the feature.
How Message Recall works: The basics
Message Recall is simple to use from a user's point of view. You simply open the Sent view of your Lotus Notes V8 mail file, highlight the mail you want to recall, and click the Recall Message button in the Action bar (see figure 1).

Figure 1. Action bar showing Recall Message button on the right

A dialog box then displays that shows the original recipients, any of which you can deselect or choose to get a response (see figure 2).

Figure 2. Recall Message dialog box

After you click OK, the recall request is acknowledged with a dialog box. If everything is set up properly, the messages are removed from the recipients’ mail. You receive a report telling you which messages were recalled (and whether they had been read) and whether any messages could not be recalled and why. Now let's investigate what’s behind this simple process.
Configuring Message Recall
Lotus Notes and Domino V8 ship with Message Recall enabled. Whether you build a new system from scratch or upgrade servers and users, this feature is ready to use. If you have a large system that will be upgraded over a period of time, you may want to disable the feature at first, to allow time to plan for user support and training.
The Server Configuration document is used to configure Message Recall. This central location lets you easily allow or disallow it for all servers. Mail Policy documents can further refine the settings, as can individual user preferences.
If you do not have any Server Configuration documents, Message Recall is still turned on by default, and the default settings, listed here, apply:
  • Message Recall: Enabled
  • Allow recall of messages with unread status: Unread Only
  • Do not allow recall of messages older than: 14 days
If your deployment plan calls for the feature to be rolled out later, you must create a Server Configuration document to turn it off for now.
To disable Message Recall, simply open the Lotus Domino V8 Server Configuration document and change the Message Recall setting from Enabled, as seen in Figure 3, to Disabled.

Figure 3. Message Recall tab in the Server Configuration document

Requirements for Message Recall
The requirements for Message Recall are:
  • A Lotus Domino V8 or later server containing the mail to be recalled.
  • A mail file based on a Lotus Notes V8 or later mail template with which to recall the message.
  • A copy of the mail message in the sender’s mail file. (This is usually found in the Sent view, but the Message Recall button is also found in the All Documents view.)
  • Permission given at the server to perform Message Recall (and optionally at the policy and the recipient’s mail-file level, where the feature can be turned off).
Message Recall not only works with these minimal requirements but also works even if the recalling user is on a server that is not Lotus Domino V8. All that is required is a Lotus Notes V8 mail template containing the button to recall the message and that the message being called resides on a Domino V8 server. Moreover, interim servers between the recaller and the recipient can be any version because the request at that point is simply an email message.
Limitations of Message Recall
Message Recall works only on mail that is routed over NRPC. This means that neither mail routed to the Internet nor internal mail routed over SMTP can be successfully recalled.
If a Mail Policy exists that limits the use of this feature for a user, this policy applies and can limit the feature’s functionality. If there is no policy, or if the policy allows a user to change the recall setting, each user can choose to disable the feature in his or her Mail Preferences.
Only mail that resides on a Lotus Domino V8 or later server can be successfully recalled. The Lotus Domino V8 router does the work, so a recall request sent to a Lotus Domino V7 server does not succeed, and the recalling user receives a report stating that the server does not support Message Recall.
A copy of the message must be saved in the sender’s mail file to recall it successfully. If the sender did not save the message when it was sent, Message Recall cannot be used. Because signatures are checked (for security purposes), mail must be recalled from the mail file from which it was sent, by the original sender. A delegatee can recall a message from another user’s mail file, but only if the delegatee sent it. This means that a Lotus Notes Administrator or an administrative assistant is not able to recall a message sent by another user, unless that person has access to the user’s ID and password.
If a message has been forwarded by a recipient, the forwarded message is not recalled by the original sender’s recall request because the UNID differs from that of the original. In other words, Message Recall does not chase down forwarded copies of an email. Any forwarded copies must be recalled by the person who forwarded them.
Message Recall is not available with IBM Lotus Domino Web Access. Mail sent to a Lotus Domino Web Access user can be recalled if it is on a Lotus Domino V8 server and the recalling user has a Lotus Notes V8 mail template to use to initiate the recall.
Message Recall works only for email, not for Calendar and To Do items. Users see a pop-up box stating "This message type cannot be recalled" if they attempt to recall a Calendar or To Do item. These items can be cancelled or changed using the Reschedule or Cancel options that exist in all versions of Lotus Notes calendars.
How the feature works: The details
When the original sender clicks the Recall Message button from the Sent (or All Documents) view, the Lotus Notes client creates a recall request for the highlighted message. This request is mailed to each recipient or group that the user chooses. (Note in figure 2 that the addresses of Internet users are displayed, and a recall request can be mailed, but Message Recall does not succeed for these users.)

Tip

If you plan to run a mixed-version system for a while, consider putting a mail rule on your pre-Lotus Domino V8 SMTP servers to reject these messages. Otherwise, recall reports are sent to the Internet, where they may cause confusion for the recipient. Such a mail rule can be set to not accept messages with a form containing "recall." You can then disable this rule when the SMTP server is upgraded to Lotus Domino V8.
The document UNID is used to identify the message in the recipient’s mail file. If the server is capable of performing a recall, the message is located and deleted. The router removes the message completely and leaves only a deletion stub, allowing the message to be removed from replica copies as well. Even if the recipient has soft deletions enabled, the message is never left in the Trash folder. Because the UNID of the document is used to locate the message, any copies that have been moved to folders are also removed.
The recalling user receives a report with the results of the recall attempt. If the message was successfully deleted, the report contains this information, noting whether or not the message was read. If the recall attempt fails, the report states the reason it failed. The recalling user has two very similar notices in his or her mail file, one in the Sent folder and one in the Inbox, after the message is recalled. Both are needed to properly inform the user which messages were recalled, so users need to be told to retain both notices until they are satisfied with the results of the recall. Both notices can then be deleted.
The recipient of the original message is not notified that a recall has occurred. The message is simply no longer in the recipient's mail file.
If you attempt to recall a message that was sent using SMTP, the Lotus Domino V8 router sends you a Non-Delivery Report stating “Message Recall Requests cannot be routed via SMTP.”
Controlling Message Recall
Although the Message Recall feature is enabled, disabled, and configured at the server level, it can be refined and controlled at other points, providing companies the flexibility to use the feature the way they want.
Controls start at the Server Configuration document, in which the feature is enabled or disabled; selections are available to allow the recall of read, unread, or both; and time limits are available ranging from weeks down to minutes.
Mail Policy control is available for Lotus Notes Administrators. The Message Recall options are as follows and are shown in figure 4:
  • User is allowed to recall sent messages: yes or no.
  • Other users are allowed to recall messages they sent to this user: yes or no.
  • Allow recall of messages with unread status: Unread only or Both read and unread.
  • Do not allow recall of messages older than: specify number of weeks, days, hours, or minutes.

Figure 4. Mail Policy settings in the Server Configuration document

The Mail Policy can be applied to a subset of users who may be under legal obligation to keep all messages, or perhaps applied to limit who is allowed to recall messages at all. The Mail Policy overrides the settings in the Server Configuration document. For instance, if the server allows unread mail to be recalled, but the Mail Policy applied to a user allows unread and read mail to be recalled, the Mail Policy applies, and both unread mail and read mail can be recalled. This provides very specific control for individual users.
Unless Mail Recall is disabled at the server or via a policy, users can also control Message Recall. Under User Preferences - Basics, they can select or deselect “Allow others to recall mail sent to me.” If the user deselects this option, the setting causes a notice to be sent back to the recalling user stating that the message cannot be recalled.
Message Recall and legal compliance
Message Recall is a new concept to many organizations that use Lotus Notes and Domino. Questions have arisen over whether this new feature conflicts with legal requirements, specifically many of the newer laws concerning the retention of email.
Solutions that are designed for compliance purposes usually require that all mail be journaled as it is sent, before it arrives at the recipient’s file. Message Recall does not affect this type of solution. In fact, not only is the original message journaled, the recall request is journaled also.
If the original message is ever needed for compliance purposes, it is in the mail journal and in the off-site storage that is part of your compliance solution. As added assurance, Message Recall requests themselves cannot be recalled. Thus, a robust mail compliance solution should have no problem dealing with Message Recall, and many companies already have custom-built or other mail recall solutions that are fully compliant.
Understanding the finer points
Because Lotus Notes mail can be read offline, with the user disconnected from a network, the read and unread status of a local replica of a mail file can be different from that of a server-based copy until replication takes place. Thus, a recall request sent to the server may report that a message was unread when, in fact, the user read it offline.
Mobile devices can present a challenge to users wishing to recall all copies of a message. Unless a handheld device is configured to process deletions from the server-based mail, messages are not removed from the handheld device. The owner of the handheld device controls this setting. Also, depending on the vendor, Message Recall may not be possible from the device itself.
Mail sent to users with ambiguous names is not recalled; the recall message cannot determine which recipient was the one selected by the sender originally.
Mail can be recalled from groups successfully, but if the group membership changes between the time the mail was sent and recalled, recall notices may not be sent to original recipients who have since been removed from the group, and recall notices may be sent to users who never got the original message. Also, you may recall messages from groups that include Internet addresses, but only those users in the group with valid Lotus Notes addresses receive a recall notice. The other requests result in failure notices.
Finally, messages can always be printed and forwarded or even have screenshots made of them. Message Recall is not effective in these cases. Message Recall is not guaranteed to get rid of any and all traces of a message; rather, it is intended to give users the ability to recover from errors made in sending messages.
Conclusion
Lotus Notes and Domino V8 Message Recall is yet another tool for users, and one that may become quite popular. When or if you choose to deploy the feature, consider offering training to your users so they are able to use it effectively.
www.ibm.com

Move a Lotus Domino server to a new certifier without a reinstall

Charles TANABAL
Suppose that there is a company name change and you must to move your Lotus Domino server to a new certifier, without changing any server settings. Learn step-by-step how to move your Lotus Domino server without having to reinstall it or change any settings using the Server1/ExistingCertifier will be moved to Server1/NewCertifier example.
  1. Create a new certifier (/Newcertifier).
  2. Be sure to cross-certify in both directions (/Newcertifier-> /ExistingCertifier and /ExistingCertifier->/Newcertifier).
  3. Create a new server registration with the new certifier (Server1/NewCertifier).
  4. In your "all servers documents" view, duplicate the Server1/ExistingCertifier server document.
  5. Related resources from SearchDomino.com:
    Securely connect Lotus Domino servers on different domainsLotus Domino Server performance-tuning pointers
    Top 10 Lotus Notes Domino administration tips of 2007
  6. Edit the duplicate document and change the server name to "Server1Temp/NewCertifier."
  7. Edit the server document for Server1/Newcertifer (the newly registered document) and copy the value of the field "Certified Public Key" to your clipboard, under the Administration tab.
  8. Next, paste in your new server document by replacing the
    • "Certified Public Key" value in the Server1Temp/NewCertifier server document.
    • Delete the Server1/NewCertifier server document.
    • Edit the document for Server1Temp/NewCertifier and change the name to Server1/Newcertifier. Save and close out.
    • Duplicate all connexion documents where Server1/ExistingCertifer is the source or destination server.
    • Next, change Server1/Existingcertifier to Server1/NewCertifier in all duplicated documents.
    • With your admin client, add Server1/NewCertifier in the access control list (ACL) of all Lotus Notes databases marked with 'Manager' access. Specify it as "Administration Server," using the advanced tab for MailFiles and other Lotus Notes databases, if needed.
    • Replicate names to all Lotus Domino servers.
    • Use a code-generated button to send an email to all Lotus Notes users with Server1 as their home server, so that they can change the Mailserver Field in their location document. Below is the LotusScript code to create this button. Note: Clicking this button will change the servername information in all location and connections documents of the local address book. Domino administrators should adapt this code to meet their own specifications.
      Sub Click(Source As Button) 
      Dim session As New NotesSession 
      Dim db As NotesDatabase 
      Dim view As NotesView 
      Dim doc As NotesDocument 
      Dim nextdoc As NotesDocument 
      Dim destLocation As Variant 
      Dim privnab As New NotesDatabase
      ("","names.nsf") 
              
      If Not (privnab.isopen) Then 
                      privnab.Open "","" 
      End If 
      Set view = db.GetView("($Connections)") 
      Set doc = view.GetFirstDocument 
      While Not doc Is Nothing 
      destServer = doc.GetItemValue("Destination") 
      Set nextdoc=view.GetNextDocument(doc) 
      'THE BELOW LINE MUST BE MODIFIED 
      TO SPECIFY THE RELATIVE SERVER NAME: 
      If destServer(0) = 
      "CN=serverName/O=ExistingCertifier" 
      Then 
      'If you want to remove the document include this line: 
      'Call doc.Remove(True) 
      'If you want to update the Destination
       field include the below instead: 
      Call doc.replaceitemvalue
       ("Destination","CN=servername/O=NewCertifier") 
      'If you want to update the Destination 
      Server Address entry include the below: 
      'Call doc.replaceitemvalue 
      ("OptionalNetworkAddress",
      "<IP address>") 
      'Include the line below if you've 
      included either of the ReplaceItemValue calls: 
      Call doc.save (True,True) 
      End If 
      Set doc=nextdoc 
      Wend 
              
      Set view = db.GetView("($Locations)") 
      Set doc = view.GetFirstDocument 
      While Not doc Is Nothing 
      destServer = Lcase
      (doc.GetItemValue("MailServer")) 
      Set nextdoc=view.GetNextDocument(doc) 
      'THE BELOW LINE MUST BE 
      MODIFIED TO SPECIFY THE 
      RELATIVE SERVER NAME: 
      If destServer(0) = 
      "cn=servername/o=existingcertifier" 
      Then 
      'If you want to remove the document
       include this line: 
      'Call doc.Remove(True) 
      'If you want to update the Destination 
      field include the below instead: 
      Call doc.replaceitemvalue ("Destination",
      "CN=Servername/O=NewCertifier") 
      'If you want to update the Destination 
      Server Address entry include the below: 
      'Call doc.replaceitemvalue 
      ("OptionalNetworkAddress","<IP address>") 
      'Include the line below if you've included 
      either of the ReplaceItemValue calls: 
      Call doc.save (True,True) 
      End If 
      Set doc=nextdoc 
      Wend 
      End Sub
      
    • Replace the "mailserver" field with "CN=Server1/O=NewCertifier" for all users specifying server1 as Homeserver. Stop the Lotus Domino server.
    • Replicate names to all Lotus Domino servers.
    • Stop Server1.
    • Rename the existing Server ID (Server.id) file under the Domino Directory to ExistingCertifier as the extension (Server.ExistingCertifier). Note: Place the ID file that was created during the registration of Server1/NewCertifer and change the name to Server.id to ensure that it corresponds with what is in your notes.ini file.
    • Copy the notes.ini file to Notes.ini.Existingcertifier.
    • Edit your notes.ini file, find Server1/ExistingCertifier and replace it with Server1/newCertifier, and then save and close.
    • Restart your Lotus Domino server.

Securely connect Lotus Domino servers on different domains

Jim MC
One of the many challenges Lotus Notes Domino administrators face is the increased workload that results from having a distributed setup of their IT organization. This is especially true if they are working on multiple domains. When exchanging information across more than one domain, security is a particular concern. This reader-submitted tip explains how to safely and securely connect Lotus Domino servers that are located on different domains through a process called cross-certification.

In the case of Domino servers being located in different domains, administrators can cross-certify the servers to communicate, connect, and exchange information with each other. Cross-certifying allows Lotus Notes users in one domain access to data in another domain -- while simultaneously maintaining security at its highest levels.
Here are the steps you should follow to carry out the cross-certification process:
  1. Create a "safe copy" of an existing user ID file and open your Lotus Notes client.
    Related resources from SearchDomino.com:
    Expert Advice: Cannot connect to new domainProtect
  1. Lotus Notes from malicious code with the Domino ECL
    Lotus Notes Domino Access, Permissions and Authentication Reference Center
  2. From the File menu, locate the User Security option. The location of this menu option will vary, depending on your installed version of Lotus Notes.
  3. Select the Your Certificates tab and also the Export Notes ID (Safe Copy) tab from the Other Actions dropdown list. When prompted, click Save to create the SAFE.ID file. This will create a safe copy of the ID file for a Lotus Notes user in the first domain.
  4. Transfer the created file to the destination Lotus Domino server (i.e. the Domino server located in second domain).
  5. Copy the file to diskette, shared directory folder, CD-ROM, or otherwise, transfer the file to the Lotus Domino server.
  6. Launch the Domino Administrator client.
  7. Select the File -> Open Server menu options to connect to the Domino server.
  8. From the main navigation window, select the Configuration tab.
  9. Now click Certification and Cross-Certify from the right-most side. If the options are not displayed, click on the Tools button to expand the list configuration options.
  10. From the Choose a Certifier dialog window, choose the Certifier ID button and select the CERT.ID file associated with the Lotus Domino server. This is a special ID file that was automatically created when the Domino server was installed. A copy of the file will probably be stored on the Domino server. Select the file and click OK to continue.
  11. When prompted, specify the password associated with the server CERT.ID file and click OK again. You must know this password to continue with the process.
  12. You will now be prompted to select the safe copy of the ID file. This will enable all Lotus Notes users in the first domain to access the Lotus Domino server in the second domain. Click OK after the SAFE.ID file has been selected.
  13. Click the Cross Certify button to generate the cross-certificate for the destination Domino Server Directory. Note that the first time you connect to the destination Domino server you will be prompted to create a digital certification for the destination server. This is a one-time event so just click on the "Yes" button when that message is displayed.

Blank Screen LED Error Codes (HP CQ)

Support details

This document pertains to HP Notebook PCs with the HP Unified Extensible Firmware Interface (UEFI) beginning in late-2008.
On startup, you may see a blank screen. If the diagnostic utilities detect a specific problem with hardware components, the fan turns on, but the screen remains blank. To help identify the cause of the problem, various LED lights on the keyboard blink a series of codes.
The diagnostic utilities use the LEDs near the Num Lock or Caps Lock keys to blink a series of error codes. At the end of the series the blinking stops. The pattern of blinks will occur any time you attempt to start the computer until the error is resolved.
Battery power LED blinks
The Battery power LED indicates the condition of the power supply. When starting the computer, or when the computer is in operation, use the chart to identify the power condition.
Battery Power LEDComponent TestedError Condition
Battery power LED off, and Caps Lock/Num Lock offBattery or AC AdapterAC adapter not connected or failure

Battery low charge or failure
Battery power LED blinkingBatteryInsufficient charge on the battery
When new computer is used for first time, the white LED light for the AC power connector blinks.Battery is still in "Shipping Mode", the light continues to blink even when AC power is connected. 

To resolve, turn off notebook, connect AC power and allow battery to charge for at least 30 minutes, then start computer.
LEDs near Caps Lock and Number Lock keys blink when starting notebook
The LED lights near the Caps Lock and Num Lock keys will blink if an error is detected during the start up process. The LEDs will blink a number of times in a sequence and then stop. The number of blinks in the sequence indicates what component caused an error when it was being tested during start up.
If the LEDs stop blinking and the computer does not start, you can press the power button again to repeat the tests. Count the number of blinks, and use the chart to identify the error condition.
Knowing the number of blinks is helpful when you contact an HP support agent for technical help.
Caps Lock/Num Lock LEDComponent TestedError Condition
LEDs blink 1 timeCPUCPU not functional
LEDs blink 2 timesBIOSBIOS corruption failure
LEDs blink 3 timesMemoryModule error not functional
LEDs blink 4 timesGraphicsGraphics controller not functional
LEDs blink 5 timesSystem boardGeneral system board failure
LEDs blink 6 timesBIOSBIOS authentication failure
Error code explanations
The sections below provide some common explanations for each error code listed in the table above.
When the computer is on battery power only and the AC adapter is disconnected, if the Battery Power and Caps Lock / Number Lock LEDs do not glow, there is either a very low change or no charge in the battery. Connect the AC power adapter, verify that the battery power LED glows, allow the battery to charge for 15 - 30 minutes and then attempt to start the computer. If it starts, run the battery test and calibrate the battery. If it does not start, if possible, connect a replacement battery to verify that the battery is the problem.
If the computer does not start after charging the battery, remove the battery and connect the AC power supply. Verify that the battery power LED glows, and then attempt to start the computer. If the LED still does not glow, either the AC adapter has failed, or there is a bad connection between the adapter and the system board. If possible, connect a different AC power adapter to verify that the adapter is the problem, or contact HP for support which may require a service event.
If the battery light LED (which looks like a lightning bolt  ) flashes, the battery has insufficient charge to start the computer. To resolve this error, try the following solutions.
  • Connect the notebook PC to AC power and attempt to start the computer again.
    • Check the AC adapter to confirm that all of the plugs are securely seated.
    • Determine if the power LED on the AC adapter is lit (if available) to verify that the computer is receiving AC power from the wall outlet.
  • If the computer operates on AC power correctly, charge the battery for thirty minutes to one hour and then restart the computer.
    NOTE:Charging the battery for this length of time is called "trickle charging". Trickle charging is a continuous constant-current charge at a low rate, which recharges the battery slowly when it is in a deep discharge state. Deep discharge occurs when a battery is left unused for extended periods of time.
The computer processor (Blink code 1) has stopped functioning properly. Contact HP for assistance.
If a BIOS corruption error occurs (Blink code 2), you may not even notice the blink codes, because as soon as the computer recognizes the error, it restarts, attempts to recover the BIOS, and then restarts again. You may notice an extra-long startup process as a result, and a message indicating that the BIOS has been recovered may display on startup. If this occurs, update the BIOS on the computer. For more information, refer to the HP Notebook PCs - Locate and Install Updated BIOS, Drivers, and Software .
support document.
If you experience a memory failure (Blink code 3), follow the guide in the table below.
If Using Original MemoryIf New Memory Is Added
Reseat the memory. 

If reseating the memory does not resolve the problem, try replacing the memory with new memory.
Reseat the memory. 

If you continue to experience this error code after reseating the memory, the problem may be with the memory itself. Take the new memory out of the computer, put the original memory back into the computer, and then retest it.
If you do not feel comfortable reseating the memory yourself, take the computer to a computer retailer and ask them to reseat it for you.
NOTE:Some memory module errors may allow the computer to start but will then cause the computer to restart and display a blinking error code.
If you experience a graphics controller failure (Blink code 4), contact HP for assistance.
A general system board failure (Blink code 5) is the failure of a component not covered by the other LED error codes. Contact HP for assistance.
The BIOS authentication error (Blink code 6) is extremely rare. It is the result of a discrepancy between the BIOS and the hardware that is installed on the computer. This error occurs when the BIOS cannot authenticate signatures from the hardware on the system. The purpose of the BIOS authentication is to be sure that no one has tampered with the BIOS on the computer.
If a BIOS authentication failure occurs, the computer automatically performs a BIOS recovery. If the computer does not automatically recover the BIOS, manually perform a BIOS recovery. To manually perform a BIOS recovery, press all four arrow keys at the same time to cause the BIOS to go to the EFI partition to find and recover the current BIOS.