Search This Blog

Friday, November 11, 2011

Domino Tuning Tips (all platforms)

The following tips will help Administrators to optimize their Notes/Domino environment. It is important for Administrators to understand that any configuration change should be closely monitored for its effects. Make single changes at a time to ensure adverse effects are easy to spot and changes can be backed-out.


View Index Updates


It is possible to tune the Domino update task to take advantage of the additional CPU resource of multi-core hardware. There are a several NOTES.INI parameters that you may use:

Update Task (View Index Updates)


If the system’s resources allow it, additional Update tasks may be started on the server, by adding the parameter, Updaters=[number], to the NOTES.INI. As a rule of thumb, do not exceed a maximum of one Update task per CPU. For example, to enable two Update tasks, apply this parameter:
  • Updaters=2

Full Text Index Updates


A separate thread can be created solely for full text updates. This separate thread will only be responsible for updating full text indexes, so view updates will continue to occur even if a large full text index is being rebuilt. Consider applying this NOTES.INI parameter if many full text indexes are present on a Domino server and sufficient CPU resource remain:
  • UPDATE_FULLTEXT_THREAD=1

Maintaining view indexes


The NOTES.INI parameter DEFAULT_INDEX_LIFETIME_DAYS=[number of days] allows administrators to set a default lifetime for database view indexes if none was selected by the database designer in the Design View Attributes dialog box.

The default value for this parameter is 45. Setting this to a lower value can have two benefits:
  • Save Disk Space
  • Reduce amount of work Update task must perform

Creating view indexes is disk intensive and requires CPU and memory resource, so it is important for Administrators and designers to strike a balance. It is not recommended to set this value to lower than 14 (two weeks).

Disable Transaction Logging For Certain Databases

Turning transaction logging may impact your enabled DAOS features relative to mail. In most cases, disabling Transaction Logging is not recommended because you lose the benefits of fast server restart. Disabling transaction logging on a database will cause Fixup to run on that database, creating the potential for slow restart. Furthermore, you will not be able to perform incremental backups using the transaction logs for that database.
For some databases, however, it might not be necessary to enable transaction logging. This is because it generates additional transaction log traffic on the disk and causes the transaction log drive to fill up quicker. Examples of databases that are suitable for disabling of transaction logs are:
  • Mail boxes (mail.box) - Do not disable Transactional Logging on Mail.boxes if you are running DAOS.
  • Server Log (log.nsf)
  • Monitoring Results (statrep.nsf)
  • Statistics Mail-in (statmail.nsf)
  • Web Administration (webadmin.nsf)
  • Schedule (clubusy.nsf or busytime.nsf)
  • Cluster Database Directory (cldbdir.nsf)

Table 1 highlights NOTES.INI parameters to force new instances of the certain system databases created on server startup to have transaction logging disabled. They do not disable transaction logging on existing databases, or if they are created manually.

    NOTES.INI Parameter
    System Database
    MailBoxDisableTXNLogging=1
    Mail.box (See note below)
    Log_DisableTXNLogging=1
    Log.nsf
    Schedule_DisableTXNLogging=1
    Clubusy.nsf or busytime.nsf
Table 1: NOTES.INI parameters to disable database transaction logging.
Note: Do not disable Transactional Logging on Mail.boxes if you are running DAOS.

Replication


If you have more than one Domino server in your environment then you will need to set up replication. The following tips should help optimize the time and resource requirements to replication data around your environment.

Multiple Replicator Tasks


The database replicator (replica) task on a server is responsible for handling scheduled replication requests, as configured in server connection documents. The Cluster Replicator (clrepl) task is event driven and responds to changes in a database. These changes are then replicated to other cluster members.

Each replicator task only replicates with one server at a time, therefore the first replication must complete before the next one can start. The number of concurrent replicator tasks running can be set with the following NOTES.INI parameter:
  • Replicators=[number]

In environments where more than two servers participate in the cluster, additional clrepl tasks can be enabled. The number of concurrent clrepl tasks running can be set with NOTES.INI parameter:
  • Cluster_Replicators=[number]

Administrators should monitor Replica.Cluster Domino statistics. For instance, the WorkQueueDepth Domino statistic indicates the number of changes waiting to be replicated to cluster members. If it is continually increasing, enable additional clrepl tasks.
The following statistics give indication on the current, average and peak Cluster work queue depth:
  • Replica.Cluster.WorkQueueDepth
  • Replica.Cluster.WorkQueueDepth.Avg
  • Replica.Cluster.WorkQueueDepth.Max

Replication Triangulation


When a client or server replicates with a remote server, it keeps a log of the name of the remote server and the time and date in the Replication History. Domino uses the replication history to determine which documents to scan for changes during the next replication. The purpose of Replication Triangulation is to make each server aware of every other server which maintains a replica of the same database, and which has had a successful replication.

In a large environment (hundreds of servers in a domain), the number of replication history events to maintain can cause a significant performance impact to the Domino server. Maintaining replication triangulation history for databases which exist on hundreds or thousands of servers is too expensive. This can manifest itself in the form of increased CPU activity for the replica task.

You can disable replication triangulation with the following server NOTES.INI parameters (available in Domino 7.0 and later)
  • NSF_REPLHIST_NO_TRI=1
  • REPL_NO_WS_TRI_HIST=1
  • REPL_NO_REMOTE_TRI_HIST=1

For local replicas, use Notes Client NOTES.INI parameters:
  • NSF_REPLHIST_NO_TRI=1 [This will prevent existing triangulated entries from being read]
  • REPL_NO_WS_TRI_HIST=1 [This will prevent new triangulated entries from being written]
Note: After setting the NOTES.INI parameters, the replication history must be purged from each replica of the databases affected.


Further information is available in IBM technote 1270104.

Internal Caches


In Domino, caches are used to store frequent lookups in memory, to prevent the data being continually read from disk. Since the mechanical hard-disk is commonly the slowest component of the server, caches help improve performance and make a more responsive user experience. The following sections provide Administrators with tips to ensure cache sizes are properly set.

NLCache


NLCache is the name lookup cache. The default is 16MB before Domino 8.5.2. Beginning in Domino 8.5.2, the default is 64MB. It can be increased as needed up to 4GB.

To determine if you need to increase the NLCache, use the show stat database or show NLcache console command. For example:
Database.NAMELookupCacheCacheSize = 16,447,205 (current size of cache)
Database.NAMELookupCacheLookups = 1,879,903
Database.NAMELookupCacheMaxSize = 16,777,216 (default maximum size)
Database.NAMELookupCacheMisses = 1,362,746

A relatively high number of misses compared to lookups indicate that you should increase the maximum allowed cache size. Administrators should increase the size of the NLCache in increments, so try doubling the default to begin with, if necessary. Modify using the ini parameter:
  • NLCache_Size= For example, set 67108864, which sets NLCache_Size to 64MB.

Group Cache


When the server needs to lookup the members of a group (For example, in the event of an authentication request) it first checks the Group Cache. It will store results in the Group Cache to optimize performance. Groups are invalidated during updates or when cache is full. The default size is 4MB and it can be increased to 15MB.

If the cache needs to rebuild frequently because not enough data can be cached, this can slow down group lookups. Therefore, frequent or very large group updates can slow down server.

Verify GroupCache statistics with “Show stat net”
NET.GroupCache.Hits = 155
NET.GroupCache.Misses = 10
NET.GroupCache.NumEntries = 9
NET.GroupCache.Size = 65,406
NET.GroupCache.Used = 2,716

NET.GroupCache.Misses indicates the number of times a group was not found in the cache and so had to be read from disk. Administrators should increase the size of the Group Cache in increments until the number of misses reduces. Modify size with the NOTES.INI parameter:
  • Group_Cache_Size= For example set 15360 , which sets Group_Cache_Size to 15MB.

Unified Buffer Manager (UBM)


The UBM (formerly referred to as the NSF Buffer Pool) is used to increase performance by caching disk I/O requests to all databases on a server. The UBM is typically the largest contribution of shared memory usage on the server. In earlier releases of Domino, the maximum size of the UMB was automatically calculated as 3/8th's of the physical RAM in a server. In a server with many gigabytes of RAM, this could result in an unacceptably high UMB upper-limit. Consider a server with 4GB RAM installed:
Max UMB Size = (4 x 3/8) = 1.5GB.
If the UMB were to grow to its maximum size of 1.5GB, there is a risk that shared memory may exceed process limits and result in a server instability.
Domino 8 now enforces a maximum UBM size of 512 MB default on Windows32, Linux, AIX, Solaris, i5/OS and z/OS platforms. On 64-bit Domino, the maximum UBM size is 1024 MB. Therefore, there is no need to take action to properly size the UBM in Domino 8. For additional details on this new behavior, see IBM technote #1268988.
For recommendations on setting NSF_BUFFER_POOL_SIZE_MB, read IBM technote 1286171.


NSF_DbCache_MaxEntries


The NSF_DbCache_MaxEntries determines the number of databases that a server can hold in its database cache at one time. If your server has sufficient memory, you can improve the performance of the server by increasing the number of databases that Lotus Domino can cache in memory at one time. The default value is 25 or the NSF_Buffer_Pool_Size divided by 300 KB, whichever value is greater.

You should monitor the Database.DbCache.Hits statistic on your server. This indicates the number of times a database open request was satisfied by finding the database in cache. A high value indicates database cache is working effectively. If the ratio of Database.DbCache.Hits to InitialDbOpen is low, you might consider increasing NSF_DbCache_Maxentries.

For detail information on how to set the number of databases cached simultaneously in Lotus Domino, read IBM technote 1279893.

NSF Monitor Pool


The Monitor Pool caches event monitors which include server and client mail rules. If set too low the Monitor Pool can become full. Side effects include causing new calendar notices not to display in the Calendar Mini View.

The default pool size is 40 MB. The maximum value for this pool is 256/512 MB.

To see the current values review stats
Database.MonitorPool.Event.Used = 52309
Database.MonitorPool.Monitors.Used = 1184
Database.MonitorPool.Size = 41943040

Check the LOG.NSF for the error “Insufficient memory – NSF monitor pool is full” to know if you should increase this pool for the Domino server. Many customers find a value between 80 and 100 MB to work best. Increase this value if you are approaching the maximum by using the NOTES.INI setting:
  • NSF_MONITOR_POOL_SIZE_MB=

Multiple Mail Boxes


Additional mailboxes are required on a server only when the amount of mail traffic prevents the router from being able to effectively access the mail box files and access conflicts are encountered. Access conflicts occur when other threads or processes lock the mailbox, while another entry tries to access the mailbox and is denied.

The typical number of mail boxes on a mail server ranges from one to four, depending on the volume of mail. IBM technote 1148438 provides Administrators with the procedure to determine whether additional mailboxes are required on busy servers. If you feel you are in need of another mailbox. Change the server configuration to 2 mail boxes.

Tips for Server Based Mail Rules


Keep the number to the absolute minimum, based on business requirements. Rules require processing by the mail router each time it delivers mail. Therefore the fewer rules, the fewer server resources are required.
Place rules most likely to be triggered at the top of list and use the “Stop processing further mail rules” option.
Searching the memo body is CPU intensive and will cause and increase in the CPU used by the SMTP task

Use the following NOTES.INI parameter to cap the number of mail rules per user's mailfile or server mail.box:
  • MailMaxFilters=<1 to 100>

This IBM Case study provides Administrators with an indication to the impact mail rules have on a server.

Tuning User Sessions


There are two NOTES.INI parameters that define how many client user sessions a server can have concurrently open and also how long a session remains open. Keeping sessions open requires memory on the server, whereas opening a new session consumes a small amount of CPU resource. Therefore a balance must be achieved.
  • Server_MaxSessions=[number] (Maximum number of concurrent open sessions)
  • Server_Session_Timeout=[number minutes] (maximum duration of idle activity before the Domino server terminates the session)

A timeout value of 30-45 minutes is recommended for most customers. For more information read IBM technote 1089879.

Domino Configuration Tuner (DCT)


The Domino Configuration Tuner (DCT) provides easy-to-use self-service configuration analysis for more robust installations and better performance of Domino servers. Make sure you are up-to-date with suggested changes from Lotus Domino by periodically updating DCT. For more information, read "Domino Configuration Tuner (DCT) provides easy-to-use self-service configuration" - IBM technote 4019358.

1 comment: