When using CUDB Consistency Check, some conditions must be met to ensure that the check runs correctly, and that PLDB or DSG replica performance degradation is avoided. Therefore, make sure that the following conditions are met before using the feature:

• No LDAP schema change is performed while a Consistency Check task is running.

• No software installation, upgrade, or rollback is executed while a Consistency Check task is running.

Immediately before and during the execution of a Consistency Check task, the checked Data Store (DS) units of the PLDB or DSG are automatically monitored for degradation, to avoid overloading the PLDB or DSG replica. The results of such degradation are as follows:

• The check task does not start and the failure is logged if either of the following happens before starting the check:

  • DS unit degradation is detected (to avoid overload).

  • Replication goes down for the checked slave (to limit output size and avoid false positive results).

• An ongoing check task stops, its results are discarded, and the failure is logged if any of the following happens during its execution:

  • DS unit degradation is detected (to avoid overload).

  • Replication stops for the checked slave (to limit output size and avoid false positive results).

  • Mastership change occurs within the PLDB or DSG (to avoid generating invalid output).

• There are a few seconds when the execution of a Consistency Check task can fail to start if either of these happens:

  • A Consistency Check task is ordered immediately after a PLDB or DSG mastership change.

  • A mastership change occurs on the PLDB or DSG just before starting a previously ordered Consistency Check task.

    Note:

    The reason for this limitation is that the replication lag information is not available in the system in the short period following the PLDB or DSG mastership change.

In case any kind of configuration change is performed on a CUDB node (for example, topology changes like adding DSGs, CUDB nodes, moving nodes between sites, changing network connection information), the previously ordered tasks that are in conflict with the changed configuration can be discarded from the PTL. This can be checked by listing the task history with the cudbConsistencyMgr -t command. See Reviewing the Task History for more information.

In addition, if the administrative operation applyConfig is executed after a configuration change, the running consistency check tasks can be aborted if the change concerns configuration data used by the Consistency Check function.

This section shows how to order the check for the replicas. To order a check task, the target slave replica must be fully operational (that is, not degraded), accessible (that is, replication must work), and the PLDB or DSG must have a working master.

Consistency check can be ordered both for DSG and PLDB replicas as described below.

• Use the following command to order a check task for a DSG replica on one of the System Controllers (SCs):

  SC_2_1# cudbConsistencyMgr --order ms --node <nodeid> --dsg <dsgid>

  In the above command, <nodeid> specifies the ID of the CUDB node to be checked, while <dsgid> specifies the DSG that is the subject of the Consistency Check.

• Use the following command to order a check task for a PLDB replica:

  SC_2_1# cudbConsistencyMgr --order ms --node <nodeid> --pl

If the ordering is successful, then cudbConsistencyMgr prints the task ID, which can be used to track the task. Along with the task ID, the command also prints the ID of the CUDB node where the task is executed. In the example below, the node ID is 42:

Task UTC_2014-09-10-11-41-32_N42_U0000000035 is put into the pending task list and will be executed on node 42.

Optional Arguments

The check tasks currently in the system can be listed by executing the following command:

SC_2_1# cudbConsistencyMgr -l

The output of the command uses two lists:

• Pending Task List (PTL) for ordered tasks.

• Running Task List (RTL) for currently running tasks.

See Example 1 below for an example output:

Example 1   Task Listing of cudbConsistencyMgr -l

CUDB_121 SC_2_1# cudbConsistencyMgr -l

[Site 1]

        RTL:
                UTC_2014-09-09-11-00-10_N121_U0000000849
                        checkType=ms,source=S1-N121-D1,check=S2-N122-D1,maxReplicaLag=10000,
                        alarmSeverityLimit=500,verboseMode=off,debugMode=off
        PTL:

[Site 2]

        RTL:
                UTC_2014-09-09-11-00-16_N122_U0000000793
                        checkType=ms,source=S2-N122-D2,check=S1-N121-D2,maxReplicaLag=10000,
                        alarmSeverityLimit=500,verboseMode=off,debugMode=off
        PTL:
                UTC_2014-09-09-11-00-17_N122_U0000000794
                        checkType=ms,source=S2-N122-D2,check=S1-N121-D2,maxReplicaLag=10000,
                        alarmSeverityLimit=500,verboseMode=off,debugMode=off
                UTC_2014-09-09-11-00-21_N122_U0000000795
                        checkType=ms,source=S2-N122-D3,check=S1-N121-D3,maxReplicaLag=10000,
                        alarmSeverityLimit=500,verboseMode=off,debugMode=off
                UTC_2014-09-09-11-00-22_N122_U0000000796
                        checkType=ms,source=S2-N122-D3,check=S1-N121-D3,maxReplicaLag=10000,
                        alarmSeverityLimit=500,verboseMode=off,debugMode=off
                UTC_2014-09-09-11-00-23_N122_U0000000797
                        checkType=ms,source=S2-N122-D3,check=S1-N121-D3,maxReplicaLag=10000,
                        alarmSeverityLimit=500,verboseMode=off,debugMode=off
                UTC_2014-09-09-11-00-23_N122_U0000000798
                        checkType=ms,source=S2-N122-D3,check=S1-N121-D3,maxReplicaLag=10000,
                        alarmSeverityLimit=500,verboseMode=off,debugMode=off
                UTC_2014-09-09-11-00-23_N122_U0000000799
                        checkType=ms,source=S2-N122-D3,check=S1-N121-D3,maxReplicaLag=10000,
                        alarmSeverityLimit=500,verboseMode=off,debugMode=off

CUDB_121 SC_2_1#

The main points of task execution are recorded in the task history locally on the PLDB or DSG master node together with a timestamp and task ID.

Execute the following command to review the node-local history of check tasks:

SC_2_1# cudbConsistencyMgr -t

The output consists of log lines describing when each task was added to the list of pending tasks, when they started and stopped, and what was the result of the execution. The results are ordered by time, and are listed from the oldest to the newest.

2015-01-23 16:39:15+01:00 SC_2_2 UTC_2015-01-23-15-39-15_N119_U0000000000: task found in PTL: checkType=ms,source=S1-N119-D66,check=S2-N120-D66,maxReplicaLag=10000,alarmSeverityLimit=500,verboseMode=off,debugMode=off

2015-01-23 16:39:15+01:00 SC_2_2 UTC_2015-01-23-15-39-15_N119_U0000000000: task execution started

2015-01-23 16:44:00+01:00 SC_2_2 UTC_2015-01-23-15-39-15_N119_U0000000000: task execution finished with result: Successful completion, no difference found. Exit code: 0.

The possible exit codes and their meanings are as follows:

• Successful completion, no difference found. Exit code: 0

• Successful completion, differences found. Exit code: 1

• Termination due to error (for example server timeout, and so on). Exit code: 2

• Successful completion, database checked only partially because the Potential Problem List limit was exceeded. Exit code: 3

Exit codes 0 and 1 mean that the PLDB or DSG database has been scanned to the end, and valid output is written.

Exit code 2 means that the output of the check task has been discarded.

Exit code 3 means that only a portion of the PLDB or DSG database has been scanned, and the output written covers the scanned part.

The list is saved in size limited log files containing about at least two weeks of data.

Execute the following command to try removing an ordered task from the pending task list:

SC_2_1# cudbConsistencyMgr --remove <taskId>

With the above command, the system attempts to remove the task from the list of pending tasks only. It does not remove the task if it is already started and running. Use the --force option to try removing a task from the running task list:

SC_2_1# cudbConsistencyMgr --remove <taskId> --force

In the above commands, <taskId> is the task ID as it appears in the output of cudbConsistencyMgr -l.

The output of the above commands must be similar to the below example:

CUDB_166 SC_2_1# cudbConsistencyMgr --remove UTC_2014-09-22-07-20-04_N166_U0000000011
--force
Task: UTC_2014-09-22-07-20-04_N166_U0000000011
successfully removed from RTL

A consistency check task can be started only if the source and target DS of the checked PLDB or DSG is alive (as displayed by the cudbSystemStatus command), and if they have not changed since the ordering. If a DS state changes from alive to something else (such as degraded, stopped, maintenance), or if the PLDB or DSG mastership changes, then any running task associated with the PLDB, DSG, or the faulty DS is stopped and removed from the RTL. If an ordered task is associated with a not alive DS, PLDB, or with a DSG that had mastership change since the ordering, then the ordered task is not moved to the RTL, and is removed from the PTL.

For the complete list of cudbConsistencyMgr options, execute cudbConsistencyMgr --help, or refer to CUDB Node Commands and Parameters for more information.

The log files produced by CUDB Consistency Check are stored on the disk storage system of the SC where the check task was executed, in the following directory:

/local/cudb_ddci/replica_check

saAmfSISUHAState."safSu=SC-1,safSg=2N,safApp=ERIC-CUDB_CUDBDDCI"."safSi=2N-1":
active(1)
saAmfSISUHAState."safSu=SC-2,safSg=2N,safApp=ERIC-CUDB_CUDBDDCI"."safSi=2N-1":
standby(2)

Each Consistency Check task writes three files to the directory mentioned above:

• The LDAP tree log with the following file name: cudbDsuDiff_tree_<TASK-ID>_S<FROM-SITE-ID>_N<FROM-NODE-ID>_S<TO-SITE-ID>_N<TO-NODE-ID>_DSG<DSG-ID>.xml

• The table log with the following file name: cudbDsuDiff_table_<TASK-ID>_S<FROM-SITE-ID>_N<FROM-NODE-ID>_S<TO-SITE-ID>_N<TO-NODE-ID>_DSG<DSG-ID>.log

• The execution log with the following file name: cudbDsuDiff_exec_<TASK-ID>_S<FROM-SITE-ID>_N<FROM-NODE-ID>_S<TO-SITE-ID>_N<TO-NODE-ID>_DSG<DSG-ID>.log

The table log and the execution log contain internal CUDB information, and are intended for Ericsson support personnel only.

If no differences are found between the two replicas, or the found inconsistencies are identified only on the LDAP entry DN level, then the table log file is removed, as it would be empty.

The above behavior is affected by the verbose mode as follows:

The Consistency Check tool raises alarms on the node that holds the slave replica, if inconsistencies are found during the check. Four alarms can be raised; the alarm selection depends on the following conditions:

• Where the check is performed (the PLDB or a DSG).

• The number of inconsistencies found on internal database level.

• Whether the LDAP Schema Sanity Check (part of the Consistency Check) finds any schema inconsistency between the two replicas.

The available alarms are as follows:

• Storage Engine, Data Inconsistency between Replicas Found in DS, Minor

• Storage Engine, Data Inconsistency between Replicas Found in DS, Major

• Storage Engine, Data Inconsistency between Replicas Found in PLDB, Minor

• Storage Engine, Data Inconsistency between Replicas Found in PLDB, Major

Avoiding Duplicate Alarms

Raising and Clearing Alarms

When using the CUDB consistency check tools, consider the following recommendations:

• The CUDB system offers two types of consistency check tools:

  • The Lightweight Consistency Check, performed by the cudbCheckConsistency command. This command performs a quick check by comparing only the number of rows in the database tables of the PLDB or DSG master and slave replicas.

  • The Consistency Check, which compares the contents of the database tables containing LDAP entry attribute values, and performs a deep check. CUDB Consistency Check is recommended to be used regularly to monitor and keep the consistency of the data between the PLDB or DSG replicas.

• The preferred way of checking is to compare each PLDB or DSG slave replica against the master replica of the PLDB, or the specific DSG. These checks must be executed sequentially, as they are all executed on the PLDB or DSG master node. Still, all the checks can be ordered for execution. In this case, the check tasks are queued in the Pending Task List (PTL).

• The execution time of the check is proportional to the amount of data stored in the PLDB or DSG, and is specific to the PLDB or that DSG. The execution time can be calculated as follows:

  1. Order a check for the concerned PLDB or DSG slave.

  2. Wait for its completion.

  3. Issue the following command:

     cudbConsistencyMgr -t

  When executed, check for check tasks whose exit code is 0 or 1. These values mean that the entire database content was scanned. See Example 10 for an example.

  The execution time can be calculated from the task execution started and task execution finished lines. See Reviewing the Task History on how to review the task history.

• The main goal of the consistency tools is to keep the database replicas identical for data consistency. In case differences are found, the recommended procedure described in the alarm OPIs (see Alarms) is to perform recovery as soon as the alarm is raised. Two scenarios can occur:

  • Small amount of differences are detected, and no schema differences are found (a minor alarm is raised). In this case, recovery or reprovisioning can be attempted manually, based on the LDAP DNs found in the LDAP tree log.

    Note:

    The reprovisioning procedure is outside the scope of this document.

  • Large amount of differences are detected, or schema differences are found (a major alarm is raised). Manual reprovisioning or recovery is not feasible in this case. The preferred way to make the replicas identical is a combined unit data backup and restore on the affected PLDB or DSG. For further information about the combined unit data backup and restore procedure, refer to CUDB Backup and Restore Procedures.

  The decision on whether the differences are minor or major is based on the --alarm-severity-limit parameter of the check specified to the cudbConsistencyMgr command. The limit is measured in database table rows, that mostly correspond 1-to-1 to LDAP attribute value differences.

  If recovery is done, it is recommended to run the check on the other replicas of the PLDB or the same DSG. Before performing the check, check the PTL with the following command to see if any of the wanted checks are already ordered:

  cudbConsistencyMgr --list

Example 10   cudbConsistencyMgr -t Output Example

# cudbConsistencyMgr -t | grep UTC_2014-10-08-11-50-41_N119_U0000000178

2014-10-08 13:50:41+02:00 SC_2_1 UTC_2014-10-08-11-50-41_N119_U0000000178: task found in PTL: checkType=ms, //
source=S1-N119-D1 check=S2-N120-D1,maxReplicaLag=10000,alarmSeverityLimit=100
2014-10-08 13:50:41+02:00 SC_2_1 UTC_2014-10-08-11-50-41_N119_U0000000178: task execution started
2014-10-08 13:55:58+02:00 SC_2_1 UTC_2014-10-08-11-50-41_N119_U0000000178: task execution finished with //
result: Successful completion, differences found. Exit code: 1.