Contents

1   Introduction

This document describes how to replace a blade in an Ericsson Centralized User Data Base (CUDB) node deployed on native BSP 8100.

1.1   Description

This Operating Instruction (OPI) describes how to replace a blade in a CUDB node. A blade replacement must be performed either because of blade fault, or due to a hardware upgrade.

1.2   Revision Information

Rev. A

Rev. B

Rev. C

Rev. D

Rev. E

Rev. F

Rev. G

Rev. H

Rev. J

Rev. K

1.3   Typographic Conventions

Typographic conventions can be found in the following document:

• Typographic Conventions

2   Node Hardware Description

Before replacing any blades, make sure to check the hardware description of the node. Refer to CUDB Node Hardware Description, Reference [1] for more information.

3   Replacing a Blade

This section describes how to identify a faulty blade, how to perform a blade replacement in a CUDB node, and how to prepare the replacement blade for operation.

Also, further actions after physical board replacement are described.

3.1   Identifying the Faulty Blade

3.1.1   Identifying the Blade Name

Perform the following steps to identify a faulty blade in a CUDB node:

1. Establish an Secure Shell (SSH) session towards the target CUDB node with the following command:

   ssh root@<CUDB_Node_OAM_VIP_Address>

   This session is established to the first or second System Controller (SC), that is either to SC_2_1 or SC_2_2.

   Refer to CUDB Users and Passwords, Reference [4] CUDB Users and Passwords for more information on the default root password.

2. Several methods are available to identify an SC, a DSG, or a PLDB in the /cluster/etc/cluster.conf file. An example is provided below.

   For instance, below is a defined node containing 2 SCs, and 8 payload blades:

   node 1 control SC_2_1
   node 2 control SC_2_2
   node 3 payload PL_2_3
   node 4 payload PL_2_4
   node 5 payload PL_2_5
   node 6 payload PL_2_6
   node 7 payload PL_2_7
   node 8 payload PL_2_8
   node 9 payload PL_2_9
   node 10 payload PL_2_10
   
   
   host all 10.22.0.1 OAM1
   host all 10.22.0.2 OAM2
   host all 10.22.0.3 PL0
   host all 10.22.0.4 PL1
   host all 10.22.0.5 DS1_0
   host all 10.22.0.6 DS1_1
   host all 10.22.0.7 DS2_0
   host all 10.22.0.8 DS2_1
   ..............
   

   The below list provides some example scenarios for various failing blades:

   • In case the failing blade is Blade 7, with the IP of 10.22.0.7 (note the last octet), then pay attention to the following two lines of the cluster.conf file:

     node 7 payload PL_2_7
     host all 10.22.0.7 DS1_0
     

     In the above lines, PL_2_7 is the name of the payload blade, the blade number is "7", while the identification number of the associated DS is "1" (DS1_0).

   • In case the failing blade is Blade 3 with the IP of 10.22.0.3 (note the last octet), then pay attention to the following two lines of the cluster.conf file:

     node 3 payload PL_2_3
     host all 10.22.0.3 PL0
     

     In the above lines, PL_2_3 is the name of the payload blade, "3" is the blade number, while the Processing Layer ID is PL0. In case the blade in the cluster needs reboot, the following command must be used:

     cluster reboot --node 3

   • In case the failing blade is Blade 1 with the IP of 10.22.0.1 (note the last octet), then pay attention to the following two lines of the cluster.conf file:

     node 1 payload SC_2_1
     host all 10.22.0.1 OAM1
     

     In the above lines SC_2_1 is the name of the blade, "1" is the blade number, while the SC ID is OAM1.

3.1.2   Identifying Blade Rack and Subrack Position

To identify the physical blade that has to be replaced, do the following:

1. Establish a BSP CLI session:

   ssh advanced@<BSP-NBI-SCX> -p2024

2. Enter the following commands:

   show-table ManagedElement=1,DmxcFunction=1,Eqm=1,VirtualEquipment=cudb -m Blade -p bladeId,userLabel

The output must be similar to the following example:

=======================
| bladeId | userLabel                               |
=======================
| 0-1      | SC-1                                   |
| 0-11     | PL-6                                   |
| 0-13     | PL-7                                   |
| 0-15     | PL-8                                   |
| 0-17     | PL-9                                   |
| 0-19     | PL-10                                  |
| 0-21     | PL-11                                  |
| 0-23     | PL-12                                  |
| 0-3      | SC-2                                   |
| 0-5      | PL-3                                   |
| 0-7      | PL-4                                   |
| 0-9      | PL-5                                   |
| 1-1      | PL-13                                  |
| 1-11     | PL-18                                  |
| 1-13     | PL-19                                  |
| 1-15     | PL-20                                  |
| 1-17     | PL-21                                  |
| 1-19     | PL-22                                  |
| 1-21     | PL-23                                  |
| 1-23     | PL-24                                  |
| 1-3      | PL-14                                  |
| 1-5      | PL-15                                  |
| 1-7      | PL-16                                  |
| 1-9      | PL-17                                  |
| 2-1      | PL-25                                  |
| 2-11     | PL-30                                  |
| 2-13     | PL-31                                  |
| 2-15     | PL-32                                  |
| 2-17     | PL-33                                  |
| 2-19     | PL-34                                  |
| 2-21     | PL-35                                  |
| 2-23     | PL-36                                  |
| 2-3      | PL-26                                  |
| 2-5      | PL-27                                  |
| 2-7      | PL-28                                  |
| 2-9      | PL-29                                  |
=======================

Note:  

LDE and BSP 8100 naming conventions are slightly different, so SC_2_1 on LDE level equals to SC-1 on BSP 8100 and so on.

The bladeId identifies the blade position in the rack, the first number meaning the subrack and the second meaning the slot within the subrack. For example, PL-14 is in the third slot of subrack 1.

3.2   Preparing the Blade Replacement

Perform the following steps to prepare the blade replacement:

Note:  

In the below commands, <name> and <blade> are used to identify blades, where:

• <blade> is a numeric identifier, for example in SC_2_1 <blade> is 1, in PL_2_3 <blade> is 3.
• <name> is the controller name (SC_2_ <blade>) or the payload blade name (PL_2_ <blade>).

1. Establish an SSH session towards the target CUDB node with the following command:

   ssh root@<CUDB_Node_OAM_VIP_Address>

   This session is established to the first or second SC, that is either to SC_2_1 or SC_2_2.

   For more information on the default root password refer to CUDB Users and Passwords, Reference [4] CUDB Users and Passwords.

2. Lock the blade at SAF level with the following command:

   cmw-node-lock <name>

3. Check if the specific blade is locked at SAF level with the following command:

   cmw-status -v node

   The output must be similar to the following example:

   safAmfNode=PL-7,safAmfCluster=myAmfCluster

   AdminState=LOCKED-INSTANTIATION(3)

   OperState=ENABLED(1)

4. Make a backup of the rpm.conf file as follows:
   1. Make a copy of the file, and rename it to rpm.conf_FULL with the following command:

      cp /cluster/nodes/<blade>/etc/rpm.conf /cluster/nodes/<blade>/etc/rpm.conf_FULL

      The rpm.conf_FULL is now contains all entries of the original rpm.conf file.

   2. Overwrite the contents of the original rpm.conf file, so that it contains only the cudbKernelTuning rpm, and the ldews-control or ldews-payload entries (depending on the type of blade to replace). Use the following command to do so:

      grep -ia 'ldews\|linux\|cudbKernelTuning' /cluster/nodes/<blade>/etc/rpm.conf_FULL > /cluster/nodes/<blade>/etc/rpm.conf

5. In case an SC is being replaced, create a symbolic link for the executing hooks with the following command:

   cd /cluster/hooks/<blade>

   ln -s /cluster/hooks/pre-installation.tar.gz pre-installation.tar.gz

   ln -s /cluster/hooks/post-installation.tar.gz post-installation.tar.gz

6. Replace the blade as described in Section 3.3.

   Note:  

   When replacing SC, alarm SAF, LOTC Disk Replication Consistency Failed, might appear. If physical replacement is taking more than 20 minutes, alarm SAF, LOTC Disk Replication Communication Failed, might appear. These alarms are expected during blade replacement procedure on SC and must be automatically cleared when all replacement steps are executed. For more information, please follow the corresponding alarm OPI.

   
   

3.3   Replacing GEP Boards

Refer to the Manage Blade document in the BSP 8100 CPI for detailed information on the procedure required to physically replace a blade.

3.4   CUDB Node Configuration Changes

This section describes the configuration changes to perform in a CUDB node in case blade replacement is needed.

3.4.1   Obtaining MAC Addresses for the New Blade

The MAC addresses are used as input to create the cluster.conf file, which is used by LDE. The MAC addresses are also required to configure the Jumpstart server before installing LDE on the SCs, as well for the blade replacement procedure.

The MAC addresses are fetched through the BSP CLI. This MAC is the MAC base, used to obtain the MAC addresses necessary to complete the cluster.conf file generation.

To obtain the MAC addresses, do the following:

1. Establish a BSP CLI session:

   ssh advanced@<BSP-NBI-SCX> -p2024

2. Execute the following commands to show the MAC addresses:

   show-table ManagedElement=1,DmxcFunction=1,Eqm=1,VirtualEquipment=cudb -m Blade -p bladeId,firstMacAddr

   The output must be similar to the below example:

   ===============================
   | bladeId | firstMacAddr      |
   ===============================
   | 0-1     | 90:55:AE:3A:CB:1D |
   | 0-11    | 90:55:AE:3A:CA:5D |
   | 0-13    | 90:55:AE:3A:C9:CD |
   | 0-15    | 90:55:AE:3A:CA:75 |
   | 0-17    | 90:55:AE:3A:C9:9D |
   | 0-19    | 90:55:AE:3A:CA:ED |
   | 0-21    | 90:55:AE:3A:CB:AD |
   | 0-23    | 90:55:AE:3A:CD:5D |
   | 0-3     | 90:55:AE:3A:C9:55 |
   | 0-5     | 90:55:AE:3A:CA:15 |
   | 0-7     | 90:55:AE:3A:C9:FD |
   | 0-9     | 90:55:AE:3A:C9:25 |
   | 1-1     | 90:55:AE:3A:B0:7D |
   | 1-11    | 90:55:AE:3A:BF:C5 |
   | 1-3     | 90:55:AE:3A:C1:45 |
   | 1-5     | 90:55:AE:3A:BF:05 |
   | 1-7     | 90:55:AE:3A:BF:1D |
   | 1-9     | 90:55:AE:3A:BF:35 |
   ===============================

The MAC shown for each shelf slot is the base MAC. All the MACs can be obtained by adding a number to the <base mac>, in accordance to the following tables. Table 1 applies to BSP 8100 (GEP3) boards, while Table 2 applies to BSP 8100 (GEP5) boards.

(1)  The resulting MAC must be in hexadecimal format.

(1)  The resulting MAC must be in hexadecimal format.

Note:  

Ports ETH-0 and ETH-1 are enabled only during the initial software installation phase from the Jumpstart server. After the LDE is installed on the blade, they remain disabled and cannot be used.

3.4.2   Obtaining Board Revision for the New Blade

In case of SC replacement in BSP 8100 systems with GEP3 hardware, perform the procedure below to check the product revision of the new blade:

1. Establish a BSP NBI CLI session:

   ssh advanced@<BSP-NBI-SCX> -p2024 -t -s cli

2. Execute the following command to show the blade hardware revisions:

   show ManagedElement=1,SystemFunctions=1,HwInventory=1 -m HwItem -p productIdentity

   The expected output must be similar to the below example:

   productIdentity="ROJ 208 840/3"
   productDesignation="GEP3-HD300"
   productRevision="R4B"
   

3.4.3   Editing the LDE installation.conf File

In case of SC replacement in BSP 8100 systems with GEP3 hardware, perform the procedure below to edit the installation.conf file.

1. Establish an SSH session towards the target CUDB node with the following command:

   ssh root@<CUDB_Node_OAM_VIP_Address>

   This session is established to the first or second SC, either to SC_2_1 or SC_2_2.

   Refer to CUDB Users and Passwords, Reference [4] CUDB Users and Passwords for more information on the default root password.

2. Locate the installation.conf file in the following directory:

   /cluster/etc/installation.conf

3. Edit the file and set parameter value depending on the hardware revision of the new blade obtained in Section 3.4.2:
   • If it is lower than R9A, use the following value:

     disk_device_path=/dev/sdb

   • If it is R9A or higher, use the following value:

     disk_device_path=/dev/sda

3.4.4   Editing the LDE cluster.conf File

Perform the following steps to edit the cluster.conf file.

1. Establish an SSH session towards the target CUDB node with the following command:

   ssh root@<CUDB_Node_OAM_VIP_Address>

   This session is established to the first or second SC, that is either to SC_2_1 or SC_2_2.

   Refer to CUDB Users and Passwords, Reference [4] CUDB Users and Passwords for more information on the default root password.

2. Locate the cluster.conf file in the following directory:

   /cluster/etc/cluster.conf

3. Open the file, and replace the old MACs with the ones. Use Table 1 or Table 2 in Section 3.4.1 as a means to calculate the actual MAC addresses.

   An example of the LDE cluster.conf file is provided below. Interfaces 1 or 2 are related to blade number: for example, if payload blade PL_2_5 is replaced, then interface 5 needs MAC addresses adaptation.

# # Example /cluster/etc/cluster.conf
########################
#  
#  Interface definition
#  

interface 1 eth3 ethernet 90:55:ae:3a:b0:7e
interface 1 eth4 ethernet 90:55:ae:3a:b0:7f
interface 1 eth5 ethernet 90:55:ae:3a:b0:82
interface 1 eth6 ethernet 90:55:ae:3a:b0:83

interface 2 eth3 ethernet 90:55:ae:3a:c1:46
interface 2 eth4 ethernet 90:55:ae:3a:c1:47
interface 2 eth5 ethernet 90:55:ae:3a:c1:4a
interface 2 eth6 ethernet 90:55:ae:3a:c1:4b

interface 3 eth3 ethernet 90:55:ae:3a:bf:06
interface 3 eth4 ethernet 90:55:ae:3a:bf:07
interface 3 eth5 ethernet 90:55:ae:3a:bf:0a
interface 3 eth6 ethernet 90:55:ae:3a:bf:0b

interface 4 eth3 ethernet 90:55:ae:3a:c9:fe
interface 4 eth4 ethernet 90:55:ae:3a:c9:ff
interface 4 eth5 ethernet 90:55:ae:3a:ca:02
interface 4 eth6 ethernet 90:55:ae:3a:ca:03

interface 5 eth3 ethernet 90:55:ae:3a:c9:26
interface 5 eth4 ethernet 90:55:ae:3a:c9:27
interface 5 eth5 ethernet 90:55:ae:3a:c9:2a
interface 5 eth6 ethernet 90:55:ae:3a:c9:2b

4. Verify the syntax of the cluster.conf file with the following command:

   cluster config -v

   In case of any error message, check the command output and correct syntax mistakes. Warning messages can be ignored.

5. Reload the configuration with the following command:

   cluster config --reload --all

   Note:  

   The command fails for a currently replaced blade, this is the expected behavior. (Node X (<name>)not responding, skipped). Continue with next step.

   
   
6. The new blade(s) start(s) booting from network.

3.5   System Controller Replacement Steps

This section describes the procedure to finalize the SC blade replacement.

The new blade is by default set to boot from network, the following procedure describes how to set it to boot from hard disk.

During this procedure, the new SC also synchronizes its replicated storage disk partition with another SC. This process can take up to one hour, depending on storage disk partition size and available network bandwidth. Use the following command on another SC to check the synchronization status:

cat /proc/drbd

Perform the following steps to finalize the SC replacement:

1. Restore the original rpm.conf file with the following command: cp /cluster/nodes/<blade>/etc/rpm.conf_FULL /cluster/nodes/<blade>/etc/rpm.conf

   In the above command, <blade> must be replaced with the blade number. For example, in case of SC_2_2, the blade number is 2.

2. Set the new SC blade to boot from hard disk. See Section 3.8 for details.
3. Reboot the new SC from console interface with the following command:

   reboot

3.6   DSG and PLDB Replacement Steps

If the blade to replace is a DSG or PLDB, then perform the following steps:

1. Log in to one of the SCs, and execute the following commands:

   ssh root@<CUDB_Node_OAM_VIP_Address>

   Refer to CUDB Users and Passwords, Reference [4] CUDB Users and Passwords for more information on the default root password.

   cd /opt/ericsson/cudb/OAM/support/bin/

   ./cudbPartTool rebuild -n <blade>

   In the above command, <blade> must be replaced with the blade number. For example, in case of PL_2_5, <blade> is 5.

2. Check if the partition is created with the following command:

   ./cudbPartTool check -n <blade>

   In the above command, <blade> must be replaced with the blade number. For example, in case of PL_2_5, <blade> is 5.

   The output must be similar to the below example:

   CUDB_82 SC_2_2# ./cudbPartTool check -n 5
   
   CUDB partitioning tool
   
   -= Cluster filesystem analysis =-
   
   Payload PL_2_5 report:
    WARNING: local storages not mounted.
   
   
   Done.
   

3. Restore the original rpm.conf file with the following command:

   cp /cluster/nodes/<blade>/etc/rpm.conf_FULL /cluster/nodes/<blade>/etc/rpm.conf

   In the above command, <blade> must be replaced with the blade number. For example, in case of PL_2_5, <blade> is 5.

3.7   Finalizing Replacement

Perform the following steps to finish blade replacement. The following steps apply to replacing every blade type (SC, DSG, and PLDB).

Note:  

In case of SC replacement, crontab jobs and their definitions, or similar tasks, which are not deployed by default in CUDB, or scheduled with data or software backup scripts, will be lost. If necessary, redeploy them after the procedure is completed.

1. Unlock the blade at SAF level with the following command:

   cmw-node-unlock <name>

   <name> is the name of the replaced blade, for example PL_2_5.

2. Reboot the newly-installed blade with the following command:

   cluster reboot -n <blade>

   <blade> is the number of the replaced blade, for example 5 for PL_2_5.

3. Wait until the blade has rebooted and joined the cluster. Use the following command to list the joined blades, and to check the operational states of the SUs:

   cmw-status -v node

   The expected output must be similar to the below example:

   safAmfNode=PL-7,safAmfCluster=myAmfCluster

   AdminState=UNLOCKED(1)

   OperState=ENABLED(1)

4. Wait until all the processes are started in the blade and check if the system has recovered without faults with the cudbSystemStatus command. In case of DS, errors related to the DS database can be ignored because of the data restore done later.

   Note:  

   If the status is not correct, stop the procedure, and contact the next level of maintenance support.

   
   
5. Exit the SSH session with the exit command.
6. Depending on the blade type, do the following:
   • If the replaced blade is an SC, the procedure is finished.
   • If the replaced blade is a DSG blade, to backup and restore a DSG replica, perform the steps described in the Performing Combined Unit Data Backup and Restore section of CUDB Backup and Restore Procedures, Reference [7].
   • If the replaced blade is any one in PLDB group, to backup and restore a PLDB replica, perform the steps described in the Performing Combined Unit Data Backup and Restore section of CUDB Backup and Restore Procedures, Reference [7].

     After the NDBs are started and the mysql server connections are OK, execute the following command:

     cudbPrepareStore --pl

     Note:  

     After finishing the rebuild procedure, the stored procedures are not restored. Recreate them with the following command:

     cudbManageStore -p -o restorestoredprocedures

     
     

Refer to the Configuring Automatic Mastership Change section of CUDB System Administrator Guide, Reference [6] for more information.

3.8   Changing the Boot Device Order

The procedure required to change boot device order on BSP 8100 (GEP5) or BSP 8100 (GEP3) hardware is described in the Boot Device Setting section of the Manage Blade document in the BSP 8100 CPI (Ericsson BSP R9 or above).

Note:  

Refer to the Set GEP3 Disk Processor Boot Device or Set GEP5 Disk processor Boot Device documents in the BSP 8100 CPI, if older version of Ericsson BSP product is used.

The default boot order configuration for new blades in BSP 8100 systems with GEP3 or GEP5 hardware is to boot from network:

boardConfiguration="0.1.255.255.255.255.255.255.255.255.255.255.255.255.255.255.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0"

The target boot order configuration for PL blades in BSP 8100 systems with GEP3 or GEP5 hardware, after blade replacement is finalized, is the same.

The target boot order configuration for SC blades in BSP 8100 systems, after blade replacement is finalized, is to boot from hard disk:

GEP3

boardConfiguration="16.255.255.255.255.255.255.255.255.255.255.255.255.255.255.255.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0"

GEP5

boardConfiguration="18.17.16.255.255.255.255.255.255.255.255.255.255.255.255.255.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0"

3.9   Replacing Multiple Blades in Parallel

This section provides instructions required to replace multiple blades in parallel on CUDB nodes.

3.9.1   Parallel Blade Replacement Procedure

Only the same group of blades can be replaced in parallel at once. In the CUDB system, blades can be grouped into three distinct groups: SC blades, PLDB blades, and DSG blades. These groups can be further divided into groups of even-numbered and odd-numbered blades, resulting six distinct groups of blades in total:

1. SC_2_2
2. SC_2_1
3. Odd-numbered PLDB blades
4. Even-numbered PLDB blades
5. Odd-numbered DSG blades
6. Even-numbered DSG blades

Perform the following steps to replace multiple blades in parallel:

Note:  

To ensure that there is enough traffic handling capacity during replacement execution, it is recommended that the maximum number of payload blades to be replaced in parallel must not be larger than the configured value of the redundancyLevel attribute of the CudbLdapAccess class.

If there are more blades to be replaced, it must be done iteratively, in a way that in each iteration, replacement is done for maximum of N blades from the same group in parallel, where N is the value of the redundancyLevel attribute. However, if replacement is done in low traffic period or in a maintenance window, when the degraded traffic handling capacity could still be sufficient, it can be decided to execute replacement for more than N blades in parallel.

1. Check the value of the redundancyLevel attribute of the CudbLdapAccess class and take special note of it. For more information, refer to the Class CudbLdapAccess section of CUDB Node Configuration Data Model Description, Reference [5].
2. Identify all faulty blades inside the node, as described in Section 3.1 to be able to group them.
3. Identify the position of all faulty blades, as described in Section 3.1.2.
4. Prepare for the replacement of all faulty blades, as described in Section 3.2.
5. In case of replacing SC group(s) or PLDB group(s), force the external applications to move their primary connections to another CUDB node. This applies in case primary connections are established, or the SC or the PLDB blades are affected.
6. Execute the blade replacement exactly in the following order, skipping any group which has no faulty blades:
   1. SC_2_2
   2. SC_2_1
   3. Odd-numbered PLDB blades
   4. Even-numbered PLDB blades
   5. Odd-numbered DSG blades
   6. Even-numbered DSG blades
   Do!

   Always follow the order of groups exactly.

7. If number of blades that have been replaced in parallel was greater than value of redundancy level parameter, please also execute cudbLdapFeRestart command. For more information, refer to the cudbLdapFeRestart section of CUDB Node Commands and Parameters, Reference [2].

Glossary

For the terms, definitions, acronyms and abbreviations used in this document, refer to CUDB Glossary of Terms and Acronyms, Reference [8].

Reference List