Adding a compute or storage node

The following image displays the processes in the VIM Overcloud scaling (storage/compute expansion) and configuration:

Figure 21: Overcloud Scaling and Configuration

Prerequisites
  • For manual switch configuration, ensure the following:
    • Compute is connected as per the Portmap.

    • Split the switch port.
      • To split the management port, use the following command:
        net add interface swp<based on portmap>breakout 4x10G
        net commit
      • To split the data port, use the following command:
        net add interface swp<based on portmap> breakout 4x
        net commit
  • Ensure to add the iLO ports of the new nodes in the management switch. For more information about the management switch configuration, see Manually configuring HPE FlexFabric 5900 (Management) switch.

  • Before adding a compute node, storge node, or an HCI node to the VIM Overcloud, ensure that the VIM Overcloud is in the OVERCLOUD_INSTALLED state by executing the following command:
    nps show --data vim --node overcloud
  • For the HCI node expansion, the new nodes must be identical with the existing HCI nodes. For example, same processors, same NICs in the same order, and the same number of NVMe disks.

  • For the storage node expansion, the new nodes must be identical with the existing storage nodes. For example, same processors, same NICs in the same order, and the same number of SAS and SSD disks.

  • For the compute node expansion:
    • All the compute nodes, if added for the first time, must be identical.

    • If the Overcloud stack already has compute nodes existing, ensure that the newly added compute nodes are identical to the existing ones. For example, the NICs in the newly added compute nodes must be placed in the same slots as in the existing compute nodes.

Procedure
  1. Configure the data switch for compute and storage addition using the following sample configuration:
    • Management NIC configuration for compute and storage
      1. Configure alias for connected interface:
        net add interface swp8s3 alias E8-C4-Intel-P1
      2. Configure Bond for management Interface:
        net add bond E8-C4-Intel-P1 bond slaves swp8s3
      3. Enable LACP BYPASS for this interface:
        net add bond E8-C4-Intel-P1 bond lacp-bypass-allow
        net add bond E8-C4-Intel-P1 bond lacp-rate slow
        
      4. Set MTU:
        net add bond E8-C4-Intel-P1l mtu 9216
      5. Allowed Management VLAN for bond interface:
        net add bond E8-C4-Intel-P1 bridge pvid 1901
        net add bond E8-C4-Intel-P1 bridge vids 1901-1903
        
      6. Add Bond interface to Bridge:
        net add bridge bridge ports E8-C4-Intel-P1
      7. Set CLAG ID for Bond interface:
        net add bond E8-C4-Intel-P1 clag id 15
    • Configuration for OVS-DPDK (Only for Compute node expansion)

      1. Configure Alias for the port:
        net add interface swp9s3 alias E81-C4-PCI-P1
      2. Configure Bond:

        net add bond ADD-C1-EL81C4-PCI bond slaves swp9s3
      3. Set the MTU to jumbo frame:
        net add bond ADD-C1-EL81C4-PCI mtu 9216
      4. Add ports to bridge:
        net add bridge bridge ports ADD-C1-EL81C4-PCI
      5. Configure CLAG ID:
        net add bond ADD-C1-EL81C4-PCI clag id 16
      6. Permit Vlan for Data NIC
        net add bond ADD-C1-EL81C4-PCI bridge vids 1949-1960
    • Configuration for SRIOV (For Compute node or node expansion)
      1. Configure Alias for the port:
        net add interface swp9s3 alias E81-C4-PCI-P1
      2. Set the MTU to jumbo frame:
        net add interface swp9s3 swp9s3 mtu 9216
      3. Add interface to Bridge:
        net add bridge bridge ports swp9s3
      4. Allow required Vlan on Data NIC:
        net add interface swp9s3 bridge vids 1949-1960
  2. Update the SRIOV or OVS-DPDK profiles for compute or HCI node expansion using the following steps:
    1. Go to step 3 when profiles are available, if HCI nodes or compute nodes are added to the Overcloud stack.
    2. Add OVS DPDK profile for computes for the first time using the following command:
      nps create -p vim_node -f <json file>
      Where the <json file> is
      NOTE:

      Following JSON files provide only sample configurations. For more information on configurations, Compute Node parameters for input JSON.

      "computeovsdpdk": {
                  "reserved_host_memory": "4096",
                  "vcpu_pin_set": "2,3,4,5,6,7,10,11,12,13,14,15",
                  "interfaces": {
                      "dpdk": [
                          {
                              "bond_members": [
                                  "nic6",
                                  "nic7"
                              ],
                              "type": "bond",
                "driver": "mlx5_core",
                              "name": "br-data",
                              "physnet": "datacentre",
                              "bond_ovs_options": "bond_mode=balance-tcp lacp=active"
                          }                    
                      ]
                  },
                  "hugepg_count": "100",
                  "type": "compute",
                  "isolated_core_list": "1,2,3,4,5,6,7,9,10,11,12,13,14,15",
                  "dpdk": {
                      "core_list": "0,8",
                      "socket_memory": "4096,4096",
                      "pmd_core_list": "1,9",
                      "memory_channels": "4"
                  }
              }
      
      NOTE:

      Ensure that the physnet value that you define here is the same as the one defined at the time of starter kit installation.

    3. Add the SR-IOV profile for computes or HCI nodes for the first time using the following command:
      nps create -p vim_node -f <json file>
      Where the <json file> is:
      NOTE:

      Following JSON files provide only sample configurations. For more information on configurations, Compute Node parameters for input JSON.

      "computesriov": {
                  "reserved_host_memory": "4096",
                  "vcpu_pin_set": "2,3,4,5,6,7,10,11,12,13,14,15",
                  "interfaces": {
                      "sriov": [
                          {
                              "name": "nic6"
                              "physnet": "datacentre",
                              "vf": "7"
                          },
                          {
                              "name": "nic7",
                              "physnet": "datacentre",
                              "vf": "7"
                          }
                      ]
                  },
                  "hugepg_count": "100",
                  "type": "compute",
                  "isolated_core_list": "1,2,3,4,5,6,7,9,10,11,12,13,14,15"
              }
      
      NOTE:

      Ensure that the physnet value that you define here is the same as the one defined at the time of starter kit installation.

  3. To add a new node, use the Addon Servers UI to generate the JSON file and run the following command with the location of the JSON file:

    nps add-infra -f <location of the server profile JSON file>

    NOTE:
    • The JSON file for adding nodes must be generated using the Addon Servers UI only. To create a JSON using the TICG, see Generating an input JSON using the TICG for Addon Kits.

    • The following is a sample format of the JSON generated using the Addon servers UI:
      {
          "topology_name": "<topology_name_same as starter kit>",
          "topology_modules": [
              <topology_modules_of_starter_kit>  ],
          "topology_data": {
              "infra": {
                  "servers": [
                      {
                          "description": "Description",
                          "hostname": ""<Name of the server>",
                          "hw_profile": "<Name of the server profile>",
                          "ilo": "<IP address of iLO>",
                          "user": "<User name>",
                          "password": "<password>",
                          "role": "<Role of the server>",
                          "model": "<Model of the server"
                      }
                  ],
                  "oobm_type": "manual",
                  "description": "<description>"
              }
          }
      }
    • The following is a sample format of the JSON for adding a Ceph node generated using the Addon servers UI:
      {
          "topology_name": "rhosp-c7",
          "topology_modules": [
              "hardware_preparation",
              "switch_automation",
              "image_service",
              "vim_deployment"
          ],
          "topology_data": {
              "infra": {
                  "servers": [
                      {
                          "description": "ceph1",
                          "hostname": "ceph1",
                          "hw_profile": "default_storage_profile_rhosp",
                          "ilo": "<IP address of iLO>",
                          "user": "admin",
                          "password": "<password>",
                          "role": "ceph-storage",
                          "model": "DL380"
                      }
                  ],
                  "oobm_type": "manual",
                  "description": "List of all physical infrastructure elements configured by code"
              }
          }
      }
  4. To apply and verify the BIOS, RAID, and NIC settings for a new server, do the following:
    1. Apply the BIOS Settings using the following command:
      nps deploy -s hw_prep -l debug -sl <server_ip> -a <action>
      where, <action> can be bios, raid, or nic.
    2. Verify the BIOS Settings using the following command
      nps show --service hw_prep

      Wait until the status is displayed as completed, success, aborted, or failed.

      NOTE:

      If the status is aborted or failed, see Rectifying hardware configuration failure.

  5. Create or update flavors using the following command:
    NOTE:

    Flavor creation is not needed for Ceph node addition.

    1. Run the following command:
      nps deploy -s  vim_overcloud -a update-flavors
    2. Check the status of the update-flavors operation using the following command:
      nps show --data vim --node overcloud

      Wait until the status is displayed as FLAVORS_UPDATED or FLAVOR_UPDATE_FAILED.

      • If the state is displayed as FLAVORS_UPDATED, log in to the Undercloud VM and verify that the flavors are created in the Undercloud by executing the following command:
        $ source stackrc
        $ openstack flavor list
      • If the state is displayed as FLAVOR_UPDATE_FAILED, analyze the nps-rhosp-<topology_name>-ansible.log and the nps-rhosp-cli-<topology_name>-date.log files in the /var/nps/logs/<topology_name>/ directory of the NPS toolkit VM. Based on the log files, complete the corrective actions.

      • Perform update-flavors operation again using the following command:

        nps deploy -s vim_overcloud -a update-flavors
  6. To register and check the new node to the Ironic service do the following:
    1. Register the new node to the Ironic service using the following command:
      nps deploy -s  vim_overcloud -a register-nodes
    2. Check the status of the register-nodes operation using the following command:
      nps show --data vim --node overcloud
      Wait until the status is displayed as NODES_REGISTERED, NODES_REGISTER_PARTIAL, or NODES_REGISTER_FAILED.
      • If the state is displayed as NODES_REGISTERED, log in to the Undercloud VM and verify that the nodes are registered with the Ironic service by executing the following command:

        $ source stackrc
        $ openstack baremetal node list
      • If the state is displayed as NODES_REGISTER_PARTIAL or NODES_REGISTER_FAILED, analyze the nps-rhosp-<topology_name>-ansible.log and the nps-rhosp-cli-<topology_name>-date.log files in the /var/nps/logs/<topology_name>/ directory of the NPS toolkit VM. Based on the log files, complete the corrective actions.

      • Perform the register-nodes operation again using the following command:

        nps deploy -s vim_overcloud -a register-nodes
  7. To introspect and check the new registered node do the following:
    1. Introspect the new registered node using the following command:
      nps deploy -s  vim_overcloud -a introspect-nodes
    2. Check the status of the introspect-nodes operation using the following command:
      nps show --data vim --node overcloud

      Wait until the status is displayed as NODES_INTROSPECTED, NODES_INTROSPECTED_PARTIAL, or NODES_INTROSPECTION_FAILED.

      • If the state is displayed as NODES_INTROSPECTED, log in to the Undercloud VM and verify that the nodes are introspected by executing the following command:

        $ source stackrc
        $ openstack baremetal introspection list
      • If the state is displayed as NODES_INTROSPECTED_PARTIAL or NODES_INTROSPECTION_FAILED, analyze the nps-rhosp-<topology_name>-ansible.log and the nps-rhosp-cli-<topology_name>-date.log files in the /var/nps/logs/<topology_name>/ directory of the NPS toolkit VM. Based on the log files, complete the corrective actions.

      • Perform the introspect-nodes operation again.

        For more information, see Rectifying hardware configuration failure.

  8. To tag and check the new node with the appropriate profiles do the following:
    1. Tag the new node with the appropriate profiles using the following command:
      nps deploy -s  vim_overcloud -a tag-nodes
    2. Check the status of the tag-nodes operation using the following command:
      nps show --data vim --node overcloud

      Wait until the status is displayed as PROFILES_UPDATED, PROFILES_UPDATED_PARTIAL, or PROFILES_UPDATE_FAILED.

      • If the state is displayed as PROFILES_UPDATED, log in to the Undercloud VM and verify that the profiles are updated by executing the following command:

        $ source stackrc
        $ openstack overcloud profiles list
      • If the state is displayed as PROFILES_UPDATED_PARTIAL or PROFILES_UPDATE_FAILED, analyze the nps-rhosp-<topology_name>-ansible.log and the nps-rhosp-cli-<topology_name>-date.log files in the /var/nps/logs/<topology_name>/ directory of the NPS toolkit VM. Based on the log files, complete the corrective actions.

      • Perform the tag-nodes operation again using the following command:

        nps deploy -s vim_overcloud -a tag-nodes
  9. Create and check the Overcloud installation templates:
    1. Create the Overcloud installation templates using the following command:
      # nps deploy -s  vim_overcloud -a create-templates
    2. Check the status of the create-templates operation using the following command:
      # nps show --data vim --node overcloud

      Wait until the status is displayed as TEMPLATES_CREATED or TEMPLATES_CREATION_FAILED.

      • If the state is displayed as TEMPLATES_CREATED, log in to the Undercloud VM and verify that the templates are created by executing the following command:

        $ cd /home/stack/templates

        Verify that the templates are available in the /home/stack/templates directory.

      • If the state is displayed as TEMPLATES_CREATION_FAILED, analyze the nps-rhosp-<topology_name>-ansible.log and the nps-rhosp-cli-<topology_name>-date.log files in the /var/nps/logs/<topology_name>/ directory of the NPS toolkit VM. Based on the log files, complete the corrective actions.

      • Perform the create-templates operation again using the following command:

        # nps deploy -s vim_overcloud -a create-templates
  10. To add a server as compute or HCI or Ceph node to the Overcloud do the following:
    1. Add a server as compute or HCI or Ceph node to the Overcloud using the following command:
      nps deploy -s  vim_overcloud -a overcloud-install
      NOTE:

      For node addition, this command will update the cluster and will not re-install the cluster.

    2. Check the status of the overcloud-install operation using the following command:
      nps show --data vim --node overcloud

      Wait until the status is displayed as OVERCLOUD_INSTALLED or OVERCLOUD_INSTALL_FAILED.

      • If the state is displayed as OVERCLOUD_INSTALLED, log in to the Undercloud VM and verify if the new nodes are added as computes or HCI nodes using the following commands:
        ssh stack@<oam ip address>
        source stackrc
        nova list
      • If the state is displayed as OVERCLOUD_INSTALL_FAILED, analyze the following log files:

        • The nps-rhosp-<topology_name>-ansible.log and the nps-rhosp-cli-<topology_name>-date.log files in the /var/nps/logs/<topology_name>/ directory of the NPS toolkit Host.

        • The overcloud-install.log file in the /home/stack/ directory in the Undercloud VM.

        Based on the log files, complete the corrective actions.

      • Run the overcloud-install operation again.

        For more information, see the Overcloud installation fails.