Viewing cluster status from the CLI

Submit Search

Viewing cluster status from the CLI

Use the get system ha status command to display information about an HA cluster. The command displays general HA configuration settings. The command also displays information about how the cluster unit that you have logged into is operating in the cluster.

Usually you would log into the primary unit CLI using SSH or telnet. In this case the get system ha status command displays information about the primary unit first, and also displays the HA state of the primary unit (the primary unit operates in the work state). However, if you log into the primary unit and then use the execute ha manage command to log into a subordinate unit, (or if you use a console connection to log into a subordinate unit) the get system status command displays information about this subordinate unit first, and also displays the HA state of this subordinate unit. The state of a subordinate unit is work for an active-active cluster and standby for an active-passive cluster.

For a virtual cluster configuration, the get system ha status command displays information about how the cluster unit that you have logged into is operating in virtual cluster 1 and virtual cluster 2. For example, if you connect to the cluster unit that is the primary unit for virtual cluster 1 and the subordinate unit for virtual cluster 2, the output of the get system ha status command shows virtual cluster 1 in the work state and virtual cluster 2 in the standby state. The get system ha status command also displays additional information about virtual cluster 1 and virtual cluster 2.

The command includes the following fields.

Fields	Description
`HA Health Status`	Indicates if all cluster units are operating normally (`OK`)or if a problem was detected with the cluster. For example, a message similar to `ERROR <serial-number> is lost @ <date> <time>` appears if one the subordinate units leaves the cluster.
`Model`	The FortiGate model number.
`Mode`	The HA mode of the cluster, for example, `HA A-P` or `HA A-A`.
`Group`	The group ID of the cluster.
`Debug`	The debug status of the cluster.
`Cluster Uptime`	The number of days, hours, minutes, and seconds that the cluster has been operating.
`Master selected using`	Shows how the primary unit was selected the last four times that the cluster negotiated. For example, when a cluster first forms this part of the command output could have one line showing that the primary unit is the cluster unit with the highest up time. Up to four lines can be included as the cluster negotiates to choose a new primary unit on different occasions. Each line includes a time stamp and the criteria used to select the primary unit.
`ses_pickup`	The status of session pickup: enable or disable.
`load_balance`	The status of the `load-balance-all` keyword: enable or disable. Active-active clusters only.
`schedule`	The active-active load balancing schedule. Active-active clusters only.
`override`	The status of the override option for the current cluster unit: enable or disable.
`Configuration Status`	Shows if the configurations of each of the cluster units are synchronized or not.
`System Usage stats`	Shows how busy each cluster unit is by showing the number of sessions being processed by the cluster unit, CPU usage, and memory usage.
`HBDEV stats`	Shows the status of each cluster unit's heartbeat interfaces. Includes whether the interfaces are up or down, how much data they have processed as well as errors found.
`Master Slave`	Displays the host name, serial number, cluster index or priority, and role of the primary unit and the subordinate units. The order in which the cluster units are listed starts with the cluster unit that you are logged into.
`number of vcluster`	The number of virtual clusters. If virtual domains are not enabled, the cluster has one virtual cluster. If virtual domains are enabled the cluster has two virtual clusters.
`vcluster 1 vcluster 2`	The heartbeat interface IP address of the primary unit in each virtual cluster. If virtual domains are not enabled there is one vcluster and this is the IP address of the primary unit. If virtual domains are enabled then each vcluster line will have an IP address. If the IP addresses are the same then the same FortiGate is the primary unit for both virtual clusters.
`vcluster 1` `Master Slave`	The HA state (hello, work, or standby) and HA heartbeat IP address of the cluster unit that you have logged into in virtual cluster 1. If virtual domains are not enabled, `vcluster 1` displays information for the cluster. If virtual domains are enabled, `vcluster 1` displays information for virtual cluster 1. The HA heartbeat IP address is 169.254.0.2 if you are logged into the primary unit of virtual cluster 1 and 169.254.0.1 if you are logged into a subordinate unit of virtual cluster 1. `vcluster 1` also lists the primary unit and subordinate units in virtual cluster 1. The list includes the cluster index and serial number of each cluster unit in virtual cluster 1. The cluster unit that you have logged into is at the top of the list. If virtual domains are not enabled and you connect to the primary unit CLI, the HA state of the cluster unit in virtual cluster 1 is work. The display lists the cluster units starting with the primary unit. If virtual domains are not enabled and you connect to a subordinate unit CLI, the HA state of the cluster unit in virtual cluster 1 is standby. The display lists the cluster units starting with the subordinate unit that you have logged into. If virtual domains are enabled and you connect to the virtual cluster 1 primary unit CLI, the HA state of the cluster unit in virtual cluster 1 is work. The display lists the cluster units starting with the virtual cluster 1 primary unit. If virtual domains are enabled and you connect to the virtual cluster 1 subordinate unit CLI, the HA state of the cluster unit in virtual cluster 1 is standby. The display lists the cluster units starting with the subordinate unit that you are logged into.
`vcluster 2` `Master Slave`	`vcluster 2` only appears if virtual domains are enabled. `vcluster 2` displays the HA state (hello, work, or standby) and HA heartbeat IP address of the cluster unit that you have logged into in virtual cluster 2. The HA heartbeat IP address is 169.254.0.2 if you are logged into the primary unit of virtual cluster 2 and 169.254.0.1 if you are logged into a subordinate unit of virtual cluster 2. `vcluster 2` also lists the primary unit and subordinate units in virtual cluster 2. The list includes the cluster index and serial number of each cluster unit in virtual cluster 2. The cluster unit that you have logged into is at the top of the list. If you connect to the virtual cluster 2 primary unit CLI, the HA state of the cluster unit in virtual cluster 2 is `work`. The display lists the cluster units starting with the virtual cluster 2 primary unit. If you connect to the virtual cluster 2 subordinate unit CLI, the HA state of the cluster unit in virtual cluster 2 is `standby`. The display lists the cluster units starting with the subordinate unit that you are logged into.

Get system ha status example - two FortiGates in active-passive mode

The following example shows get system ha status output for a cluster of two FortiGate-600Ds operating in active-passive mode. The cluster is healthy and has been running for 88 days. Primary unit select took place once and the cluster has been stable since then.

The following command output was produced by connecting to the primary unit CLI (host name External-Primary).

get system ha status
HA Health Status: OK
Model: FortiGate-600D
Mode: HA A-P
Group: 0
Debug: 0
Cluster Uptime: 88 days 07:55:15
Master selected using:
    <2016/09/20 11:45:53> FGT6HD3916800525 is selected as the master because it has the largest value of override priority.
ses_pickup: disable
override: disable
Configuration Status:
    FGT6HD3916800525(updated 4 seconds ago): in-sync
    FGT6HD3916801195(updated 4 seconds ago): out-of-sync
System Usage stats:
    FGT6HD3916800525(updated 4 seconds ago):
        sessions=91, average-cpu-user/nice/system/idle=0%/0%/0%/100%, memory=25%
    FGT6HD3916801195(updated 4 seconds ago):
        sessions=4, average-cpu-user/nice/system/idle=0%/0%/0%/100%, memory=24%
HBDEV stats:
    FGT6HD3916800525(updated 4 seconds ago):
        port3: physical/1000full, up, rx-bytes/packets/dropped/errors=7188802679/14764658/121/0, tx=14537036237/17393987/0/0
        port4: physical/1000full, up, rx-bytes/packets/dropped/errors=6599589671/9550781/122/0, tx=6599535969/9550705/0/0
    FGT6HD3916801195(updated 4 seconds ago):
        port3: physical/1000full, up, rx-bytes/packets/dropped/errors=14537164810/17394279/118/0, tx=7188884284/14764852/0/0
        port4: physical/1000full, up, rx-bytes/packets/dropped/errors=6599649293/9550869/118/0, tx=6599632709/9550845/0/0
Master: External-Primary, FGT6HD3916800525
Slave : External-Backup , FGT6HD3916801195
number of vcluster: 1
vcluster 1: work 169.254.0.2
Master:0 FGT6HD3916800525
Slave :1 FGT6HD3916801195

The following command output was produced by using execute HA manage 0 to log into the subordinate unit CLI of the cluster shown in the previous example. The host name of the subordinate unit is External-Primary.

get system ha status
HA Health Status: OK
Model: FortiGate-600D
Mode: HA A-P
Group: 0
Debug: 0
Cluster Uptime: 88 days 08:05:42
Master selected using:
    <2016/09/20 11:45:54> FGT6HD3916800525 is selected as the master because it has the largest value of override priority.
    <2016/09/20 11:44:23> FGT6HD3916801195 is selected as the master because it's the only member in the cluster.
    <2016/09/20 11:44:18> FGT6HD3916801195 is selected as the master because the peer member FGT6HD3916800525 has UPGRADE_SLAVE flag set.
    <2016/09/20 11:44:14> FGT6HD3916800525 is selected as the master because because it has UPGRADE_MASTER flag set.
ses_pickup: disable
override: disable
Configuration Status:
    FGT6HD3916801195(updated 1 seconds ago): out-of-sync
    FGT6HD3916800525(updated 1 seconds ago): in-sync
System Usage stats:
    FGT6HD3916801195(updated 1 seconds ago):
        sessions=4, average-cpu-user/nice/system/idle=0%/0%/0%/99%, memory=25%
    FGT6HD3916800525(updated 1 seconds ago):
        sessions=90, average-cpu-user/nice/system/idle=0%/0%/0%/100%, memory=25%
HBDEV stats:
    FGT6HD3916801195(updated 1 seconds ago):
        port3: physical/1000full, up, rx-bytes/packets/dropped/errors=14539719170/17400214/118/0, tx=7191448101/14770621/0/0
        port4: physical/1000full, up, rx-bytes/packets/dropped/errors=6601825943/9554019/118/0, tx=6601809359/9553995/0/0
    FGT6HD3916800525(updated 1 seconds ago):
        port3: physical/1000full, up, rx-bytes/packets/dropped/errors=7191366421/14770426/121/0, tx=14539590448/17399920/0/0
        port4: physical/1000full, up, rx-bytes/packets/dropped/errors=6601766321/9553931/122/0, tx=6601712619/9553855/0/0
Slave : External-Backup , FGT6HD3916801195
Master: External-Primary, FGT6HD3916800525
number of vcluster: 1
vcluster 1: standby 169.254.0.2
Slave :1 FGT6HD3916801195
Master:0 FGT6HD3916800525

Get system ha status example - three FortiGates in active-active mode

The following example shows get system ha status output for a cluster of three FortiGate-5001Ds operating in active-active mode. The cluster group ID is set to 20 and session pickup is enabled. Load balance all and the load balancing schedule are set to the default value.

get system ha status
HA Health Status: OK
Model: FortiGate-5001D
Mode: HA A-A
Group: 20
Debug: 0
Cluster Uptime: 7 days 04:50:43
Master selected using:
    <2016/10/12 14:36:03> FG-5KD3914800284 is selected as the master because it has the largest value of override priority.
    <2016/10/12 14:36:03> FG-5KD3914800284 is selected as the master because it has the largest value of override priority.
    <2016/10/12 13:42:46> FG-5KD3914800284 is selected as the master because it has the largest value of override priority.
    <2016/10/12 13:42:43> FG-5KD3914800353 is selected as the master because it has the largest value of uptime.
ses_pickup: enable, ses_pickup_delay=disable
load_balance: disable
load_balance_udp: disable
schedule: Round robin.
upgrade_mode: unset
override: disable
Configuration Status:
    FG-5KD3914800284(updated 4 seconds ago): in-sync
    FG-5KD3914800353(updated 3 seconds ago): in-sync
    FG-5KD3914800344(updated 3 seconds ago): in-sync
System Usage stats:
    FG-5KD3914800284(updated 4 seconds ago):
        sessions=10, average-cpu-user/nice/system/idle=0%/0%/0%/99%, memory=15%
    FG-5KD3914800353(updated 3 seconds ago):
        sessions=0, average-cpu-user/nice/system/idle=0%/0%/0%/99%, memory=14%
    FG-5KD3914800344(updated 3 seconds ago):
        sessions=0, average-cpu-user/nice/system/idle=0%/0%/0%/99%, memory=14%
HBDEV stats:
    FG-5KD3914800284(updated 4 seconds ago):
        base1: physical/1000full, up, rx-bytes/packets/dropped/errors=76249501/186982/230/0, tx=42415292/124396/0/0
        base2: physical/1000full, up, rx-bytes/packets/dropped/errors=63858640/120488/224/0, tx=31951552/60290/0/0
    FG-5KD3914800353(updated 3 seconds ago):
        base1: physical/1000full, up, rx-bytes/packets/dropped/errors=3698904361/7099198/4636/0, tx=1750757534/3891809/0/0
        base2: physical/1000full, up, rx-bytes/packets/dropped/errors=3215777622/6067509/26/0, tx=1608101090/3034153/0/0
    FG-5KD3914800344(updated 3 seconds ago):
        base1: physical/1000full, up, rx-bytes/packets/dropped/errors=58355667/132320/210/0, tx=30477028/82533/0/0
        base2: physical/1000full, up, rx-bytes/packets/dropped/errors=50349972/95004/203/0, tx=25126240/47408/0/0
Master: 5001d-slot4     , FG-5KD3914800284
Slave : 5001d-slot5     , FG-5KD3914800353
Slave : 5001d-slot3     , FG-5KD3914800344
number of vcluster: 1
vcluster 1: work 169.254.0.3
Master:0 FG-5KD3914800284
Slave :2 FG-5KD3914800344
Slave :1 FG-5KD3914800353

Get system ha status example - virtual cluster

The following example shows get system ha status output for a cluster of two FortiGate-5001Ds with virtual clustering enabled. The host names of the FortiGates are 5001d-slot4 and 5001d-slot5.

In this first example the get system ha status command was entered from 5001d_slot5. The output shows that 5001d-slot5 (serial number FG-5KD3914800353) is operating as the primary unit for virtual cluster 1 and the subordinate unit for virtual cluster 2.

get system ha status
HA Health Status: OK
Model: FortiGate-5001D
Mode: HA A-P
Group: 20
Debug: 0
Cluster Uptime: 8 days 00:17:25
Master selected using:
  virtual cluster 1:
    <2016/10/13 09:41:42> FG-5KD3914800353 is selected as the master because it has the largest value of serialno.
    <2016/10/12 15:12:52> FG-5KD3914800284 is selected as the master because it has the largest value of override priority.
    <2016/10/12 15:12:49> FG-5KD3914800353 is selected as the master because it has the largest value of uptime.
    <2016/10/12 15:12:49> FG-5KD3914800353 is selected as the master because it's the only member in the cluster.
  virtual cluster 2:
    <2016/10/13 09:48:37> FG-5KD3914800284 is selected as the master because it has the largest value of override priority.
    <2016/10/13 09:40:51> FG-5KD3914800353 is selected as the master because it has the largest value of serialno.
ses_pickup: enable, ses_pickup_delay=disable
override: vcluster1 enable, vcluster2 enable
Configuration Status:
    FG-5KD3914800353(updated 4 seconds ago): in-sync
    FG-5KD3914800284(updated 4 seconds ago): in-sync
System Usage stats:
    FG-5KD3914800353(updated 4 seconds ago):
        sessions=10, average-cpu-user/nice/system/idle=0%/0%/0%/100%, memory=14%
    FG-5KD3914800284(updated 4 seconds ago):
        sessions=0, average-cpu-user/nice/system/idle=0%/0%/0%/100%, memory=14%
HBDEV stats:
    FG-5KD3914800353(updated 4 seconds ago):
        base1: physical/1000full, up, rx-bytes/packets/dropped/errors=3965558103/7546034/4638/0, tx=1981129551/4331175/0/0
        base2: physical/1000full, up, rx-bytes/packets/dropped/errors=3432622053/6417717/27/0, tx=1823376423/3381402/0/0
    FG-5KD3914800284(updated 4 seconds ago):
        base1: physical/1000full, up, rx-bytes/packets/dropped/errors=308666532/632360/235/0, tx=307993533/571074/0/0
        base2: physical/1000full, up, rx-bytes/packets/dropped/errors=280717442/470725/229/0, tx=247222071/407529/0/0
Master: 5001d-slot5     , FG-5KD3914800353
Slave : 5001d-slot4     , FG-5KD3914800284
number of vcluster: 2
vcluster 1: work 169.254.0.1
Master:0 FG-5KD3914800353
Slave :1 FG-5KD3914800284
vcluster 2: standby 169.254.0.2
Slave :1 FG-5KD3914800353
Master:0 FG-5KD3914800284

The following example shows get system ha status output for the same cluster as shown in the previous example after using execute ha manage 1 to log into 5001d-slot4 (serial number FG-5KD3914800284).

get system ha status
HA Health Status: 
    ERROR: FG-5KD3914800344 is lost @ 2016/10/12 14:46:05
Model: FortiGate-5001D
Mode: HA A-P
Group: 20
Debug: 0
Cluster Uptime: 8 days 00:28:40
Master selected using:
  virtual cluster 1:
    <2016/10/13 09:41:42> FG-5KD3914800353 is selected as the master because it has the largest value of serialno.
    <2016/10/12 15:12:52> FG-5KD3914800284 is selected as the master because it has the largest value of override priority.
    <2016/10/12 15:12:49> FG-5KD3914800353 is selected as the master because it has the largest value of uptime.
    <2016/10/12 14:46:05> FG-5KD3914800284 is selected as the master because it has the largest value of override priority.
  virtual cluster 2:
    <2016/10/13 09:48:37> FG-5KD3914800284 is selected as the master because it has the largest value of override priority.
    <2016/10/13 09:40:51> FG-5KD3914800353 is selected as the master because it has the largest value of serialno.
ses_pickup: enable, ses_pickup_delay=disable

override: vcluster1 enable, vcluster2 enable
Configuration Status:
    FG-5KD3914800284(updated 4 seconds ago): in-sync
    FG-5KD3914800353(updated 4 seconds ago): in-sync
System Usage stats:
    FG-5KD3914800284(updated 4 seconds ago):
        sessions=0, average-cpu-user/nice/system/idle=0%/0%/0%/100%, memory=14%
    FG-5KD3914800353(updated 4 seconds ago):
        sessions=10, average-cpu-user/nice/system/idle=0%/0%/0%/99%, memory=14%
HBDEV stats:
    FG-5KD3914800284(updated 4 seconds ago):
        base1: physical/1000full, up, rx-bytes/packets/dropped/errors=311389146/636565/235/0, tx=310492903/575295/0/0
        base2: physical/1000full, up, rx-bytes/packets/dropped/errors=283083317/474100/229/0, tx=249587946/410904/0/0
    FG-5KD3914800353(updated 4 seconds ago):
        base1: physical/1000full, up, rx-bytes/packets/dropped/errors=3968057548/7550256/4638/0, tx=1983852314/4335382/0/0
        base2: physical/1000full, up, rx-bytes/packets/dropped/errors=3434987928/6421092/27/0, tx=1825742298/3384777/0/0
Slave : 5001d-slot4     , FG-5KD3914800284
Master: 5001d-slot5     , FG-5KD3914800353
number of vcluster: 2
vcluster 1: standby 169.254.0.1
Slave :1 FG-5KD3914800284
Master:0 FG-5KD3914800353
vcluster 2: work 169.254.0.2
Master:0 FG-5KD3914800284
Slave :1 FG-5KD3914800353

About the HA cluster index and the execute ha manage command

When a cluster starts up, the FortiGate Cluster Protocol (FGCP) assigns a cluster index and a HA heartbeat IP address to each cluster unit based on the serial number of the cluster unit. The FGCP selects the cluster unit with the highest serial number to become the primary unit. The FGCP assigns a cluster index of 0 and an HA heartbeat IP address of 169.254.0.1 to this unit. The FGCP assigns a cluster index of 1 and an HA heartbeat IP address of 169.254.0.2 to the cluster unit with the second highest serial number. If the cluster contains more units, the cluster unit with the third highest serial number is assigned a cluster index of 2 and an HA heartbeat IP address of 169.254.0.3, and so on. You can display the cluster index assigned to each cluster unit using the get system ha status command. Also when you use the execute ha manage command you select a cluster unit to log into by entering its cluster index.

The cluster index and HA heartbeat IP address only change if a unit leaves the cluster or if a new unit joins the cluster. When one of these events happens, the FGCP resets the cluster index and HA heartbeat IP address of each cluster unit according to serial number in the same way as when the cluster first starts up.

Each cluster unit keeps its assigned cluster index and HA heartbeat IP address even as the units take on different roles in the cluster. After the initial cluster index and HA heartbeat IP addresses are set according to serial number, the FGCP checks other primary unit selection criteria such as device priority and monitored interfaces. Checking these criteria could result in selecting a cluster unit without the highest serial number to operate as the primary unit.

Even if the cluster unit without the highest serial number now becomes the primary unit, the cluster indexes and HA heartbeat IP addresses assigned to the individual cluster units do not change. Instead the FGCP assigns a second cluster index, which could be called the operating cluster index, to reflect this role change. The operating cluster index is 0 for the primary unit and 1 and higher for the other units in the cluster. By default both sets of cluster indexes are the same. But if primary unit selection selects the cluster unit that does not have the highest serial number to be the primary unit then this cluster unit is assigned an operating cluster index of 0. The operating cluster index is used by the FGCP only. You can display the operating cluster index assigned to each cluster unit using the get system ha status command. There are no CLI commands that reference the operating cluster index.

Even though there are two cluster indexes there is only one HA heartbeat IP address and the HA heartbeat address is not affected by a change in the operating cluster index.

Using the execute ha manage command

When you use the CLI command execute ha manage <index_integer> to connect to the CLI of another cluster unit, the <index_integer> that you enter is the cluster index of the unit that you want to connect to.

Using get system ha status to display cluster indexes

You can display the cluster index assigned to each cluster unit using the CLI command get system ha status. The following example shows the information displayed by the get system ha status command for a cluster consisting of two FortiGates operating in active-passive HA mode with virtual domains not enabled and without virtual clustering.

get system ha status
.
.
.
Slave :1 FGT6HD3916801195
Master:0 FGT6HD3916800525

In this example, the cluster unit with serial number FG50012205400050 has the highest serial number and so has a cluster index of 0 and the cluster unit with serial number FG50012204400045 has a cluster index of 1. From the CLI of the primary unit of this cluster you can connect to the CLI of the subordinate unit using the following command:

execute ha manage 1

This works because the cluster unit with serial number FG50012204400045 has a cluster index of 1.

The get system ha status command output shows two similar lists of indexes and serial numbers. The listing on the sixth and seventh lines of the command output are the cluster indexes assigned according to cluster unit serial number. These are the cluster indexes that you enter when using the execute ha manage command. The cluster indexes shown in the last two lines of the command output are the operating cluster indexes that reflect how the cluster units are actually operating in the cluster. In this example both sets of cluster indexes are the same.

The last three lines of the command output display the status of vcluster 1. In a cluster consisting of two cluster units operating without virtual domains enabled all clustering actually takes place in virtual cluster 1. HA is designed to work this way to support virtual clustering. If this cluster was operating with virtual domains enabled, adding virtual cluster 2 is similar to adding a new copy of virtual cluster 1. Virtual cluster 2 is visible in the get system ha status command output when you add virtual domains to virtual cluster 2.

The HA heartbeat IP address displayed on line 8 is the HA heartbeat IP address of the cluster unit that is actually operating as the primary unit. For a default configuration this IP address will always be 169.254.0.1 because the cluster unit with the highest serial number will be the primary unit. This IP address changes if the operating primary unit is not the primary unit with the highest serial number.

Example actual and operating cluster indexes do not match

This example shows get system ha status command output for same cluster of two FortiGate-5001SX units. However, in this example the device priority of the cluster unit with the serial number FG50012204400045 is increased to 200. As a result the cluster unit with the lowest serial number becomes the primary unit. This means the actual and operating cluster indexes of the cluster units do not match.

get system ha status

Master:1 FG50012205400050

Slave :0 FG50012204400045

The actual cluster indexes have not changed but the operating cluster indexes have. Also, the HA heartbeat IP address displayed for vcluster 1 has changed to 169.254.0.2.

Virtual clustering example output

The get system ha status command output is the same if a cluster is operating with virtual clustering turned on but with all virtual domains in virtual cluster 1. The following get system ha status command output example shows the same cluster operating as a virtual cluster with virtual domains in virtual cluster 1 and added to virtual cluster 2. In this example the cluster unit with serial number FG50012204400045 is the primary unit for virtual cluster 1 and the cluster unit with serial number FG50012205400050 is the primary unit for virtual cluster 2.

get system ha status

number of vcluster: 2

vcluster 1: work 169.254.0.2

Master:1 FG50012205400050

Slave :0 FG50012204400045

vcluster 2: standby 169.254.0.1

Master:0 FG50012205400050

Slave :1 FG50012204400045

This example shows three sets of indexes. The indexes in lines six and seven are still used by the execute ha manage command. The indexes on lines ten and eleven are for the primary and subordinate units in virtual cluster 1 and the indexes on the last two lines are for virtual cluster 2.