Handling ClickHouse Node IP Change

IP address are embedded in various ClickHouse configuration files and Redis. If you change the IP address of any ClickHouse node, then take these following steps to restore communication.

Step 1: If Keeper node IP changed, then update ClickHouse Keeper registry file on all Keeper nodes to reflect the new IP address.

1. Login to the node.

2. Open the file data-clickhouse-hot-1/clickhouse-keeper/conf/keeper.xml.

3. Change the following section for the node whose IP changed.
   

      <raft_configuration>
         <server>
           <id>1</id>
           <hostname>172.30.57.174</hostname>
           <port>3888</port>
         </server>
         <server>
           <id>2</id>
           <hostname>172.30.57.175</hostname>
           <port>3888</port>
         </server>
       </raft_configuration>
   
   

Step 2: If Data/Query node IP changed, then update ClickHouse Keeper file on all ClickHouse Data/Query nodes.

1. Login to the node.

2. Open the file /etc/clickhouse-server/config.d/zookeeper.xml.

3. Change the following section.
   

   <zookeeper>
       <node>
         <host>172.30.57.174</host>
         <port>2181</port>
       </node>
       <node>
         <host>172.30.57.175</host>
         <port>2181</port>
       </node>
     </zookeeper>
   
   

Step 3: If Keeper node IP changed, then update Redis key of ClickHouse Keeper node from Supervisor.

1. Login to Supervisor node.

2. Run the following command to find out the list of ClickHouse Keeper IP addresses.
   

   #redis-cli -p 6666 -a `phLicenseTool --showRedisPassword` get cache:ClickHouse:clickhouseKeeperNodes
   

   Example Execution and Output:

    

   # redis-cli -p 6666 -a `phLicenseTool --showRedisPassword` get cache:ClickHouse:clickhouseKeeperNodes Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. "172.30.57.235"

3. Update the IP list with the new IP address.
   

   #redis-cli -p 6666 -a `phLicenseTool --showRedisPassword` set cache:ClickHouse:clickhouseKeeperNodes '<new_ip_list>'
   

   Example Execution and Output:

   # redis-cli -p 6666 -a `phLicenseTool --showRedisPassword` set cache:ClickHouse:clickhouseKeeperNodes '172.30.57.230' Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. OK

Step 4: If Data/Query node IP changed, then update Redis key of ClickHouse Data/Query node.

1. Login to Supervisor node.

2. Run the following command to find out the list of ClickHouse Keeper IP addresses.
   

   #redis-cli -p 6666 -a `phLicenseTool --showRedisPassword` get cache:ClickHouse:clickhouseNodes
   

   Example Execution and Output:

   # redis-cli -p 6666 -a `phLicenseTool --showRedisPassword` get cache:ClickHouse:clickhouseNodes Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. "172.30.57.235"

3. Update the IP list with the new IP address.
   

   #redis-cli -p 6666 -a `phLicenseTool --showRedisPassword` set cache:ClickHouse:clickhouseNodes '<new_ip_list>'

   Example Execution and Output:

   # redis-cli -p 6666 -a `phLicenseTool --showRedisPassword` set cache:ClickHouse:clickhouseNodes '172.30.57.230' Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. OK

Step 5: If Data/Query node IP changed, then update interserver_http_host and cluster file on this node where the IP changed.

interserver_http_host is the ip/hostname that can be used by other ClickHouse node to access this node.

1. Login to the Data/Query node where the IP changed.

2. Open /etc/clickhouse-server/config.d/interserver_http_host.xml file.

3. Change the following section to reflect the new IP.
   

   <yandex>
   
     <interserver_http_host>172.30.58.241</interserver_http_host>
   
   </yandex>
   

4. Update the IP address by opening the /etc/clickhouse-server/config.d/cluster.xml file.
   

5. Change the following section to reflect the new IP. In this example: 172.30.58.241.

   <yandex>
   
     <remote_servers>
       <fsiem_cluster>
         <shard>
           <internal_replication>true</internal_replication>
           <replica>
             <host>172.30.58.241</host>
             <port>9000</port>
           </replica>
         </shard>
       </fsiem_cluster>
     </remote_servers>
   
   </yandex>
   

Step 6: Test and Save from FortiSIEM GUI

1. Login to the GUI.

2. Go to ADMIN > Settings > Database > ClickHouse Config.

3. Click Test.

4. If successful, then click Deploy.