Can clickhouse-backup use HDFS as remote_storage? Recommended approach?

[qiuyangliu]

We are running clickhouse-backup inside a Kubernetes pod and would like to store backup archives on HDFS.

Does clickhouse-backup support HDFS as a remote_storage backend (i.e. set remote_storage: hdfs or similar)?

Thanks

[Slach]

you have following options

HDFS disk as backup storage + BACKUP sql command

/etc/clickhouse-server/config.d/hdfs_backup_disk.xml

<clickhouse>
    <!-- optional params for authorization -->
    <backups_hdfs>
        <hadoop_kerberos_keytab>/tmp/keytab/clickhouse.keytab</hadoop_kerberos_keytab>
        <hadoop_kerberos_principal>clickuser@TEST.CLICKHOUSE.TECH</hadoop_kerberos_principal>
        <hadoop_security_authentication>kerberos</hadoop_security_authentication>
    </backups_hdfs>

  <storage_policy>
    <storage_configuration>
      <disks>
        <hdfs>
          <type>hdfs</type>
          <endpoint>hdfs://hdfs1:9000/clickhouse/</endpoint>
          <skip_access_check>true</skip_access_check>
        </hdfs>
      </disks>
   </storage_configuration>
    <backups>
        <allowed_disk>backups_hdfs</allowed_disk>
    </backups>
</clickhouse>

then

BACKUP ALL TO Disk('backups_hdfs','backup_name')

remote_storage: custom and hdfs dfs command

look similar examples for rsync in
https://github.com/Altinity/clickhouse-backup/tree/master/test/integration/rsync
and
https://github.com/Altinity/clickhouse-backup/blob/master/test/integration/configs/config-custom-rsync.yml

hdfs_list.sh could look like that

#!/bin/bash

# List directories in HDFS

directories=`/home/hadoop/hadoop/bin/hdfs dfs -ls hdfs://hostname:9000/clickhouse/ | grep "^d"`

# Iterate over directories

echo $directories | while read -r line; do

  # Extract backup name and creation date from line

  backup_name=`echo $line | awk '{print \$8}'`

  creation_date=`echo $line | awk '{print \$6"T"\$7":00Z"}'`

  # Read metadata.json from HDFS

  metadata=`/home/hadoop/hadoop/bin/hdfs dfs -Ddfs.client.use.datanode.hostname=true -cat $backup_name/metadata.json`

  # Extract data_size and metadata_size from metadata.json

  data_size=`echo $metadata | jq '.data_size'`

  metadata_size=`echo $metadata | jq '.metadata_size'`

  # Print JSON

  echo "{\"backup_name\":\"$backup_name\",\"creation_date\":\"$creation_date\",\"data_size\":$data_size,\"metadata_size\":$metadata_size}"

done