CHFS is a parallel caching file system that is created instantly using node-local storage, such as persistent memory and NVMe SSD, across multiple compute nodes. It leverages the performance of persistent memory using pmemkv, a persistent in-memory key-value store. For NVMe SSDs, it uses the POSIX interface. CHFS leverages RDMA for high-performance data access.
CHFS provides a caching mechanism against a backend parallel file system. Files in the backend parallel file system are automatically cached on demand or manually staged in. Output files are automatically flushed by I/O-aware flushing that maximizes the performance of CHFS.
# apt install gcc g++ automake cmake libtool pkgconf \
rdma-core librdmacm-dev \
libfuse3-dev fuse3 pandoc \
git python3 bzip2 xz-utils vim% git clone --depth=2 https://github.com/spack/spack.git
% . spack/share/spack/setup-env.shFor details, see https://spack.readthedocs.io/
% spack external find autoconf automake libtool cmake m4 rdma-core pkgconf gmake
% spack install mochi-margo ^mercury~boostsys ^libfabric fabrics=rxm,sockets,tcp,udp,verbsFor details, see https://mochi.readthedocs.io/
% git clone https://github.com/otatebe/chfs.git
% cd chfs
% autoreconf -i
% mkdir build
% cd build
% spack load mochi-margo
% ../configure [--prefix=PREFIX]
% make
# make installIf you use the pmemkv backend, install pmemkv and specify --with-pmemkv in configure.
% eval `chfsctl [-h hostfile] [-p verbs] [-c /dev/dax0.0] [-b /back/end/path] [-m /mount/point] start`This executes chfsd servers and mounts CHFS at /mount/point on hosts specified by the hostfile. The -p option specifies a communication protocol. The -c option specifies a devdax device or a scratch directory on each host.
The backend directory, typically in a parallel file system, can be specified by the -b option. Files in the backend directory can be transparently accessed at the CHFS mount directory when the -m option is specified. If all files in the backend file system need to be accessed, you can specify / as a backend directory. For efficient access, files can be staged in beforehand by the chstagein command. The following example stages in all files in the backend directory.
% cd /back/end/path
% find . | xargs [ mpirun ... ] chstageinchstagein can be executed with and without mpirun. The output files will be flushed automatically to the backend directory. To ensure that all dirty files are flushed, use chfs_sync() or chfsctl stop.
The -m option specifies the mount directory of CHFS. When this option is specified, the CHFS subdirectory with the same path as the mount directory is mounted. That is, files in CHFS can be accessed using the same path both through the POSIX interface via chfuse and through the CHFS API.
For a devdax device, a pmem obj pool should be created with the layout pmemkv by pmempool create -l pmemkv obj /dev/dax0.0. For user-level access, the permissions of the device should be changed, and bad block check should be disabled by pmempool feature --disable CHECK_BAD_BLOCKS /dev/dax0.0.
chfsctl outputs the settings of the CHFS_SERVER, CHFS_BACKEND_PATH, and CHFS_SUBDIR_PATH environment variables, which are used to execute chfuse and CHFS commands.
For details, see manual page of chfsctl.
By default, communication between a client and a server on the same node also uses the protocol specified by the -p option. When the -a option is specified to chfsd, shared memory is used within a node (Mercury auto_sm). Clients detect it from the server address and use it automatically.
% eval `chfsctl [-h hostfile] [-p verbs] -O -a [-m /mount/point] start`chfsd and clients on the same node need to be executed by the same user. Data transfers within a node use cross-memory attach (process_vm_readv and process_vm_writev), which needs to be permitted for both chfsd and clients. Otherwise, transfers larger than several KiB within a node fail. Many distributions including Ubuntu set kernel.yama.ptrace_scope to 1, which does not permit it. Set it to 0 on every node as follows. Note that this allows a process to access the memory of other processes of the same user.
# sysctl kernel.yama.ptrace_scope=0To keep the setting after reboot, add a file that is read after /etc/sysctl.d/10-ptrace.conf.
# echo kernel.yama.ptrace_scope=0 > /etc/sysctl.d/60-ptrace.confContainers share the kernel setting of the host, so set it on the host.
CHFS is mounted by the -m option of the chfsctl command. To mount it on other hosts, use the chfuse command.
% chfuse -o direct_io /mount/pointTo mount the CHFS subdirectory in the same way as chfsctl, use the subdir module as follows.
% export CHFS_SUBDIR_PATH=/mount/point
% chfuse -o direct_io,modules=subdir,subdir=$CHFS_SUBDIR_PATH $CHFS_SUBDIR_PATHCHFS_SERVER and other environment variables output by the chfsctl command should be defined.
For details, see manual page of chfuse.
POSIX programs can access CHFS using the CHFS-zpoline system call interception library without source-code modification and without mounting CHFS by chfuse.
% git clone --recursive https://github.com/otatebe/chfs-zpoline.git
% cd chfs-zpoline
% autoreconf -i
% ./configure [--prefix=PREFIX]
% make
# make installWhen using CHFS-zpoline, CHFS is virtually mounted on /chfs, or / when specifying cached directories using LIBZPDIRS. When LIBZPDIRS is specified, accesses to files in the specified directories are hooked. CHFS can also be accessed by a relative path.
% sudo sysctl vm.mmap_min_addr=0
% export LIBZPHOOK=$PREFIX/lib/libcz.so
% export LIBZPDIRS="$PWD /data"
% LD_PRELOAD=$PREFIX/lib/libzpoline.so program ...- chlist(1) - list CHFS servers
- chmkdir(1) - create a directory in CHFS
- chrmdir(1) - remove a directory in CHFS
- chfind(1) - search for files in CHFS
- chstagein(1) - stage-in files to CHFS
- chfsd(8) - CHFS server, which is usually executed by chfsctl
- CHFS_SERVER - server addresses separated by ','
- CHFS_BACKEND_PATH - backend path
- CHFS_SUBDIR_PATH - current working directory in CHFS
- CHFS_CHUNK_SIZE - chunk size
- CHFS_ASYNC_ACCESS - set 1 to enable asynchronous accesses
- CHFS_BUF_SIZE - buffer size
- CHFS_LOOKUP_LOCAL - set 1 to connect to a local chfsd only
- CHFS_LOOKUP_RELAY_GROUP - group number for relayed connections to chfsd
- CHFS_LOOKUP_RELAY_GROUP_AUTO - set 1 for setting group number automatically for relayed connections to chfsd
- CHFS_RDMA_THRESH - RDMA transfer is used when the size is larger than CHFS_RDMA_THRESH
- CHFS_RPC_TIMEOUT_MSEC - RPC timeout in milliseconds
- CHFS_NODE_LIST_CACHE_TIMEOUT - node list cache timeout in seconds
- CHFS_LOG_PRIORITY - maximum log priority to report
When you use pmemkv, devdax is preferable. When you use fsdax, the following environment variable is recommended to touch every page of the persistent memory pool, although it slows down the startup of chfsd.
- PMEMOBJ_CONF="prefault.at_open=1;prefault.at_create=1"
ROMIO ADIO for CHFS is available. With the ROMIO ADIO for CHFS, MPI-IO applications can access CHFS without any source code modification. You can access CHFS by chfs:$MDIR/path/name or by specifying ROMIO_FSTYPE_FORCE=chfs:, where $MDIR is the mount directory of CHFS.
% apt install gfortran wget
% wget https://download.open-mpi.org/release/open-mpi/v5.0/openmpi-5.0.11.tar.bz2
% tar xfp openmpi-5.0.11.tar.bz2
% cd openmpi-5.0.11
% wget https://raw.githubusercontent.com/otatebe/chfs/cache/dev/ompi/ad_chfs-5.0.patch
% patch -p1 < ad_chfs-5.0.patch
% (cd 3rd-party/romio341 && ./autogen.sh)
% mkdir build && cd build
% ../configure --enable-prte-prefix-by-default --with-io-romio-flags=--with-file-system=chfs+ufs+testfs
% make -j $(nproc)
# make installFor Open MPI 4.1.x, use ad_chfs-4.1.patch instead, run ./autogen.sh in ompi/mca/io/romio321/romio, and specify --mca io romio321 instead of --mca io romio341 in the following.
Make sure that the mpicc of the installed Open MPI is in your PATH.
% cd chfs/build
% spack load mochi-margo
% ../configure [--prefix=PREFIX] [--enable-zero-copy-read-rdma]
% make
# make install% mpirun --mca io romio341 -x CHFS_SERVER -x CHFS_BACKEND_PATH -x CHFS_SUBDIR_PATH ...Since CHFS does not support file locking, the following features are not supported. Shared file pointer operations return MPI_ERR_UNSUPPORTED_OPERATION, and MPI_File_set_atomicity() returns MPI_ERR_ARG.
- Shared file pointers - MPI_File_{read,write}shared(), MPI_File{iread,iwrite}shared(), MPI_File{read,write}ordered(), MPI_File{read,write}ordered{begin,end}(), MPI_File_seek_shared(), MPI_File_get_position_shared(), and MPI_MODE_SEQUENTIAL
- Atomic mode - MPI_File_set_atomicity()
- Data sieving for writes - the romio_ds_write hint is always disabled
% git clone https://github.com/hpc/ior.git
% cd ior
% ./bootstrap
% ./configure [--prefix=PREFIX]
% make
# make install% mpirun -x CHFS_SERVER -x CHFS_BACKEND_PATH -x CHFS_SUBDIR_PATH ior -a CHFS [--chfs.chunk_size=SIZE]A large chunk size, e.g., 1 MiB, should be specified for the best performance. If you are using Open MPI with CHFS ADIO, you can use the MPIIO backend by -a MPIIO with chfs:$MDIR/file, or with -x ROMIO_FSTYPE_FORCE=chfs: and a relative path, where $MDIR is the mount directory of CHFS. If you are using CHFS-zpoline, you can use the POSIX backend with a relative path or /chfs$MDIR/file.
The following APIs are supported.
int chfs_init(const char *server);
int chfs_initialized();
int chfs_term();
int chfs_term_without_sync();
int chfs_size();
const char *chfs_version();
void chfs_set_chunk_size(int chunk_size);
void chfs_set_async_access(int enable);
void chfs_set_buf_size(int buf_size);
void chfs_set_stagein_buf_size(int buf_size);
void chfs_set_rdma_thresh(size_t thresh);
void chfs_set_rpc_timeout_msec(int timeout_msec);
void chfs_set_node_list_cache_timeout(int timeout_sec);
int chfs_create(const char *path, int32_t flags, mode_t mode);
int chfs_create_chunk_size(const char *path, int32_t flags, mode_t mode,
int chunk_size);
int chfs_open(const char *path, int32_t flags);
int chfs_close(int fd);
char *chfs_path_at(int fd, const char *path);
int chfs_chdir(const char *path);
int chfs_fchdir(int fd);
ssize_t chfs_pwrite(int fd, const void *buf, size_t size, off_t offset);
ssize_t chfs_write(int fd, const void *buf, size_t size);
ssize_t chfs_pread(int fd, void *buf, size_t size, off_t offset);
ssize_t chfs_read(int fd, void *buf, size_t size);
off_t chfs_seek(int fd, off_t off, int whence);
int chfs_fsync(int fd);
int chfs_truncate(const char *path, off_t len);
int chfs_ftruncate(int fd, off_t len);
int chfs_unlink(const char *path);
int chfs_mkdir(const char *path, mode_t mode);
int chfs_rmdir(const char *path);
int chfs_stat(const char *path, struct stat *st);
int chfs_fstat(int fd, struct stat *st);
int chfs_lstat(const char *path, struct stat *st);
int chfs_access(const char *path, int mode);
int chfs_readdir(const char *path, void *buf,
int (*filler)(void *, const char *, const struct stat *, off_t));
int chfs_readdir_index(const char *path, int index, void *buf,
int (*filler)(void *, const char *, const struct stat *, off_t));
int chfs_linux_getdents64(int fd, char dirp[], size_t count);
int chfs_symlink(const char *target, const char *path);
int chfs_readlink(const char *path, char *buf, size_t size);
void chfs_sync();
int chfs_stagein(const char *path);-
Osamu Tatebe, Kazuki Obata, Kohei Hiraga, Hiroki Ohtsuji, "CHFS: Parallel Consistent Hashing File System for Node-local Persistent Memory", Proceedings of the ACM International Conference on High Performance Computing in Asia-Pacific Region (HPC Asia 2022), pp.115-124, 2022
-
Osamu Tatebe, Hiroki Ohtsuji, "Caching Support for CHFS Node-local Persistent Memory File System", Proceedings of 3rd Workshop on Extreme-Scale Storage and Analysis (ESSA 2022), pp.1103-1110, 2022
-
Osamu Tatebe, Kohei Hiraga, Hiroki Ohtsuji, "I/O-Aware Flushing for HPC Caching Filesystem", Proceedings of 3rd Workshop on Re-envisioning Extreme-Scale I/O for Emerging Hybrid HPC Workloads (REX-IO), pp.11-17, 2023
-
Haruka Miyauchi, Sohei Koyama (advisor), Osamu Tatebe (advisor), "Design of Reliable and Efficient Syscall Hooking Library for a Parallel File System", The International Conference for High Performance Computing, Networking, Storage, and Analysis (SC), ACM Student Research Competition Undergraduate, Atlanta GA, November 19-21, 2024