Improving Remote File Access in Distributed Object Stores by Decoupling Metadata and Data Paths Using NVMe-oF

Storage network protocols such as NVMe-oF operate below the file system layer. Therefore, even when NVMe-oF allows storage volumes to be shared across the network, compute nodes cannot access remote files managed by another node’s file system without a cluster file system. In conventional distributed systems, accessing files owned by a remote node requires communication with the remote node via Remote Procedure Call (RPC). The remote node then retrieves the data from a disaggregated storage node and transfers it to the requesting node. To reduce redundant network traffic, this study proposes remote direct I/O (Rdio), which separates RPC-based remote data access into two distinct planes: a metadata plane for file mapping and a data plane for direct access to remote storage. The data plane ensures data flows only through the storage network. We integrate Rdio into MinIO and show that Rdio significantly improves performance by reducing remote data movement between nodes.