Quantum Myriad All-Flash Scale-Out Architecture

Ben Jarvis, Lead Storage Architect, discusses the innovative architecture of the Quantum Myriad all-flash scale-out storage system. Myriad was designed from the ground up to handle high-performance video and HPC workloads, leveraging large scale-out internet-facing service paradigms. It utilizes a microservices container-based system running on Kubernetes, deployed on white box commodity hardware, and is optimized for NVMe flash storage and high-performance RDMA Ethernet networks.
The system is designed to invert traditional storage models by optimizing software to fully utilize the hardware’s capabilities, focusing on memory bandwidth and minimizing context switches. Myriad’s architecture supports inline data protection, data reduction through compression and deduplication, and presents a self-contained cluster to customers.
Jarvis explains the integration with Kubernetes for state management and the system’s approach to handling physical storage on nodes. Myriad operates on a transactional key-value store basis, supporting transactions necessary for a POSIX-compliant file system. It employs a redirected write system with data protection through mirroring and erasure encoding, optimizing for low-latency, high-concurrency operations. The system’s design avoids performance bottlenecks common in traditional storage systems by adopting a work-at-risk approach for managing transactions. This method allows for independent operations with synchronization only at the end, significantly reducing the probability of conflicts and enabling the system to scale efficiently. Myriad is optimized for workloads that do not involve simultaneous writes to the same file by multiple nodes, steering clear of legacy HPC workloads that do not align with its architectural advantages.

Presenter: Ben Jarvis   / ben-jarvis-132b2220  

Delegates:
Andrea Mauro   / andrea_mauro  
David Klee   / kleegeek  
Justin Warren   / jpwarren  
Max Mortillaro   / maxmortillaro