Test Lustre Cluster Throughput and High Availability options

Post published:7 January 2025
Post category:Cloud
Post comments:0 Comments

In this video (English subtitled), I demonstrate the aggregate performance of a 4-node Lustre cluster (3 OSS + 1 MDS/MGS) and the block volumes provided by Oracle OCI. Of course, this is a functionality test and not a production POC. In addition to understanding and observing the aggregate performance that only 3 active OSS nodes can deliver, I wanted to test 2 failover systems between OSS1 and OSS2 nodes (OSS3 is unpaired and works without failover).

1 – The first test, provided by Pacemaker (best practice), delivers performance while ensuring reliability between the two nodes by using block volumes shared attached between the hosts. In the event of a failure, the second node (and vice versa) can take over the shared OSTs and continue providing the service without any interruption. When the host in a down state becomes available again, the OSTs (Block volumes) will be reassigned and managed by the original host.

2 – In the second test, I used DRBD to synchronize the two block volumes, each exclusively attached to its respective OSS node. The synchronization is synchronous, so the network bandwidth consumption it requires must be considered. Additionally, in this case, there is an active primary node and a secondary standby node (from the local block volume ‘OST’ perspective) , meaning the standby node does not deliver usable throughput. I conducted this test to simulate the use of a local storage resource (such as an NVMe disk) as support for the Lustre cluster.

Conclusion:

We can see how both failover systems are capable of handling a potential node failure, allowing the implementation of one system or the other depending on the requirements and the desired infrastructure. Additionally, we can observe that with only 3 active data nodes (OSS), this Lustre cluster can deliver nearly 14 GB/s of aggregate throughput. By adding a fourth OSS and pairing it with OSS3 using Pacemaker, approximately 20 GB/s could be achieved keeping the service in high availability.

If you enjoyed this, don’t miss the other posts about Lustre:

At the time of publishing this post, I am an Oracle employee, but he views expressed on this blog are my own and do not necessarily reflect the views of Oracle.