Download a PDF formatted version of this article

FSx for Lustre — Is There a Better Alternative for Fast File Storage on AWS?

FSx for Lustre is a frequent, and often expensive, cloud file storage choice for high performance computing (HPC) clusters on Amazon Web Services (AWS). It’s based on the Lustre open-source parallel distributed file system, which splits file data into chunks and spreads them across a cluster of storage servers with directly attached SSDs, enabling file data to be read and written in parallel. FSx for Lustre can provide good performance, but it comes with significant drawbacks — and costs:

Aggregate throughput¹ is directly proportional to deployed capacity. FSx for Lustre does have selectable levels of throughput per terabyte. The highest tier, the one most likely to be used in HPC at scale, is 1000 MBps/TiB. Increasing throughput beyond that is done solely by adding capacity — which requires adding servers, each with the same fixed amount of directly-attached storage. FSx for Lustre capacities include 1.2 TiB, 2.4 TiB, and increments of 2.4 TiB. Latencies are consistently sub-millisecond regardless of file system size.

FSx for Lustre: to increase throughput you must increase capacity (add file servers) — manually

FSx for Lustre can quickly become expensive. Especially when you’re forced to buy more capacity than needed just to obtain needed throughput. To reach an aggregate throughput of 17,000 MBps, for example, you should plan on deploying 19.2 TiB of capacity — regardless of whether you need that much storage space. Compression doesn’t help in this situation either, because throughput is a function of deployed capacity — the number of storage servers and their attached SSDs — not the space used.

Over-provisioning capacity is just the beginning.

Operational costs. Capacity and throughput planning — along with regular monitoring and configuration adjustments — are required because FSx for Lustre is not elastic. Increasing or decreasing capacity and throughput is done manually. The good news is that when you initiate a capacity increase (via GUI, command line, etc.), FSx for Lustre automatically adds new storage servers, adds more metadata nodes if needed, and then rebalances file data across the existing and newly added storage servers in the background.

The bad news comes when you want to decrease capacity or throughput. It’s not possible to shrink an FSx for Lustre file system. The only way to accomplish a capacity decrease is to create a new file system volume at the smaller desired size, stop all production access to the old volume, move all the files, then restart production — making sure application are accessing the new volume. Then delete the old volume.

This kind of manual operational effort and expense negates a major reason for using public Cloud.

Backups needed to protect Lustre data. AWS offers options for automatic daily backups and manually initiated backups of FSx for Lustre file systems — if they’re not linked to Amazon S3 (more on that in a moment). This enables you to restore data to the way it was when a previous backup was taken, which can be important if files were accidentally changed or deleted. Backups are stored in Amazon S3 to ensure they won’t be lost forever because of a system failure. (S3 has 99.999999999% or “eleven nines” durability.)

You can link an FSx for Lustre file system to an S3 bucket, making S3 the primary data repository for Lustre data. Using S3 this way provides eleven nines durability to Lustre data (more accurately, the linked copy), so Amazon doesn’t offer backups of S3-linked Luster file systems. Want to restore data to a previous point in time? If you’re running Amazon S3 Backup, you can restore data from one of those backups to the Lustre-linked S3 bucket.

Amazon implements Lustre-to-S3 linking using two mechanisms: (1) “automatic import” or “import data repository tasks” (or both) to copy S3 data into Lustre, and (2) “automatic export” or “export data repository tasks” (or both) to copy data changes made in Lustre (e.g., the results of an HPC workflow) back to S3.

At this point you’re probably thinking, “Wait. I’d be storing two copies of my data. Would I have to pay for both copies?”

Yes, you would. The good news is there’s a way store less than the entire S3 bucket in Lustre. The bad news is you do it manually using another type of “data repository task” that enables you to “release” inactive data — i.e., remove it from Lustre but leave it in S3. You specify the paths of directories or files you want to release and (optionally) a minimum amount of time since last access for a file to be eligible for release. The system will remove the selected data from Lustre but keep it in S3. However, the Lustre file system can still run out of space. So, you’ll need to regularly monitor Lustre and adjust repository tasks as needed.

In other words, even more operational costs.

Lustre file system and client kernel compatibility. Lustre uses kernel modules, which have dependencies on other kernel modules to operate correctly. That means each version of FSx for Lustre supports specific versions of the Linux kernel. You must select the correct Lustre version to ensure access to Lustre storage. When you decide to upgrade Linux, you must also check whether you need a corresponding upgrade for Lustre. And vice-versa.

So, what’s the alternative? Is there something that can give me the performance I need without the management headaches and big price tag?

Yes, there is: flexFS™, a high-performance POSIX-compliant file system that — instead of creating and operating its own object storage infrastructure — leverages the low cost, scale, and aggregate throughput of hyperscale object storage such as AWS S3². The result is truly elastic file storage that outperforms conventional cloud file systems — for significantly less cost.

flexFS also splits file data into chunks, but instead of spreading them across a private storage cluster, it stores the chunks as objects in hyperscale object storage. FlexFS achieves hyperscale throughput by having compute instances perform file I/O directly with the hyperscale object store (#1 in the figure above). Throughput and latency performance are thus inherited directly from the S3 object store, which has virtually limitless throughput capacity that is truly elastic.

flexFS: throughput and capacity are not linked — and are both fully elastic

S3 does have a weakness: latency performance for file metadata I/O, data I/O to many small files, and random I/O — which flexFS overcomes in two ways:

1. Metadata needed to map file blocks to objects is managed outside the object store in a separate, low-latency persistent service (#2 in the figure above).
2. An optional, distributed “proxy group” forms a write-back cache for handling I/O to many small files, especially random I/O (#3 in the figure above). The proxy group can be set to cache a specific number of blocks per file in each volume (for example, the first ten blocks). I/O to those blocks are the only operations that have another server in the path. All other I/O remains directly between the compute instance and the object store. (flexFS mount clients decide when to use each path. They don’t check with some other entity first.)

Customers such as Alnylam, Amgen, Bristol Myers Squibb and NASA have been using flexFS in production for HPC workloads, and for general-purpose file sharing by hundreds of concurrent users, for years.

Okay, how do these two file systems compare? Let’s contrast flexFS with each of the FSx for Lustre attributes we outlined above:

Aggregate throughput is thoroughly elastic and limited only by aggregate network bandwidth. Even better, throughput is completely unrelated to storage capacity. You can get the throughput you need regardless of how much storage you use. For latency-sensitive workloads, flexFS performance is tunable over a range from fully throughput-optimized to fully latency-optimized — or both — using a “proxy group”, a cluster of servers that form a distributed write-back cache.

flexFS costs significantly less than FSx for Lustre.The greater the capacity, the greater the cost relief. Even better, you avoid the need to over-provision storage just to get needed throughput!

Operational costs disappear. Because FlexFS capacity and throughput are both elastic and independent of each other, you avoid Lustre’s capacity planning, monitoring, and manual configuration — time your technical staff can better spend on higher-value work.

Continuous data protection and eleven-nines durability. flexFS creates a snapshot for every data modification, enabling you to restore a volume or file to any previous point in time — as far back as your settable retention policy allows — without needed to restore from a separate backup. Because flexFS stores data in S3, it inherits the latter’s 99.999999999% data durability.

Independence from Linux kernel versions. flexFS is a FUSE file system, making it independent of Linux kernel versions — freeing you from the complexity of checking a supported version matrix before any upgrade³.

In addition, flexFS includes data at-rest and in-flight encryption security that’s activated by default. Adding optional end-to-end encryption ensures no one outside your organization — not even the cloud providers themselves — can access your data.

flexFS has full support for extended file attributes (xAttr), extended Access Control Lists (xACL) and Linux advisory file locking.

Intrigued? Find more info at flexfs.io, documentation site docs.flexfs.io, or contact the flexFS team at info@flexfs.io. You can also download a PDF of this article.

¹ The combined data throughput for an entire HPC cluster. In I/O-intense workloads, adding compute nodes translates directly to a proportional demand for higher throughput.

² Other examples include Azure Blob, Google Cloud Storage, OCI Object Storage, etc.

³ There have been countless discussions of in-kernel file systems having lower latency than FUSE file systems. That argument holds water when the storage is locally attached NVMe drives. And only in-kernel file systems can be used on boot media. But the added latency of networks — such as those used by flexFS and FSx for Lustre on AWS — dwarfs any difference caused by where file system code resides.