On paper, these three do the same job. They give many clients a shared POSIX filesystem. You mount them, you see files, your application does not need to change.
In practice they behave in completely different ways, and the difference does not show up in the datasheet. It shows up two months later, when the job that ran fine on test data slows down on the real dataset and nobody can explain why.
I have written about all three separately. This post is the part I kept skipping: how to decide which one you need before you build it.
Ask about the workload first
Most storage decisions start from the wrong question. People ask how many terabytes and how many euros per terabyte. Those are the last two questions, not the first.
Start with five things about your workload. You can usually get these numbers in half a day. Once you have them, most of the architecture is already decided.
File size distribution. Are you reading a few very large files, or millions of small ones? This single number can move the bottleneck from bandwidth to metadata, and no amount of extra bandwidth will help you once it has moved.
Read/write pattern. Are you reading or writing sequentially, randomly, or both? A training job streaming through shards is not the same as an application reading small pieces from the middle of many files. The same distinction matters on writes: large sequential writes, random updates and append heavy workloads stress different parts of the storage stack.
Metadata intensity. How many opens, stats and lookups do you do for every megabyte you actually read or write? A job that opens ten million files reads mostly metadata, even if it thinks it is reading data.
Concurrent clients. Ten clients and five hundred clients are different systems, not the same system with a bigger number.
Working set and cache reuse. Does the dataset get read once, or do you read the same data repeatedly? A workload with high cache reuse behaves very differently from one that streams through data once and never touches it again.
Write those five answers down. Everything below is easier once you have them.
Lustre
Lustre splits the filesystem in two. Metadata lives on the metadata servers, data lives on the object storage servers, and the client talks to both directly. A single file can be striped across several data servers, so one client can pull from many machines at the same time.
That design explains both the strength and the weakness.
Where it wins. Large files, sequential reads and writes, many clients at once. Aggregate throughput scales as you add data servers, and a single client is not necessarily limited to the speed of one storage node. This is why it is the default choice for HPC and for feeding large GPU clusters. If your requirement is expressed in gigabytes per second, this is one of the families you should be looking at.
I covered the throughput side in Understanding Lustre Performance and the design patterns in Optimizing HPC Storage.
Where it hurts, and what you can do about it. The usual complaint about Lustre is small files. The reason is in the design. For a large file, the open request to the metadata server is a tiny part of the total traffic. For a small file it is not. You pay metadata overhead and then data access overhead for a few kilobytes of payload. Multiply that by ten million files and the data servers can sit almost idle while the job crawls.
That was the whole story for a long time. It is not the whole story now.
Data on MDT stores the data of small files directly on the metadata target, so the extra requests to the data servers go away. The metadata target usually sits on fast storage tuned for small I/O, so you can get the benefit twice. Distributed Namespace spreads the namespace across several metadata targets, which is what you need when the problem is the number of operations rather than the size of each one. Progressive File Layout lets a file start on one storage tier and continue on other storage as it grows.
So the honest version is this. Lustre is weak on small files with a default layout, and can be decent with a layout designed for them. The catch is that this is a design decision. You can take it with lfs setstripe on directories, and you take it before the workload is already established. It is not something you casually change on a Friday afternoon on a filesystem that already holds your data.
Two warnings before you rely on any of this. If you are using a managed Lustre service, check whether the provider lets you control those layouts at all, because that is where the difference between the open source project and the service you actually bought shows up. And check the version, because the behaviour changed over releases and early measurements of Data on MDT showed a smaller gain on writes than on reads.
There is also a cheaper fix that people skip. If your dataset is ten million small files, the best answer is often not to tune the filesystem. It is to stop having ten million small files. Packing them into larger sequential archives puts the workload back into the shape Lustre is good at, and it helps every other filesystem in this post as well.
What it costs to run. This is the part that gets underestimated. Lustre is a cluster. Clients need a matching module, versions have to line up, failover has to be configured and tested, and someone has to own it. I tested the failover behaviour myself in Test Lustre Cluster Throughput and High Availability options and it works well, but it works well because it was set up on purpose.
Managed Lustre services from cloud providers remove most of that work, and they are usually the right answer if you need Lustre but do not want to run it. You pay for it in flexibility and in price.
Managed NFS
Every cloud has one. You create a filesystem, you get an endpoint, you mount it. Capacity grows on its own and you never think about a node again.
Where it wins. Everything that is shared but not extreme. Home directories, configuration, scripts, models being copied around, application data with moderate concurrency. It is POSIX, it is simple, it needs no operations team, and it is available in an afternoon.
For a large part of enterprise workloads this is the correct answer and people skip it because it sounds boring.
The ceiling. There is one, and you should find out where it is before you design around it, not after. Throughput is usually limited per filesystem and per client, but the actual ceiling depends heavily on how the managed service implements NFS, which NFS version and features the client uses, and how the service distributes data behind the endpoint.
Some services can scale well across clients and some support parallel data access through features such as pNFS. That still does not make every managed NFS service equivalent to a parallel filesystem such as Lustre.
If your requirement is one job reading many gigabytes per second through a single mount, this is where you need to understand the service limits before you commit to the architecture. Not because NFS is bad, but because managed NFS exposes a different performance model from a filesystem designed around application visible parallelism and striping.
Metadata. Fine for normal directory trees, but it can become the bottleneck when the workload is dominated by massive numbers of metadata operations. Millions of files in one directory are a very different workload from a normal enterprise directory tree.
JuiceFS
JuiceFS is a POSIX filesystem whose data lives in object storage and whose metadata lives in a separate database. The client splits files into chunks, writes the chunks as objects, and keeps the file structure in the metadata engine.
I went through it in detail in JuiceFS: a smart, distributed and free filesystem for the cloud era.
Where it wins. Capacity at object storage prices, with a real filesystem on top. Teams spread across locations that need the same namespace. Warm data that has to stay accessible without paying for high performance storage. Elastic growth without planning.
The namespace can be shared globally, but latency still matters. A global namespace does not turn WAN latency into LAN latency.
The price you pay. Two things.
First, metadata operations ultimately depend on the metadata engine. For workloads dominated by opens, stats and directory traversal, the latency and scalability of that engine can become the bottleneck. Client and kernel metadata caches can reduce the number of requests, but they do not remove the dependency.
Second, the metadata engine is now a database you have to run, back up and protect. If you lose it, the objects are still there and the filesystem is not. People notice this too late. Whatever engine you pick, treat it as production from the first day.
Cache changes the equation
On a cold path, data has to come from object storage. Subsequent reads may be served from the local cache or kernel page cache.
This matters because the same dataset can produce very different numbers depending on cache state.
If you benchmark only the first pass you will underestimate it, and if you benchmark only the second pass you will overestimate it. Measure both and say which one you are quoting.
For workloads that read data only once, caching may provide little benefit and can add its own overhead. For workloads with high data reuse, the cache can completely change the performance profile.
The short version
| Lustre | Managed NFS | JuiceFS | |
|---|---|---|---|
| Best file size | large | mixed | medium and large |
| Aggregate throughput | very high | medium to high, service dependent | medium to high, cache dependent |
| Single client throughput | high | service dependent | cache and network dependent |
| Small file handling | weak by default, decent if you design the layout for it | usually acceptable until metadata dominates | highly dependent on metadata engine and cache |
| Metadata heavy workloads | strong with the right design | usually good until metadata becomes dominant | highly dependent on metadata engine and cache |
| Operational effort | high, or pay for managed | almost none | medium, plus a database |
| Raw capacity cost | high | medium | low at scale |
| Good fit | GPU training, HPC | shared enterprise data | large warm datasets, distributed namespaces |
Read the table as a starting point, not as an answer. The answer comes from your five numbers.
Most real systems use two of them
The choice is rarely exclusive, and treating it as exclusive is how people end up paying for the wrong thing at the wrong size.
A common and sensible layout: cold and warm data sits in object storage, reachable through JuiceFS or through direct object access. The active dataset is staged onto a fast parallel filesystem for the duration of the job. Shared configuration, code and small files sit on managed NFS because that is what it is good at.
Three systems sounds like more complexity, and it is. It is also usually cheaper and faster than forcing one of them to do all three jobs. This is the same argument I made in Stop Buying the Wrong Storage and in Ceph is amazing, just don’t ask it to be Lustre. A platform being excellent does not make it right for a specific workload.
Three mistakes I keep seeing
Choosing by price per terabyte. It is the easiest number to compare and the least useful one. If the cheap capacity makes an expensive compute cluster wait, you did not save anything.
Benchmarking with the wrong data. Testing with a hundred large files and then running the job on ten million small ones tells you nothing. Test with a sample of the real dataset, with the real file size distribution.
Ignoring the operating cost. A filesystem you have to run has a cost measured in someone’s time, every week, forever. If that person is you, count it honestly before you choose the option with the better numbers.
Final Thoughts
There is no best filesystem here. There is a workload, and there is the storage that matches it.
Answer the five questions first. File size, access pattern, metadata intensity, concurrency, cache reuse. Once you have those, the choice usually makes itself, and you will be able to explain it to someone who asks why you did not just buy the cheap one.
Do not choose the filesystem first. Choose the workload first. The filesystem comes after.
Related articles:
