Daniel's got a whole thing about NFS mounting this week — the way he describes it, it's the glue between Linux file systems, the thing you reach for when you've got two Linux machines and you want them to share files. And he's got this impression a lot of us probably share: that for Linux-to-Linux, NFS is the gold standard, the right way to do it. But then he starts pulling at the thread. NFS isn't one protocol, it's a succession of them. There are gotchas that ambush even experienced sysadmins. And he's wondering whether he's using something that's almost aggregated without knowing it — just running on inertia — or whether this thing actually has a steady future.
The aggregated question is what grabbed me. Because he's not wrong — most people running NFS don't know which version they're on, don't know what mount options they're using, and definitely don't know what the kernel is doing to paper over the differences. And it works anyway, until it doesn't.
Until it doesn't. And then it really doesn't.
So let's pull back the curtain on NFS — what it actually is, what it isn't, and when you should think twice before using it.
Start with what it's actually doing under the hood. I mount a remote directory and it looks like it's local. What's the trick?
The trick is a kernel-level protocol that translates local filesystem calls into network requests. Your application calls open, read, write — standard POSIX stuff — and the NFS client in the kernel intercepts those, packages them up as RPC calls, and ships them off to the server. The server's NFS daemon unpacks them, performs the actual filesystem operation, and sends the result back. The application never knows the file isn't local.
So the glue metaphor is literal — it's a translation layer that makes two separate kernel filesystems pretend to be one.
And the illusion is remarkably good when it works. But the key word there is kernel. NFS is kernel-to-kernel. That's where the performance comes from, and it's also where the brittleness comes from. If the NFS kernel module has a problem, your entire mount point is gone. Not just one application — everything touching that mount.
Alright, so Daniel mentioned it's a succession of protocols. How different are we talking?
Radically different. NFSv2 came out in 1984. It was stateless — the server didn't remember anything about your client between requests. UDP only. And it had a 32-bit file size limit, which capped files at two gigabytes. Fine for 1984, absurd today.
Two gigs. I've got sleep logs bigger than that.
NFSv3 dropped in 1995 and it's still the most widely deployed version. Added TCP support, moved to 64-bit file sizes, and introduced asynchronous writes — the server could acknowledge a write before it actually hit disk, which was a huge performance win but also opened the door to data loss if the server crashed before flushing.
So v3 is where the tradeoffs start getting real.
And then NFSv4 in the year 2000 was a complete rethink. Stateful. The server now maintains session state with each client. Integrated file locking into the main protocol — in v3, locking was handled by a separate daemon, rpc.lockd, which meant you could have a perfectly functional NFS mount that silently couldn't lock anything. v4 also brought Kerberos authentication, which was a massive security upgrade over the v3 model of... basically trusting the network.
The v3 security model being, what, "you're on the network, you're probably fine"?
That's not far off. NFSv3 authenticates by IP address and UID. If I can spoof your IP and claim to be UID zero, I'm root on your NFS server. v4 with Kerberos actually cryptographically verifies identity.
And then it kept going.
v4.1 in 2010 brought pNFS — parallel NFS — which lets a client stripe reads and writes across multiple storage servers. Think of it as RAID at the protocol level. v4.2 in 2014 added server-side copy, so if you want to duplicate a file on the server, the data never has to travel to the client and back. Also added sparse file awareness and some SELinux label support. And as of right now, the IETF NFSv4 working group is actively drafting v4.3.
So it's not dead. It's still being developed.
Very actively. The question is whether anyone's adopting the newer versions. Most NFS deployments I see are still v3, some v4. Almost nobody is running v4.1 or v4.2 in production unless they have a specific need for pNFS or server-side copy.
Which gets at Daniel's aggregated point. People are running something and they don't even know which version they're on because the kernel negotiates it transparently.
And the negotiation is silent. You mount with nfs as the filesystem type, the client and server figure out the highest version they both support, and that's what you get. You can override it with a mount option, but most people don't. So you could be running v3 and think you're getting v4 features like integrated locking, and you're not.
Let's dig into the gotchas then. Because this is where NFS stops being a nice abstraction and starts being the thing that ruins your weekend.
The stale file handle error. ESTALE. This is the one that makes people stare at the ceiling at three in the morning.
What does it actually mean?
Every file on a Unix filesystem has an inode — a data structure that describes the file's metadata and points to its data blocks. When an NFS client opens a file, the server gives it a file handle that references that inode. If something changes on the server side — the file gets deleted, the filesystem gets unmounted and remounted, the export configuration changes — that inode reference becomes invalid. The client's file handle now points to nothing. And the kernel returns ESTALE.
And the error message gives you nothing.
"Stale file handle." That's it. No indication of what file, what changed, or when. And the really insidious version is when you've got a process that opened a file, the file got deleted on the server, but the process still has the file handle open. Every subsequent read returns ESTALE. The data is gone, the handle is invalid, and the process has no way to recover.
I've seen a developer do rm -rf on an NFS mount from one machine while another machine still had files open. The directory disappears but the handles live on as ghosts.
And they're useless ghosts. You can't read, you can't write, you can't even stat the file to see what it was. The only fix is to unmount and remount, which kills every open handle across the entire mount point.
So one deleted directory takes down every process touching that mount.
If they try to access the stale handles, yes. And they will, because they don't know the handle is stale until they try to use it.
What about permissions? Daniel didn't mention it directly but this is where NFS gets weird compared to local filesystems.
NFS doesn't use usernames. It uses raw UID and GID numbers. So if Alice on the client has UID 1001 and Bob on the server has UID 1001, Alice gets Bob's files. The server has no idea who "Alice" is — it only sees the number.
Which works fine in a tightly managed environment where UIDs are synchronized across machines. And is a complete disaster everywhere else.
LDAP or NIS can centralize UID management, but a lot of small deployments just... don't. And then they wonder why permissions are wrong. There's also root_squash, which maps UID zero from the client to nobody on the server — prevents a root user on a client from becoming root on the NFS server. But there's also no_root_squash, which disables that protection, and all_squash, which maps every client UID to a single anonymous user. all_squash is useful for public shares but poorly understood — people turn it on without realizing it means every client user now has the same permissions.
And nobody_squash?
That's not actually a standard option. People confuse it with all_squash or root_squash. Which is kind of the point — the squash options are confusing enough that sysadmins misconfigure them regularly.
Let's talk about hard versus soft mounts. This is the one where the wrong choice can corrupt your data.
Hard mount is the default. When an NFS operation fails — say the server goes down — a hard mount retries indefinitely. Your process hangs. It's unkillable, depending on the kernel and mount options. The entire process tree that touches that mount can lock up. But when the server comes back, the operation completes and your data is intact.
And soft mount?
Soft mount returns an error after a configurable number of retries. Your process doesn't hang. It gets an I/O error and can handle it gracefully — or not. The problem is that a soft mount can return a partial write. The server acknowledged some bytes but not all of them, the network blipped, the retry count was exceeded, and the client gets an error. The application thinks the write failed, but part of it might have succeeded. That's silent data corruption.
So hard mounts preserve data integrity at the cost of availability. Soft mounts preserve availability at the cost of data integrity.
And there's no good third option. The intr flag — interruptible — was supposed to let you kill a hung hard-mount process with a signal. But it doesn't work reliably on a lot of kernels. You can sit there sending SIGKILL to a process and it just... ignores you.
Which brings us to NFS over WiFi.
Don't.
I mean, that's the advice. But people do it.
NFS was designed for reliable, low-latency networks. WiFi is neither. Packet loss, latency spikes, brief disconnections — all of those are normal WiFi behavior and all of them are catastrophic for NFS. A hard mount over WiFi will hang constantly. A soft mount over WiFi will corrupt data constantly. There's no winning.
And the tuning knobs — rsize and wsize — those are the read and write buffer sizes?
Right. The defaults are usually 1 megabyte for NFSv4, smaller for v3. Over a high-latency link, you want larger buffers to amortize the round-trip cost. Over a lossy link, smaller buffers reduce the amount of data you have to retransmit. But most people never touch these. They mount with defaults and wonder why performance is terrible over their VPN link.
What about locking? You mentioned rpc.lockd for v3.
In NFSv3, file locking is handled by a completely separate protocol — NLM, the Network Lock Manager — running in a daemon called rpc.lockd. The NFS mount itself can be working perfectly while lockd is dead, and you won't know until a process tries to acquire a lock and hangs. Or doesn't hang, depending on how the application handles lock timeouts. Either way, you've got silent lock failures.
So you think you're protected against concurrent writes and you're not.
And in NFSv4, locking is integrated into the main protocol. It's stateful. The server knows which client holds which lock. But now you've got a new problem: lock recovery. If the server reboots, all the locks are gone. The server enters a grace period where it waits for clients to reclaim their locks. If a client doesn't reclaim in time — or if the client also rebooted and doesn't know it had locks — those locks are permanently lost. Any process waiting on a lock that never gets reclaimed is stuck.
The server reboot cascade. Server goes down, comes back up, grace period expires before all clients check in, half the locks evaporate, and now you've got processes on five machines all stuck waiting for locks that don't exist anymore.
And the only fix is to restart those processes. If you can find them all.
So if NFS has all these sharp edges, why do people still use it? And what are the alternatives actually like?
The main alternative in Linux environments is Samba, which implements SMB — the Server Message Block protocol, also called CIFS. SMB was designed for Windows, and the authentication model is completely different. Username and password, or Kerberos tickets against a domain controller. No UID mapping nonsense.
Which is better for mixed environments.
Much better. If you've got Windows and Linux machines sharing files, Samba is the obvious choice. But SMB has its own problems. SMB signing — which cryptographically signs every packet — kills performance on high-throughput links. And Samba's implementation of SMB has historically lagged behind Microsoft's, though it's gotten much better. The bigger issue is that SMB wasn't designed for POSIX semantics. Things like symlinks, hard links, and Unix permissions don't map cleanly to SMB. Samba has extensions to handle them, but it's a translation layer on top of a translation layer.
And SSHFS?
SSHFS runs over SFTP, which is a subsystem of SSH. No kernel module needed — it's a FUSE filesystem, entirely in userspace. Encryption by default, works over the public internet, no server configuration beyond having SSH running. It's the easiest thing to set up.
And the tradeoff?
Performance. SSHFS is single-threaded. Every stat call — every metadata operation — is a separate SFTP request over the SSH connection. For metadata-heavy workloads, like listing a directory with thousands of files, it's painfully slow. NFS batches metadata operations and runs in the kernel, so it's orders of magnitude faster for that kind of workload. SSHFS also doesn't support file locking properly — there's no lock manager, so concurrent access is basically unprotected.
So SSHFS is the quick-and-dirty solution for occasional access, but you wouldn't run a database on it.
You absolutely would not. The latency per operation is too high, and the lack of locking means you're one concurrent write away from corruption.
What about the newer distributed filesystems? Ceph, GlusterFS?
Those solve a different problem. NFS assumes a single server. If that server goes down, everything is unavailable. Ceph and GlusterFS distribute data across multiple nodes — no single point of failure, horizontal scalability, the whole cloud-native pitch. But they're significantly more complex to set up and manage. Ceph in particular has a reputation for being powerful and also being a full-time job to operate.
And object storage — S3 and its clones?
Not POSIX. S3 is eventually consistent — or was, until recently — and the API is get, put, list, delete. No partial writes, no locking, no directory hierarchy in the filesystem sense. You can mount S3 as a filesystem with something like s3fs or rclone, but it's a FUSE layer translating POSIX calls to HTTP requests. The performance and consistency characteristics are completely different from a real filesystem.
So NFS still has a niche where you need POSIX compliance, low latency, and high throughput on a trusted network.
It's still the best tool for that specific job. If you're running a render farm, or a database that needs shared storage, or a cluster of compute nodes that all need to see the same filesystem — NFS is hard to beat. The kernel-to-kernel path is fast, the protocol is mature, and the POSIX semantics are correct.
But Daniel's aggregated question keeps nagging at me. Most people using NFS don't know any of this. They mount it, it works, and they never dig deeper until something breaks.
And the kernel's transparency is a double-edged sword. It's wonderful that you can mount NFS without understanding the protocol. Right up until you can't. The stale file handle error doesn't tell you what went wrong. The hard mount hang doesn't tell you why your process is stuck. The UID mismatch doesn't tell you that Alice and Bob share a UID.
The protocol assumes a level of understanding that the tooling doesn't enforce.
That's well put. NFS is a protocol that rewards understanding. The more you know about which version you're running and what assumptions it makes, the fewer surprises you'll get. But the surprises never fully go away.
What about the future? Daniel asked whether this thing is running on inertia or has a real trajectory.
Both. The protocol is still being actively developed — NFSv4.3 is in draft at the IETF. The features they're adding are real and useful. But the world is moving toward cloud-native storage. Kubernetes has CSI drivers. Persistent volumes are provisioned dynamically. Object storage is the default for new applications. NFS is increasingly a legacy integration — the thing you use to connect your new Kubernetes cluster to your old NetApp filer.
The NFS-to-S3 gateway pattern.
You've got data on S3 but some legacy application needs POSIX access, so you run an NFS gateway that translates NFS calls to S3 API calls. It works, but it's a compatibility shim. NFS isn't the primary storage layer anymore — it's the adapter.
That's a very different role than what it was designed for.
It is. But it's also a testament to how deeply embedded NFS is. You can't rip it out. Too many applications assume a POSIX filesystem. Too many workflows depend on NFS mounts. It's going to be with us for decades, even if new deployments are increasingly choosing object storage or distributed filesystems.
NFS isn't dying. It's ossifying.
That's... actually a good way to put it. It's becoming infrastructure — like DNS or NTP. Nobody gets excited about deploying a new NFS server, but everybody expects it to be there when they need it.
Speaking of real-world NFS disasters — Hilbert, you've been quiet.
Hilbert: HP ProLiant DL380. Generation two. Six of them in a rack in Bristol, 1998.
...
Go on.
Hilbert: Small animation studio. They were doing stop-motion, believe it or not. Wallace and Gromit type stuff. All the render nodes were SGI Irix machines, but the file server was Linux because SGI storage was priced like it was made of gold. NFSv3, hard mounts, no intr that actually worked.
Irix and Linux. Different UID ranges?
Hilbert: Irix started user IDs at zero for root and then the first real user was... I want to say four hundred something. Linux started at five hundred. The render farm ran as a service account that was UID four-twelve on Irix and UID five-twelve on Linux.
And nobody mapped them.
Hilbert: Nobody mapped them. The render nodes wrote their output frames to the NFS mount. The Linux server saw UID four-twelve writing files. The director's editing workstation — also Irix — was UID four-twelve. Which was the director's actual user account.
Oh no.
Hilbert: A render job overwrote the director's final cut. Not the whole thing — just enough frames that the edit was ruined. They had backups, but the backup was from the day before. Lost about fourteen hours of editing work.
That's the UID mismatch problem in one story.
Hilbert: That wasn't the bad one.
There's a worse one.
Hilbert: The bad one was the cat. Director's cat. Walked across the keyboard of the editing workstation and hit — we never figured out exactly what — but the result was rm -rf on the NFS mount. The render farm was mid-job. Every render node had files open on that mount. Hard mounts, no working intr.
The rm -rf deleted the directories on the server, the render nodes still had open file handles, and every subsequent read returned stale file handle errors.
Hilbert: Worse. The render nodes didn't just get stale handles. They locked up. The hard mount meant every process that touched the mount hung waiting for a response that was never going to come. All six render nodes. We lost three days of render time before we figured out what happened and hard-rebooted everything.
Three days.
Hilbert: The cat was fine.
Of course the cat was fine.
The intr flag — you said it didn't work. Was this a kernel bug?
Hilbert: It worked on paper. The documentation said you could kill a hung NFS process with SIGKILL if you used intr. In practice, on that kernel — I think it was 2.0.something — the signal would get delivered but the process was stuck in an uninterruptible sleep in the kernel. The kill signal just sat in a queue.
You had effectively unkillable processes and didn't know it until the cat demonstrated the problem.
Hilbert: We'd been running that configuration for eight months. Could have happened any time. The cat just got there first.
That's the thing about NFS that documentation doesn't capture. The protocol spec is precise. The kernel implementation is... sometimes less precise. And the gap between what the man page says and what actually happens is where the disasters live.
Hilbert: I still have one of those DL380s. It's in a cupboard.
Still running?
Hilbert: No. I keep cables in it.
The UID mismatch thing sticks with me though. That's not a protocol bug — it's a design assumption. NFS assumes you've got centralized identity management, and if you don't, it silently does the wrong thing.
The wrong thing isn't a permissions error. It's data destruction.
Because it doesn't know it's wrong. The server sees UID four-twelve and serves the files. It has no way to know that UID four-twelve on the client is a render service account and UID four-twelve on the server is the director. They're just numbers.
Where does that leave us? NFS is old, it's weird, it's full of traps — but it's also not going anywhere. The question is what role it plays going forward.
I think the open question is whether the kernel-to-kernel filesystem model still makes sense. Object storage and cloud-native abstractions are taking over. NFS is becoming a compatibility layer — the NFS-to-S3 gateway, the legacy mount point on a Kubernetes cluster. That's a very different job than being the primary shared filesystem for a render farm.
It's not dying, but it's being pushed to the edges.
The edges are where the weirdest bugs live. When you're running NFS as a translation layer on top of object storage, the POSIX semantics that NFS provides don't match the eventual consistency model of the object store underneath. You get behavior that's correct according to NFS but wrong according to reality.
The protocol rewards understanding. The more layers, the harder the understanding.
The surprises never fully go away.
This has been My Weird Prompts, produced by Hilbert Flumingtop. We'll be back soon.
Find us at my weird prompts dot com.
