Network Filesystems

January 7, 2003 / Kurt Nelson / 0 Comments

Table of Contents

1. Overview of Network Filesystems

Network filesystems are used to allow a machine to access another machine’s
files remotely. There are a plethora of ways to do this in Linux. The most
common way today is NFS, however many people are shifting to different methods
because of either security or administration reasons.

2. NFS

2.2. Server Setup

To set up a server, one must have the portmap and nfs-utils installed
(see resources for links). To configure your exported filesystems,
edit the file /etc/exports. for example, if your fileserver BARNEY
wanted to share folders /mnt/mp3 and /mnt/work with system JOHN, and
also share /mnt/work with systems MARY and BETTY, with betty also
having read-only access to /mnt/mp3, your exports file would look like
this:

/etc/exports:

# Shares on BARNEY
/mnt/mp3 JOHN(async,rw) BETTY(async,ro)
/mnt/work JOHN(async,rw) MARY(async,rw) BETTY(async,rw)

As you can see, you merely specify the directory you want to share,
and then follow it by the hosts you want to be able to access it (IP’s
are fine too) and the permissions for that user. Now you are ready to
share your files. start portmap, and then start the nfs service (rc
scripts are included with most distributions to do this)

2.4. Secure NFS over SSH

Okay, so that’s great, now what happens when John is on a business
trip in Paris and wants to mount his network share to grab some files
he forgot for a presentation he’s doing? It’s not very smart to set
up the NFS server to allow connections from outside IP’s, and even if
it was, it would be even stupider to actually access it, since all the
data goes in the clear, and can be inspected at any point by anyone
with half a brain. Well, John uses SSH to get into a secure shell, so
why not use it to forward nfs? Well, NFS uses UDP, and SSH can only
forward TCP, it does not know what to do with UDP datagrams. Enter
SNFS and sec_rpc. How does it work? you ask. sec_rpc basically
translates the UDP datagrams into something that SSH can forward,
and then translates them back on the other side. so you’re still
using NFS, but through a tunnel.

Here’s how you start out. on the server, you create an /etc/exports
file like so:

/etc/exports:

# SSH exports file
/mnt/mp3 localhost(async,rw)
/mnt/work localhost(async,rw)

Now that you’ve exported the filesystems to the local host, you need
to install sec_rpc on both the client and server machines. you do it
the standard autoconf way, ./configure; make; make install. so now on
BARNEY you need to be running nfsd like normal.

On John’s pc is where all the complex stuff is happening. John runs
at his PC:

# snfshost REMOTE:MOUNTPROG

Here MOUNTPROG is a six-digit number chosen between 200000 and
249999 such that both MOUNTPROG and NFSPROG=MOUNTPROG+50000 are
unassigned RPC program numbers (e.g. MOUNTPROG=201000,
NFSPROG=251000). REMOTE is the remote host name.

That will create the config file needed for the RPC numbers. now you
add a line to john’s fstab:

/etc/fstab:

LOCAL.DOMAIN:/DIR /REMOTE/DIR nfs user, noauto, hard, intr, rsize=8192, wsize=8192, mountprog=MOUNTPROG, nfsprog=NFSPROG 0 0

(sorry if that wraps weird, it should go on one line)
in that case above, MOUNTPROG and NFSPROG are the numbers you figured
out before. LOCAL.DOMAIN is john’s fully qualified domain or
localhost.

Now that John is thoroughly confused, he creates the local mount
directory, starts portmap, and now has some more weird commands to
throw at the host:

# smkdirall
# rpc_psrv -r -d /usr/local/etc/snfs/REMOTE
(where REMOTE is the remote host name)

Now john types in the root password for BARNEY, and he should get the
message: "ID String correctly read: RPC Proxy Client". Now he is
ready to mount:

# mount /REMOTE/DIR
# df

woo, you have a mount or something. And now you are thorougly
confused. As you may have noticed, a major disadvantage of SNFS is
that you need to know the host’s root password. also, you need to
have remote root SSH enabled. Not to mention, this is a messy setup.
but it works (sorta).

It is possible to set it up so that the remote
mount runs as non-root, with the correct setuid binaries, but that is
still messy. The people at SFS agree with you. the next section will
show you how to do it using SFS, a more elegant solution.

3. SFS

3.1. Background

NFS was originally developed by Sun for the purpose of mounting a disk
partition on a remote machine as if it was on a local hard drive.
this allowed for fast seamless sharing of files across a network. NFS
is an RPC service, and works through the RPC portmapper daemon. Most
unix-like systems ship with an NFS daemon included, or a NFS daemon
can be obtained, which is why it is the de-facto standard for network
file sharing.

The problem with NFS is that it is inherently insecure. The current
NFS protocol does not support any authentication, and any validation
is only done by IP, which can be easily spoofed. There is no
user-level restriction, so if anyone can exploit a client machine on a
NFS network or in some cases just plug another machine into the
network, they can gain access to the NFS server’s shares.

Linux supports NFS protocols v. 2 and 3. the implementation of NFS in
the linux kernel is partly user-space, and partly in the kernel space,
in order to keep things running fast.

SFS was created because the insecurities of NFS made it easily spoofed
and not very insecure. not to mention, if a client was connecting
with a dynamic IP, the NFS would have to change its exports as per
each time a client’s ip changed, or (very insecure) export it to an
entire IP range.

SFS version uses shared-key authentication at 1024 bits default
(higher bit keys are allowed)

3.2. Server set-up

To set up an SFS server, you create a user sfs and a group sfs, and
then get the sfs-0.6 source from www.fs.net and compile it. then you
set up your /etc/exports to export filesystems to localhost similar to
like you would SNFS:

/etc/exports:

/var/sfs/root localhost(async,rw)
/mnt/mp3 localhost(async,rw)
/mnt/work localhost(async,rw)

Then you create the file /etc/sfs/sfsrwsd_config as follows:

/etc/sfs/sfsrwsd_config:

Export /var/sfs/root /
Export /mnt/mp3 /mp3
Export /mnt/work /work

note sfs requires a ‘root’ for the server. the exports will be under
this root. you now create /var/sfs/root , /var/sfs/root/mp3 , and
/var/sfs/root/work. then chown these to the user sfs group sfs. you
generate a host key for the server like so:

# sfskey gen -P /etc/sfs/sfs_host_key

After doing this, start your nfsd, and sfssd.

4. OpenAFS

4.1. Introduction

AFS was pioneered at Carnegie Mellon University, supported and
developed by TransArc corporation (now owned by IBM). AFS, or the
Andrew File System, is a distributed filesystem. IBM in 2001 branched
out AFS and created OpenAFS, an open source version of AFS. OpenAFS
is unique among this bunch in that it works on windows systems as well
as Linux.

Because openAFS is a distributed filesystem, it allows for high
availability. a cluster of servers can mirror the same AFS cell. Any
client will not know which server they are connected to, and when they
write to a file, the changes are propagated to the other servers in
the cell. if any server goes down, the cell can continue to operate.

4.2. Server setup

Gentoo has a good guide on how to set up OpenAFS. I will put up my own
guide to OpenAFS setup soon.

4.3. Client setup

To be a NFS client, you don’t need to edit any config files. all you
need to do is start portmap, and then provided you have the nfs tools
installed, simply mount the partitions. Let’s say JOHN from the
previous example wanted to mount the mp3 and work folders now. he
simply runs at the command line:

# mount -t nfs BARNEY:/mnt/mp3 /mnt/mp3
# mount -t nfs BARNEY:/mnt/work /mnt/work

It is very simple to mount NFS partitions. because no further
authentication is needed, on a fixed network with fixed hosts, one
can simply set a bootup script to mount these on startup.

Now John wants to connect to sfs. Here’s what he does: on the server
BARNEY (not his local client box) john creates a key from the shell
like this: sfskey register . he can do this from an ssh shell if he
wants to. it’s a one-time process. Now what john does from paris
is, he starts portmap and sfscd from his client box. Now as non-root,
he runs:

# sfsagent john@barney.office.domain.com

sfs will not read your /etc/hosts file, it will totally ignore it (to
prevent spoofing, only a globally-verifiable DNS name will be
accepted) now john logs in with the passcode he put on his key, and
now he is into barney. he now proceeds to access his share:

# cd /sfs/barney.office.domain.com/
# ls
work/
mp3/

All his files will be read over a 1024-bit secure encrypted channel.
SFS is probably the best way for sharing files over an insecure
network, because it uses the tried-and-true nfs protocol, and once a
key is made, john can log in from anywhere.

Check for the next version of this howto to learn about OpenAFS client
setup.

5. Coda

Coda is another distributed filesystem from carnegie-mellon based on
AFS. Coda was open source from the beginning however, and it has been
available in the Linux Kernel since 2.2. Coda has one important
feature which NFS and OpenAFS do not, and that is disconnected
operation. disconnected operation allows a client to disconnect from
the network, and continue to access and modify files. when the client
reconnects to the network, that user’s changes are propagated back to
the coda cell. This is excellent for laptops, because then a user can
take the laptop away, modify his own home directory files, and then
put it back on the network and have all the files be resynched with
the global fileserver. Coda does this by locally caching files.
Because of aggressive local caching, Coda is also faster than OpenAFS.

One downside of all these features is that, since Coda needs to keep
so much metadata on the filesystems, it needs its own raw partition
for data storage.

Check back for the next version of this Howto for more info on setting
up coda.

6. Intermezzo

Intermezzo is another distributed file system, loosely based on coda.
Like coda, it allows disconnected operation. Intermezzo is meant to
be lightweight. It also needs a dedicated raw partition, which can be
any of ext2/3, xfs, or reiserfs. The intermezzo protocol is based on
HTTP, and it can use apache or any cgi-supporting http daemon for
synchronization.

Intermezzo is still in its early stages. However, linus made sure it
was included in the 2.4 kernel starting with 2.4.14, just before he
went off to work on the 2.5 kernel. Intermezzo is still in early
beta, but it shows a lot of promise.

In my tests with Intermezzo, i found it is a little flaky in its
setup, and the setup process is not very well documented. Whatever
documentation exists is inconsistent with the most recent version.

7. Summmation

There are many alternatives to NFS. NFS is a very good protocol, and
it has been tried and tested, but it is too insecure to be used over
wide area networks, unless encapsulated inside a VPN tunnel, or
another more secure protocol such as SNFS or SFS is used. Also, NFS
is limited in that a volume can only be exported by one server, so
that you cannot distribute the load across multiple servers or
implement failover. Coda, Intermezzo, and OpenAFS aim to leverage
that by distributing the filesystem. As of now, Coda looks to be the
best choice for a distributed FS, although IBM is working hard to
improve OpenAFS. Intermezzo is a little too unstable to be considered
usable, and not enough documentation exists.

Which one you choose is dependent on how you want to do things. If
you have a home directory you want to access when on the road and not
connected, then a caching system like Coda is probably for you.
However, if you simply want to access files from two connected systems
over the net (over Resnet or LAWN for example), you probably want to
use SFS as SNFS is a little tricky to setup, and SFS allows connection
from anywhere with the same public key using strong encryption.

8. Resources

#nfs