Editing BZFS in a chroot jail

BZFS in a chroot jail is a concept where a [[BZFS]] server is run on a host computer in a very secure environment, preventing it from having access to any other parts of the host system.

==Overview==
Installing the BZFLAG Server (bzfs) in a ‘sandbox’ or a ‘jail’ on Linux by using the features provided with chroot is an advanced method for setting up a server. It offers additional security for the hosting server. For general information on setting up a server see [[Creating_A_Server]].

This method has been tested on Redhat 8 and 9 systems. Other Linux distributions should work in a similar manner.

==Background==
Before attempting to set up BZFS in a chroot jail it is best to have a little background in using the chroot command. The best place to start is reading the man page
 man chroot
but the basic concept is to run a program in a directory and force it to think that it is the root (the top) of the filesystem (the ‘sandbox’ or ‘jail’), so that if the application was ever compromised, only it’s folder would be accessible to the attacker and not the entire filesystem. User root, or any program with root privileges can break out of a chroot  jail. USers should be ware that a badly configured chroot jail might even be a larger security problem then not running one. The application jk_check from http://olivier.sessink.nl/jailkit does checks to verify if your chroot jail is safe.

Before a program can be run in a jail, the user must make sure that the program has all dependencies that it may need. This means creating a smaller copy of the root filesystem so that the program can access the files that it needs, and knows where to find them. This means that if a program requires a library in /lib, then we will need a lib directory with those libraries in our jail. (if this is confusing – hang in there, you will see an example below).

== Just tell me how to do it!! ==

Let’s say we have already compiled and installed (via RPM or source) bzflag in /usr/share/bzflag (or wherever you installed it), and we want to have it run in a jail.

I like to create a folder in the root of the filesystem called chroot, and then place folders in there for everything that I am running in a jail. This helps me to keep track of which programs/services/applications I am running in a jail easily. I also don’t like creating jails in system directories (in this case it would be /usr/share or wherever you installed it).

I would, for example, create a folder like /chroot/bzflag. 
I would then copy the actual executable in a bin folder. So know we have:
 /chroot/bzflag/
 /chroot/bzflag/bin 
 /chroot/bzflag/bin/bzfs (the bzfs executable in the jail’s bin directory).
Now we need the dependency files – how do we know which files bzfs depends on? Run ‘ldd –v /usr/bin/bzfs’ (or where ever you installed bzfs to when you compiled it. NOTE: that this is the real one and not the one you copied to the jail) and you should get something like (on Redhat 9 systems):
 $ ldd -v bzfs
        libstdc++.so.5 => /usr/lib/libstdc++.so.5 (0x40034000)
        libm.so.6 => /lib/tls/libm.so.6 (0x400e7000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x40109000)
        libc.so.6 => /lib/tls/libc.so.6 (0x42000000)
        /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x40000000)
 
        Version information:
        ./bzfs:
                libc.so.6 (GLIBC_2.1.3) => /lib/tls/libc.so.6
                libc.so.6 (GLIBC_2.1) => /lib/tls/libc.so.6
                libc.so.6 (GLIBC_2.0) => /lib/tls/libc.so.6
                libstdc++.so.5 (CXXABI_1.2) => /usr/lib/libstdc++.so.5
                libstdc++.so.5 (GLIBCPP_3.2) => /usr/lib/libstdc++.so.5
        /usr/lib/libstdc++.so.5:
                libgcc_s.so.1 (GCC_3.0) => /lib/libgcc_s.so.1
                libc.so.6 (GLIBC_2.2) => /lib/tls/libc.so.6
                libc.so.6 (GLIBC_2.1.3) => /lib/tls/libc.so.6
                libc.so.6 (GLIBC_2.3) => /lib/tls/libc.so.6
                libc.so.6 (GLIBC_2.0) => /lib/tls/libc.so.6
                libc.so.6 (GLIBC_2.1) => /lib/tls/libc.so.6
        /lib/tls/libm.so.6:
                libc.so.6 (GLIBC_2.1.3) => /lib/tls/libc.so.6
                libc.so.6 (GLIBC_2.0) => /lib/tls/libc.so.6
        /lib/libgcc_s.so.1:
                libc.so.6 (GLIBC_2.2.4) => /lib/tls/libc.so.6
                libc.so.6 (GLIBC_2.1.3) => /lib/tls/libc.so.6
                libc.so.6 (GLIBC_2.0) => /lib/tls/libc.so.6
        /lib/tls/libc.so.6:
                ld-linux.so.2 (GLIBC_2.3) => /lib/ld-linux.so.2
                ld-linux.so.2 (GLIBC_2.1) => /lib/ld-linux.so.2
                ld-linux.so.2 (GLIBC_2.0) => /lib/ld-linux.so.2
                ld-linux.so.2 (GLIBC_PRIVATE) => /lib/ld-linux.so.2}}}
This is where the ‘mini root’ filesystem comes in. For every file listed in the output of the ldd command – copy those files to the jail into the appropriate directory. The jailkit package at http://olivier.sessink.nl/jailkit/ contains a utility jk_cp that can be used to automatically copy all dependencies of a file into a jail. 
 mkdir /chroot/
 jk_cp /chroot/ /bin/bzfs

Make sure you get all of them or bzfs will complain that it is missing something later on. On Redhat 9 there is a /lib/tls directory, on Redhat 8 these same libraries were in /lib/i686 – so you will have to substitute the proper directories here for wherever your system expects them. Use the exact same path as outputted by ldd. So you should copy something similar to (create the directories as needed):
 /chroot/bzflag/usr/lib/libstdc++.so.5   (copied from /usr/lib)
 /chroot/bzflag/lib/libgcc_s.so.1        (copied from /lib)
 /chroot/bzflag/lib/tls/libm.so.6        (copied from /lib/tls)
 /chroot/bzflag/lib/tls/libc.so.6        (copied from /lib/tls)
 /chroot/bzflag/lib/ld-linux.so.2        (copied from /lib)

BZFS also needs libraries to resolve DNS names to IP addresses (to register itself with the list server), and so you will need these extra libraries that are not reported through ldd (copy from /lib to your corresponding jail directory):
 /lib/libnss_dns.so.2  -> /chroot/bzflag/lib/libnss_dns.so.2
 /lib/libresolv.so.2   -> /chroot/bzflag/lib/libresolv.so.2

You will also need an etc directory with the resolv.conf file in it so that the above libraries know which DNS server they need to contact in order to resolve things, so copy from /etc into your corresponding jail directory:
 /etc/resolv.conf -> /chroot/bzflag/etc/resolv.conf

and now the hard part is done…

Again this bit can be done easier using jailkit:
 jk_init /chroot/ netbasics

Here is the complete listing of what your jail should look like or be similar to (in the case of the lib directory which will change from Linux version/distro I am sure – it already changed just from Redhat 8 to 9):
 $ ls -alFR ./bzflag/
 ./bzflag/:
 total 24
 drwxr-xr-x    6 root     root         4096 Jul 12 08:12 ./
 drwxr-xr-x    3 root     root         4096 Jul 12 07:24 ../
 drwxr-xr-x    2 root     root         4096 Jul 12 07:25 bin/
 drwxr-xr-x    2 root     root         4096 Jul 12 08:29 etc/
 drwxr-xr-x    3 root     root         4096 Jul 12 08:34 lib/
 drwxr-xr-x    3 root     root         4096 Jul 12 08:26 usr/
 
 ./bzflag/bin:
 total 188
 drwxr-xr-x    2 root     root         4096 Jul 12 07:25 ./
 drwxr-xr-x    6 root     root         4096 Jul 12 08:12 ../
 -rwsr-sr-x    1 nobody   nobody     179160 Jul 12 07:25 bzfs*
 
 ./bzflag/etc:
 total 12
 drwxr-xr-x    2 root     root         4096 Jul 12 08:29 ./
 drwxr-xr-x    6 root     root         4096 Jul 12 08:12 ../
 -rw-r--r--    1 root     root           54 Jul 12 08:29 resolv.conf
 
 ./bzflag/lib:
 total 252
 drwxr-xr-x    3 root     root         4096 Jul 12 08:34 ./
 drwxr-xr-x    6 root     root         4096 Jul 12 08:12 ../
 -rwxr-xr-x    1 root     root       103044 Jul 12 08:20 ld-linux.so.2*
 -rwxr-xr-x    1 root     root        30324 Jul 12 08:20 libgcc_s.so.1*
 -rwxr-xr-x    1 root     root        18416 Jul 12 08:25 libnss_dns.so.2*
 -rwxr-xr-x    1 root     root        76552 Jul 12 08:23 libresolv.so.2*
 drwxr-xr-x    2 root     root         4096 Jul 12 08:16 tls/
 
 ./bzflag/lib/tls:
 total 1740
 drwxr-xr-x    2 root     root         4096 Jul 12 08:16 ./
 drwxr-xr-x    3 root     root         4096 Jul 12 08:34 ../
 -rwxr-xr-x    1 root     root      1549556 Jul 12 08:16 libc.so.6*
 -rwxr-xr-x    1 root     root       211876 Jul 12 08:16 libm.so.6*
 
 ./bzflag/usr:
 total 80
 drwxr-xr-x    3 root     root         4096 Jul 12 08:26 ./
 drwxr-xr-x    6 root     root         4096 Jul 12 08:12 ../
 drwxr-xr-x    2 root     root        69632 Jul 12 08:02 lib/
 
 ./bzflag/usr/lib:
 total 776
 drwxr-xr-x    2 root     root        69632 Jul 12 08:02 ./
 drwxr-xr-x    3 root     root         4096 Jul 12 08:26 ../
 -rwxr-xr-x    1 root     root       710608 Jul 12 08:02 libstdc++.so.5*
== More Security ==

Since you have to execute chroot as root, bzfs will run as root – which is not what we want. We can force bzfs to run as nobody by changing his ownership to nobody, and setting the sticky bit on him. This way, when root executes /chroot/bzflag/bin/bzfs – it is executed as nobody, and therefore has very little privileges on the system. 

This is how we do that:
 chown nobody.nobody /chroot/bzflag/bin/bzfs
 chmod +s /chroot/bzflag/bin/bzfs

But this changes only the effective user id, not the real user id. A safer way to do this is to make use of the jk_chrootlaunch utility in jailkit (http://olivier.sessink.nl/jailkit/), it will drop all privileges, and then execute as user nobody, without any possibility to regain root privileges, because both the effective AND the real user id are now changed to nobody.

 jk_chrootlaunch -j /chroot/bzflag/ -x /bin/bzfs –d –d –d –d –public TestServer –publicaddr <your_public_ip_address:port>

== Testing ==

Run a command like the following (as root),
 chroot /chroot/bzflag/ /bin/bzfs –d –d –d –d –public TestServer –publicaddr <your_public_ip_address:port>

It will log any errors to /var/log/daemon.log

This is where all the magic of chroot happens. And this command should make sense to you as you read the man page, right :) If not, the command basically takes 2 arguments. The first is the "new root" (/chroot/bzflag/). The second is the command to run. Notice that we specify /bin/bzfs to run bzfs, and not /chroot/bzflag/bin.

You should see something similar to:
 $ chroot /chroot/bzflag/ /bin/bzfs -d -d -d -d -public test -publicaddr 65.39.216.149:5155
 style: 0
 Opening List Server 0
 listening on 0.0.0.0:5155
 
 ADD 65.39.216.149:5155 2 BZFS107e 000000c800010000000000000000000000c800c800c800c800c800000000000000000000 test
 
 Closing List server 0
 Player [0] accept() from 208.186.182.172:47076 on 7
 Player [0] shutdownAcceptClient: close(7) 

If you don’t or get an error like “failed http ...” – then a resolv/dns library or the resolv.conf in etc can’t be found – copy them again check your spelling and any paths and try again…

The next thing to verify is that bzfs is really running as nobody and not as root. Start bzfs in the background this time ($ chroot /chroot/bzflag/ /bin/bzfs -d -d -d -d -public test -publicaddr <your_public_ip_address_here>:<your_server_port> &), and then ‘ps aux | grep –i bz’ and look for at the user bzfs is running as.

You should see something like:
 $ ps aux | grep -i bz
 nobody    2930  0.1  1.3  3032 1668 pts/1    S    09:49   0:00 [bzfs]
 root      2932  0.0  0.5  3568  628 pts/1    S    09:49   0:00 grep -i bz
 
if all this works, then you are done.

You might also want to double check that your server did register itself with the main listserver – there are 2 ways to do this.

The first is through bzflag – open the client and browse through the servers with the “find server” otion, and see if yours is listed.

The second – and the method I prefer as it is faster, is to just open a web browser up to the list server. The current one is http://208.186.182.172:5156/. You should see your server listed in this list when you start (register) and stop (unregister) your bzfs server. I keep this open in a browser and just refresh it when stopping and starting the bzfs server.

== Misc ==

I use config files for each server that I run, and keep all the download the maps in separate directories.

I keep all my config file in a conf dir. For example /chroot/bzflag/conf/

I also keep all my (not mine, but the ones I downloaded :) ) in a map dir like: /chroot/bzflag/maps 

You may want to create some startup scripts that start your server up automatically in case it reboots. I use a separate rc file for this for simplicity. One could create an init script and then use chkconfig to call it in rc3.d, but I am lazy and haven’t done that.

I simply declared in my rc.local (which is loaded last anyways and as such all network systems will be available when it is called) to execute rc.bzfs which I have in /etc/rc.d/rc.bzfs.

In my rc.bzfs, I have sections for each server I am running. One such section I have looks like:
 #FreeForAll config
 echo >>/var/log/bzfsFFA.log
 echo >>/var/log/bzfsFFA.log
 echo `date` >>/var/log/bzfsFFA.log
 chroot /chroot/bzflag/ /bin/bzfs -d -conf /conf/q-net.conf >> /var/log/bzfsFFA.log 2>&1 & 

This way I can ‘tail –f’ the log file in /var/log, and see what’s going on as all output (errors, standard output – everything) is directed to this file. This will also start it up in the background, and so it will keep going even when you log off. Every time I kill the process and start it using this script, I get 2 empty lines and the date when I issued the command in the log file – this way I can trace it and see. I also start bzfs using the –d option. This outputs more stuff, but not too much, to the log files. I get to see who connected and their current IP’s so if I get a complaint about a person cheating, I can see their callsign and their last IP and monitor it. I can also see who issues the /password command and know if my admin password is compromised.

The weakness that this rc.bzfs startup file has is that you can’t just shut down one server (if you host multiple servers on your box) and start it back up again easily. I have started creating sub scripts for this reason. One script for each server that I run. That way all I have to do is to kill the process and run that one script and everything is OK. I keep these scripts in the bin folder of the jail along with bzfs and make them executable. Since they all call bzfs, you don’t have to make their owner nobody.nobody or have their sticky bit set..

== The End ==
I hope this is useful to people out there, as a bit of time went into it to figure out just which files are needed to run bzfs in a jail. I started out with the entire /lib copied over, and got it working with the minimal files listed above.


==See also==
[[BZFS]]

==External Links==
http://olivier.sessink.nl/jailkit

[[Category:Server]]
[[Category:Server_security]]