Difference between revisions of "BZFS in a chroot jail"

From BZFlagWiki
Jump to: navigation, search
(Port BZFS_in_a_chroot_jail from old wiki)
m (Add to Server and Support categories)
Line 214: Line 214:
Quol (quolsimo@hotmail.com)
Quol (quolsimo@hotmail.com)

Revision as of 15:12, 20 February 2007


The purpose of this doc is to show how to install the BZFLAG Server (bzfs) in a ‘sandbox’ or a ‘jail’ on Linux by using the features provided with chroot. For general information on setting up a server see Creating_A_Server.

This has been tested on Redhat 8 and 9 systems, although it should be fairly similar, if not identical on other Linux distributions.

If when you read it you see some errors or had some issues not mentioned in here and have the answers to them, you can e-mail me or create a new section at the end with the updates (after my signature - and sign the updates yourself so I know whom to credit if the new info gets merged). I can always merge them into the doc or create a new section (keeping the credits of course) for smoother reading at a later date.

OK - let's get started...


Before we begin, a little background in using the chroot command is needed. The best place to start is reading the man page (man chroot), but basically the concept is to run a program in a folder and force the program to think that it is the root (the top) of the filesystem (the ‘sandbox’ or ‘jail’), so that if the application was ever compromised (like the shell access vulnerability scare that went around for bzfs a little while ago), only it’s folder would be accessible to the attacker and not the entire filesystem. User root, or any program with root privileges can break out of a chroot jail. And beware, a badly configured chroot jail might even be a security problem! jk_check from jailkit (http://olivier.sessink.nl/jailkit) does check if your chroot jail is safe.

Before we can run a program in a jail, we have to make sure that it has everything it needs to run, and this means creating a mini root filesystem so that the program can access the files that it needs, and knows where to find them. This means that if a program requires a library in /lib, then we will need a lib directory with those libraries in our jail. (if this is confusing – hang in there, you will see an example below).

Just tell me how to do it!!

Let’s say we have already compiled and installed (via RPM or source) bzflag in /usr/share/bzflag (or wherever you installed it), and we want to have it run in a jail.

I like to create a folder in the root of the filesystem called chroot, and then place folders in there for everything that I am running in a jail. This helps me to keep track of which programs/services/applications I am running in a jail easily. I also don’t like creating jails in system directories (in this case it would be /usr/share or wherever you installed it).

I would, for example, create a folder like /chroot/bzflag. I would then copy the actual executable in a bin folder. So know we have:

/chroot/bzflag/bin/bzfs (the bzfs executable in the jail’s bin directory).

Now we need the dependency files – how do we know which files bzfs depends on? Run ‘ldd –v /usr/bin/bzfs’ (or where ever you installed bzfs to when you compiled it. NOTE: that this is the real one and not the one you copied to the jail) and you should get something like (on Redhat 9 systems):

$ ldd -v bzfs
       libstdc++.so.5 => /usr/lib/libstdc++.so.5 (0x40034000)
       libm.so.6 => /lib/tls/libm.so.6 (0x400e7000)
       libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x40109000)
       libc.so.6 => /lib/tls/libc.so.6 (0x42000000)
       /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x40000000)

       Version information:
               libc.so.6 (GLIBC_2.1.3) => /lib/tls/libc.so.6
               libc.so.6 (GLIBC_2.1) => /lib/tls/libc.so.6
               libc.so.6 (GLIBC_2.0) => /lib/tls/libc.so.6
               libstdc++.so.5 (CXXABI_1.2) => /usr/lib/libstdc++.so.5
               libstdc++.so.5 (GLIBCPP_3.2) => /usr/lib/libstdc++.so.5
               libgcc_s.so.1 (GCC_3.0) => /lib/libgcc_s.so.1
               libc.so.6 (GLIBC_2.2) => /lib/tls/libc.so.6
               libc.so.6 (GLIBC_2.1.3) => /lib/tls/libc.so.6
               libc.so.6 (GLIBC_2.3) => /lib/tls/libc.so.6
               libc.so.6 (GLIBC_2.0) => /lib/tls/libc.so.6
               libc.so.6 (GLIBC_2.1) => /lib/tls/libc.so.6
               libc.so.6 (GLIBC_2.1.3) => /lib/tls/libc.so.6
               libc.so.6 (GLIBC_2.0) => /lib/tls/libc.so.6
               libc.so.6 (GLIBC_2.2.4) => /lib/tls/libc.so.6
               libc.so.6 (GLIBC_2.1.3) => /lib/tls/libc.so.6
               libc.so.6 (GLIBC_2.0) => /lib/tls/libc.so.6
               ld-linux.so.2 (GLIBC_2.3) => /lib/ld-linux.so.2
               ld-linux.so.2 (GLIBC_2.1) => /lib/ld-linux.so.2
               ld-linux.so.2 (GLIBC_2.0) => /lib/ld-linux.so.2
               ld-linux.so.2 (GLIBC_PRIVATE) => /lib/ld-linux.so.2}}}

This is where the ‘mini root’ filesystem comes in. For every file listed in the output of the ldd command – copy those files to the jail into the appropriate directory. The jailkit package at http://olivier.sessink.nl/jailkit/ contains a utility jk_cp that can be used to automatically copy all dependencies of a file into a jail.

mkdir /chroot/
jk_cp /chroot/ /bin/bzfs

Make sure you get all of them or bzfs will complain that it is missing something later on. On Redhat 9 there is a /lib/tls directory, on Redhat 8 these same libraries were in /lib/i686 – so you will have to substitute the proper directories here for wherever your system expects them. Use the exact same path as outputted by ldd. So you should copy something similar to (create the directories as needed):

/chroot/bzflag/usr/lib/libstdc++.so.5   (copied from /usr/lib)
/chroot/bzflag/lib/libgcc_s.so.1        (copied from /lib)
/chroot/bzflag/lib/tls/libm.so.6        (copied from /lib/tls)
/chroot/bzflag/lib/tls/libc.so.6        (copied from /lib/tls)
/chroot/bzflag/lib/ld-linux.so.2        (copied from /lib)

BZFS also needs libraries to resolve DNS names to IP addresses (to register itself with the list server), and so you will need these extra libraries that are not reported through ldd (copy from /lib to your corresponding jail directory):

/lib/libnss_dns.so.2  -> /chroot/bzflag/lib/libnss_dns.so.2
/lib/libresolv.so.2   -> /chroot/bzflag/lib/libresolv.so.2

You will also need an etc directory with the resolv.conf file in it so that the above libraries know which DNS server they need to contact in order to resolve things, so copy from /etc into your corresponding jail directory:

/etc/resolv.conf -> /chroot/bzflag/etc/resolv.conf

and now the hard part is done…

Again this bit can be done easier using jailkit:

jk_init /chroot/ netbasics

Here is the complete listing of what your jail should look like or be similar to (in the case of the lib directory which will change from Linux version/distro I am sure – it already changed just from Redhat 8 to 9):

$ ls -alFR ./bzflag/
total 24
drwxr-xr-x    6 root     root         4096 Jul 12 08:12 ./
drwxr-xr-x    3 root     root         4096 Jul 12 07:24 ../
drwxr-xr-x    2 root     root         4096 Jul 12 07:25 bin/
drwxr-xr-x    2 root     root         4096 Jul 12 08:29 etc/
drwxr-xr-x    3 root     root         4096 Jul 12 08:34 lib/
drwxr-xr-x    3 root     root         4096 Jul 12 08:26 usr/

total 188
drwxr-xr-x    2 root     root         4096 Jul 12 07:25 ./
drwxr-xr-x    6 root     root         4096 Jul 12 08:12 ../
-rwsr-sr-x    1 nobody   nobody     179160 Jul 12 07:25 bzfs*

total 12
drwxr-xr-x    2 root     root         4096 Jul 12 08:29 ./
drwxr-xr-x    6 root     root         4096 Jul 12 08:12 ../
-rw-r--r--    1 root     root           54 Jul 12 08:29 resolv.conf

total 252
drwxr-xr-x    3 root     root         4096 Jul 12 08:34 ./
drwxr-xr-x    6 root     root         4096 Jul 12 08:12 ../
-rwxr-xr-x    1 root     root       103044 Jul 12 08:20 ld-linux.so.2*
-rwxr-xr-x    1 root     root        30324 Jul 12 08:20 libgcc_s.so.1*
-rwxr-xr-x    1 root     root        18416 Jul 12 08:25 libnss_dns.so.2*
-rwxr-xr-x    1 root     root        76552 Jul 12 08:23 libresolv.so.2*
drwxr-xr-x    2 root     root         4096 Jul 12 08:16 tls/

total 1740
drwxr-xr-x    2 root     root         4096 Jul 12 08:16 ./
drwxr-xr-x    3 root     root         4096 Jul 12 08:34 ../
-rwxr-xr-x    1 root     root      1549556 Jul 12 08:16 libc.so.6*
-rwxr-xr-x    1 root     root       211876 Jul 12 08:16 libm.so.6*

total 80
drwxr-xr-x    3 root     root         4096 Jul 12 08:26 ./
drwxr-xr-x    6 root     root         4096 Jul 12 08:12 ../
drwxr-xr-x    2 root     root        69632 Jul 12 08:02 lib/

total 776
drwxr-xr-x    2 root     root        69632 Jul 12 08:02 ./
drwxr-xr-x    3 root     root         4096 Jul 12 08:26 ../
-rwxr-xr-x    1 root     root       710608 Jul 12 08:02 libstdc++.so.5*

More Security

Since you have to execute chroot as root, bzfs will run as root – which is not what we want. We can force bzfs to run as nobody by changing his ownership to nobody, and setting the sticky bit on him. This way, when root executes /chroot/bzflag/bin/bzfs – it is executed as nobody, and therefore has very little privileges on the system.

This is how we do that:

chown nobody.nobody /chroot/bzflag/bin/bzfs
chmod +s /chroot/bzflag/bin/bzfs

But this changes only the effective user id, not the real user id. A safer way to do this is to make use of the jk_chrootlaunch utility in jailkit (http://olivier.sessink.nl/jailkit/), it will drop all privileges, and then execute as user nobody, without any possibility to regain root privileges, because both the effective AND the real user id are now changed to nobody.

jk_chrootlaunch -j /chroot/bzflag/ -x /bin/bzfs –d –d –d –d –public TestServer –publicaddr <your_public_ip_address:port>


Run a command like the following (as root),

chroot /chroot/bzflag/ /bin/bzfs –d –d –d –d –public TestServer –publicaddr <your_public_ip_address:port>

It will log any errors to /var/log/daemon.log

This is where all the magic of chroot happens. And this command should make sense to you as you read the man page, right :) If not, the command basically takes 2 arguments. The first is the "new root" (/chroot/bzflag/). The second is the command to run. Notice that we specify /bin/bzfs to run bzfs, and not /chroot/bzflag/bin.

You should see something similar to:

$ chroot /chroot/bzflag/ /bin/bzfs -d -d -d -d -public test -publicaddr
style: 0
Opening List Server 0
listening on

ADD 2 BZFS107e 000000c800010000000000000000000000c800c800c800c800c800000000000000000000 test

Closing List server 0
Player [0] accept() from on 7
Player [0] shutdownAcceptClient: close(7) 

If you don’t or get an error like “failed http ...” – then a resolv/dns library or the resolv.conf in etc can’t be found – copy them again check your spelling and any paths and try again…

The next thing to verify is that bzfs is really running as nobody and not as root. Start bzfs in the background this time ($ chroot /chroot/bzflag/ /bin/bzfs -d -d -d -d -public test -publicaddr <your_public_ip_address_here>:<your_server_port> &), and then ‘ps aux | grep –i bz’ and look for at the user bzfs is running as.

You should see something like:

$ ps aux | grep -i bz
nobody    2930  0.1  1.3  3032 1668 pts/1    S    09:49   0:00 [bzfs]
root      2932  0.0  0.5  3568  628 pts/1    S    09:49   0:00 grep -i bz

if all this works, then you are done.

You might also want to double check that your server did register itself with the main listserver – there are 2 ways to do this.

The first is through bzflag – open the client and browse through the servers with the “find server” otion, and see if yours is listed.

The second – and the method I prefer as it is faster, is to just open a web browser up to the list server. The current one is You should see your server listed in this list when you start (register) and stop (unregister) your bzfs server. I keep this open in a browser and just refresh it when stopping and starting the bzfs server.


I use config files for each server that I run, and keep all the download the maps in separate directories.

I keep all my config file in a conf dir. For example /chroot/bzflag/conf/

I also keep all my (not mine, but the ones I downloaded :) ) in a map dir like: /chroot/bzflag/maps

You may want to create some startup scripts that start your server up automatically in case it reboots. I use a separate rc file for this for simplicity. One could create an init script and then use chkconfig to call it in rc3.d, but I am lazzy and haven’t done that.

I simply declared in my rc.local (which is loaded last anyways and as such all network systems will be available when it is called) to execute rc.bzfs which I have in /etc/rc.d/rc.bzfs.

In my rc.bzfs, I have sections for each server I am running. One such section I have looks like:

#FreeForAll config
echo >>/var/log/bzfsFFA.log
echo >>/var/log/bzfsFFA.log
echo `date` >>/var/log/bzfsFFA.log
chroot /chroot/bzflag/ /bin/bzfs -d -conf /conf/q-net.conf >> /var/log/bzfsFFA.log 2>&1 & 

This way I can ‘tail –f’ the log file in /var/log, and see what’s going on as all output (errors, standard output – everything) is directed to this file. This will also start it up in the background, and so it will keep going even when you log off. Every time I kill the process and start it using this script, I get 2 empty lines and the date when I issued the command in the log file – this way I can trace it and see. I also start bzfs using the –d option. This outputs more stuff, but not too much, to the log files. I get to see who connected and their current IP’s so if I get a complaint about a person cheating, I can see their callsign and their last IP and monitor it. I can also see who issues the /password command and know if my admin password is compromised.

The weakness that this rc.bzfs startup file has is that you can’t just shut down one server (if you host multiple servers on your box) and start it back up again easily. I have started creating sub scripts for this reason. One script for each server that I run. That way all I have to do is to kill the process and run that one script and everything is OK. I keep these scripts in the bin folder of the jail along with bzfs and make them executable. Since they all call bzfs, you don’t have to make their owner nobody.nobody or have their sticky bit set..

The End

I hope this is useful to people out there, as a bit of time went into it to figure out just which files are needed to run bzfs in a jail. I started out with the entire /lib copied over, and got it working with the minimal files listed above.

See you out there

Quol (quolsimo@hotmail.com)