Vserver administration

Introduction

Official homepage: Linux VServer Project

Good introduction:

• Linux-VServer Technology
• La Technologie Linux-VServer

Debian support:

apt-cache search vserver
  kernel-patch-vserver - context switching virtual private servers - kernel patch
  util-vserver - tools for Virtual private servers and context switching
  vserver-debiantools - Tools to manage debian virtual servers

Misc:

• Benchmark Comparisons between UML, VMWare, vserver and Xen (pdf)

Kernel compilation

The Debian way

I followed instructions given in

• /usr/share/doc/kernel-patch-vserver/README.Debian
• Step-by-step 2.6
• Debian vservers
• Debian and vserver, french howto
• Debian and vserver

apt-get install kernel-patch-vserver linux-source-2.6.16 kernel-package fakeroot
cd /usr/src
tar xjf linux-source-2.6.16.tar.bz2
cd /usr/src/linux-source-2.6.16
cp config-2.6.16-1-amd64-k8 .config
export PATCH_THE_KERNEL=YES
make-kpkg --rootcmd fakeroot \
        --revision custom01 \
        --added-patches vserver \
        --append-to-version +vserver \
        --initrd \
        binary-arch
"Virtual root device support" -> **y**
"Legacy kernel API" -> y
"Show a Legacy Version ID" -> n
"Disable Legacy Networking Kernel API" -> n
"Enable Proc Security" -> y
"Enable Hard CPU Limits" -> y
"Limit the IDLE task" -> n
"Persistent Inode Context Tagging" -> UID24/GID24 (32/32 probably not yet supported on Reiserfs)
"Tag NFSD User Auth and Files" -> n
"VServer Debugging Code" -> n

Install kernel and reboot

Vanilla with GrSec, still the Debian way

I used linux-2.6.17.14.tar.bz2 + patch-2.6.17.14-vs2.0.2.1-grsec2.1.9.diff
and the config of the Debian kernel config-2.6.17-2-vserver-amd64

make oldconfig

I activated HARDCPU limits and misc PAX & GRSEC stuff (this page can help):

CONFIG_VSERVER_HARDCPU=y
CONFIG_VSERVER_HARDCPU_IDLE=y
CONFIG_PAX=y
CONFIG_PAX_SOFTMODE=y
CONFIG_PAX_PT_PAX_FLAGS=y
CONFIG_PAX_HAVE_ACL_FLAGS=y
CONFIG_PAX_NOEXEC=y
CONFIG_PAX_PAGEEXEC=y
CONFIG_PAX_MPROTECT=y
CONFIG_PAX_ASLR=y
CONFIG_PAX_RANDUSTACK=y
CONFIG_PAX_RANDMMAP=y
CONFIG_PAX_MEMORY_SANITIZE=y
CONFIG_GRKERNSEC=y
CONFIG_GRKERNSEC_CUSTOM=y
CONFIG_GRKERNSEC_KMEM=y
CONFIG_GRKERNSEC_IO=y
CONFIG_GRKERNSEC_PROC_MEMMAP=y
CONFIG_GRKERNSEC_BRUTE=y
CONFIG_GRKERNSEC_MODSTOP=y
CONFIG_GRKERNSEC_ACL_HIDEKERN=y
CONFIG_GRKERNSEC_ACL_MAXTRIES=3
CONFIG_GRKERNSEC_ACL_TIMEOUT=30
CONFIG_GRKERNSEC_PROC=y
CONFIG_GRKERNSEC_PROC_USER=y
CONFIG_GRKERNSEC_PROC_ADD=y
CONFIG_GRKERNSEC_LINK=y
CONFIG_GRKERNSEC_FIFO=y
CONFIG_GRKERNSEC_CHROOT=y
CONFIG_GRKERNSEC_CHROOT_PIVOT=y
CONFIG_GRKERNSEC_CHROOT_CHDIR=y
CONFIG_GRKERNSEC_CHROOT_FCHDIR=y
CONFIG_GRKERNSEC_CHROOT_MKNOD=y
CONFIG_GRKERNSEC_CHROOT_SHMAT=y
CONFIG_GRKERNSEC_CHROOT_UNIX=y
CONFIG_GRKERNSEC_CHROOT_NICE=y
CONFIG_GRKERNSEC_CHROOT_SYSCTL=y
CONFIG_GRKERNSEC_RESLOG=y
CONFIG_GRKERNSEC_SIGNAL=y
CONFIG_GRKERNSEC_FORKFAIL=y
CONFIG_GRKERNSEC_PROC_IPADDR=y
CONFIG_GRKERNSEC_EXECVE=y
CONFIG_GRKERNSEC_SHM=y
CONFIG_GRKERNSEC_DMESG=y
CONFIG_GRKERNSEC_RANDPID=y
CONFIG_GRKERNSEC_RANDNET=y
CONFIG_GRKERNSEC_SYSCTL=y
CONFIG_GRKERNSEC_FLOODTIME=10
CONFIG_GRKERNSEC_FLOODBURST=4

make-kpkg --rootcmd fakeroot --us --uc --initrd kernel-image

And I got a linux-image-2.6.17.14-grsec2.1.9-vs2.0.2.1_2.6.17.14-grsec2.1.9-vs2.0.2.1-10.00.Custom_amd64.deb

Host preparation

apt-get install util-vserver vserver-debiantools
wget http://vserver.13thfloor.at/Stuff/SCRIPT/testme.sh
chmod +x testme.sh
./testme.sh
dd bs=1024k count=1024 if=/dev/zero of=1gb.test
modprobe loop
losetup /dev/loop0 ./1gb.test
./testfs.sh [ -F reiser ] -D /dev/loop0 -M /mnt
losetup -d /dev/loop0
modprobe -r loop

There is no error at this point but as I'm using Reiserfs, I have to activate manually the extended attributes (for lsattr/chattr) by adding the following option to /etc/fstab lines: "attrs" (?? also option acl ??)
Test: lsattr <mount point of a Reiserfs>

Change the vserver base path

• /etc/vservers/.defaults/vdirbase -> /var/lib/vservers
• I change it to /home/vservers, fix the above symlink
• Re-create the "chroot barrier":
  setattr --barrier /home/vservers
  showattr /home -> B for vservers
• Some tools could have /var/lib/vservers hardcoded, for safety I create a symlink /var/lib/vservers pointing to /home/vservers

Manipulating vservers

Create a vserver

Edit /etc/vservers/newvserver-vars:

# cf http://amd64.debian.net/README.mirrors.html
MIRROR="http://ftp.belnet.be/debian-amd64/debian"
INTERFACE="<my_if>"
ARCH="amd64"

Create a vserver with 64bits:

LANG=C newvserver --hostname template64 --domain teuwen.org --ip <new_ip>/24 --dist etch

Create a vserver with 32bits emulation:

LANG=C newvserver --hostname template32 --domain teuwen.org --ip <new_ip>/24 --dist etch --arch i386 --mirror "http://<i386_debian_mirror>"

Tuning:

• take care of the config duplication!
• enter the vserver and run tzconfig to choose the proper timezone
• fix /etc/apt/sources.list
• delete rcX.d links to umountroot
• Warning! If you use newvserver as such, it will overwrite the host /etc/motd due to a symlink
• See [Vserver tools] for a patch for newvserver

Removing unnecessary progs (check if you really don't need them!!):

• aptitude apt-utils base-config cpio dselect tasksel libncursesw5 libsigc++-1.2-5c2 libsigc++-2.0-0c2a
• dmidecode laptop-detect module-init-tools
• bsdmainutils ed nano nvi
• groff-base man-db manpages info libgdbm3
• netcat traceroute wget libssl0.9.8
• gettext-base libconsole libgnutls11 liblzo2-2 libtasn1-2-bin

Automatic start at bootup

echo default > /etc/vservers/<my_vserver>/apps/init/mark

Note that at shotdown all vservers will be stopped

Delete a vserver

Remove dirs /home/vservers/<my_vserver> (depends on the setting of vdirbase, cf. above), /etc/vservers/<my_vserver> and /var/run/vservers/<my_vserver> and the corresponding symlink in /var/run/vservers.rev

Config of a vserver

TODO

?? /etc/vservers/<my_vserver>.conf
?? S_CAPS

see Detailed config page (better choosing boring CSS...)

If you don't assign unique IPs to the vservers but reuse the one of the host:

touch /etc/vservers/<vserver>/interfaces/<N>/nodev

When this file exists, the interface will be assumed to exist already. This can be used to assign primary interfaces which are created by the host or another vserver.

Run a vserver

vserver <my_vserver> start
vserver <my_vserver> enter

If you get "mesg: /dev/pts/1: Operation not permitted", be root on the host with "su -"

vserver <my_vserver> stop

Other tools

vserver <my_vserver> status
vserver-stat
vtop, vps, vpstree, vkill

/etc/rc.d/init.d/rebootmgr is a daemon which can be called from vservers via vreboot and vhalt to stop/restart the vserver from inside

See also compatibility of util-vserver alpha branch

See Vserver tools for my own/modified scripts

Duplicate a vserver

vserver <my_vserver1> stop
dupvserver --from <my_vserver1> --to <my_vserver2> --ip <new_ip>

dupvserver is broken with the new configuration structure /etc/vservers/<my_vserver>/
See Vserver tools for a patch for dupvserver

Move/copy a vserver

Basically stop the vserver and copy /etc/vservers/<my_vserver> and /home/vservers/<my_vserver>
E.g. rsync -e ssh -avHl /vservers/XX new-server:/vserver/XX

Share directories

To mount a directory from one vserver into another from the host:

vnamespace -e <vserver> mount --rbind /directory/to/mount/somewhere /where/to/mount/it
vnamespace -e <vserver> umount /where/it/was/mounted

or

mount --bind /home /var/lib/vservers/vserver1/home
mount --bind /home /var/lib/vservers/vserver2/home

The second method had the disavantage to require a reboot of the vserver

To mount an NFS share in a vserver:
Add the nfs share to /etc/vservers/<vserver>/fstab
If you want the user to be able to do it from the vserver itself, you've to add some capabilities, apparently sth like SECURE_MOUNT, SECURE_REMOUNT and/or BINARY_MOUNT to /etc/vservers/<vserver>/ccapabilities (didn't try)

Apt-get

LANG=C vapt-get <my_vserver1> <my_vserver2> <...> -- install <pkg1> <pkg2>

Unify

cf immutable-linkage-invert flag

Preparation:

mkdir /etc/vservers/template64/apps/vunify
mkdir /etc/vservers/<my_vserver>/apps/vunify
ln -s /etc/vservers/template64 /etc/vservers/<my_vserver>/apps/vunify/refserver.template64

Unification:
Be sure both vservers are running

vserver <my_vserver> unify [-n] [-R]

-n for dry run, no change
-R for de-unifying

When using tar, add option -U to unlink & recreate files instead of overwriting.
Manual set/unset of the immutable-linkage-invert flag:

setattr --iunlink /my/file
setattr --~iunlink /my/file

Disk limits

cf http://linux-vserver.org/Disk+Limits

• Assign static contexts for the vservers (i.e. have a value between 2 and 49151 in /etc/vservers/<name>/context)
• Mount the filesystem holding the vserver(s) with the tagxid option
  • Check if this is mounted properly: use cat /proc/mounts
    Ex.: /dev/mapper/Zeus-home /home reiserfs rw,tagxid 0 0
  • WARNING: if the filesystem is already in use with vservers, nothing prevent you to umount the filesystem while the vservers are still running, which is VERY BAD! Be careful.
  • I could only get the tagxid taken properly into account after a reboot
  • To set tagxid on / you need to do it from initrd as tagxid cannot be set at remount: add to the kernel params the option "rootflags=tagxid", e.g. via /boot/grub/menu.lst #kopts=...
• Change the xid of already existing files:

chxid -c <my_vserver> -R /home/vservers/<my_vserver>

• Set limits, first method: here limit to 5Gb, 100000 inodes and 5% for the root user
  For info as I could not get it working properly yet

mkdir /var/cache/vservers
ln -s /var/cache/vservers /etc/vservers/.defaults/cachebase
mkdir /etc/vservers/.defaults/cachebase/<my_server>
ln -s /etc/vservers/.defaults/cachebase/<my_server> /etc/vservers/<my_server>/cache
mkdir -p /etc/vservers/<my_vserver>/dlimits/0
echo /home/vservers/<my_vserver> > /etc/vservers/<my_vserver>/dlimits/0/directory
echo $(( 5 * 1024 * 1024 )) > /etc/vservers/<my_vserver>/dlimits/0/space_total
echo 100000 > /etc/vservers/<my_vserver>/dlimits/0/inodes_total
echo 5 > /etc/vservers/<my_vserver>/dlimits/0/reserved

• Set limits, second method:
  • Install my vdlimit_ script in /usr/local/sbin: Vserver tools

ln -s /usr/local/sbin/vdlimit_ /etc/vservers/<my_vserver>/scripts/post-start.d/vdlimit_$((5*1024))

• • To change the limit on-the-fly simply rename the link and execute

./vdlimit_<new_size> pre-stop <my_vserver>;./vdlimit_<new_size> post-start <my_vserver>;

Network

Intern network

For pure loopback, use dummy interface, cf http://mirabellug.org/wikini/wakka.php?wiki=VServers

For usable dummy interface, us permanent taps as the uml tools allow:

apt-get install uml-utilities

• Create a pseudo-interface:

auto tap0
iface tap0 inet static
    address 192.168.2.1
    netmask 255.255.255.0
    tunctl_user uml-net

And configure vservers with the same dev=tap0

Update: to check but actually all traffic with private or public IP will anyway be done through lo so this is probably not required

Note that if you use openvpn, you can create tun/tap with

openvpn --mktun --dev tap0

Configure daemons to listen only to the IP-address of the mothersystem

• openbsd-inetd: (not netkit-inetd) in file /etc/inetd.conf:
  Prepend the service with <IP pub>:
  Example

<IP pub>:cvspserver       stream  tcp     nowait  root    /usr/sbin/tcpd  /usr/sbin/cvs-pserver

• xinetd: (not inetd) in file /etc/xinetd.conf:

defaults
{ bind = <IP pub> }

/etc/init.d/xinetd restart

• sshd: in file /etc/ssh/sshd_config:

ListenAddress <IP pub>

/etc/init.d/ssh restart

• exim4: in file /etc/exim4/update-exim4.conf.conf:

dc_local_interfaces='<IP pub>'

/etc/init.d/exim4 restart

Better to do it through debconf to avoid surprises at update time: dpkg-reconfigure exim4-config

• courier-imap: in file /etc/courier/imapd:

ADDRESS=<IP pub>

/etc/init.d/courier-imap restart

• courier-imap-ssl: in file /etc/courier-ssl/imapd:

ADDRESS=<IP pub>

/etc/init.d/courier-imap-ssl restart

• imapproxy: in file /etc/imapproxy.conf:

listen_address <IP pub>

Within a vserver, you'll probably hav to reduce the cache_size or give capability to the vserver to raise the setrlimit.

• mysql: in file /etc/mysql/my.cnf:

bind-address = <IP pub>

• vsFtpd: in file /etc/vsftpd.conf:

listen_address=<IP pub>

• postgresql: in file /etc/postgresql/postgresql.conf:

virtual_host = '<IP pub>'

• apache2: in file /etc/apache2/ports.conf:

Listen <IP pub>:80

• zope2.9: in file /etc/zope2.9/<instance>/zope.conf:

ip-address <IP pub>

• portmap: in file /etc/default/portmap:

OPTIONS="-i <IP pub/loopback>"

• dnsmasq: in file /etc/dnsmasq.conf:

listen-address=<IP pub>
bind-interfaces

• openvpn in file /etc/openvpn/server.conf:

local <IP pub>

• slapd: in file /etc/default/slapd

SLAPD_SERVICES="ldap://<IP priv>/ ldaps://<IP pub>/"

• oidentd in file /etc/default/oidentd

Add to OIDENT_OPTIONS list: "-a <IP pub>"

• netstat -lp -> other greedy daemons?
• Seems that this is possible via another method, here it will bind the daemon to the first IP of the interface:
  exec /usr/sbin/chbind --ip eth0 /path/to/daemon

Add an interface without rebooting the vserver

• add the ip to the host (ip addr add ...)
• add the ip to the guest's network context

# naddress --add --nid <nid> --ip <ip>/<mask>

• enter the guest (best via ssh)
• restart the services if required
  (most services will automatically start using the new addresses)
• update the config to reflect the changes for the next guest restart (if desired)

Thanks Herbert!

Understanding vservers

Security contextes

• Find security context of process N:

chcontext --ctx 1 cat /proc/N/status|grep s_context

• Be in the same context:

chcontext --ctx X /bin/sh

• Master context: 1, example to get all listening ports:

chcontext --ctx 1 netstat -lpn

See also Virtual private servers and security contexts

Ceiling capabilities

• As non-root, check capBset:

cat /proc/self/status

• Reduce ceiling caps:

reducecap --secure /bin/sh

• Now capBset is reduced:

cat /proc/self/status
su

• capEff raised a bit but not enough to do for example /sbin/ifconfig eth0 down
• See also [Capabilities in Linux|http://www.lids.org/lids-howto/node34.html]

Security

Not necessarily related to vserver but always useful to consider :-)

• ssh
  • Use the AllowUsers option to give ssh rights only to those who need it.
  • Brute-force protection: apt-get install denyhosts
    Edit /etc/denyhosts.conf to get email reports
    Un case someone forgot his pwd and got banned, to remove the ban directly: remove it from /var/lib/denyhosts files and /etc/hosts.deny of course
• iptables (on the host)
  • cf --uid-owner and other --XXX-owner options
    on OUTPUT table to avoid download of malicious code
    on INPUT table to avoid bindshells
• resource limits
  • cpu/mem

GrSec

• http://pax.grsecurity.net/
• http://people.linux-vserver.org/~harry/_README_
• http://www.zataz.net/docs/8024/introduction-grsecurity.html
• http://linux-vserver.org/grsecurityHowto
• http://ludit.kuleuven.be/software/vserver/_README_

apt-get install paxctl gradm2

Iptables Proxy

• http://www.virtuaserver.com.br/forum/viewtopic.php?t=130

Other tricks

• For other tweaks, see http://deb.riseup.net/vserver/usage/ :
  • What if I accidentally removed a vserver while it was running?
  • Howto convert legacy vservers to the new format
  • Howto add an IP to a running vserver, without restarting it?
  • Howto make the host interface and IP available in a vserver
  • Howto impose disk limits in each vserver
• http://www.paul.sladen.org/vserver/faq
• Problematic programs
• If you drop files from "outside of the vserver context" (from the host e.g.) you've to reassign the correct xid to the files:

chxid -c <vserver> -R /home/vservers/<vserver>
# all at once:
for i in $(ls /etc/vservers/); do echo $i; chxid -c $i -R /home/vservers/$i;done

• If you drop files from "outside of the vserver context" (from the host e.g.) you've to regenerate the disk usage and limit of the vserver if you use my vdlimit_ script:

vserver <vserver> stop
rm /var/cache/vservers/<vserver>_vdlimit_
vserver <vserver> start

• To run a script (e.g. an /etc/init.d/start_my_daemon) in ctx 1, e.g. to start ntop and be sure it can see all the traffic, simply add at the begin of the script:

if cat /proc/self/vinfo|grep -q -v ":[^0-9]1$"; then                                                                                                                      
   /usr/sbin/chcontext --ctx 1 $0 $*                                                                                                                                     
   exit                                                                                                                                                                  
fi

• To "mount" a samba shared drive from a vserver is not possible or at least when running grsec but you can still use the good old ftp-styled smbclient

smbclient //machine/share -U domain/user

TODO

• http://www.nongnu.org/util-vserver/doc/conf/compatibility.html
• http://linux-vserver.derjohn.de/
• [VServer wiki|http://vserver.strahlungsfrei.de/tiki-index.php]
• [Administrator Guide|http://linux-vserver.org/linux-vserver_administrators_gide]
• [Debian newvserver|http://www.paul.sladen.org/vserver/debian/]
• [Howto Debian vserver|http://www.howtoforge.com/linux_vserver_debian]
• ?? apt-get install vlan
• ?? ipac-ng
• CPU limit
  • http://linux-vserver.org/Linux-VServer-Paper-06
  • http://list.linux-vserver.org/archive/vserver/msg08134.html
• BW limit
  • http://lartc.org/howto/
• http://linux-vserver.org/HowTo+Read+ProcFS
• http://linux-vserver.org/HistoryList?full=1
• Publish Munin scripts
• http://linux-vserver.org/VServer+installation+Fedora+Core+5
• http://vserver.13thfloor.at/Experimental/
• http://www.archivesat.com/Linux-VServer/
• http://www.solucorp.qc.ca/miscprj/s_context.hc?s1=1&s2=0&s3=0&s4=0&full=0&prjstate=1&nodoc=0
• (fr) http://fr.wikibooks.org/wiki/Vserver