Vserver administration

From YobiWiki
Jump to navigation Jump to search

Introduction

Official homepage: Linux VServer Project

Good introduction:

Debian support: 

apt-cache search vserver
  kernel-patch-vserver - context switching virtual private servers - kernel patch
  util-vserver - tools for Virtual private servers and context switching
  vserver-debiantools - Tools to manage debian virtual servers

Misc:

Kernel compilation

The Debian way

I followed instructions given in 

apt-get install kernel-patch-vserver linux-source-2.6.16 kernel-package fakeroot
cd /usr/src
tar xjf linux-source-2.6.16.tar.bz2
cd /usr/src/linux-source-2.6.16
cp config-2.6.16-1-amd64-k8 .config
export PATCH_THE_KERNEL=YES
make-kpkg --rootcmd fakeroot \
        --revision custom01 \
        --added-patches vserver \
        --append-to-version +vserver \
        --initrd \
        binary-arch
"Virtual root device support" -> **y**
"Legacy kernel API" -> y
"Show a Legacy Version ID" -> n
"Disable Legacy Networking Kernel API" -> n
"Enable Proc Security" -> y
"Enable Hard CPU Limits" -> y
"Limit the IDLE task" -> n
"Persistent Inode Context Tagging" -> UID24/GID24 (32/32 probably not yet supported on Reiserfs)
"Tag NFSD User Auth and Files" -> n
"VServer Debugging Code" -> n

Install kernel and reboot

Vanilla with GrSec, still the Debian way

I used linux-2.6.17.14.tar.bz2 + patch-2.6.17.14-vs2.0.2.1-grsec2.1.9.diff
and the config of the Debian kernel config-2.6.17-2-vserver-amd64 

make oldconfig

I activated HARDCPU limits and misc PAX & GRSEC stuff (this page can help): 

CONFIG_VSERVER_HARDCPU=y
CONFIG_VSERVER_HARDCPU_IDLE=y
CONFIG_PAX=y
CONFIG_PAX_SOFTMODE=y
CONFIG_PAX_PT_PAX_FLAGS=y
CONFIG_PAX_HAVE_ACL_FLAGS=y
CONFIG_PAX_NOEXEC=y
CONFIG_PAX_PAGEEXEC=y
CONFIG_PAX_MPROTECT=y
CONFIG_PAX_ASLR=y
CONFIG_PAX_RANDUSTACK=y
CONFIG_PAX_RANDMMAP=y
CONFIG_PAX_MEMORY_SANITIZE=y
CONFIG_GRKERNSEC=y
CONFIG_GRKERNSEC_CUSTOM=y
CONFIG_GRKERNSEC_KMEM=y
CONFIG_GRKERNSEC_IO=y
CONFIG_GRKERNSEC_PROC_MEMMAP=y
CONFIG_GRKERNSEC_BRUTE=y
CONFIG_GRKERNSEC_MODSTOP=y
CONFIG_GRKERNSEC_ACL_HIDEKERN=y
CONFIG_GRKERNSEC_ACL_MAXTRIES=3
CONFIG_GRKERNSEC_ACL_TIMEOUT=30
CONFIG_GRKERNSEC_PROC=y
CONFIG_GRKERNSEC_PROC_USER=y
CONFIG_GRKERNSEC_PROC_ADD=y
CONFIG_GRKERNSEC_LINK=y
CONFIG_GRKERNSEC_FIFO=y
CONFIG_GRKERNSEC_CHROOT=y
CONFIG_GRKERNSEC_CHROOT_PIVOT=y
CONFIG_GRKERNSEC_CHROOT_CHDIR=y
CONFIG_GRKERNSEC_CHROOT_FCHDIR=y
CONFIG_GRKERNSEC_CHROOT_MKNOD=y
CONFIG_GRKERNSEC_CHROOT_SHMAT=y
CONFIG_GRKERNSEC_CHROOT_UNIX=y
CONFIG_GRKERNSEC_CHROOT_NICE=y
CONFIG_GRKERNSEC_CHROOT_SYSCTL=y
CONFIG_GRKERNSEC_RESLOG=y
CONFIG_GRKERNSEC_SIGNAL=y
CONFIG_GRKERNSEC_FORKFAIL=y
CONFIG_GRKERNSEC_PROC_IPADDR=y
CONFIG_GRKERNSEC_EXECVE=y
CONFIG_GRKERNSEC_SHM=y
CONFIG_GRKERNSEC_DMESG=y
CONFIG_GRKERNSEC_RANDPID=y
CONFIG_GRKERNSEC_RANDNET=y
CONFIG_GRKERNSEC_SYSCTL=y
CONFIG_GRKERNSEC_FLOODTIME=10
CONFIG_GRKERNSEC_FLOODBURST=4

make-kpkg --rootcmd fakeroot --us --uc --initrd kernel-image

And I got a linux-image-2.6.17.14-grsec2.1.9-vs2.0.2.1_2.6.17.14-grsec2.1.9-vs2.0.2.1-10.00.Custom_amd64.deb

Host preparation

apt-get install util-vserver vserver-debiantools
wget http://vserver.13thfloor.at/Stuff/SCRIPT/testme.sh
chmod +x testme.sh
./testme.sh
dd bs=1024k count=1024 if=/dev/zero of=1gb.test
modprobe loop
losetup /dev/loop0 ./1gb.test
./testfs.sh [ -F reiser ] -D /dev/loop0 -M /mnt
losetup -d /dev/loop0
modprobe -r loop

There is no error at this point but as I'm using Reiserfs, I have to activate manually the extended attributes (for lsattr/chattr) by adding the following option to /etc/fstab lines: "attrs" (?? also option acl ??)
Test: lsattr <mount point of a Reiserfs>

Change the vserver base path

  • /etc/vservers/.defaults/vdirbase -> /var/lib/vservers
  • I change it to /home/vservers, fix the above symlink
  • Re-create the "chroot barrier":
    setattr --barrier /home/vservers%%%showattr /home -> B for vservers
  • Some tools could have /var/lib/vservers hardcoded, for safety I create a symlink /var/lib/vservers pointing to /home/vservers

Manipulating vservers

Create a vserver

Edit /etc/vservers/newvserver-vars: 

# cf http://amd64.debian.net/README.mirrors.html
MIRROR="http://ftp.belnet.be/debian-amd64/debian"
INTERFACE="<my_if>"
ARCH="amd64"

Create a vserver with 64bits: 

LANG=C newvserver --hostname template64 --domain teuwen.org --ip <new_ip>/24 --dist etch

Create a vserver with 32bits emulation: 

LANG=C newvserver --hostname template32 --domain teuwen.org --ip <new_ip>/24 --dist etch --arch i386 --mirror "http://<i386_debian_mirror>"

Tuning:

  • take care of the config duplication!
  • enter the vserver and run tzconfig to choose the proper timezone
  • fix /etc/apt/sources.list
  • delete rcX.d links to umountroot
  • Warning! If you use newvserver as such, it will overwrite the host /etc/motd due to a symlink
  • See [Vserver tools] for a patch for newvserver

Removing unnecessary progs (check if you really don't need them!!):

  • aptitude apt-utils base-config cpio dselect tasksel libncursesw5 libsigc++-1.2-5c2 libsigc++-2.0-0c2a
  • dmidecode laptop-detect module-init-tools
  • bsdmainutils ed nano nvi
  • groff-base man-db manpages info libgdbm3
  • netcat traceroute wget libssl0.9.8
  • gettext-base libconsole libgnutls11 liblzo2-2 libtasn1-2-bin

Automatic start at bootup

echo default > /etc/vservers/<my_vserver>/apps/init/mark

Note that at shotdown all vservers will be stopped

Delete a vserver

Remove dirs /home/vservers/<my_vserver> (depends on the setting of vdirbase, cf. above), /etc/vservers/<my_vserver> and /var/run/vservers/<my_vserver> and the corresponding symlink in /var/run/vservers.rev

Config of a vserver

TODO 

?? /etc/vservers/<my_vserver>.conf
?? S_CAPS

see Detailed config page (better choosing boring CSS...)

Run a vserver

vserver <my_vserver> start
vserver <my_vserver> enter

If you get "mesg: /dev/pts/1: Operation not permitted", be root on the host with "su -" 

vserver <my_vserver> stop

Other tools

vserver <my_vserver> status
vserver-stat
vtop, vps, vpstree, vkill

/etc/rc.d/init.d/rebootmgr is a daemon which can be called from vservers via vreboot and vhalt to stop/restart the vserver from inside

See also compatibility of util-vserver alpha branch

See Vserver tools for my own/modified scripts

Duplicate a vserver

vserver <my_vserver1> stop
dupvserver --from <my_vserver1> --to <my_vserver2> --ip <new_ip>

dupvserver is broken with the new configuration structure /etc/vservers/<my_vserver>/
See Vserver tools for a patch for dupvserver

Move/copy a vserver

Basically stop the vserver and copy /etc/vservers/<my_vserver> and /home/vservers/<my_vserver>
E.g. rsync -e ssh -avHl /vservers/XX new-server:/vserver/XX

Share directories

To mount a directory from one vserver into another from the host: 

vnamespace -e <vserver> mount --rbind /directory/to/mount/somewhere /where/to/mount/it
vnamespace -e <vserver> umount /where/it/was/mounted

or 

mount --bind /home /var/lib/vservers/vserver1/home
mount --bind /home /var/lib/vservers/vserver2/home

The second method had the disavantage to require a reboot of the vserver

Apt-get

LANG=C vapt-get <my_vserver1> <my_vserver2> <...> -- install <pkg1> <pkg2>

Unify

cf immutable-linkage-invert flag

Preparation: 

mkdir /etc/vservers/template64/apps/vunify
mkdir /etc/vservers/<my_vserver>/apps/vunify
ln -s /etc/vservers/template64 /etc/vservers/<my_vserver>/apps/vunify/refserver.template64

Unification:
Be sure both vservers are running 

vserver <my_vserver> unify [-n] [-R]

-n for dry run, no change
-R for de-unifying

When using tar, add option -U to unlink & recreate files instead of overwriting.
Manual set/unset of the immutable-linkage-invert flag: 

setattr --iunlink /my/file
setattr --~iunlink /my/file

Disk limits

cf http://linux-vserver.org/Disk+Limits

  • Assign static contexts for the vservers (i.e. have a value between 2 and 49151 in /etc/vservers/<name>/context)
  • Mount the filesystem holding the vserver(s) with the tagxid option
    • Check if this is mounted properly: use cat /proc/mounts
      Ex.: /dev/mapper/Zeus-home /home reiserfs rw,tagxid 0 0
    • WARNING: if the filesystem is already in use with vservers, nothing prevent you to umount the filesystem while the vservers are still running, which is VERY BAD! Be careful.
    • I could only get the tagxid taken properly into account after a reboot
  • Change the xid of already existing files:
chxid -c <my_vserver> -R /home/vservers/<my_vserver>
  • Set limits, first method: here limit to 5Gb, 100000 inodes and 5% for the root user
    For info as I could not get it working properly yet
mkdir /var/cache/vservers
ln -s /var/cache/vservers /etc/vservers/.defaults/cachebase
mkdir /etc/vservers/.defaults/cachebase/<my_server>
ln -s /etc/vservers/.defaults/cachebase/<my_server> /etc/vservers/<my_server>/cache
mkdir -p /etc/vservers/<my_vserver>/dlimits/0
echo /home/vservers/<my_vserver> > /etc/vservers/<my_vserver>/dlimits/0/directory
echo $(( 5 * 1024 * 1024 )) > /etc/vservers/<my_vserver>/dlimits/0/space_total
echo 100000 > /etc/vservers/<my_vserver>/dlimits/0/inodes_total
echo 5 > /etc/vservers/<my_vserver>/dlimits/0/reserved
  • Set limits, second method: 
ln -s /usr/local/sbin/vdlimit_ /etc/vservers/<my_vserver>/scripts/post-start.d/vdlimit_$((5*1024))

Network

Intern network

For pure loopback, use dummy interface, cf http://mirabellug.org/wikini/wakka.php?wiki=VServers

For usable dummy interface, us permanent taps as the uml tools allow: 

apt-get install uml-utilities
  • Create a pseudo-interface:
auto tap0
iface tap0 inet static
    address 192.168.2.1
    netmask 255.255.255.0
    tunctl_user uml-net

And configure vservers with the same dev=tap0

Update: to check but actually all traffic with private or public IP will anyway be done through lo so this is probably not required

Configure daemons to listen only to the IP-address of the mothersystem

  • openbsd-inetd: (not netkit-inetd) in file /etc/inetd.conf:
    Prepend the service with <IP pub>:
    Example
<IP pub>:cvspserver       stream  tcp     nowait  root    /usr/sbin/tcpd  /usr/sbin/cvs-pserver
  • xinetd: (not inetd) in file /etc/xinetd.conf:
defaults
{ bind = <IP pub> }

/etc/init.d/xinetd restart
  • sshd: in file /etc/ssh/sshd_config:
ListenAddress <IP pub>

/etc/init.d/ssh restart
  • exim4: in file /etc/exim4/update-exim4.conf.conf:
dc_local_interfaces='<IP pub>'

/etc/init.d/exim4 restart

Better to do it through debconf to avoid surprises at update time: dpkg-reconfigure exim4-config

  • courier-imap: in file /etc/courier/imapd:
ADDRESS=<IP pub>

/etc/init.d/courier-imap restart
  • courier-imap-ssl: in file /etc/courier-ssl/imapd:
ADDRESS=<IP pub>

/etc/init.d/courier-imap-ssl restart
  • imapproxy: in file /etc/imapproxy.conf:
listen_address <IP pub>

Within a vserver, you'll probably hav to reduce the cache_size or give capability to the vserver to raise the setrlimit.

  • mysql: in file /etc/mysql/my.cnf:
bind-address = <IP pub>
  • vsFtpd: in file /etc/vsftpd.conf:
listen_address=<IP pub>
  • postgresql: in file /etc/postgresql/postgresql.conf:
virtual_host = '<IP pub>'
  • apache2: in file /etc/apache2/ports.conf:
Listen <IP pub>:80
  • zope2.9: in file /etc/zope2.9/<instance>/zope.conf:
ip-address <IP pub>
  • portmap: in file /etc/default/portmap:
OPTIONS="-i <IP pub/loopback>"
  • dnsmasq: in file /etc/dnsmasq.conf:
listen-address=<IP pub>
bind-interfaces
  • netstat -lp -> other greedy daemons?
  • Seems that this is possible via another method, here it will bind the daemon to the first IP of the interface:
    exec /usr/sbin/chbind --ip eth0 /path/to/daemon

Add an interface without rebooting the vserver

  • add the ip to the host (ip addr add ...)
  • add the ip to the guest's network context
# naddress --add --nid <nid> --ip <ip>/<mask>
  • enter the guest (best via ssh)
  • restart the services if required
    (most services will automatically start using the new addresses)
  • update the config to reflect the changes for the next guest restart (if desired)

Thanks Herbert!

Understanding vservers

Security contextes

  • Find security context of process N:
chcontext --ctx 1 cat /proc/N/status|grep s_context
  • Be in the same context:
chcontext --ctx X /bin/sh
  • Master context: 1, example to get all listening ports:
chcontext --ctx 1 netstat -lpn

See also Virtual private servers and security contexts

Ceiling capabilities

  • As non-root, check capBset:
cat /proc/self/status
  • Reduce ceiling caps:
reducecap --secure /bin/sh
  • Now capBset is reduced:
cat /proc/self/status
su

Security

Not necessarily related to vserver but always useful to consider :-)

  • ssh
    • Use the AllowUsers option to give ssh rights only to those who need it.
    • Brute-force protection: apt-get install denyhosts
      Edit /etc/denyhosts.conf to get email reports
      Un case someone forgot his pwd and got banned, to remove the ban directly: remove it from /var/lib/denyhosts files and /etc/hosts.deny of course
  • iptables (on the host)
    • cf --uid-owner and other --XXX-owner options
      on OUTPUT table to avoid download of malicious code
      on INPUT table to avoid bindshells
  • resource limits
    • cpu/mem

GrSec

apt-get install paxctl gradm2

Iptables Proxy

Other tricks

TODO

Retrieved from "https://wiki.yobi.be/index.php?title=Vserver_administration&oldid=2138"