Difference between revisions of "Vserver administration"

From YobiWiki
Jump to navigation Jump to search
m
 
m (Reverted edits by Etegohy (Talk) to last revision by PhilippeTeuwen)
 
(25 intermediate revisions by 3 users not shown)
Line 1: Line 1:
−
==Errors==
 +
==Introduction==
  +
Official homepage: [http://linux-vserver.org/ Linux VServer Project]
−
In some situations my CPU was burning like hell for days.
  
−
<br>This was the vcontext/login process when using the command "vserver XX enter".
  
−
<br>If it happens that the process is disconnected from its terminal it eats all the CPU :-(
  
−
<br>I could reproduce the problem as follow:
  
−
* ssh to the server as normal user
  
−
* sudo su -
  
−
* vserver XX enter
  
−
* from another term, kill -9 one of the 2 su processes ("su -" or "-su")
  
−
* quit the current terminal
  
−
Now from another terminal you can see that
  
−
vps ax|grep " login"
  
−
shows a "login" process not attached to any tty (cf the "?") and using top you see it's running 100% cpu.
  
   
  +
Good introduction:
−
Actually nature of this process is a bit strange because the real process is called vcontext as seen by top, pgrep and as you can see in /proc/XXX/status but its command line was overwritten with "login\0\0\0\0\0\0\0\0\0\0\0..." and commands like vps or pgrep -f show you this name "login" as you can see in /proc/XXX/cmdline.
  
  +
* [http://linux-vserver.org/index.php?page=Linux-VServer-Paper Linux-VServer Technology]
  +
* [http://linux-vserver.org/index.php?page=Linux-VServer-Paper-French La Technologie Linux-VServer]
   
  +
Debian support:
−
So I wrote a script and installed it in a cron to kill any such dandling process:
  
  +
apt-cache search vserver
  +
kernel-patch-vserver - context switching virtual private servers - kernel patch
  +
[http://www.nongnu.org/util-vserver/ util-vserver] - tools for Virtual private servers and context switching
  +
vserver-debiantools - Tools to manage debian virtual servers
  +
  +
Misc:
  +
* [http://www.lri.fr/~fragile/IMG/pdf/Quetier.pdf Benchmark Comparisons between UML, VMWare, vserver and Xen (pdf)]
  +
  +
==Kernel compilation==
  +
===The Debian way===
  +
I followed instructions given in
  +
* /usr/share/doc/kernel-patch-vserver/README.Debian
  +
* [http://linux-vserver.org/Step-by-Step+Guide+2.6 Step-by-step 2.6]
  +
* [http://deb.riseup.net/vserver/preparing/ Debian vservers]
  +
* [http://arnofear.free.fr/linux/vserver-1.php Debian and vserver, french howto]
  +
* [http://lena.franken.de/linux/debian_and_vserver/ Debian and vserver]
 
<pre>
  
<pre>
  +
apt-get install kernel-patch-vserver linux-source-2.6.16 kernel-package fakeroot
−
pids=$(/usr/sbin/vps axc|\
  
  +
cd /usr/src
−
egrep ' ? +[RS] +[1-9][0-9]*:[0-9]+ +vcontext'|\
  
  +
tar xjf linux-source-2.6.16.tar.bz2
−
sed 's/^ *\([0-9]\+\) .*$/\1/')
  
  +
cd /usr/src/linux-source-2.6.16
−
[ "$pids" != "" ] && \
  
  +
cp config-2.6.16-1-amd64-k8 .config
−
echo "Killing dandling login" && \
  
  +
export PATCH_THE_KERNEL=YES
−
kill $pids
  
  +
make-kpkg --rootcmd fakeroot \
  +
--revision custom01 \
  +
--added-patches vserver \
  +
--append-to-version +vserver \
  +
--initrd \
  +
binary-arch
  +
"Virtual root device support" -> **y**
  +
"Legacy kernel API" -> y
  +
"Show a Legacy Version ID" -> n
  +
"Disable Legacy Networking Kernel API" -> n
  +
"Enable Proc Security" -> y
  +
"Enable Hard CPU Limits" -> y
  +
"Limit the IDLE task" -> n
  +
"Persistent Inode Context Tagging" -> UID24/GID24 (32/32 probably not yet supported on Reiserfs)
  +
"Tag NFSD User Auth and Files" -> n
  +
"VServer Debugging Code" -> n
 
</pre>
  
</pre>
  +
Install kernel and reboot
−
And /etc/cron.d/watch-vcontext:
  
  +
===Vanilla with GrSec, still the Debian way===
  +
I used linux-2.6.17.14.tar.bz2 + patch-2.6.17.14-vs2.0.2.1-grsec2.1.9.diff
  +
<br>and the config of the Debian kernel config-2.6.17-2-vserver-amd64
  +
make oldconfig
  +
I activated HARDCPU limits and misc PAX & GRSEC stuff ([http://people.linux-vserver.org/~harry/_README_ this page] can help):
 
<pre>
  
<pre>
  +
CONFIG_VSERVER_HARDCPU=y
−
MAILTO=root
  
  +
CONFIG_VSERVER_HARDCPU_IDLE=y
−
*/5 * * * * root /home/vservers/tools/kill-vcontext
  
  +
CONFIG_PAX=y
  +
CONFIG_PAX_SOFTMODE=y
  +
CONFIG_PAX_PT_PAX_FLAGS=y
  +
CONFIG_PAX_HAVE_ACL_FLAGS=y
  +
CONFIG_PAX_NOEXEC=y
  +
CONFIG_PAX_PAGEEXEC=y
  +
CONFIG_PAX_MPROTECT=y
  +
CONFIG_PAX_ASLR=y
  +
CONFIG_PAX_RANDUSTACK=y
  +
CONFIG_PAX_RANDMMAP=y
  +
CONFIG_PAX_MEMORY_SANITIZE=y
  +
CONFIG_GRKERNSEC=y
  +
CONFIG_GRKERNSEC_CUSTOM=y
  +
CONFIG_GRKERNSEC_KMEM=y
  +
CONFIG_GRKERNSEC_IO=y
  +
CONFIG_GRKERNSEC_PROC_MEMMAP=y
  +
CONFIG_GRKERNSEC_BRUTE=y
  +
CONFIG_GRKERNSEC_MODSTOP=y
  +
CONFIG_GRKERNSEC_ACL_HIDEKERN=y
  +
CONFIG_GRKERNSEC_ACL_MAXTRIES=3
  +
CONFIG_GRKERNSEC_ACL_TIMEOUT=30
  +
CONFIG_GRKERNSEC_PROC=y
  +
CONFIG_GRKERNSEC_PROC_USER=y
  +
CONFIG_GRKERNSEC_PROC_ADD=y
  +
CONFIG_GRKERNSEC_LINK=y
  +
CONFIG_GRKERNSEC_FIFO=y
  +
CONFIG_GRKERNSEC_CHROOT=y
  +
CONFIG_GRKERNSEC_CHROOT_PIVOT=y
  +
CONFIG_GRKERNSEC_CHROOT_CHDIR=y
  +
CONFIG_GRKERNSEC_CHROOT_FCHDIR=y
  +
CONFIG_GRKERNSEC_CHROOT_MKNOD=y
  +
CONFIG_GRKERNSEC_CHROOT_SHMAT=y
  +
CONFIG_GRKERNSEC_CHROOT_UNIX=y
  +
CONFIG_GRKERNSEC_CHROOT_NICE=y
  +
CONFIG_GRKERNSEC_CHROOT_SYSCTL=y
  +
CONFIG_GRKERNSEC_RESLOG=y
  +
CONFIG_GRKERNSEC_SIGNAL=y
  +
CONFIG_GRKERNSEC_FORKFAIL=y
  +
CONFIG_GRKERNSEC_PROC_IPADDR=y
  +
CONFIG_GRKERNSEC_EXECVE=y
  +
CONFIG_GRKERNSEC_SHM=y
  +
CONFIG_GRKERNSEC_DMESG=y
  +
CONFIG_GRKERNSEC_RANDPID=y
  +
CONFIG_GRKERNSEC_RANDNET=y
  +
CONFIG_GRKERNSEC_SYSCTL=y
  +
CONFIG_GRKERNSEC_FLOODTIME=10
  +
CONFIG_GRKERNSEC_FLOODBURST=4
 
</pre>
  
</pre>
  +
make-kpkg --rootcmd fakeroot --us --uc --initrd kernel-image
−
This also shows something weird: processes in vservers don't appear with ps ax, ok, they appear with vps ax.
  
  +
And I got a linux-image-2.6.17.14-grsec2.1.9-vs2.0.2.1_2.6.17.14-grsec2.1.9-vs2.0.2.1-10.00.Custom_amd64.deb
−
<br>Now if I want to kill one of them from the host:
  
  +
==Host preparation==
−
vkill 29825
  
  +
<pre>
−
vkill: vc_ctx_kill(): No such process
  
  +
apt-get install util-vserver vserver-debiantools
−
chcontext --ctx 1 kill 29825
  
  +
wget http://vserver.13thfloor.at/Stuff/SCRIPT/testme.sh
−
<no error but nothing happen neither>
  
  +
chmod +x testme.sh
−
kill 29825
  
  +
./testme.sh
−
<works!!>
  
  +
dd bs=1024k count=1024 if=/dev/zero of=1gb.test
  +
modprobe loop
  +
losetup /dev/loop0 ./1gb.test
  +
./testfs.sh [ -F reiser ] -D /dev/loop0 -M /mnt
  +
losetup -d /dev/loop0
  +
modprobe -r loop
  +
</pre>
  +
There is no error at this point but as I'm using Reiserfs, I have to activate manually the extended attributes (for lsattr/chattr) by adding the following option to /etc/fstab lines: "attrs" (?? also option acl ??)
  +
<br>Test: lsattr <mount point of a Reiserfs>
  +
===Change the vserver base path===
  +
* /etc/vservers/.defaults/vdirbase -> /var/lib/vservers
  +
* I change it to /home/vservers, fix the above symlink
  +
* Re-create the "chroot barrier": <br>setattr --barrier /home/vservers<br>showattr /home -> B for vservers
  +
* Some tools could have /var/lib/vservers hardcoded, for safety I create a symlink /var/lib/vservers pointing to /home/vservers
  +
  +
==Manipulating vservers==
  +
===Create a vserver===
  +
Edit /etc/vservers/newvserver-vars:
  +
<pre>
  +
# cf http://amd64.debian.net/README.mirrors.html
  +
MIRROR="http://ftp.belnet.be/debian-amd64/debian"
  +
INTERFACE="<my_if>"
  +
ARCH="amd64"
  +
</pre>
  +
Create a vserver with 64bits:
  +
LANG=C newvserver --hostname template64 --domain teuwen.org --ip <new_ip>/24 --dist etch
  +
Create a vserver with 32bits emulation:
  +
LANG=C newvserver --hostname template32 --domain teuwen.org --ip <new_ip>/24 --dist etch --arch i386 --mirror "http://<i386_debian_mirror>"
  +
Tuning:
  +
* take care of the config duplication!
  +
* enter the vserver and run tzconfig to choose the proper timezone
  +
* fix /etc/apt/sources.list
  +
* delete rcX.d links to umountroot
  +
* Warning! If you use newvserver as such, it will overwrite the host /etc/motd due to a symlink
  +
* See [Vserver tools] for a patch for newvserver
  +
Removing unnecessary progs (check if you really don't need them!!):
  +
* aptitude apt-utils base-config cpio dselect tasksel libncursesw5 libsigc++-1.2-5c2 libsigc++-2.0-0c2a
  +
* dmidecode laptop-detect module-init-tools
  +
* bsdmainutils ed nano nvi
  +
* groff-base man-db manpages info libgdbm3
  +
* netcat traceroute wget libssl0.9.8
  +
* gettext-base libconsole libgnutls11 liblzo2-2 libtasn1-2-bin
  +
  +
===Automatic start at bootup===
  +
echo default > /etc/vservers/<my_vserver>/apps/init/mark
  +
Note that at shotdown all vservers will be stopped
  +
===Delete a vserver===
  +
Remove dirs /home/vservers/<my_vserver> (depends on the setting of vdirbase, cf. above), /etc/vservers/<my_vserver> and /var/run/vservers/<my_vserver> and the corresponding symlink in /var/run/vservers.rev
  +
===Config of a vserver===
  +
''TODO''
  +
?? /etc/vservers/<my_vserver>.conf
  +
?? S_CAPS
  +
see [http://www.nongnu.org/util-vserver/doc/conf/configuration.html Detailed config page (better choosing boring CSS...)]
  +
  +
If you don't assign unique IPs to the vservers but reuse the one of the host:
  +
touch /etc/vservers/<vserver>/interfaces/<N>/nodev
  +
''When this file exists, the interface will be assumed to exist already. This can be used to assign primary interfaces which are created by the host or another vserver.''
  +
  +
===Run a vserver===
  +
vserver <my_vserver> start
  +
vserver <my_vserver> enter
  +
If you get "mesg: /dev/pts/1: Operation not permitted", be root on the host with "su -"
  +
vserver <my_vserver> stop
  +
===Other tools===
  +
vserver <my_vserver> status
  +
vserver-stat
  +
vtop, vps, vpstree, vkill
  +
/etc/rc.d/init.d/rebootmgr is a daemon which can be called from vservers via vreboot and vhalt to stop/restart the vserver from inside
  +
  +
See also [http://www.nongnu.org/util-vserver/doc/conf/compatibility.html compatibility of util-vserver alpha branch]
  +
  +
See [[Vserver tools]] for my own/modified scripts
  +
  +
===Duplicate a vserver===
  +
vserver <my_vserver1> stop
  +
dupvserver --from <my_vserver1> --to <my_vserver2> --ip <new_ip>
  +
dupvserver is broken with the new configuration structure /etc/vservers/<my_vserver>/
  +
<br>See [[Vserver tools]] for a patch for dupvserver
  +
===Move/copy a vserver===
  +
Basically stop the vserver and copy /etc/vservers/<my_vserver> and /home/vservers/<my_vserver>
  +
<br>E.g. rsync -e ssh -avHl /vservers/XX new-server:/vserver/XX
  +
==Share directories==
  +
To mount a directory from one vserver into another from the host:
  +
vnamespace -e <vserver> mount --rbind /directory/to/mount/somewhere /where/to/mount/it
  +
vnamespace -e <vserver> umount /where/it/was/mounted
  +
  +
or
  +
mount --bind /home /var/lib/vservers/vserver1/home
  +
mount --bind /home /var/lib/vservers/vserver2/home
  +
The second method had the disavantage to require a reboot of the vserver
  +
  +
To mount an NFS share in a vserver:
  +
<br>Add the nfs share to /etc/vservers/<vserver>/fstab
  +
<br>If you want the user to be able to do it from the vserver itself, you've to add some capabilities, apparently sth like SECURE_MOUNT, SECURE_REMOUNT and/or BINARY_MOUNT to /etc/vservers/<vserver>/ccapabilities (didn't try)
  +
  +
==Apt-get==
  +
LANG=C vapt-get <my_vserver1> <my_vserver2> <...> -- install <pkg1> <pkg2>
  +
==Unify==
  +
cf immutable-linkage-invert flag
  +
  +
Preparation:
  +
mkdir /etc/vservers/template64/apps/vunify
  +
mkdir /etc/vservers/<my_vserver>/apps/vunify
  +
ln -s /etc/vservers/template64 /etc/vservers/<my_vserver>/apps/vunify/refserver.template64
  +
Unification:
  +
<br>Be sure both vservers are running
  +
vserver <my_vserver> unify [-n] [-R]
  +
-n for dry run, no change
  +
<br>-R for de-unifying
  +
  +
When using tar, add option -U to unlink & recreate files instead of overwriting.
  +
<br>Manual set/unset of the immutable-linkage-invert flag:
  +
setattr --iunlink /my/file
  +
setattr --~iunlink /my/file
  +
==Disk limits==
  +
cf http://linux-vserver.org/Disk+Limits
  +
  +
* Assign static contexts for the vservers (i.e. have a value between 2 and 49151 in /etc/vservers/<name>/context)
  +
* Mount the filesystem holding the vserver(s) with the tagxid option
  +
** Check if this is mounted properly: use cat /proc/mounts<br>Ex.: /dev/mapper/Zeus-home /home reiserfs rw,tagxid 0 0
  +
** WARNING: if the filesystem is already in use with vservers, nothing prevent you to umount the filesystem while the vservers are still running, which is VERY BAD! Be careful.
  +
** I could only get the tagxid taken properly into account after a reboot
  +
** To set tagxid on / you need to do it from initrd as tagxid cannot be set at remount: add to the kernel params the option "rootflags=tagxid", e.g. via /boot/grub/menu.lst #kopts=...
  +
* Change the xid of already existing files:
  +
chxid -c <my_vserver> -R /home/vservers/<my_vserver>
  +
* Set limits, first method: here limit to 5Gb, 100000 inodes and 5% for the root user<br>For info as I could not get it working properly yet
  +
mkdir /var/cache/vservers
  +
ln -s /var/cache/vservers /etc/vservers/.defaults/cachebase
  +
mkdir /etc/vservers/.defaults/cachebase/<my_server>
  +
ln -s /etc/vservers/.defaults/cachebase/<my_server> /etc/vservers/<my_server>/cache
  +
mkdir -p /etc/vservers/<my_vserver>/dlimits/0
  +
echo /home/vservers/<my_vserver> > /etc/vservers/<my_vserver>/dlimits/0/directory
  +
echo $(( 5 * 1024 * 1024 )) > /etc/vservers/<my_vserver>/dlimits/0/space_total
  +
echo 100000 > /etc/vservers/<my_vserver>/dlimits/0/inodes_total
  +
echo 5 > /etc/vservers/<my_vserver>/dlimits/0/reserved
  +
* Set limits, second method:
  +
** Install my vdlimit_ script in /usr/local/sbin: [[Vserver tools]]
  +
ln -s /usr/local/sbin/vdlimit_ /etc/vservers/<my_vserver>/scripts/post-start.d/vdlimit_$((5*1024))
  +
** To change the limit on-the-fly simply rename the link and execute
  +
./vdlimit_<new_size> pre-stop <my_vserver>;./vdlimit_<new_size> post-start <my_vserver>;
  +
  +
==Network==
  +
===Intern network===
  +
For pure loopback, use dummy interface, cf http://mirabellug.org/wikini/wakka.php?wiki=VServers
  +
  +
For usable dummy interface, us permanent taps as the uml tools allow:
  +
apt-get install uml-utilities
  +
* Create a pseudo-interface:
  +
<pre>
  +
auto tap0
  +
iface tap0 inet static
  +
address 192.168.2.1
  +
netmask 255.255.255.0
  +
tunctl_user uml-net
  +
</pre>
  +
And configure vservers with the same dev=tap0
  +
  +
Update: to check but actually all traffic with private or public IP will anyway be done through lo so this is probably not required
  +
  +
Note that if you use openvpn, you can create tun/tap with
  +
openvpn --mktun --dev tap0
  +
  +
===Configure daemons to listen only to the IP-address of the mothersystem===
  +
* ''openbsd-inetd:'' (not netkit-inetd) in file /etc/inetd.conf:<br>Prepend the service with <IP pub>:<br>Example
  +
<IP pub>:cvspserver stream tcp nowait root /usr/sbin/tcpd /usr/sbin/cvs-pserver
  +
* ''xinetd:'' (not inetd) in file /etc/xinetd.conf:
  +
defaults
  +
{ bind = <IP pub> }
  +
  +
/etc/init.d/xinetd restart
  +
* ''sshd:'' in file /etc/ssh/sshd_config:
  +
ListenAddress <IP pub>
  +
  +
/etc/init.d/ssh restart
  +
* ''exim4:'' in file /etc/exim4/update-exim4.conf.conf:
  +
dc_local_interfaces='<IP pub>'
  +
  +
/etc/init.d/exim4 restart
  +
Better to do it through debconf to avoid surprises at update time: dpkg-reconfigure exim4-config
  +
* ''courier-imap:'' in file /etc/courier/imapd:
  +
ADDRESS=<IP pub>
  +
  +
/etc/init.d/courier-imap restart
  +
* ''courier-imap-ssl:'' in file /etc/courier-ssl/imapd:
  +
ADDRESS=<IP pub>
  +
  +
/etc/init.d/courier-imap-ssl restart
  +
* ''imapproxy:'' in file /etc/imapproxy.conf:
  +
listen_address <IP pub>
  +
Within a vserver, you'll probably hav to reduce the cache_size or give capability to the vserver to raise the setrlimit.
  +
* ''mysql:'' in file /etc/mysql/my.cnf:
  +
bind-address = <IP pub>
  +
* ''vsFtpd:'' in file /etc/vsftpd.conf:
  +
listen_address=<IP pub>
  +
* ''postgresql:'' in file /etc/postgresql/postgresql.conf:
  +
virtual_host = '<IP pub>'
  +
* ''apache2:'' in file /etc/apache2/ports.conf:
  +
Listen <IP pub>:80
  +
* ''zope2.9:'' in file /etc/zope2.9/<instance>/zope.conf:
  +
ip-address <IP pub>
  +
* ''portmap:'' in file /etc/default/portmap:
  +
OPTIONS="-i <IP pub/loopback>"
  +
* ''dnsmasq:'' in file /etc/dnsmasq.conf:
  +
listen-address=<IP pub>
  +
bind-interfaces
  +
* ''[[Virtual_Private_Networks|openvpn]]'' in file /etc/openvpn/server.conf:
  +
local <IP pub>
  +
* ''slapd:'' in file /etc/default/slapd
  +
SLAPD_SERVICES="ldap://<IP priv>/ ldaps://<IP pub>/"
  +
* ''oidentd'' in file /etc/default/oidentd
  +
Add to OIDENT_OPTIONS list: "-a <IP pub>"
  +
* netstat -lp -> other greedy daemons?
  +
* Seems that this is possible via another method, here it will bind the daemon to the first IP of the interface:<br>exec /usr/sbin/chbind --ip eth0 /path/to/daemon
  +
  +
===Add an interface without rebooting the vserver===
  +
* add the ip to the host (ip addr add ...)
  +
* add the ip to the guest's network context
  +
# naddress --add --nid <nid> --ip <ip>/<mask>
  +
* enter the guest (best via ssh)
  +
* restart the services if required<br>(most services will automatically start using the new addresses)
  +
* update the config to reflect the changes for the next guest restart (if desired)
  +
Thanks Herbert!
  +
==Understanding vservers==
  +
===Security contextes===
  +
* Find security context of process N:
  +
chcontext --ctx 1 cat /proc/N/status|grep s_context
  +
* Be in the same context:
  +
chcontext --ctx X /bin/sh
  +
* Master context: 1, example to get all listening ports:
  +
chcontext --ctx 1 netstat -lpn
  +
See also [http://www.solucorp.qc.ca/miscprj/s_context.hc Virtual private servers and security contexts]
  +
===Ceiling capabilities===
  +
* As non-root, check capBset:
  +
cat /proc/self/status
  +
* Reduce ceiling caps:
  +
reducecap --secure /bin/sh
  +
* Now capBset is reduced:
  +
cat /proc/self/status
  +
su
  +
* capEff raised a bit but not enough to do for example /sbin/ifconfig eth0 down
  +
* See also [Capabilities in Linux|http://www.lids.org/lids-howto/node34.html]
  +
  +
==Security==
  +
Not necessarily related to vserver but always useful to consider :-)
  +
*ssh
  +
**Use the AllowUsers option to give ssh rights only to those who need it.
  +
**Brute-force protection: apt-get install denyhosts<br>Edit /etc/denyhosts.conf to get email reports<br>Un case someone forgot his pwd and got banned, to remove the ban directly: remove it from /var/lib/denyhosts files and /etc/hosts.deny of course
  +
*iptables (on the host)
  +
**cf --uid-owner and other --XXX-owner options<br>on OUTPUT table to avoid download of malicious code<br>on INPUT table to avoid bindshells
  +
*resource limits
  +
** cpu/mem
  +
  +
===GrSec===
  +
* http://pax.grsecurity.net/
  +
* http://people.linux-vserver.org/~harry/_README_
  +
* http://www.zataz.net/docs/8024/introduction-grsecurity.html
  +
* http://linux-vserver.org/grsecurityHowto
  +
* http://ludit.kuleuven.be/software/vserver/_README_
  +
apt-get install paxctl gradm2
  +
  +
==Iptables Proxy==
  +
* http://www.virtuaserver.com.br/forum/viewtopic.php?t=130
  +
  +
==Other tricks==
  +
* For other tweaks, see http://deb.riseup.net/vserver/usage/ :
  +
** What if I accidentally removed a vserver while it was running?
  +
** Howto convert legacy vservers to the new format
  +
** Howto add an IP to a running vserver, without restarting it?
  +
** Howto make the host interface and IP available in a vserver
  +
** Howto impose disk limits in each vserver
  +
* http://www.paul.sladen.org/vserver/faq
  +
* [http://linux-vserver.org/ProblematicPrograms Problematic programs]
  +
* If you drop files from "outside of the vserver context" (from the host e.g.) you've to reassign the correct xid to the files:
  +
chxid -c <vserver> -R /home/vservers/<vserver>
  +
# all at once:
  +
for i in $(ls /etc/vservers/); do echo $i; chxid -c $i -R /home/vservers/$i;done
  +
* If you drop files from "outside of the vserver context" (from the host e.g.) you've to regenerate the disk usage and limit of the vserver if you use my vdlimit_ script:
  +
vserver <vserver> stop
  +
rm /var/cache/vservers/<vserver>_vdlimit_
  +
vserver <vserver> start
  +
* To run a script (e.g. an /etc/init.d/start_my_daemon) in ctx 1, e.g. to start ntop and be sure it can see all the traffic, simply add at the begin of the script:
  +
if cat /proc/self/vinfo|grep -q -v ":[^0-9]1$"; then
  +
/usr/sbin/chcontext --ctx 1 $0 $*
  +
exit
  +
fi
  +
* To "mount" a samba shared drive from a vserver is not possible or at least when running grsec but you can still use the good old ftp-styled smbclient
  +
smbclient //machine/share -U domain/user
  +
  +
==TODO==
  +
* http://www.nongnu.org/util-vserver/doc/conf/compatibility.html
  +
* http://linux-vserver.derjohn.de/
  +
* [VServer wiki|http://vserver.strahlungsfrei.de/tiki-index.php]
  +
* [Administrator Guide|http://linux-vserver.org/linux-vserver_administrators_gide]
  +
* [Debian newvserver|http://www.paul.sladen.org/vserver/debian/]
  +
* [Howto Debian vserver|http://www.howtoforge.com/linux_vserver_debian]
  +
* ?? apt-get install vlan
  +
* ?? ipac-ng
  +
* CPU limit
  +
** http://linux-vserver.org/Linux-VServer-Paper-06
  +
** http://list.linux-vserver.org/archive/vserver/msg08134.html
  +
* BW limit
  +
** http://lartc.org/howto/
  +
* http://linux-vserver.org/HowTo+Read+ProcFS
  +
* http://linux-vserver.org/HistoryList?full=1
  +
* Publish Munin scripts
  +
* http://linux-vserver.org/VServer+installation+Fedora+Core+5
  +
* http://vserver.13thfloor.at/Experimental/
  +
* http://www.archivesat.com/Linux-VServer/
  +
* http://www.solucorp.qc.ca/miscprj/s_context.hc?s1=1&s2=0&s3=0&s4=0&full=0&prjstate=1&nodoc=0
  +
* (fr) http://fr.wikibooks.org/wiki/Vserver

Latest revision as of 21:35, 24 November 2010

Introduction

Official homepage: Linux VServer Project

Good introduction:

Debian support: 

apt-cache search vserver
  kernel-patch-vserver - context switching virtual private servers - kernel patch
  util-vserver - tools for Virtual private servers and context switching
  vserver-debiantools - Tools to manage debian virtual servers

Misc:

Kernel compilation

The Debian way

I followed instructions given in 

apt-get install kernel-patch-vserver linux-source-2.6.16 kernel-package fakeroot
cd /usr/src
tar xjf linux-source-2.6.16.tar.bz2
cd /usr/src/linux-source-2.6.16
cp config-2.6.16-1-amd64-k8 .config
export PATCH_THE_KERNEL=YES
make-kpkg --rootcmd fakeroot \
        --revision custom01 \
        --added-patches vserver \
        --append-to-version +vserver \
        --initrd \
        binary-arch
"Virtual root device support" -> **y**
"Legacy kernel API" -> y
"Show a Legacy Version ID" -> n
"Disable Legacy Networking Kernel API" -> n
"Enable Proc Security" -> y
"Enable Hard CPU Limits" -> y
"Limit the IDLE task" -> n
"Persistent Inode Context Tagging" -> UID24/GID24 (32/32 probably not yet supported on Reiserfs)
"Tag NFSD User Auth and Files" -> n
"VServer Debugging Code" -> n

Install kernel and reboot

Vanilla with GrSec, still the Debian way

I used linux-2.6.17.14.tar.bz2 + patch-2.6.17.14-vs2.0.2.1-grsec2.1.9.diff
and the config of the Debian kernel config-2.6.17-2-vserver-amd64 

make oldconfig

I activated HARDCPU limits and misc PAX & GRSEC stuff (this page can help): 

CONFIG_VSERVER_HARDCPU=y
CONFIG_VSERVER_HARDCPU_IDLE=y
CONFIG_PAX=y
CONFIG_PAX_SOFTMODE=y
CONFIG_PAX_PT_PAX_FLAGS=y
CONFIG_PAX_HAVE_ACL_FLAGS=y
CONFIG_PAX_NOEXEC=y
CONFIG_PAX_PAGEEXEC=y
CONFIG_PAX_MPROTECT=y
CONFIG_PAX_ASLR=y
CONFIG_PAX_RANDUSTACK=y
CONFIG_PAX_RANDMMAP=y
CONFIG_PAX_MEMORY_SANITIZE=y
CONFIG_GRKERNSEC=y
CONFIG_GRKERNSEC_CUSTOM=y
CONFIG_GRKERNSEC_KMEM=y
CONFIG_GRKERNSEC_IO=y
CONFIG_GRKERNSEC_PROC_MEMMAP=y
CONFIG_GRKERNSEC_BRUTE=y
CONFIG_GRKERNSEC_MODSTOP=y
CONFIG_GRKERNSEC_ACL_HIDEKERN=y
CONFIG_GRKERNSEC_ACL_MAXTRIES=3
CONFIG_GRKERNSEC_ACL_TIMEOUT=30
CONFIG_GRKERNSEC_PROC=y
CONFIG_GRKERNSEC_PROC_USER=y
CONFIG_GRKERNSEC_PROC_ADD=y
CONFIG_GRKERNSEC_LINK=y
CONFIG_GRKERNSEC_FIFO=y
CONFIG_GRKERNSEC_CHROOT=y
CONFIG_GRKERNSEC_CHROOT_PIVOT=y
CONFIG_GRKERNSEC_CHROOT_CHDIR=y
CONFIG_GRKERNSEC_CHROOT_FCHDIR=y
CONFIG_GRKERNSEC_CHROOT_MKNOD=y
CONFIG_GRKERNSEC_CHROOT_SHMAT=y
CONFIG_GRKERNSEC_CHROOT_UNIX=y
CONFIG_GRKERNSEC_CHROOT_NICE=y
CONFIG_GRKERNSEC_CHROOT_SYSCTL=y
CONFIG_GRKERNSEC_RESLOG=y
CONFIG_GRKERNSEC_SIGNAL=y
CONFIG_GRKERNSEC_FORKFAIL=y
CONFIG_GRKERNSEC_PROC_IPADDR=y
CONFIG_GRKERNSEC_EXECVE=y
CONFIG_GRKERNSEC_SHM=y
CONFIG_GRKERNSEC_DMESG=y
CONFIG_GRKERNSEC_RANDPID=y
CONFIG_GRKERNSEC_RANDNET=y
CONFIG_GRKERNSEC_SYSCTL=y
CONFIG_GRKERNSEC_FLOODTIME=10
CONFIG_GRKERNSEC_FLOODBURST=4

make-kpkg --rootcmd fakeroot --us --uc --initrd kernel-image

And I got a linux-image-2.6.17.14-grsec2.1.9-vs2.0.2.1_2.6.17.14-grsec2.1.9-vs2.0.2.1-10.00.Custom_amd64.deb

Host preparation

apt-get install util-vserver vserver-debiantools
wget http://vserver.13thfloor.at/Stuff/SCRIPT/testme.sh
chmod +x testme.sh
./testme.sh
dd bs=1024k count=1024 if=/dev/zero of=1gb.test
modprobe loop
losetup /dev/loop0 ./1gb.test
./testfs.sh [ -F reiser ] -D /dev/loop0 -M /mnt
losetup -d /dev/loop0
modprobe -r loop

There is no error at this point but as I'm using Reiserfs, I have to activate manually the extended attributes (for lsattr/chattr) by adding the following option to /etc/fstab lines: "attrs" (?? also option acl ??)
Test: lsattr <mount point of a Reiserfs>

Change the vserver base path

  • /etc/vservers/.defaults/vdirbase -> /var/lib/vservers
  • I change it to /home/vservers, fix the above symlink
  • Re-create the "chroot barrier":
    setattr --barrier /home/vservers
    showattr /home -> B for vservers
  • Some tools could have /var/lib/vservers hardcoded, for safety I create a symlink /var/lib/vservers pointing to /home/vservers

Manipulating vservers

Create a vserver

Edit /etc/vservers/newvserver-vars: 

# cf http://amd64.debian.net/README.mirrors.html
MIRROR="http://ftp.belnet.be/debian-amd64/debian"
INTERFACE="<my_if>"
ARCH="amd64"

Create a vserver with 64bits: 

LANG=C newvserver --hostname template64 --domain teuwen.org --ip <new_ip>/24 --dist etch

Create a vserver with 32bits emulation: 

LANG=C newvserver --hostname template32 --domain teuwen.org --ip <new_ip>/24 --dist etch --arch i386 --mirror "http://<i386_debian_mirror>"

Tuning:

  • take care of the config duplication!
  • enter the vserver and run tzconfig to choose the proper timezone
  • fix /etc/apt/sources.list
  • delete rcX.d links to umountroot
  • Warning! If you use newvserver as such, it will overwrite the host /etc/motd due to a symlink
  • See [Vserver tools] for a patch for newvserver

Removing unnecessary progs (check if you really don't need them!!):

  • aptitude apt-utils base-config cpio dselect tasksel libncursesw5 libsigc++-1.2-5c2 libsigc++-2.0-0c2a
  • dmidecode laptop-detect module-init-tools
  • bsdmainutils ed nano nvi
  • groff-base man-db manpages info libgdbm3
  • netcat traceroute wget libssl0.9.8
  • gettext-base libconsole libgnutls11 liblzo2-2 libtasn1-2-bin

Automatic start at bootup

echo default > /etc/vservers/<my_vserver>/apps/init/mark

Note that at shotdown all vservers will be stopped

Delete a vserver

Remove dirs /home/vservers/<my_vserver> (depends on the setting of vdirbase, cf. above), /etc/vservers/<my_vserver> and /var/run/vservers/<my_vserver> and the corresponding symlink in /var/run/vservers.rev

Config of a vserver

TODO 

?? /etc/vservers/<my_vserver>.conf
?? S_CAPS

see Detailed config page (better choosing boring CSS...)

If you don't assign unique IPs to the vservers but reuse the one of the host: 

touch /etc/vservers/<vserver>/interfaces/<N>/nodev

When this file exists, the interface will be assumed to exist already. This can be used to assign primary interfaces which are created by the host or another vserver.

Run a vserver

vserver <my_vserver> start
vserver <my_vserver> enter

If you get "mesg: /dev/pts/1: Operation not permitted", be root on the host with "su -" 

vserver <my_vserver> stop

Other tools

vserver <my_vserver> status
vserver-stat
vtop, vps, vpstree, vkill

/etc/rc.d/init.d/rebootmgr is a daemon which can be called from vservers via vreboot and vhalt to stop/restart the vserver from inside

See also compatibility of util-vserver alpha branch

See Vserver tools for my own/modified scripts

Duplicate a vserver

vserver <my_vserver1> stop
dupvserver --from <my_vserver1> --to <my_vserver2> --ip <new_ip>

dupvserver is broken with the new configuration structure /etc/vservers/<my_vserver>/
See Vserver tools for a patch for dupvserver

Move/copy a vserver

Basically stop the vserver and copy /etc/vservers/<my_vserver> and /home/vservers/<my_vserver>
E.g. rsync -e ssh -avHl /vservers/XX new-server:/vserver/XX

Share directories

To mount a directory from one vserver into another from the host: 

vnamespace -e <vserver> mount --rbind /directory/to/mount/somewhere /where/to/mount/it
vnamespace -e <vserver> umount /where/it/was/mounted

or 

mount --bind /home /var/lib/vservers/vserver1/home
mount --bind /home /var/lib/vservers/vserver2/home

The second method had the disavantage to require a reboot of the vserver

To mount an NFS share in a vserver:
Add the nfs share to /etc/vservers/<vserver>/fstab
If you want the user to be able to do it from the vserver itself, you've to add some capabilities, apparently sth like SECURE_MOUNT, SECURE_REMOUNT and/or BINARY_MOUNT to /etc/vservers/<vserver>/ccapabilities (didn't try)

Apt-get

LANG=C vapt-get <my_vserver1> <my_vserver2> <...> -- install <pkg1> <pkg2>

Unify

cf immutable-linkage-invert flag

Preparation: 

mkdir /etc/vservers/template64/apps/vunify
mkdir /etc/vservers/<my_vserver>/apps/vunify
ln -s /etc/vservers/template64 /etc/vservers/<my_vserver>/apps/vunify/refserver.template64

Unification:
Be sure both vservers are running 

vserver <my_vserver> unify [-n] [-R]

-n for dry run, no change
-R for de-unifying

When using tar, add option -U to unlink & recreate files instead of overwriting.
Manual set/unset of the immutable-linkage-invert flag: 

setattr --iunlink /my/file
setattr --~iunlink /my/file

Disk limits

cf http://linux-vserver.org/Disk+Limits

  • Assign static contexts for the vservers (i.e. have a value between 2 and 49151 in /etc/vservers/<name>/context)
  • Mount the filesystem holding the vserver(s) with the tagxid option
    • Check if this is mounted properly: use cat /proc/mounts
      Ex.: /dev/mapper/Zeus-home /home reiserfs rw,tagxid 0 0
    • WARNING: if the filesystem is already in use with vservers, nothing prevent you to umount the filesystem while the vservers are still running, which is VERY BAD! Be careful.
    • I could only get the tagxid taken properly into account after a reboot
    • To set tagxid on / you need to do it from initrd as tagxid cannot be set at remount: add to the kernel params the option "rootflags=tagxid", e.g. via /boot/grub/menu.lst #kopts=...
  • Change the xid of already existing files:
chxid -c <my_vserver> -R /home/vservers/<my_vserver>
  • Set limits, first method: here limit to 5Gb, 100000 inodes and 5% for the root user
    For info as I could not get it working properly yet
mkdir /var/cache/vservers
ln -s /var/cache/vservers /etc/vservers/.defaults/cachebase
mkdir /etc/vservers/.defaults/cachebase/<my_server>
ln -s /etc/vservers/.defaults/cachebase/<my_server> /etc/vservers/<my_server>/cache
mkdir -p /etc/vservers/<my_vserver>/dlimits/0
echo /home/vservers/<my_vserver> > /etc/vservers/<my_vserver>/dlimits/0/directory
echo $(( 5 * 1024 * 1024 )) > /etc/vservers/<my_vserver>/dlimits/0/space_total
echo 100000 > /etc/vservers/<my_vserver>/dlimits/0/inodes_total
echo 5 > /etc/vservers/<my_vserver>/dlimits/0/reserved
  • Set limits, second method: 
ln -s /usr/local/sbin/vdlimit_ /etc/vservers/<my_vserver>/scripts/post-start.d/vdlimit_$((5*1024))
    • To change the limit on-the-fly simply rename the link and execute
./vdlimit_<new_size> pre-stop <my_vserver>;./vdlimit_<new_size> post-start <my_vserver>;

Network

Intern network

For pure loopback, use dummy interface, cf http://mirabellug.org/wikini/wakka.php?wiki=VServers

For usable dummy interface, us permanent taps as the uml tools allow: 

apt-get install uml-utilities
  • Create a pseudo-interface:
auto tap0
iface tap0 inet static
    address 192.168.2.1
    netmask 255.255.255.0
    tunctl_user uml-net

And configure vservers with the same dev=tap0

Update: to check but actually all traffic with private or public IP will anyway be done through lo so this is probably not required

Note that if you use openvpn, you can create tun/tap with 

openvpn --mktun --dev tap0

Configure daemons to listen only to the IP-address of the mothersystem

  • openbsd-inetd: (not netkit-inetd) in file /etc/inetd.conf:
    Prepend the service with <IP pub>:
    Example
<IP pub>:cvspserver       stream  tcp     nowait  root    /usr/sbin/tcpd  /usr/sbin/cvs-pserver
  • xinetd: (not inetd) in file /etc/xinetd.conf:
defaults
{ bind = <IP pub> }

/etc/init.d/xinetd restart
  • sshd: in file /etc/ssh/sshd_config:
ListenAddress <IP pub>

/etc/init.d/ssh restart
  • exim4: in file /etc/exim4/update-exim4.conf.conf:
dc_local_interfaces='<IP pub>'

/etc/init.d/exim4 restart

Better to do it through debconf to avoid surprises at update time: dpkg-reconfigure exim4-config

  • courier-imap: in file /etc/courier/imapd:
ADDRESS=<IP pub>

/etc/init.d/courier-imap restart
  • courier-imap-ssl: in file /etc/courier-ssl/imapd:
ADDRESS=<IP pub>

/etc/init.d/courier-imap-ssl restart
  • imapproxy: in file /etc/imapproxy.conf:
listen_address <IP pub>

Within a vserver, you'll probably hav to reduce the cache_size or give capability to the vserver to raise the setrlimit.

  • mysql: in file /etc/mysql/my.cnf:
bind-address = <IP pub>
  • vsFtpd: in file /etc/vsftpd.conf:
listen_address=<IP pub>
  • postgresql: in file /etc/postgresql/postgresql.conf:
virtual_host = '<IP pub>'
  • apache2: in file /etc/apache2/ports.conf:
Listen <IP pub>:80
  • zope2.9: in file /etc/zope2.9/<instance>/zope.conf:
ip-address <IP pub>
  • portmap: in file /etc/default/portmap:
OPTIONS="-i <IP pub/loopback>"
  • dnsmasq: in file /etc/dnsmasq.conf:
listen-address=<IP pub>
bind-interfaces
  • openvpn in file /etc/openvpn/server.conf:
local <IP pub>
  • slapd: in file /etc/default/slapd
SLAPD_SERVICES="ldap://<IP priv>/ ldaps://<IP pub>/"
  • oidentd in file /etc/default/oidentd
Add to OIDENT_OPTIONS list: "-a <IP pub>"
  • netstat -lp -> other greedy daemons?
  • Seems that this is possible via another method, here it will bind the daemon to the first IP of the interface:
    exec /usr/sbin/chbind --ip eth0 /path/to/daemon

Add an interface without rebooting the vserver

  • add the ip to the host (ip addr add ...)
  • add the ip to the guest's network context
# naddress --add --nid <nid> --ip <ip>/<mask>
  • enter the guest (best via ssh)
  • restart the services if required
    (most services will automatically start using the new addresses)
  • update the config to reflect the changes for the next guest restart (if desired)

Thanks Herbert!

Understanding vservers

Security contextes

  • Find security context of process N:
chcontext --ctx 1 cat /proc/N/status|grep s_context
  • Be in the same context:
chcontext --ctx X /bin/sh
  • Master context: 1, example to get all listening ports:
chcontext --ctx 1 netstat -lpn

See also Virtual private servers and security contexts

Ceiling capabilities

  • As non-root, check capBset:
cat /proc/self/status
  • Reduce ceiling caps:
reducecap --secure /bin/sh
  • Now capBset is reduced:
cat /proc/self/status
su

Security

Not necessarily related to vserver but always useful to consider :-)

  • ssh
    • Use the AllowUsers option to give ssh rights only to those who need it.
    • Brute-force protection: apt-get install denyhosts
      Edit /etc/denyhosts.conf to get email reports
      Un case someone forgot his pwd and got banned, to remove the ban directly: remove it from /var/lib/denyhosts files and /etc/hosts.deny of course
  • iptables (on the host)
    • cf --uid-owner and other --XXX-owner options
      on OUTPUT table to avoid download of malicious code
      on INPUT table to avoid bindshells
  • resource limits
    • cpu/mem

GrSec

apt-get install paxctl gradm2

Iptables Proxy

Other tricks

  • For other tweaks, see http://deb.riseup.net/vserver/usage/ :
    • What if I accidentally removed a vserver while it was running?
    • Howto convert legacy vservers to the new format
    • Howto add an IP to a running vserver, without restarting it?
    • Howto make the host interface and IP available in a vserver
    • Howto impose disk limits in each vserver
  • http://www.paul.sladen.org/vserver/faq
  • Problematic programs
  • If you drop files from "outside of the vserver context" (from the host e.g.) you've to reassign the correct xid to the files:
chxid -c <vserver> -R /home/vservers/<vserver>
# all at once:
for i in $(ls /etc/vservers/); do echo $i; chxid -c $i -R /home/vservers/$i;done
  • If you drop files from "outside of the vserver context" (from the host e.g.) you've to regenerate the disk usage and limit of the vserver if you use my vdlimit_ script:
vserver <vserver> stop
rm /var/cache/vservers/<vserver>_vdlimit_
vserver <vserver> start
  • To run a script (e.g. an /etc/init.d/start_my_daemon) in ctx 1, e.g. to start ntop and be sure it can see all the traffic, simply add at the begin of the script:
if cat /proc/self/vinfo|grep -q -v ":[^0-9]1$"; then                                                                                                                      
   /usr/sbin/chcontext --ctx 1 $0 $*                                                                                                                                     
   exit                                                                                                                                                                  
fi
  • To "mount" a samba shared drive from a vserver is not possible or at least when running grsec but you can still use the good old ftp-styled smbclient
smbclient //machine/share -U domain/user

TODO

Retrieved from "https://wiki.yobi.be/index.php?title=Vserver_administration&oldid=6849"