Mediawiki LocalServer

Mediawiki offline support thanks to Google Gears

This is an attempt to provide an offline version of the wiki using Google Gears LocalServer.
Inspired by this blog post

To be able to use Mediawiki offline, the Mediawiki website has to have this extension installed and the browser has to have the Google Gears extension.

The original blog post was quick and dirty (do it in less than one hour), but too dirty for me:

• There was always a complete replication of the full wiki as soon as some change occurred
• There was no way to know if we were viewing the offline or the online version of a page
• There was no way to temporarily disable the offline pages
• It implied hacking the Mediawiki code

So I rewrote it as a Mediawiki extension and tried to overcome the mentioned problems.
Note that it is done for the default style MonoBook.

Note that I don't advertise GGears for the users who don't have the extension or don't enable javascript:
By default the GGears toolbox is hidden, only the users having already GGears will see it.

Documentation

Bits I found useful for this task:

• the original idea
• http://code.google.com/apis/gears/api_localserver.html
• The Google Gears inspector:

svn checkout http://google-gears.googlecode.com/svn/trunk/gears/inspector google-gears-read-only

• http://www.mediawiki.org/wiki/Manual:MediaWiki_hooks

Google Gears browser extension

First you need to install the Google Gears extension
Note that it does a stupid browser detection and refuses my Iceweasel 2.0 so I had to cheat with my User Agent Switch extension and mimick Firefox 2.0 ;-) If you've some troubles with those prerequisites, like Iceweasel/Iceape not being recognized, you can also install it by clicking on this link

Google Gears LocalServer extension

/extensions/GGears/

• Create a folder GGears in your MediaWiki extensions directory
• As I'm using rewriting rules, I had to add to /etc/apache2/sites-available/wiki.yobi.be (for accessing the .js):

RewriteCond %{REQUEST_URI} !/GGears/

gears_init.js

• Download the gears_init.js from the Google Gears web site and place it in the newly created folder

localserver.js

• Create localserver.js:

// Change this to set the name of the managed resource store to create.
// You use the name with the createManagedStore, and removeManagedStore,
// and openManagedStore APIs. It isn't visible to the user.
var STORE_NAME = "yobiwiki";

var MANIFEST_FILENAME = "index.php?action=manifest";

var localServer;
var store;
var blink=false;

// Called onload to initialize local server and store variables
function initStore() {
  if (!window.google || !google.gears) {
//    textOut("Google Gears not found");
  return;
  }
  localServer = google.gears.factory.create("beta.localserver");
  store = localServer.openManagedStore(STORE_NAME);
  if (!store) {
    textOut("No store found");
  } else {
    // Here is the place to touch the manifestUrl, if required
    textOut("");
  }
  menuStore();
}

// Create (or open&update) the managed resource store
function createStore() {
  if (!window.google || !google.gears) {
    alert("You must install Google Gears first.");
    return;
  }
  store = localServer.createManagedStore(STORE_NAME);
  if (!store) {
    textOut("Error creating store");
  } else {
    store.manifestUrl = MANIFEST_FILENAME;
    textOut("Downloading...");
    hideItem("t-createStore");
    showItem("t-removeStore");
  }
  updateStore();
}

// Update the managed resource store
function updateStore() {
  if (!window.google || !google.gears) {
    alert("You must install Google Gears first.");
    return;
  }

  store.checkForUpdate();

  var timerId = window.setInterval(function() {
    // When the currentVersion property has a value, all of the resources
    // listed in the manifest file for that version are captured. There is
    // an open bug to surface this state change as an event.
    if (store.currentVersion) {
      window.clearInterval(timerId);
      textOut("Sync done!\n" +
              "Version: " +
              store.currentVersion + "\n" +
              "Refreshing...");
      menuStore();
      setTimeout("location.reload(true)", 2000);
    } else if (store.updateStatus == 3) {
      window.clearInterval(timerId);
      textOut("Error: " + store.lastErrorMessage);
    } else {
      textBlink();
    }
  }, 500);
}

// Remove the managed resource store.
function removeStore() {
  if (!window.google || !google.gears || !store) {
    return;
  }
  localServer.removeManagedStore(STORE_NAME);
  textOut("Store removed\n" +
          "Refreshing...");
  hideItem("t-createStore");
  hideItem("t-updateStore");
  hideItem("t-removeStore");
  hideItem("t-enableStore");
  hideItem("t-disableStore");
  setTimeout("location.reload(true)", 2000);
}

function enableStore() {
  if (!window.google || !google.gears || !store) {
    return;
  }
  store.enabled = true;
  menuStore();
  setTimeout("location.reload(true)", 5);
}

function disableStore() {
  if (!window.google || !google.gears || !store) {
    return;
  }
  store.enabled = false;
  menuStore();
  setTimeout("location.reload(true)", 5);
}

// Utility function to output some status text.
function textOut(s) {
 var elm = document.getElementById("t-textStore");
  while (elm.firstChild) {
    elm.removeChild(elm.firstChild);
  } 
  elm.appendChild(document.createTextNode(s));
  elm.style.fontWeight='normal';
  blink=false;
}

function textBlink() {
 var elm = document.getElementById("t-textStore");
  if (blink) {
    elm.style.fontWeight='normal';
    blink=false;
  } else {
    elm.style.fontWeight='bold';
    blink=true;
  }
}

// update the menu entries according to the current state
function menuStore() {
  var elm = document.getElementById("p-ggears");
  if (elm) {
    elm.style.visibility='visible';
  }
  if (!store) {
    showItem("t-createStore");
    hideItem("t-updateStore");
    hideItem("t-removeStore");
    hideItem("t-enableStore");
    hideItem("t-disableStore");
  } else {
    hideItem("t-createStore");
    showItem("t-updateStore");
    showItem("t-removeStore");
    if (store.enabled) {
      hideItem("t-enableStore");
      showItem("t-disableStore");
    } else {
      hideItem("t-disableStore");
      showItem("t-enableStore");
    }
  }
}

function showItem(s) {
 var elm = document.getElementById(s);
  if (elm) {
    elm.style.display='list-item';
  }
}

function hideItem(s) {
 var elm = document.getElementById(s);
  if (elm) {
    elm.style.display='none';
  }
}

• Modify the store name in this localserver.js to what you want:

var STORE_NAME = "mymediawiki";

localserver.php

• Create the localserver.php as our Mediawiki extension:

<?php

if (defined('MEDIAWIKI')) {


function fnMyGGearsHookStyle() {
    return <<<EOS
<style type='text/css'>
#t-offStore {
  color: #7d7d7d;
}
#t-textStore {
  color: #ba0000;
}
#p-ggears { 
  visibility: hidden;
}
</style>
EOS;
}

function fnMyGGearsHookJS($out) {
    $gearjs='<script type="text/javascript" src="/extensions/GGears/gears_init.js"></script>';
    $out->addScript($gearjs);
    $gearjs='<script type="text/javascript" src="/extensions/GGears/localserver.js"></script>';
    $out->addScript($gearjs);
    $out->addHeadItem('myggearsstyle', fnMyGGearsHookStyle());
    return true;
}

function fnMyGGearsHookBox() {
    echo "\n".'</ul>'."\n";
    echo '</div>'."\n";
    echo '</div>'."\n";
    echo '<div class="portlet" id="p-ggears">'."\n";
    echo '<h5>Gears LocalServer</h5>'."\n";
    echo '<div id="t-ggears" class="pBody">'."\n";
    echo '<ul>'."\n";
    echo '<li id="t-createStore"><a href="#" onclick="createStore();">create Store</a></li>'."\n";
    echo '<li id="t-updateStore"><a href="#" onclick="createStore();">update Store</a></li>'."\n";
    echo '<li id="t-enableStore"><a href="#" onclick="enableStore();">enable Store</a></li>'."\n";
    echo '<li id="t-disableStore"><a href="#" onclick="disableStore();">disable Store</a></li>'."\n";
    echo '<li id="t-removeStore"><a href="#" onclick="removeStore();">remove Store</a></li>'."\n";
    echo '<span id="t-offStore">'."\n";
    // If we are called by Google Gears, let's stamp the page as "OFFLINE"
    if (isset($_SERVER['HTTP_X_GEARS_GOOGLE'])) {
        define('GGearsOffline');
        echo 'OFFLINE copy<br>'."\n";
    }
    echo '</span>'."\n";
    echo '<span id="t-textStore"></span>'."\n";
    echo '<script type="text/javascript">/*<![CDATA[*/ initStore(); /*]]>*/</script>'."\n";
    return true;
}

function fnMyGGearsHookManifest($action, $article) {
    if ($action != 'manifest')
        return true;
    $dbr =& wfGetDB( DB_SLAVE );
    $res = $dbr->select('recentchanges', 'max(rc_timestamp)', '', 'Database::select', array() );
    $s = $dbr->fetchRow( $res );
    $dbr->freeResult( $res );
    echo '{'."\n";
    echo '  "betaManifestVersion": 1,'."\n";
    echo '  "version": "' . $s['max(rc_timestamp)'] . '",'."\n";
    echo '  "entries": ['."\n";
    $b = false;
    $res = $dbr->select('page', 'page_title');
    while ( $row = $dbr->fetchObject( $res ) ) {
        $titles[] = $row->page_title;
    }
    $dbr->freeResult( $res );
    foreach( $titles as $title ) {
        if ($b)
            echo ','."\n";
        echo '    { "url": "/wiki/' . str_replace("%2F","/", urlencode($title)) . '" }';
        $b = true;
    }

    // adding static resources...
    $f="extensions/GGears/static_urls.txt";
    if (file_exists($f) && is_readable($f) && $h = fopen($f, 'r')) {
        while (!feof($h)) {
            $u = fgets($h);
            if ($u) {
                if ($b)
                    echo ','."\n";
                echo '    { "url": "'.trim($u).'" }';
                $b = true;
            }
        }
        fclose($h);
    }
    echo "\n".'  ]'."\n".'}'."\n";
    // A bit hacky: here we've output our content and we don't want
    // mediawiki to add anything so we simply exit instead of return
    exit;
}

$wgExtensionCredits['other'][] = array('name' => 'GGears',
                           'version' => '0.1',
                           'author' => 'Philippe Teuwen',
//                         'url' => 'http://www.mediawiki.org/wiki/Extension:LocalServer',
                           'url' => 'http://wiki.yobi.be/Mediawiki_LocalServer',
                           'description' => 'Allows to work offline with Google Gears');
$wgHooks['UnknownAction'][] = 'fnMyGGearsHookManifest';
$wgHooks['BeforePageDisplay'][] = 'fnMyGGearsHookJS';
$wgHooks['MonoBookTemplateToolboxEnd'][] = 'fnMyGGearsHookBox';
}

?>

• At the end of your LocalSettings.php, add a link to the new extension:

require_once("$IP/extensions/GGears/localserver.php");

static_urls.txt

Make sure to list all your static resources (css, js, images,...)

/skins/common/shared.css?97
/skins/common/commonPrint.css?97
/skins/common/wikibits.js?97
/skins/common/ajax.js?97
/skins/common/ajaxsearch.js?97
/skins/common/ajaxwatch.js?97
/skins/common/images/poweredby_mediawiki_88x31.png
/skins/common/images/gnu-fdl.png
/skins/monobook/main.css?97
/skins/monobook/headbg.jpg
/skins/monobook/external.png
/skins/monobook/bullet.gif
/skins/monobook/user.gif
/index.php?title=-&action=raw&smaxage=0&gen=js&useskin=monobook
/index.php?title=MediaWiki:Common.css&usemsgcache=yes&action=raw&ctype=text/css&smaxage=18000
/index.php?title=MediaWiki:Monobook.css&usemsgcache=yes&action=raw&ctype=text/css&smaxage=18000
/index.php?title=-&action=raw&gen=css&maxage=18000&smaxage=0
/local/yobiwiki-pub.png
/extensions/GGears/localserver.js
/extensions/GGears/gears_init.js

What is missing

What I wanted

What is crudely missing in the Google Gears API is a way to avoid to fetch all the pages at each new revision of the manifest.
Initially I made a manifest which accepts as argument the current version of the local Store and produces a list of the pages new or changed since.
But then GGears deletes all the other pages from the Store :-(

See this discussion and the proposals here below

Current hack

So the current solution cannot avoid fetching all pages, the best we can achieve is to return a HTTP/1.0 304 Not Modified as often as possible.

By default Mediawiki honors properly If-Modified-Since except that for logged users it invalidates the cache for all the pages every time one page is edited :-(
This is supposedly to refresh all the things around a wikipage (headers, menus, footers,...) but for our offline version we prefer not to fetch the whole wiki just because we saved one single page.
So the ugly hack (ugly because not possible from the extension hooks): modify includes/User.php:

function validateCache( $timestamp ) {
    $this->load();
    // If we are called by Google Gears, don't care about minor modifs
    // concerning the current user.
    if (isset($_SERVER['HTTP_X_GEARS_GOOGLE']))
        return true;
    return ($timestamp >= $this->mTouched);
}

This still requires about 30s to query 200 of my wiki pages (instead of 85s + some more bandwidth for the initial download)
That means after every single edition I've to wait 30s then to reload the current page to see my changes :-(

LocalServer v2

As apparently the developers of Google Gears recognize also that it would be an interesting feature to add to LocalServer, what are the different possibilities to help us in the context of wikis or other usages?
Note that I consider here the solutions from a user perspective, not from the GGears developer perspective (how complex is to implement the feature)...

The goal is to get a feature that avoids us to fetch all the pages of a given resource (think about a 2000-page wiki) everytime a subset of them changed.
Because even with a properly handled "If-Modified-Since" header, it's still 2000 requests...

Delta Manifest

Local Google Gears calls the ManifestURL in such a way that the currentVersion is transmitted to the server

• By a GET parameter
  • Pros:
    • Nothing to change in the GGears call as today the ManifestURL can already contain a parameter
  • Cons:
    • The ManifestURL must be updated manually from js during init, e.g. store.manifestUrl = MANIFEST_FILENAME + '?v=' + store.currentVersion;
• By a new X-header
  • Pros:
    • Completely transparent when requesting the ManifestURL which can be again static, no need to change the ManifestURL at init()

Then the servers replies with a Manifest file that informs the browser of the changes since its local version.

• By mentioning only the changed files
  • Pros:
    • Lightweight Manifest file
  • Cons:
    • How to handle deleted files? A new field { "URL":..., "del"=true }?
• By mentioning all the files but indicating those who didn't change, e.g. { "URL":..., "keep"=true }
  • Pros:
    • This handles cleanly the deletion of files, just by removing them for the Manifest file, as today
  • Cons:
    • The Manifest file will contain always the list of all the files, so it will be rather big

Versioned Manifest

Another solution is to have an optional version field e.g. { "URL":...,"src":...,"version":... } in the Manifest file

• Pros:
  • ManifestURL is static
  • The call in GGears is static
  • Manifest file is the same for all clients, so can be cached on the server
  • This handles cleanly the deletion of files, just by removing them for the Manifest file, as today
• Cons:
  • The Manifest file will contain always the list of all the files, so it will be rather big

Archived manifest.php, for the future?

Just in case, here is the manifest.php I tried to make with those assumptions in mind:

{
  "betaManifestVersion": 1,
<?php
require_once("../../DBSettings.php");
// We can get the version of the current DB to output only the
// pages changes since then, otherwise we dump all pages
$since = 0;
if (isset($_GET['since']) && is_numeric($_GET['since'])) {
    $since = $_GET['since'];
}
//error_log(date(DATE_RFC822).": since: ". $since . "\n", 3, './log.txt');

$l = mysql_connect($wgDBserver,$wgDBuser,$wgDBpassword);
mysql_selectdb($wgDBname);
$q = "SELECT max(rc_timestamp) from ".$wgDBprefix."recentchanges";
$r = mysql_query($q);
$t = mysql_fetch_row($r);
echo '  "version": "' . $t[0] . 'a",';
?>

  "entries": [
<?php
$q = "SELECT page_title FROM ".$wgDBprefix."page WHERE page_touched > ".$since.";";
$r = mysql_query($q);
while ($t = mysql_fetch_row($r))
{
    echo '    { "url": "/wiki/' . $t[0] . '" },'."\n";
}

// Just removing unchanged pages is not ok as they will be removed from the Store :-(
$q = "SELECT page_title FROM ".$wgDBprefix."page WHERE page_touched <= ".$since.";";
$r = mysql_query($q);
while ($t = mysql_fetch_row($r))
{
// Ideally I'd like to see sth like that to tell to keep the current page from the Store:
//    echo '    { "url": "/wiki/' . $t[0] . '","refresh": false },'."\n";
// But since the option doesn't exist in GGears, we've to fetch all pages again...
    echo '    { "url": "/wiki/' . $t[0] . '" },'."\n";
}
// Add also all the static ressources: css, images, js,...
?>
    { "url": "/extensions/GGears/localserver.js"},
    { "url": "/extensions/GGears/gears_init.js"}
  ]
}

With in initStore() and createStore() (localserver.js):

store.manifestUrl = MANIFEST_FILENAME + '?since=' + store.currentVersion;

BTW I tried another hack with some

   { "url": "/wiki/OnePage", "src": ".../notmodified.php" },

and notmodified.php being a simple script returning a 304 error

<php header( "HTTP/1.0 304 Not Modified" ); ?>

but GGears stores locally under the name of the src and not the name of the url (which makes some sense), so this attempt failed too.

TODO

• check on clients who have ggears but don't use it on this wiki:

I want to avoid the popup for visitors with GGears but no intention of sync.
The following command is the trigger:

 localServer = google.gears.factory.create("beta.localserver");

At that stage we know GGears is installed but we don't know if the user already allowed us or not
If yes -> as usual otherwise propose to create the store with a link
To remember when a user already said yes, we could create a cookie as we'll store more than one cookie anyway ;-)

• replace manifest.php by a new action of the index.php, cf UnknownAction hook
  This is to avoid direct access to the /extensions/ dir, which shouldn't be required
• on saving page: the page displayed is not up-to-date, we've to wait for the full sync and then only a refresh will display the latest revision.
  We can get it faster by disabling the Store but then subsequent editions won't update the store anymore.
  Maybe we could work (when online) always with Store disabled and with forced uploadStore() on save page...
• could we find weird but cleaner way to return the error 304 via mediawiki extension hooks?
• see Google Gears + Greasemonkey to take Wikipedia offline