Mediawiki LocalServer

MediaWiki offline support thanks to Google Gears

This is an attempt to provide an offline browsing (not editing) of main article space (no talk pages, no categories) of any MediaWiki-based wiki using LocalServer API of Google Gears.
Inspired by this blog post

To be able to use Mediawiki offline, the Mediawiki website has to have this extension installed and the browser has to have the Google Gears extension.

The original blog post was quick and dirty (do it in less than one hour), but too dirty for me:

• There was always a complete replication of the full wiki as soon as some change occurred
• There was no way to know if we were viewing the offline or the online version of a page
• There was no way to temporarily disable the offline pages
• It implied hacking the Mediawiki code

So I rewrote it as a Mediawiki extension and tried to overcome the mentioned problems.
Note that it is done for the default style MonoBook.

Browser requirements:

• JavaScript enabled
• Google Gears extension enabled. This extension does not display the wiki's offline capability ("GGears toolbox") for the users who don't have the extension or don't enable javascript. You should see "Gears LocalServer" panel on the left on this wiki if your browser is ok to go.
• user must click "create Store". Google Gears dialog will appear and ask user for confirmation.
• users have to accept a cookie if they want to have the store (which is a kind of mega cookie anyway...)

Unlimited number of users and browsers can get your wiki offline. No state is stored on your MediaWiki server. Usual access control still apply.

Documentation

Bits I found useful for this task:

• the original idea
• http://code.google.com/apis/gears/api_localserver.html
• http://code.google.com/p/google-gears/
• The Google Gears inspector:

svn checkout http://google-gears.googlecode.com/svn/trunk/gears/inspector google-gears-read-only

• http://www.mediawiki.org/wiki/Manual:MediaWiki_hooks

A good practice is also to keep the javascript console open to see quickly any warning or error maybe not directly visible in the page layout.

• MW Manual: Extensions
• Firefox 3 starts to develop an offline capability for websites as well, based on HTML 5. Firefox's Offline resources feature is not used here. Any browser with Google Gears will do.
• Firebug - Firefox extension for obtaining list of static objects (pictures, CSS styles, etc.). Optional.

Google Gears on client side

Installation

First you need to install the Google Gears extension
Note that it does a stupid browser detection and refuses my Iceweasel 2.0 so I had to cheat with my User Agent Switch extension and mimick Firefox 2.0 ;-) If you've some troubles with those prerequisites, like Iceweasel/Iceape not being recognized, you can also install it by clicking on this link

Usage with Mediawiki

creating the local store

Once you've Google Gears you'll see an extra "gears localserver" toolbox in the left menu of the wiki.
You can then choose to create the offline store by clicking on "create Store".
You will be prompted by Google Gears to allow the site to be stored on your computer.
You've also to accept a permanent cookie to remember this authorization.
This will take about 1 our 2 mins to replicate.

working in offline mode

You'll then be in offline mode.
When in this mode you can still do whatever you want, if you're looking to a regular article you'll see the local copy but if you're doing sth else (editing, etc) you'll be in direct contact with the server.
The mention "OFFLINE" with the timestamp tells you when you're seeing the local copy instead of the remote one.
Google Gears will take care of updating your local store in the background from time to time.

working in offline mode, being really offline

If you're disconnected from the Net and you've enabled the store (work Offline), you'll be able to browse the wiki and see the main page of each regular article.
But not more! No search, edit, talk, etc as those require a real connection.

working in online mode (e.g. for edition)

For now there is still a small refreshing issue when editing pages so it's better to switch to online mode when editing the wiki.
You can switch mode by clicking on "work Online", it will disable the local store.
In that mode Google Gears will not update your local copy anymore but you can force it by clicking on "update Store".

the magic URL

If it happens that you're disconnected but with the store disabled (online mode) and you're stuck as the browser tries to reach the server, there is a magic URL that will re-enable the store: the pseudo-page "offline".
So for this wiki this means http://wiki.yobi.be/wiki/offline
The link works only for those who have a local replicate of the wiki as this URL is honored properly only locally.
Tips: Store that URL in your bookmark to access the wiki so it will always work, no matter if you're online or offline.

troubles?

If you've any trouble you can contact me.

Google Gears LocalServer extension on server side

The extension is composed of 4 files.

• gears_init.js is the init file provided by Google to initiate Google Gears
• localserver.js is the part of the extension that will run on the client side
• localserver.php is the part of the extension that will run on the server side
• static_urls.txt is used to populate the manifest file

Prerequisites

This extension was designed under MediaWiki with the following characteristics, it could work with previous versions or other setups but was not tested, so please report any success/failure, thanks!

• MW 1.11.2
• standard Short URL e.g. http://wiki.yobi.be/wiki/Main_Page
• Monobook skin

Installation

/extensions/GGears/

• Create a folder GGears in your MediaWiki extensions directory
• Add the following 4 files to it

gears_init.js

• Download the gears_init.js from the Google Gears web site and place it in the newly created folder

localserver.js

• Create [{{#file: localserver.js}} localserver.js (you can download it here)]:
• Adjust MANIFEST_FILENAME and HELPME_FILENAME paths to point to your wiki

// Change this to set the name of the managed resource store to create.
// You use the name with the createManagedStore, and removeManagedStore,
// and openManagedStore APIs. It isn't visible to the user.
var STORE_NAME = "yobiwiki";

var MANIFEST_FILENAME = "/index.php?action=ggears_manifest";
var HELPME_FILENAME = "/index.php?action=ggears_helpme";
// In case we are offline with a disabled store, that's the relative part
// of the URL to type to activate the local store
var HELPME_URL = "offline";

var localServer;
var store;
var store2;
var blink=false;

function createCookie(name,value,days) {
	if (days) {
		var date = new Date();
		date.setTime(date.getTime()+(days*24*60*60*1000));
		var expires = "; expires="+date.toGMTString();
	}
	else var expires = "";
	document.cookie = name+"="+value+expires+"; path=/";
}

function readCookie(name) {
	var nameEQ = name + "=";
	var ca = document.cookie.split(';');
	for(var i=0;i < ca.length;i++) {
		var c = ca[i];
		while (c.charAt(0)==' ') c = c.substring(1,c.length);
		if (c.indexOf(nameEQ) == 0) return c.substring(nameEQ.length,c.length);
	}
	return null;
}

function eraseCookie(name) {
	createCookie(name,"",-1);
}

function allowedCookie() {
   createCookie(GGCOOKIE_NAME+'test', 'test');
   if (readCookie(GGCOOKIE_NAME+'test')) {
      eraseCookie(GGCOOKIE_NAME+'test');
      return true;
   }
   return false;
}

// Called onload to initialize local server and store variables
function initStore() {
  if (!window.google || !google.gears) {
    //    textOut("Google Gears not found");
      return;
  }
  //Lets see if the user already authorized GGears
  if (readCookie(GGCOOKIE_NAME)) {
    localServer = google.gears.factory.create("beta.localserver");
    store = localServer.openManagedStore(STORE_NAME);
    // if (store) {Here is the place to touch the manifestUrl, if required}
    menuStore();
    // add the version if it's the offline page
    var elm = document.getElementById("t-offStore");
    if (elm.firstChild) {
      elm.appendChild(document.createTextNode('v: '+store.currentVersion));
    }
  } else {
    menuStore();
  }
}

// Functions to create an offline helper in case
// we are stuck offline with a disabled store
// As it's optional we won't throw any error
function createHelpmeStoreCallback(url, success, id) {
  store2.rename(HELPME_FILENAME, HELPME_URL);
}
function createHelpmeStore() {
  if (!window.google || !google.gears || !localServer) {
    return;
  }
  store2 = localServer.createStore(STORE_NAME);
  if (!store2) {
    return;
  }
  store2.capture(HELPME_FILENAME, createHelpmeStoreCallback);
}
function removeHelpmeStore() {
  if (!window.google || !google.gears || !localServer) {
    return;
  }
  localServer.removeStore(STORE_NAME);
}

// Create (or open&update) the managed resource store
function createStore() {
  if (!window.google || !google.gears) {
    alert("You must install Google Gears first.");
    return;
  }
  if (!allowedCookie()) {
    alert("You must allow cookies!\nAborting...");
    return;
  } else {
    createCookie(GGCOOKIE_NAME, 1, 365*10);
  }
  if (!localServer) {
    localServer = google.gears.factory.create("beta.localserver");
  }
  store = localServer.createManagedStore(STORE_NAME);
  if (!store) {
    textOut("Error creating store");
  } else {
    store.manifestUrl = MANIFEST_FILENAME;
    hideItem("t-createStore");
    showItem("t-removeStore");
    // To not switch to offline mode by default:
    //store.enabled = false;
    updateStore();
  }
}

// Update the managed resource store
function updateStore() {
  if (!window.google || !google.gears) {
    alert("You must install Google Gears first.");
    return;
  }
  textOut("Downloading...");
  // Let's also create/refresh our helper page
  createHelpmeStore();
  store.checkForUpdate();
  var timerId = window.setInterval(function() {
    // if store was removed during sync:
    if (!store) {
      window.clearInterval(timerId);
    } else if (store.updateStatus == 0) {
      window.clearInterval(timerId);
      textOut("Sync done!\n" +
              "Version: " +
              store.currentVersion);
      menuStore();
      setTimeout('textOut("")', 2000);
    } else if (store.updateStatus == 3) {
      window.clearInterval(timerId);
      textOut("Error: " + store.lastErrorMessage);
    } else {
      textBlink();
    }
  }, 500);
}

// Remove the managed resource store.
function removeStore() {
  if (!window.google || !google.gears || !store) {
    return;
  }
  var enabled = store.enabled;
  localServer.removeManagedStore(STORE_NAME);
  eraseCookie(GGCOOKIE_NAME);
  removeHelpmeStore();
  // Were we working offline?
  if (enabled) {
    hideItem("t-createStore");
    hideItem("t-updateStore");
    hideItem("t-removeStore");
    hideItem("t-enableStore");
    hideItem("t-disableStore");
    textOut("Store removed\n" +
            "Going online...");
    setTimeout("location.reload(true)", 2000);
  } else {
    store = null;
    menuStore();
    textOut("Store removed");
    setTimeout('textOut("");', 2000);
  }
}

function enableStore() {
  if (!window.google || !google.gears || !store) {
    return;
  }
  store.enabled = true;
  menuStore();
  setTimeout("location.reload(true)", 5);
}

function disableStore() {
  if (!window.google || !google.gears || !store) {
    return;
  }
  store.enabled = false;
  menuStore();
  setTimeout("location.reload(true)", 5);
}

// Utility function to output some status text.
function textOut(s) {
 var elm = document.getElementById("t-textStore");
  while (elm.firstChild) {
    elm.removeChild(elm.firstChild);
  }
  elm.appendChild(document.createTextNode(s));
  elm.style.fontWeight='normal';
  blink=false;
}

function textBlink() {
 var elm = document.getElementById("t-textStore");
  if (blink) {
    elm.style.fontWeight='normal';
    blink=false;
  } else {
    elm.style.fontWeight='bold';
    blink=true;
  }
}

// update the menu entries according to the current state
function menuStore() {
  var elm = document.getElementById("p-ggears");
  if (elm) {
    elm.style.visibility='visible';
  }
  if (!readCookie(GGCOOKIE_NAME) || !store) {
    showItem("t-createStore");
    hideItem("t-updateStore");
    hideItem("t-removeStore");
    hideItem("t-enableStore");
    hideItem("t-disableStore");
  } else {
    hideItem("t-createStore");
    showItem("t-updateStore");
    showItem("t-removeStore");
    if (store.enabled) {
      hideItem("t-enableStore");
      showItem("t-disableStore");
    } else {
      hideItem("t-disableStore");
      showItem("t-enableStore");
    }
  }
}

function showItem(s) {
  var elm = document.getElementById(s);
  if (elm) {
    elm.style.display='list-item';
  }
}

function hideItem(s) {
  var elm = document.getElementById(s);
  if (elm) {
    elm.style.display='none';
  }
}

• Modify the store name in this localserver.js to what you want:

 var STORE_NAME = "mymediawiki";

• In principle this file is portable to any other kind of wiki or CMS, blog,... website, the only modifications are the few variables defined at the top: STORE_NAME, MANIFEST_FILENAME etc.

localserver.php

• Create the [{{#file: localserver.php}} localserver.php (you can download it here)] as our Mediawiki extension:

<?php

if (defined('MEDIAWIKI')) {

function fnMyGGearsHookStyle() {
    return <<<EOS
<style type='text/css'>
#t-offStore {
  color: #7d7d7d;
}
#t-textStore {
  color: #ba0000;
}
#p-ggears {
  visibility: hidden;
}
</style>
EOS;
}

function fnMyGGearsHookJS($out) {
    global $wgScriptPath;
    $gearjs="<script type='text/javascript' src='$wgScriptPath/index.php?action=ggears_js'></script>\n";
    $out->addScript($gearjs);
    $out->addHeadItem('myggearsstyle', fnMyGGearsHookStyle());
    return true;
}

function fnMyGGearsHookBox() {
?>

</ul>
</div>
</div>
<div class="portlet" id="p-ggears">
<h5>Gears LocalServer</h5>
<div id="t-ggears" class="pBody">
<ul>
<li id="t-createStore"><a href="#" onclick="createStore();">create Store</a></li>
<li id="t-updateStore"><a href="#" onclick="updateStore();">update Store</a></li>
<li id="t-enableStore"><a href="#" onclick="enableStore();">work Offline</a></li>
<li id="t-disableStore"><a href="#" onclick="disableStore();">work Online</a></li>
<li id="t-removeStore"><a href="#" onclick="removeStore();">remove Store</a></li>
<?php
    echo '<span id="t-offStore">';
    // If we are called by Google Gears, let's stamp the page as "OFFLINE"
    if (isset($_SERVER['HTTP_X_GEARS_GOOGLE'])) {
        define('GGearsOffline');
        echo 'OFFLINE copy<br>'."\n";
    }
    echo '</span>'."\n";
?>
<span id="t-textStore"></span>
<script type="text/javascript">/*<![CDATA[*/ initStore(); /*]]>*/</script>
<?php
    return true;
}

function fnMyGGearsHookActions($action, $article) {
    $ret = true;
    switch ($action) {
        // A bit hacky: here we've output our content and we don't want
        // mediawiki to add anything so we simply exit instead of return
        case 'ggears_manifest':
            fnMyGGearsHookActionManifest($article);
            exit;
            break;
        case 'ggears_js':
            fnMyGGearsHookActionJS($article);
            exit;
            break;
        case 'ggears_helpme':
            fnMyGGearsHookActionHelpme($article);
            exit;
            break;
    }
    return $ret;
}

function fnMyGGearsHookActionManifest($article) {
    $dbr =& wfGetDB( DB_SLAVE );
    $res = $dbr->select('recentchanges', 'max(rc_timestamp)', '', 'Database::select', array() );
    $s = $dbr->fetchRow( $res );
    $dbr->freeResult( $res );
    echo '{'."\n";
    echo '  "betaManifestVersion": 1,'."\n";
    echo '  "version": "' . $s['max(rc_timestamp)'] . '",'."\n";
    echo '  "entries": ['."\n";
    $b = false;
    $res = $dbr->select('page', array('page_title', 'page_namespace'));
    while ( $row = $dbr->fetchObject( $res ) ) {
        $title = Title::makeTitle( $row->page_namespace, $row->page_title );
        $urls[] = $title->getLocalUrl();
    }
    $dbr->freeResult( $res );
    foreach( $urls as $url ) {
        if ($b) echo ','."\n";
        echo '    { "url": "'.$url.'" }';
        $b = true;
    }

    // adding our JS
    if ($b) echo ','."\n";
    global $wgScriptPath;
    echo '    { "url": "'. $wgScriptPath .'/index.php?action=ggears_js" }';

    // adding other static resources...
    $f="extensions/GGears/static_urls.txt";
    if (file_exists($f) && is_readable($f) && $h = fopen($f, 'r')) {
        while (!feof($h)) {
            $u = fgets($h);
            if ($u) {
                echo ','."\n";
                echo '    { "url": "'.trim($u).'" }';
            }
        }
        fclose($h);
    }
    echo "\n".'  ]'."\n".'}'."\n";
}

function fnMyGGearsHookActionJS($article) {
    global $wgCookiePrefix;
    $f = 'extensions/GGears/gears_init.js';
    if (file_exists($f) && is_readable($f)) {
        readfile($f);
    }
    echo "\n".'var GGCOOKIE_NAME = "'.$wgCookiePrefix.'GGears";'."\n";
    $f = 'extensions/GGears/localserver.js';
    if (file_exists($f) && is_readable($f)) {
        readfile($f);
    }
}

function fnMyGGearsHookActionHelpme($article) {
    // the helpme page is standalone in its own store, so we inline our JS in it
    echo "<html>\n<head>\n<script type=\"$wgJsMimeType\">/*<![CDATA[*/\n";
    fnMyGGearsHookActionJS($article);
    echo "\n/*]]>*/</script>\n</head>\n<body>\n<script type=\"$wgJsMimeType\">/*<![CDATA[*/\n";
    echo 'localServer = google.gears.factory.create("beta.localserver");'."\n";
    echo 'store = localServer.openManagedStore(STORE_NAME);'."\n";
    echo 'store.enabled = true;'."\n";
    echo 'setTimeout("window.location = \"'.$article->mTitle->mUrlform.'\"", 50);'."\n";
    echo "\n/*]]>*/</script>\n</body>\n</html>";
}

$wgExtensionCredits['other'][] = array('name' => 'GGears',
                           'version' => '0.1',
                           'author' => 'Philippe Teuwen',
//                         'url' => 'http://www.mediawiki.org/wiki/Extension:LocalServer',
                           'url' => 'http://wiki.yobi.be/wiki/Mediawiki_LocalServer',
                           'description' => 'Allows to work offline with Google Gears');
$wgHooks['UnknownAction'][] = 'fnMyGGearsHookActions';
$wgHooks['BeforePageDisplay'][] = 'fnMyGGearsHookJS';
$wgHooks['MonoBookTemplateToolboxEnd'][] = 'fnMyGGearsHookBox';
}

?>

• At the end of your LocalSettings.php, add a link to the new extension:

 require_once("$IP/extensions/GGears/localserver.php");

static_urls.txt

Make sure to list all your static resources (css, js, images,...)
This list is specific to my wiki, make yours!.

/skins/common/shared.css?97
/skins/common/commonPrint.css?97
/skins/common/wikibits.js?97
/skins/common/ajax.js?97
/skins/common/ajaxsearch.js?97
/skins/common/ajaxwatch.js?97
/skins/common/images/poweredby_mediawiki_88x31.png
/skins/common/images/gnu-fdl.png
/skins/monobook/main.css?97
/skins/monobook/headbg.jpg
/skins/monobook/external.png
/skins/monobook/bullet.gif
/skins/monobook/user.gif
/index.php?title=-&action=raw&smaxage=0&gen=js&useskin=monobook
/index.php?title=MediaWiki:Common.css&usemsgcache=yes&action=raw&ctype=text/css&smaxage=18000
/index.php?title=MediaWiki:Monobook.css&usemsgcache=yes&action=raw&ctype=text/css&smaxage=18000
/index.php?title=-&action=raw&gen=css&maxage=18000&smaxage=0
/local/yobiwiki-pub.png
/local/login-bg.gif
/images/4/40/Pinguinalulu-panoramix---panoratux.png

Architecture

server side

localserver.php is composed of the following functions:

• fnMyGGearsHookJS()
  • injected to MW with $wgHooks['BeforePageDisplay'][] = 'fnMyGGearsHookJS';
  • adds an include of the JS via /index.php?action=ggears_js, cf action hooks below
  • calls fnMyGGearsHookStyle()
• fnMyGGearsHookStyle()
  • adds some CSS: by default the special menu will be hidden and uncovered by JS so clients without JS won't see the menu
• fnMyGGearsHookBox()
  • injected to MW with $wgHooks['MonoBookTemplateToolboxEnd'][] = 'fnMyGGearsHookBox';
  • adds the HTML to create the special menu
  • if called by Google Gears, adds an "OFFLINE copy" string below the menu
  • adds a JS call to initStore()
• fnMyGGearsHookActions()
  • injected to MW with $wgHooks['UnknownAction'][] = 'fnMyGGearsHookActions';
  • handles new special MW actions, it's actually a trick to be able to serve special content via the main index.php as by default the /extensions/ directory is not reachable directly
  • ?action=ggears_manifest calls fnMyGGearsHookActionManifest()
  • ?action=ggears_js calls fnMyGGearsHookActionJS()
  • ?action=ggears_helpme calls fnMyGGearsHookActionHelpme()
• fnMyGGearsHookActionManifest()
  • creates the Manifest file required for a ManagedResourceStore
  • global version is the timestamp of the latest modified article
  • generates the list of pages
  • adds our JS file /index.php?action=ggears_js
  • adds all URLs found in static_urls.txt and required for a proper offline experience (css, js, images,...)
• fnMyGGearsHookActionJS()
  • streams both gears_init.js and localserver.js contents concatenated.
  • provides a cookie name based on $wgCookiePrefix to JS
• fnMyGGearsHookActionHelpme()
  • generates a standalone HTML page to enable the store and redirect to the main wiki page
  • the page embeds also the JS via a call to fnMyGGearsHookActionJS()

client side

The client receives the bits generated by localserver.php

• the wiki page, with some extra JS, CSS and a special menu Gears
• JS functions called on page loading:
  • initStore()
    • if no Google Gears extension, do nothing
    • if no cookie, presents "create Store" link
    • otherwise, opens the local Managed Store, presents the contextual menu and, if we're looking at the local copy, adds a version string in the menu below "OFFLINE copy"
      By that way we avoided to trigger the Google Gears popup as we first checked for the cookie.
• JS functions called from the special menu:
  • createStore
    • creates the cookie so next time we know we can try to open the local Managed Store
    • creates the Managed Store
    • stores the Manifest URL: /index.php?action=ggears_manifest
    • calls updateStore()
  • updateStore
    • contextual message "Downloading"
    • calls createHelpmeStore(), same operation to create or update...
    • calls store.checkForUpdate(); to trigger Google Gears
    • actively checks the status every 500ms, blinks the msg if still downloading else displays a success/error message
  • removeStore
    • removes the ManagedResourceStore
    • calls removeHelpmeStore()
    • removes the cookie
    • reload the page if we were offline & cleans the special menu
  • enableStore: store.enabled = true;
  • disableStore: store.enabled = false;
• JS functions to handle the magic page "offline"
  • createHelpmeStoreCallback, createHelpmeStore, removeHelpmeStore
    It manages a (non-managed) Resource Store where the content of /index.php?action=helpme is stored and is made available under the pseudo-page "offline", e.g. http://wiki.yobi.be/wiki/offline
• Menu helper functions: textOut, textBlink, menuStore, showItem, hideItem to display some contextual messages and toggle the content of the special menu.
• Cookie management: createCookie, readCookie, eraseCookie, allowedCookie

What is missing

What I wanted

What is crudely missing in the Google Gears API is a way to avoid to fetch all the pages at each new revision of the manifest.
Initially I made a manifest which accepts as argument the current version of the local Store and produces a list of the pages new or changed since.
But then GGears deletes all the other pages from the Store :-(

See this discussion and the proposals here below

Current hack

So the current solution cannot avoid fetching all pages, the best we can achieve is to return a HTTP/1.0 304 Not Modified as often as possible.

By default Mediawiki honors properly If-Modified-Since except that for logged users it invalidates the cache for all the pages every time one page is edited :-(
This is supposedly to refresh all the things around a wikipage (headers, menus, footers,...) but for our offline version we prefer not to fetch the whole wiki just because we saved one single page.
So the ugly hack (ugly because not possible from the extension hooks): modify includes/User.php:

function validateCache( $timestamp ) {
    $this->load();
    // If we are called by Google Gears, don't care about minor modifs
    // concerning the current user.
    if (isset($_SERVER['HTTP_X_GEARS_GOOGLE']))
        return true;
    return ($timestamp >= $this->mTouched);
}

This still requires about 30s to query 200 of my wiki pages (instead of 85s + some more bandwidth for the initial download)
That means after every single edition I've to wait 30s then to reload the current page to see my changes :-(

To integrate this hack to the extension, we need to redefine User::validateCache() which is not possible in PHP unless the runkit is installed :-(
Or Mediawiki has to provide a new hook but this is too specialized to become a standard hook...

LocalServer v2

As apparently the developers of Google Gears recognize also that it would be an interesting feature to add to LocalServer, what are the different possibilities to help us in the context of wikis or other usages?
Note that I consider here the solutions from a user perspective, not from the GGears developer perspective (how complex is to implement the feature)...

The goal is to get a feature that avoids us to fetch all the pages of a given resource (think about a 2000-page wiki) everytime a subset of them changed.
Because even with a properly handled "If-Modified-Since" header, it's still 2000 requests...

Delta Manifest

Local Google Gears calls the ManifestURL in such a way that the currentVersion is transmitted to the server

• By a GET parameter
  • Pros:
    • Nothing to change in the GGears call as today the ManifestURL can already contain a parameter
  • Cons:
    • The ManifestURL must be updated manually from js during init, e.g. store.manifestUrl = MANIFEST_FILENAME + '?v=' + store.currentVersion;
• By a new X-header
  • Pros:
    • Completely transparent when requesting the ManifestURL which can be again static, no need to change the ManifestURL at init()

Then the servers replies with a Manifest file that informs the browser of the changes since its local version.

• By mentioning only the changed files
  • Pros:
    • Lightweight Manifest file
  • Cons:
    • How to handle deleted files? A new field { "URL":..., "del"=true }?
• By mentioning all the files but indicating those who didn't change, e.g. { "URL":..., "keep"=true }
  • Pros:
    • This handles cleanly the deletion of files, just by removing them for the Manifest file, as today
  • Cons:
    • The Manifest file will contain always the list of all the files, so it will be rather big

Versioned Manifest

Another solution is to have an optional version field e.g. { "URL":...,"src":...,"version":... } in the Manifest file

• Pros:
  • ManifestURL is static
  • The call in GGears is static
  • Manifest file is the same for all clients, so can be cached on the server
  • This handles cleanly the deletion of files, just by removing them for the Manifest file, as today
• Cons:
  • The Manifest file will contain always the list of all the files, so it will be rather big

Archived manifest.php, for the future?

Just in case, here is the manifest.php I tried to make with those assumptions in mind:

{
  "betaManifestVersion": 1,
<?php
require_once("../../DBSettings.php");
// We can get the version of the current DB to output only the
// pages changes since then, otherwise we dump all pages
$since = 0;
if (isset($_GET['since']) && is_numeric($_GET['since'])) {
    $since = $_GET['since'];
}
//error_log(date(DATE_RFC822).": since: ". $since . "\n", 3, './log.txt');

$l = mysql_connect($wgDBserver,$wgDBuser,$wgDBpassword);
mysql_selectdb($wgDBname);
$q = "SELECT max(rc_timestamp) from ".$wgDBprefix."recentchanges";
$r = mysql_query($q);
$t = mysql_fetch_row($r);
echo '  "version": "' . $t[0] . 'a",';
?>

  "entries": [
<?php
$q = "SELECT page_title FROM ".$wgDBprefix."page WHERE page_touched > ".$since.";";
$r = mysql_query($q);
while ($t = mysql_fetch_row($r))
{
    echo '    { "url": "/wiki/' . $t[0] . '" },'."\n";
}

// Just removing unchanged pages is not ok as they will be removed from the Store :-(
$q = "SELECT page_title FROM ".$wgDBprefix."page WHERE page_touched <= ".$since.";";
$r = mysql_query($q);
while ($t = mysql_fetch_row($r))
{
// Ideally I'd like to see sth like that to tell to keep the current page from the Store:
//    echo '    { "url": "/wiki/' . $t[0] . '","refresh": false },'."\n";
// But since the option doesn't exist in GGears, we've to fetch all pages again...
    echo '    { "url": "/wiki/' . $t[0] . '" },'."\n";
}
// Add also all the static ressources: css, images, js,...
?>
    { "url": "/extensions/GGears/localserver.js"},
    { "url": "/extensions/GGears/gears_init.js"}
  ]
}

With in initStore() and createStore() (localserver.js):

 store.manifestUrl = MANIFEST_FILENAME + '?since=' + store.currentVersion;

BTW I tried another hack with some

   { "url": "/wiki/OnePage", "src": ".../notmodified.php" },

and notmodified.php being a simple script returning a 304 error

 <?php header( "HTTP/1.0 304 Not Modified" ); ?>

but GGears stores locally under the name of the src and not the name of the url (which makes some sense), so this attempt failed too.

TODO

• on saving page: the page displayed is not up-to-date, we've to wait for the full sync and then only a refresh will display the latest revision.
  We can get it faster by disabling the Store during the update then re-enabling the store once update is done
• could we find weird but cleaner way to return the error 304 via mediawiki extension hooks?
• see Google Gears + Greasemonkey to take Wikipedia offline
• have also fqdn and fqdn/wiki/ in the local store
• save more? User:* Talk:* ...
  • UPDATE: for Category:* see the patch below
• use workerpool to sync in background?
• check http://www.mediawiki.org/wiki/Manual:Extensions
• work also with other themes than monobook? cleaner hookbox

Alternatives

• Building a (fast) Wikipedia offline reader
• Stardict: Wikipedia dictionaries

Patches & external contributions

Extending local store

This patch provided by (name will come) enhances the local store by including also the special Category:* pages and all the media files stored under /images/

Download [{{#file: localserver.php.cats.diff}} the patch here]

--- localserver.php	2008-11-23 01:12:17.000000000 +0100
+++ localserver.php	2008-11-23 01:16:54.000000000 +0100
@@ -18,6 +18,27 @@
 EOS;
 }
 
+# from http://snippets.dzone.com/posts/show/4147
+function find_files($path) {
+       $path = rtrim(str_replace("\\", "/", $path), '/') . '/';
+       $entries = Array();
+       $result = Array();
+       $dir = dir($path);
+       while (false !== ($entry = $dir->read())) {
+         $entries[] = $entry;
+       }
+      $dir->close();
+      foreach ($entries as $entry) {
+        $fullname = $path . $entry;
+        if ($entry != '.' && $entry != '..' && is_dir($fullname)) {
+          $result = array_merge($result, find_files($fullname));
+        } else if (is_file($fullname)) {
+          $result[] = $fullname;
+        }
+      }
+    return $result;
+    }
+
 function fnMyGGearsHookJS($out) {
     $gearjs='<script type="text/javascript" src="/index.php?action=ggears_js"></script>';
     $out->addScript($gearjs);
@@ -91,12 +112,33 @@
         $titles[] = $row->page_title;
     }
     $dbr->freeResult( $res );
+
     foreach( $titles as $title ) {
         if ($b) echo ','."\n";
         echo '    { "url": "/wiki/' . str_replace("%2F","/", urlencode($title)) . '" }';
         $b = true;
     }
 
+    # now get categories
+    $res = $dbr->select('categorylinks', "cl_to", '', 'Database::select', array('DISTINCT') );
+    while ( $row = $dbr->fetchRow($res) ) {
+      $cattitles[] = $row['cl_to'];
+    }
+    $dbr->freeResult($res);
+
+    foreach( $cattitles as $title ) {
+        if ($b) echo ','."\n";
+        echo '    { "url": "/wiki/Category:' . str_replace("%2F","/", urlencode($title)) . '" }';
+        $b = true;
+    }
+
+    # now find all images
+    foreach ( find_files('images') as $image) {
+      if (stripos($image, 'images/deleted') === false) { // Deleted Images directory is protected by a local .htaccess file. Will fail if included.
+        if ($b) echo ','."\n";
+        echo '    { "url": "/' . str_replace("%2F","/", urlencode($image)) . '" }';
+        $b = true;
+      }
+    }
+
     // adding our JS
     if ($b) echo ','."\n";
     echo '    { "url": "/index.php?action=ggears_js" }';

Normal (not short) URLs

To enable the extension to work with URLs in the form http://wiki.domain.com/index.php/Main_Page, simply change line 98 of localserver.php as follows:

       echo '    { "url": "' . $wgScriptPath . '/index.php/' . str_replace("%2F","/", urlencode($title)) . '" }';

This was tested and works under the following conditions:

• MediaWiki: 1.11.0
• PHP: 5.1.6 (apache2handler)
• MySQL: 5.0.27-log
• normalURL e.g. http://wiki.yobi.be/index.php/Main_Page
• Monobook skin

NOTE: I suspect (but did not test) that if your configuration uses index.php?, that the replacement string above should be changed to '/index.php?'.