Mediawiki RawFile

Very short introduction

Just have a look to the 2 examples to see how to use the extension
and to the Installation section to see how to install the extension in your MediaWiki server

Introduction

Originally the idea was to be able to download directly a portion of code as a file.
I've numerous code examples in my wiki and I wanted an easy way to download them, easier than a copy/paste!
But from there it was rather easy to get something very close to literate programming just by allowing multiple blocks referring to the same file, which will be concatenated together at download time.

• It must work with pre, nowiki, js, css, code, source, so let's make it general: take the tag that comes after the parser function we'll create and select data up to the closing tag.
• There are two distinct functionalities provided by the extension:
  • the parser that will convert a magic word into a link to the download URL
  • an extended ?action=raw that will strip the raw output to keep the desired code

Syntax

There are 2 kinds of elements to add to the wiki language:

• anchors that will flag which code blocks belong to a specific file
  • {{#fileAnchor: myscript.sh}}
  • Not visible in the regular wiki display
• links that will allow to download the file
  • {{#fileLink: myscript.sh}}
  • Transformed into new regular wikicode that will be eventually transformed to real URLs:
    {{fullurl:{{PAGENAME}}|action=raw&file=myscript.sh}}
http://wiki.yobi.be/index.php?title=Mediawiki_RawFile&action=raw&file=myscript.sh

For regular use, when a single code block is used and when the download link can be at the same position as the anchor, there is a shortcut notation mixing both anchor & link properties: {{#file: myscript.sh}}

Short example

The extension works with any block such as pre, nowiki, js, css, code, source,...
This example is using the < source > provided by the extension

Let's save the following code [{{#file: myscript.sh}} as myscript.sh]
<source lang=bash>
#!/bin/bash

echo 'Hello world!'
exit 0
</source>

will give:

---

Let's save the following code [{{#file: myscript.sh}} as myscript.sh]

#!/bin/bash

echo 'Hello world!'
exit 0

Complete example

And a full example with anchors & link:

Let's start with the Bash usual header:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
#!/bin/bash
</source>
Then we'll display a welcome message:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
echo 'Welcome on earth!'
</source>
And we finally exit cleanly:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
exit 0
</source>
[{{#filelink: myotherscript.sh}} myotherscript.sh is now available for download below the code]

will give:

---

Let's start with the Bash usual header: {{#fileanchor: myotherscript.sh}}

#!/bin/bash

Then we'll display a welcome message: {{#fileanchor: myotherscript.sh}}

echo 'Welcome on earth!'

And we finally exit cleanly: {{#fileanchor: myotherscript.sh}}

exit 0

[{{#filelink: myotherscript.sh}} myotherscript.sh is now available for download below the code]

The code (the ultimate example)

Which you can of course download just by following [{{#filelink: RawFile.php}} this link :-)]

So let's explain a bit the code in a Literate Programming way...

Hooks

First some hooks for our functions...

We will create:

• a Parser Function (see also here), with help of
  • $wgExtensionFunctions to define the setup function
  • Magic Words
  • LanguageGetMagic hook to initialize the magic words
• a RawPageViewBeforeOutput hook to intercept the raw output

{{#fileanchor: RawFile.php}}

<?php

if (defined('MEDIAWIKI')) {

$wgExtensionFunctions[] = 'efRawFile_Setup';
$wgHooks['LanguageGetMagic'][]       = 'efRawFile_Magic';
$wgHooks['RawPageViewBeforeOutput'][] = 'fnRawFile_Strip';

Setup function

For the wiki parsing to create download links, file and fileLink are equally treated, while fileAnchor will be simply left out. {{#fileanchor: RawFile.php}}

function efRawFile_Setup() {
    global $wgParser;
    $wgParser->setFunctionHook( 'file', 'efRawFile_Render' );
    $wgParser->setFunctionHook( 'filelink', 'efRawFile_Render' );
    $wgParser->setFunctionHook( 'fileanchor', 'efRawFile_Empty' );
}

Hook to initialize the magic words

We add the magic words here: the first array element indicates if it is case sensitive, in this case it is not case sensitive. We could add extra elements to create synonyms for our parser function.
Unless we return true, other parser functions extensions will not get loaded. {{#fileanchor: RawFile.php}}

function efRawFile_Magic( &$magicWords, $langCode ) {
    $magicWords['file'] = array( 0, 'file' );
    $magicWords['filelink'] = array( 0, 'filelink' );
    $magicWords['fileanchor'] = array( 0, 'fileanchor' );
    return true;
}

Parser functions of the magic words

The transformation rule to replace link shortcuts to actual links for download
The input parameters are wikitext with templates expanded, the output should be wikitext too
TODO: what error to send out if there is no filename given?
TODO: supports links to files located in other local wiki pages, sth like 2nd arg default to $pagename='Mediawiki RawFile'
EDIT: It seems that commit 27667 (1.11 -> 1.12) changed the default parser, which breaks the recursive parsing. Thanks to Tim Starling for helping me to get around the problem! {{#fileanchor: RawFile.php}}

function efRawFile_Render( &$parser, $filename = '') {
    return $parser->mTitle->getFullURL( 'action=raw&file='.urlencode( $filename ) );
}

And the other one, just removing the anchors from the rendered wiki page.
Curiously enough if the function doesn't exist at all the effect is exactly the same, MW doesn't throw any error.
But let's keep things clean... {{#fileanchor: RawFile.php}}

function efRawFile_Empty( &$parser, $filename = '') {
    return '';
}

Hook to intercept the raw output

This part of the code doesn't look that nice because we've to parse the raw wiki page ourselves to retrieve the code sections we want.

First let's see if ?action=raw was used in the context of this extension: in that case we receive the filename as GET parameter, otherwise we simply return from our extension with return value=true which means we authorize the raw display (originally the hook was created to add an authentication point) {{#fileanchor: RawFile.php}}

function fnRawFile_Strip(&$rawPage, &$text) {
    if (!isset($_GET['file']))
        return true;
    $filename=$_GET['file'];

By default the downloadable file will still be handled by the ob_gzhandler session made by Mediawiki. To avoid output buffering and gzipping, one can uncomment the following line: {{#fileanchor: RawFile.php}}

    // Uncomment the following line to avoid output buffering and gzipping:
    // wfResetOutputBuffers();

Raw action already set the headers with some client cache pragmas and is supposed to be displayed in the browser but in our case we want to make this "page" a downloadable file so we overwrite the headers which were defined and we add a few more, to ensure there is no caching on the client (it's very hard for the client to force a refresh on a file download, contrary to a web page) and to provide the adequate filename.

{{#fileanchor: RawFile.php}}

    header("Content-disposition: attachment;filename={$filename}");
    header("Content-type: application/octetstream"); 
    header("Content-Transfer-Encoding: binary"); 
    header("Expires: 0");
    header("Pragma: no-cache"); 
    header("Cache-Control: no-store");

Then we'll strip the output, first we've to locate the anchors but there are anchors that could be protected in literal blocks like nowiki.
So we'll mask the literal blocks before searching for the anchors (we mask with the same string length because we'll retrieve an offset that we will use on the initial string and offsets must match)
TODO: should we care also of source, js, css, pre,... blocks? {{#fileanchor: RawFile.php}}

    $maskedtext=preg_replace_callback('/<nowiki>(.*?)<\/nowiki>/', 
        create_function(
           '$matches',
           'return ereg_replace(".","X",$matches[0]);'
        ),
        $text);

Now we can search for the anchors (or the short version, in which case we only keep the first hit, no multiple blocks support)
And we free the memory used for the masked version
TODO: instead of cowardly returning if we don't find our anchors, we should cancel the headers and return a proper error page {{#fileanchor: RawFile.php}}

    if (preg_match_all('/{{#fileanchor: *'.$filename.' *}}/i', $maskedtext, $matches, PREG_OFFSET_CAPTURE))
        $offsets=$matches[0];
    else if (preg_match_all('/{{#file: *'.$filename.' *}}/i', $maskedtext, $matches, PREG_OFFSET_CAPTURE))
        $offsets=array($matches[0][0]);
    else
        // We didn't find our anchor, let's output all the raw...
        return true;
    unset($maskedtext);

$text is both input & output so we copy it and start with an empty output. {{#fileanchor: RawFile.php}}

    $textorig=$text;
    $text='';

For each anchor found we've to isolate the content of the next block. {{#fileanchor: RawFile.php}}

    foreach ($offsets as $offset) {

Let's remove the text up to the tag following the anchor
TODO: the next tag could be a < br >, which we should skip {{#fileanchor: RawFile.php}}

        $out = substr($textorig, $offset[1]);
        $out = substr($out, strpos($out, '<'));

What type of tag do we have?
Note that we're looking to the word directly following '<' up to '>' or a space, e.g. if there are arguments to the tag.
TODO: once again, better handling of errors than just returning. {{#fileanchor: RawFile.php}}

        if (!preg_match('/^<([^> ]+)/', $out, $matches))
            return true;
        $key = $matches[1];

OK, let's extract the text up to the closing tag
We skip the first carriage return after the opening tag, if any
We look for the closing tag and we take what's in between.
TODO: once again, better handling of errors than just returning. {{#fileanchor: RawFile.php}}

        $begin = strpos($out, '>')+1;
        if (ord(substr($out,$begin,1))==10)
            $begin++;
        if (preg_match_all('/<\/'.$key.'>/', $out, $matches, PREG_OFFSET_CAPTURE))
            $text .= substr($out, $begin, $matches[0][0][1]-$begin);
        else
            // error, we could not find end of bloc
            $text .= substr($out, $begin);
    }

No need to deal with a Content-Length header because Mediawiki will do it for us, moreover more properly than we could if the output is sent gzipped, which is the default.
So that's it, $text contains our file! {{#fileanchor: RawFile.php}}

    return true;
}

Credits

There is an official way to register the extension in a Mediawiki installation, so that it will be visible on the Special:Version page.
Let's say the extension is in the category of parser hooks even if there is also a hook on Raw action. {{#fileanchor: RawFile.php}}

$wgExtensionCredits['parserhook'][] = array('name' => 'RawFile',
                           'version' => '0.2',
                           'author' => 'Philippe Teuwen',
                           'url' => 'http://www.mediawiki.org/wiki/Extension:RawFile',
//                         'url' => 'http://wiki.yobi.be/wiki/Mediawiki_RawFile',
                           'description' => 'Downloads a RAW copy of <nowiki><tag>data</tag></nowiki> in a file<br>'.
                                            'Useful e.g. to download a script or a patch<br>'.
                                            'It also allows what is called [http://en.wikipedia.org/wiki/Literate_programming Literate Programming]');
}

?>

And finally registration of the extension at the Mediawiki website according to the Extensions Manual.

So this extension has now its own page on the official Mediawiki site.

Installation

Download [{{#filelink: RawFile.php}} RawFile.php] and save it under the MediaWiki directory as extensions/RawFile/RawFile.php

Add at the end of LocalSettings.php:

require_once("$IP/extensions/RawFile/RawFile.php");

Status

If you use the extension properly the code is fully functional but it's rather raw on error handling.

ChangeLog

0.2

• Fix problem with Content-Length mismatch when transport is gzipped (default for Mediawiki if client supports it)

0.1

• Initial version

Questions and feedback

If you've any trouble, questions or suggestions, you can contact me.