YobiWiki - User contributions [en]

Mediawiki RawFile

2014-02-17T10:14:29Z

Fuujuhi: /* ChangeLog */ Say have a work-around for 0.5.1

==Very short introduction==
Just have a look to the 2 [[Mediawiki_RawFile#Short_example|examples]] to see how to use the extension
 and to the [[Mediawiki_RawFile#Installation|Installation]] section to see how to install the extension in your [[MediaWiki]] server
==Introduction==
Originally the idea was to be able to download directly a portion of code as a file.
 I've numerous code examples in my wiki and I wanted an easy way to download them, easier than a copy/paste!
 But from there it was rather easy to get something very close to [http://en.wikipedia.org/wiki/Literate_programming literate programming] just by allowing multiple blocks referring to the same file, which will be concatenated together at download time.

* It must work with pre, nowiki, js, css, code, source, so let's make it general: take the tag that comes after the parser function we'll create and select data up to the closing tag.
* There are two distinct functionalities provided by the extension:
** the parser that will convert a magic word into a link to the download URL
** an extended ?action=raw that will strip the raw output to keep the desired code

==Syntax==
The extension introduces 3 elements:
;Anchor
: Used to flag that the next code block in the wiki text belongs to a specific file. The code block can be any wiki block (such as '''<code><pre></code>''', '''<code><code></code>''', '''<code><tt></code>''', '''<code><source></code>'''...). '''<code> </code>''' tags are ignored. Note that anchors are invisible in the wiki display.
;Link
: They are transformed by the extension into links that allows for downloading all blocks attached to a given anchor name.
;Anchor-link
: A shortcut notation mixing both an anchor and download link, handy for regular use, when a single code block is used and when the download link can be at the same position as the anchor.

The syntax is as follows. The syntax using tag <code><file></code> and tag attribute <code>class</code> is new since v0.4. Note that elements of both syntaxes can be mixed in a same page.
{| border="2" cellspacing="4" cellpadding="3" style="margin: 1em 1em 1em 0; background: #f9f9f9; border: 1px #aaaaaa solid; border-collapse: collapse; empty-cells:show;"
!width="80em" style="background: #8da7d6;"|Element!!style="background: #8da7d6;"|Syntax and description
|-
|'''Anchor'''
|<pre><nowiki>
{{#fileAnchor: anchorname}}
<pre class='anchorname'>...</pre>
<code class="anchorname">...</code>
<code class="cssclass anchorname">...</code>
...
</nowiki></pre>
Indicates that the next wiki block is attached to an anchor ''anchorname''. The content of that block will be downloaded (possibly appended with other blocks if there are several blocks attached to the same ''anchorname'') when a file link is clicked on. 
'''(since v0.4)''' To attach an anchor ''anchorname'' to a wiki block, simply add an attribute <code>class="anchorname"</code> to it. The extension supports multi-class specification, meaning that a same block can be associated to different files, and that the <code>class</code> attribute can still be used to specify custom CSS properties as in standard wiki text.
; ''anchorname''
; class="''anchorname''"
: The name of the anchor to which the wiki block is attached
|-
|'''Link'''
|<pre><nowiki>
[{{#fileLink: anchorname}} link text]
[{{#fileLink: anchorname|pagetitle}} link text]
<file anchor="anchorname" [name="filename"] [title="pagetitle"]>link text</file>
</nowiki></pre>
Creates a link to download all blocks that are attached to an anchor ''anchorname''.
;''anchorname''
;anchor="''anchorname''"
: The name of the anchor to look for. All blocks attached to an anchor ''anchorname'' will be downloaded.
;name="''filename''"
:''Optional'' - Specifies the name of the file to download. If absent, ''anchorname'' is then used as the name of the downloaded file.
; ''pagetitle''
;title="''pagetitle''"
: ''Optional'' - Indicates that the blocks to download are on the wiki page titled ''pagetitle''. If absent, blocks are looked for on the current page.
; ''link text''
: The text of the link to display.
|-
|'''Anchor-link'''
|<pre><nowiki>
[{{#file: filename}} link text]
<file name="filename" [tag="''tagname''"]>link text</file>
</nowiki></pre>
Creates a link to download the next wiki block as a file named ''filename''. 
'''(since v0.4)''' The attribute <code>tag</code> can be used to specify the ''tagname'' of the block to download. 
; ''filename''
; name="''filename''"
: The name of the file to download.
;tag="''tagname''"
:''Optional'' - When set, the extension only looks for blocks whose name matches the given ''tagname''. This attribute is particularly useful when there are some irrelevant blocks between the '''anchor-link''' and the block you want to download. If absent, the first encountered block following the anchor is downloaded.
; ''link text''
: The text of the link to display.
|}

===Short example===
The extension works with any block such as pre, nowiki, js, css, code, source,...
 This example is using the syntax highlighting <nowiki><source></nowiki> tag provided by [http://www.mediawiki.org/wiki/Extension:SyntaxHighlight_GeSHi SyntaxHighlight extension] (using [http://qbnz.com/highlighter/ GeSHi Highlighter])
 If you didn't install that extension on your MediaWiki, you can try the example by using <nowiki><pre></nowiki> instead of <nowiki><source></nowiki>.

<pre><nowiki>Let's save the following code [{{#file: myscript.sh}} as myscript.sh]
<source lang=bash>
#!/bin/bash

echo 'Hello world!'
exit 0
</source></nowiki>
</pre>
will give:
----
Let's save the following code [{{#file: myscript.sh}} as myscript.sh]
<source lang=bash>
#!/bin/bash

echo 'Hello world!'
exit 0
</source>

===Complete example===
And a full example with anchors & link:

<pre><nowiki>
Let's start with the Bash usual header:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
#!/bin/bash
</source>
Then we'll display a welcome message:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
echo 'Welcome on Earth!'
</source>
And we finally exit cleanly:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
exit 0
</source>
[{{#filelink: myotherscript.sh}} myotherscript.sh is now available for download below the code]
</nowiki></pre>
will give:
----
Let's start with the Bash usual header:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
#!/bin/bash
</source>
Then we'll display a welcome message:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
echo 'Welcome on Earth!'
</source>
And we finally exit cleanly:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
exit 0
</source>
[{{#filelink: myotherscript.sh}} myotherscript.sh is now available for download below the code]

=== Example using templates ===

The new syntax <code><file name="..." [tag="..."]>...</file></code> allows for using the RawFile extension in templates as well.

The example below uses the template [[Template:Rawfiledownloadexample]] to avoid duplication between the file name in the text and as a parameter in the <code><file></code> tag.

See the template source for instructions on how to create the template.

<pre>
{{Rawfiledownloadexample|name=myfile.txt|content=Once upon a time
There was a Tag
Tag was clickable
And clicked it was}}
</pre>

The code above gives
<hr/>
{{Rawfiledownloadexample|name=myfile.txt|content=Once upon a time
There was a Tag
Tag was clickable
And clicked it was}}

==The code (the ultimate example)==
Which you can of course download just by following [{{#filelink: RawFile.php}} this link :-)]

So let's explain a bit the code in a Literate Programming way...
===Hooks===
First some hooks for our functions...

We will create:
* a [http://www.mediawiki.org/wiki/Manual:Parser_functions Parser Function] (see also [http://meta.wikimedia.org/wiki/Help:Parser_function here]), with help of
** [http://www.mediawiki.org/wiki/Manual:%24wgExtensionFunctions $wgExtensionFunctions] or [http://www.mediawiki.org/wiki/Manual:Hooks/ParserFirstCallInit ParserFirstCallInit global hook] to define the setup function
** [http://www.mediawiki.org/wiki/Manual:Magic_words Magic Words]
** [http://www.mediawiki.org/wiki/Manual:Tag_extensions Tag extensions]
** [http://www.mediawiki.org/wiki/Manual:Hooks/LanguageGetMagic LanguageGetMagic] hook to initialize the magic words
* a [http://www.mediawiki.org/wiki/Manual:Hooks/RawPageViewBeforeOutput RawPageViewBeforeOutput] hook to intercept the raw output

{{#fileanchor: RawFile.php}}
<source lang=php>
<?php

if (defined('MEDIAWIKI')) {

//Avoid unstubbing $wgParser on setHook() too early on modern (1.12+) MW versions, as per r35980
if ( defined( 'MW_SUPPORTS_PARSERFIRSTCALLINIT' ) ) {
$wgHooks['ParserFirstCallInit'][] = 'efRawFile_Setup';
} else { // Otherwise do things the old fashioned way
$wgExtensionFunctions[] = 'efRawFile_Setup';
}
$wgHooks['LanguageGetMagic'][] = 'efRawFile_Magic';
$wgHooks['RawPageViewBeforeOutput'][] = 'fnRawFile_Strip';

</source>

===Setup function===
For the wiki parsing to create download links, the parser functions '''file''' and '''fileLink''' are equally treated, while '''fileAnchor''' will be simply left out. We also create a new tag '''file''' as explained [http://www.mediawiki.org/wiki/Manual:Tag_extensions here].
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Setup() {
global $wgParser;
$wgParser->setFunctionHook( 'file', 'efRawFile_Render' );
$wgParser->setFunctionHook( 'filelink', 'efRawFile_Render' );
$wgParser->setFunctionHook( 'fileanchor', 'efRawFile_Empty' );
$wgParser->setHook( 'file', 'efRawFile_FileTagRender' );
return true;
}

</source>

===Hook to initialize the magic words===
We add the magic words here: the first array element indicates if it is case sensitive, in this case it is not case sensitive. We could add extra elements to create synonyms for our parser function.
 Unless we return true, other parser functions extensions will not get loaded.
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Magic( &$magicWords, $langCode ) {
$magicWords['file'] = array( 0, 'file' );
$magicWords['filelink'] = array( 0, 'filelink' );
$magicWords['fileanchor'] = array( 0, 'fileanchor' );
return true;
}

</source>

===Parser functions of the magic words===
The transformation rule to replace link shortcuts to actual links for download, handling an optional local wiki page title if present.
 The input parameters are wikitext with templates expanded, the output should be wikitext too
 '''TODO''': what error to send out if there is no filename given?
 '''EDIT''': It seems that [http://svn.wikimedia.org/viewvc/mediawiki?view=rev&revision=27667 commit 27667] (1.11 -> 1.12) changed the default parser, which breaks the recursive parsing. Thanks to Tim Starling for helping me to get around the problem!
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Render( &$parser, $filename = '', $titleText = '') {
if( $titleText == '' )
$title = $parser->mTitle;
else
$title = Title::newFromText( $titleText );
//Don't expand templates or we'll lose our anchors {{#...}}
return $title->getFullURL( 'action=raw&anchor='.urlencode( $filename ) );
}

</source>
And the other one, just removing the anchors from the rendered wiki page.
 Curiously enough if the function doesn't exist at all the effect is exactly the same, MW doesn't throw any error.
 But let's keep things clean...
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Empty( &$parser, $filename = '') {
return '';
}

</source>

===Parser functions of the new tag <tt><file></tt>===
The transformation rule to replace <code><file></code> tag to actual links for download. The same parser function is used for both '''anchors''' and '''anchor-links'''. Since the link text may contain wiki text, we generate the link as wiki text that we ask the parser to parse again.
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_FileTagRender( $input, $args, $parser, $frame ) {
if( $args['title'] == '' )
$title = $parser->mTitle;
else
$title = Title::newFromText($parser->recursiveTagParse( $args['title'], $frame ));

//We expand templates, so <file> tag cannot be mixed with {{#fileanchor}} anchors
$link=$title->getFullURL( 'action=raw&templates=expand' );
if( $args['name'] != '' )
$link.='&name='.urlencode( $parser->recursiveTagParse( $args['name'], $frame ) );
if( $args['anchor'] != '' )
$link.='&anchor='.urlencode( $parser->recursiveTagParse( $args['anchor'], $frame ) );
if( $args['tag'] != '' )
$link.='&tag='.urlencode( $parser->recursiveTagParse( $args['tag'], $frame ) );

return $parser->recursiveTagParse( "[$link $input]", $frame );
}

</source>

===Hook to intercept the raw output===
This part of the code doesn't look that nice because we've to parse the raw wiki page ourselves to retrieve the code sections we want.

First we define a helper function that we will use to report error messages. This is simply done by replacing the content of the downloaded file with the error message and when necessary a copy of the raw text relevant to the error.
 '''TODO''': Cancel the file download header and return a proper error page
{{#fileanchor: RawFile.php}}
<source lang=php>
function fnRawFile_Strip_Error($msg,$out,&$text) {
$text=$msg;
if($out != '')
$text.="\nCandidate match: $out";
return true;
}

</source>

Next let's see if <code>?action=raw</code> was used in the context of this extension: in that case we receive the filename as GET parameter, otherwise we simply return from our extension with return value=true which means we authorize the raw display (originally the hook was created to add an authentication point)
{{#fileanchor: RawFile.php}}
<source lang=php>
function fnRawFile_Strip(&$rawPage, &$text) {
$filename=$_GET['name'];
$anchor=$_GET['anchor'];
// for backward compatibility, accept also URLs with parameter 'file'
if( $anchor=='' )
$anchor=$_GET['file'];
$tag=$_GET['tag'];
// Either anchor or name must be specified
if( $filename=='' )
$filename=$anchor;
if ( $filename=='' )
return true;
</source>
By default the downloadable file will still be handled by the ob_gzhandler session made by Mediawiki. To avoid output buffering and gzipping, one can uncomment the following line:
{{#fileanchor: RawFile.php}}
<source lang=php>
// Uncomment the following line to avoid output buffering and gzipping:
// wfResetOutputBuffers();
</source>
Raw action already set the headers with some client cache pragmas and is supposed to be displayed in the browser but in our case we want to make this "page" a downloadable file so we overwrite the headers which were defined and we add a few more, to ensure there is no caching on the client (it's very hard for the client to force a refresh on a file download, contrary to a web page) and to provide the adequate filename.
{{#fileanchor: RawFile.php}}
<source lang=php>
header("Content-disposition: attachment;filename={$filename}");
header("Content-type: application/octet-stream");
header("Content-Transfer-Encoding: binary");
header("Expires: 0");
header("Pragma: no-cache");
header("Cache-Control: no-store");
</source>
Then we'll strip the output, first we've to locate the anchors but there are anchors that could be protected in literal blocks like <code>nowiki</code>.
 So we'll mask the literal blocks before searching for the anchors (we mask with the same string length because we'll retrieve an offset that we will use on the initial string and offsets must match). This is done with the scary regex below:
* we use <code>!</code> instead of <code>/</code> as pattern indicator so that the pattern string is self-matching. This is necessary since we will apply the extension on this page as well.
* we use option <code>s</code> (multiline) and <code>e</code> (evaluate replace expression)
* Evaluated expression replaces all characters in the matched string with X's. However if there are single quote (<code>'</code>) in the matched string, they will be escaped with <code>\</code>. So we need to search for <code>\'|.</code>. The many back-slashes is because the expression is evaluated several times.
'''TODO''': should we care also of source, js, css, pre,... blocks?
{{#fileanchor: RawFile.php}}
<source lang=php>
$maskedtext=preg_replace_callback('!<nowiki>.*?</nowiki>!s',
function($m) { return ereg_replace(".","X",$m[0]); },
$text);
</source>
Now we can search for the anchors:
* If an anchor name is specified, we looked for '''all''' magic words <code><nowiki>{{#fileanchor:...}}</nowiki></code> or blocks with attribute <code>class="[someclass ]anchorname"</code>
* Otherwise we look for the first magic word <code><nowiki>{{#file:...}}</nowiki></code> with specified file name,
* And finally for the first <code><file></code> tag with the specified file name (no multiple blocks support)
And we free the memory used for the masked version
{{#fileanchor: RawFile.php}}
<source lang=php>
if (($anchor!='') && preg_match_all('/({{#fileanchor: *'.$anchor.' *}})|(<[^>]+ class *= *"([^"]*\w)?'.$anchor.'(\w[^"]*)?"[^>]*>)/i', $maskedtext, $matches, PREG_OFFSET_CAPTURE))
$offsets=$matches[0];
else if (preg_match_all('/{{#file: *'.$anchor.' *}}/i', $maskedtext, $matches, PREG_OFFSET_CAPTURE))
$offsets=array($matches[0][0]);
else if (preg_match_all('/<file( [^>]*)? name *= *"'.$filename.'"[^>]*>/i', $maskedtext, $matches, PREG_OFFSET_CAPTURE))
$offsets=array($matches[0][0]);
else {
// We didn't find our anchor
return fnRawFile_Strip_Error("ERROR - RawFile: anchor not found (anchor=$anchor, name=$filename, tag=$tag)","",$text);
}
unset($maskedtext);
</source>
$text is both input & output so we copy it and start with an empty output.
{{#fileanchor: RawFile.php}}
<source lang=php>
$textorig=$text;
$text='';
</source>
For each anchor found we've to isolate the content of the next block.
{{#fileanchor: RawFile.php}}
<source lang=php>
foreach ($offsets as $offset) {
</source>
We start from the position of the current anchor. If the tag name of the block attached to the anchor is not specified, we look for the first block that follows the anchor, excluding <code> </code> and <code><file></code> block. The search can be easily done with a regular expression, using the ''lookahead negative assertion'' <code>(?!br\b|file\b)</code> to exclude the tags to ignore. Note that we need to ignore the anchor-link block <code><file></code> since the anchor starts right before that tag, and so the regular expression would match the anchor-link block it that tag is not specifically excluded.
{{#fileanchor: RawFile.php}}
<source lang=php>
$out = substr($textorig, $offset[1]);
// If no tag specified, we take the first one
if ($tag == '')
{
// With a regex assertion, we can easily ignore 'br' and 'file' tags
if (!preg_match('/<((?!br\b|file\b)\w+\b)/', $out, $matches))
return fnRawFile_Strip_Error ("ERROR - RawFile: Can't find opening tag after anchor '$offset[0]' (anchor=$anchor, name=$filename, tag=$tag)",$out,$text);
$tag=$matches[1];
}
</source>
Now, we know the tag name of the block to download, either because it was already specified as a GET attribute in the URL, or because we've found it in the search above. Again, using a regular expression, we look for the first block matching the specified tag name that follows the current anchor, and extract the content of the blocks. Note the use of the regex option <code>/.../s</code> to tell the regex engine that the matched text can span on multiple lines (with that option, <code>.</code> does match any character or a newline character). Also, we skip the first carriage return after the opening tag, if any (with <code>\n?</code>).
{{#fileanchor: RawFile.php}}
<source lang=php>
// Find the first tag matching $tag, and return enclosed text
if (!preg_match('/<'.$tag.'( [^>]*)?>\n?(.*?)<\/'.$tag.'>/s', $out, $matches))
return fnRawFile_Strip_Error ("ERROR - RawFile: no closing '$tag' found after anchor '$offset[0]' (anchor=$anchor, name=$filename, tag=$tag)",$out,$text);
$text .= $matches[2];
}
</source>
No need to deal with a Content-Length header because Mediawiki will do it for us, moreover more properly than we could if the output is sent gzipped, which is the default.
 So that's it, $text contains our file!
{{#fileanchor: RawFile.php}}
<source lang=php>
return true;
}

</source>

===Credits===
There is an official way to register the extension in a Mediawiki installation, so that it will be visible on the [[Special:Version]] page.
 Let's say the extension is in the category of parser hooks even if there is also a hook on Raw action.
{{#fileanchor: RawFile.php}}
<source lang=php>
$wgExtensionCredits['parserhook'][] = array('name' => 'RawFile',
'version' => '0.5.1',
'author' => 'Philippe Teuwen, Michael Peeters',
'url' => 'http://www.mediawiki.org/wiki/Extension:RawFile',
// 'url' => 'http://wiki.yobi.be/wiki/Mediawiki_RawFile',
'description' => 'Downloads a RAW copy of <nowiki><tag>data</tag></nowiki> in a file '.
'Useful e.g. to download a script or a patch '.
'It also allows what is called [http://en.wikipedia.org/wiki/Literate_programming Literate Programming]');
}

?>
</source>
And finally registration of the extension at the Mediawiki website according to the [http://www.mediawiki.org/wiki/Manual:Extensions Extensions Manual].

So this extension has now [http://www.mediawiki.org/wiki/Extension:RawFile its own page on the official Mediawiki site].

==Installation==
Download [{{#filelink: RawFile.php}} RawFile.php] and save it under the MediaWiki directory as extensions/RawFile/RawFile.php

Add at the end of LocalSettings.php:
<source lang=php>
require_once("$IP/extensions/RawFile/RawFile.php");
</source>
==Status==
If you use the extension properly the code is fully functional but it's rather raw on error handling.
==ChangeLog==
'''0.5.1'''
* Integrate patch from Jani Uusitalo to recursively parse tag attribute values in case they contain [https://www.mediawiki.org/wiki/Help:Magic_words magic words] such as wiki templates or parameters. This is useful when using <code><file>...</file></code> in wiki templates.
* Prepare fix for <code>anchor not found</code> bug when using short notation <code><file name="filename" [tag="''tagname''"]>link text</file></code> in wiki templates. This still doesn't work because of the way MediaWiki (v1.22.1) expands templates in raw output (see [https://bugzilla.wikimedia.org/show_bug.cgi?id=61341 bugzilla 61341]).
** Work-around available. See bugzilla bug report, or example section above.
'''0.5'''
* Fix since PHP 5.5.0: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead. Thanks to Stephen Kent for reporting.
* '''WARNING''' you should upgrade ASAP as the previous versions 0.4 and 0.4.1 are vulnerable to remote PHP code injection!!! See [[Talk:Mediawiki_RawFile]] for more info.
* Fix for MediaWiki 1.22.1.
'''0.4.1'''
* Fix octet-stream MIME type bug which was affecting Epiphany & Opera 11. Thanks to Jani Uusitalo for reporting & [[User:Sicvolo|Sicvolo]] for finding the solution
'''0.4'''
* '''Anchors''' can be specified using html '''class''' attribute
* New syntax for '''Links''' and '''Anchor-links''':
:<code><nowiki><file [name="..."] [anchor="..."] [tag="..."] [title="..."] >Link text</file></nowiki></code>
* Support multiple files on the same page with same name (differentiated by their anchor name) or even common blocks in multiple files.
* Can specify the tag name of the block to download (to skip some irrelevant blocks when using an '''anchor-link''').
* Ignore '''<code> </code>''' tag.
* Some error reporting.
'''0.3'''
* Added optional parameter to <code>#fileLink</code> to indicate that the file is on another local wiki page
'''0.2'''
* Fix problem with Content-Length mismatch when transport is gzipped (default for Mediawiki if client supports it)
'''0.1'''
* Initial version

==Known bugs==
==Questions and feedback==
If you've any trouble, questions or suggestions, you can [[User:PhilippeTeuwen|contact me]].
==Known sites using the extension==



* [http://wiki.hpc.ufl.edu University of Florida Research Computing Wiki]
* [http://wiki.scribus.net Scribus wiki]
* [http://nfc-tools.org/ NFC-Tools]
* [https://smkent.net/wiki/ smkent.net], website of Stephen Kent
* [http://mummila.net/wiki/ mummila.net], website of Jani Uusitalo
* [http://www.richud.com/wiki/ richud.com], website of Richard Moore


See also wikiapiary to follow usage of extensions [https://wikiapiary.com/wiki/Extension:RawFile RawFile] and [https://wikiapiary.com/wiki/Extension:Fram Fram]

Mediawiki RawFile

2014-02-17T10:12:44Z

Fuujuhi: /* Example using templates */ Match with style of other examples

==Very short introduction==
Just have a look to the 2 [[Mediawiki_RawFile#Short_example|examples]] to see how to use the extension
 and to the [[Mediawiki_RawFile#Installation|Installation]] section to see how to install the extension in your [[MediaWiki]] server
==Introduction==
Originally the idea was to be able to download directly a portion of code as a file.
 I've numerous code examples in my wiki and I wanted an easy way to download them, easier than a copy/paste!
 But from there it was rather easy to get something very close to [http://en.wikipedia.org/wiki/Literate_programming literate programming] just by allowing multiple blocks referring to the same file, which will be concatenated together at download time.

* It must work with pre, nowiki, js, css, code, source, so let's make it general: take the tag that comes after the parser function we'll create and select data up to the closing tag.
* There are two distinct functionalities provided by the extension:
** the parser that will convert a magic word into a link to the download URL
** an extended ?action=raw that will strip the raw output to keep the desired code

==Syntax==
The extension introduces 3 elements:
;Anchor
: Used to flag that the next code block in the wiki text belongs to a specific file. The code block can be any wiki block (such as '''<code><pre></code>''', '''<code><code></code>''', '''<code><tt></code>''', '''<code><source></code>'''...). '''<code> </code>''' tags are ignored. Note that anchors are invisible in the wiki display.
;Link
: They are transformed by the extension into links that allows for downloading all blocks attached to a given anchor name.
;Anchor-link
: A shortcut notation mixing both an anchor and download link, handy for regular use, when a single code block is used and when the download link can be at the same position as the anchor.

The syntax is as follows. The syntax using tag <code><file></code> and tag attribute <code>class</code> is new since v0.4. Note that elements of both syntaxes can be mixed in a same page.
{| border="2" cellspacing="4" cellpadding="3" style="margin: 1em 1em 1em 0; background: #f9f9f9; border: 1px #aaaaaa solid; border-collapse: collapse; empty-cells:show;"
!width="80em" style="background: #8da7d6;"|Element!!style="background: #8da7d6;"|Syntax and description
|-
|'''Anchor'''
|<pre><nowiki>
{{#fileAnchor: anchorname}}
<pre class='anchorname'>...</pre>
<code class="anchorname">...</code>
<code class="cssclass anchorname">...</code>
...
</nowiki></pre>
Indicates that the next wiki block is attached to an anchor ''anchorname''. The content of that block will be downloaded (possibly appended with other blocks if there are several blocks attached to the same ''anchorname'') when a file link is clicked on. 
'''(since v0.4)''' To attach an anchor ''anchorname'' to a wiki block, simply add an attribute <code>class="anchorname"</code> to it. The extension supports multi-class specification, meaning that a same block can be associated to different files, and that the <code>class</code> attribute can still be used to specify custom CSS properties as in standard wiki text.
; ''anchorname''
; class="''anchorname''"
: The name of the anchor to which the wiki block is attached
|-
|'''Link'''
|<pre><nowiki>
[{{#fileLink: anchorname}} link text]
[{{#fileLink: anchorname|pagetitle}} link text]
<file anchor="anchorname" [name="filename"] [title="pagetitle"]>link text</file>
</nowiki></pre>
Creates a link to download all blocks that are attached to an anchor ''anchorname''.
;''anchorname''
;anchor="''anchorname''"
: The name of the anchor to look for. All blocks attached to an anchor ''anchorname'' will be downloaded.
;name="''filename''"
:''Optional'' - Specifies the name of the file to download. If absent, ''anchorname'' is then used as the name of the downloaded file.
; ''pagetitle''
;title="''pagetitle''"
: ''Optional'' - Indicates that the blocks to download are on the wiki page titled ''pagetitle''. If absent, blocks are looked for on the current page.
; ''link text''
: The text of the link to display.
|-
|'''Anchor-link'''
|<pre><nowiki>
[{{#file: filename}} link text]
<file name="filename" [tag="''tagname''"]>link text</file>
</nowiki></pre>
Creates a link to download the next wiki block as a file named ''filename''. 
'''(since v0.4)''' The attribute <code>tag</code> can be used to specify the ''tagname'' of the block to download. 
; ''filename''
; name="''filename''"
: The name of the file to download.
;tag="''tagname''"
:''Optional'' - When set, the extension only looks for blocks whose name matches the given ''tagname''. This attribute is particularly useful when there are some irrelevant blocks between the '''anchor-link''' and the block you want to download. If absent, the first encountered block following the anchor is downloaded.
; ''link text''
: The text of the link to display.
|}

===Short example===
The extension works with any block such as pre, nowiki, js, css, code, source,...
 This example is using the syntax highlighting <nowiki><source></nowiki> tag provided by [http://www.mediawiki.org/wiki/Extension:SyntaxHighlight_GeSHi SyntaxHighlight extension] (using [http://qbnz.com/highlighter/ GeSHi Highlighter])
 If you didn't install that extension on your MediaWiki, you can try the example by using <nowiki><pre></nowiki> instead of <nowiki><source></nowiki>.

<pre><nowiki>Let's save the following code [{{#file: myscript.sh}} as myscript.sh]
<source lang=bash>
#!/bin/bash

echo 'Hello world!'
exit 0
</source></nowiki>
</pre>
will give:
----
Let's save the following code [{{#file: myscript.sh}} as myscript.sh]
<source lang=bash>
#!/bin/bash

echo 'Hello world!'
exit 0
</source>

===Complete example===
And a full example with anchors & link:

<pre><nowiki>
Let's start with the Bash usual header:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
#!/bin/bash
</source>
Then we'll display a welcome message:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
echo 'Welcome on Earth!'
</source>
And we finally exit cleanly:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
exit 0
</source>
[{{#filelink: myotherscript.sh}} myotherscript.sh is now available for download below the code]
</nowiki></pre>
will give:
----
Let's start with the Bash usual header:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
#!/bin/bash
</source>
Then we'll display a welcome message:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
echo 'Welcome on Earth!'
</source>
And we finally exit cleanly:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
exit 0
</source>
[{{#filelink: myotherscript.sh}} myotherscript.sh is now available for download below the code]

=== Example using templates ===

The new syntax <code><file name="..." [tag="..."]>...</file></code> allows for using the RawFile extension in templates as well.

The example below uses the template [[Template:Rawfiledownloadexample]] to avoid duplication between the file name in the text and as a parameter in the <code><file></code> tag.

See the template source for instructions on how to create the template.

<pre>
{{Rawfiledownloadexample|name=myfile.txt|content=Once upon a time
There was a Tag
Tag was clickable
And clicked it was}}
</pre>

The code above gives
<hr/>
{{Rawfiledownloadexample|name=myfile.txt|content=Once upon a time
There was a Tag
Tag was clickable
And clicked it was}}

==The code (the ultimate example)==
Which you can of course download just by following [{{#filelink: RawFile.php}} this link :-)]

So let's explain a bit the code in a Literate Programming way...
===Hooks===
First some hooks for our functions...

We will create:
* a [http://www.mediawiki.org/wiki/Manual:Parser_functions Parser Function] (see also [http://meta.wikimedia.org/wiki/Help:Parser_function here]), with help of
** [http://www.mediawiki.org/wiki/Manual:%24wgExtensionFunctions $wgExtensionFunctions] or [http://www.mediawiki.org/wiki/Manual:Hooks/ParserFirstCallInit ParserFirstCallInit global hook] to define the setup function
** [http://www.mediawiki.org/wiki/Manual:Magic_words Magic Words]
** [http://www.mediawiki.org/wiki/Manual:Tag_extensions Tag extensions]
** [http://www.mediawiki.org/wiki/Manual:Hooks/LanguageGetMagic LanguageGetMagic] hook to initialize the magic words
* a [http://www.mediawiki.org/wiki/Manual:Hooks/RawPageViewBeforeOutput RawPageViewBeforeOutput] hook to intercept the raw output

{{#fileanchor: RawFile.php}}
<source lang=php>
<?php

if (defined('MEDIAWIKI')) {

//Avoid unstubbing $wgParser on setHook() too early on modern (1.12+) MW versions, as per r35980
if ( defined( 'MW_SUPPORTS_PARSERFIRSTCALLINIT' ) ) {
$wgHooks['ParserFirstCallInit'][] = 'efRawFile_Setup';
} else { // Otherwise do things the old fashioned way
$wgExtensionFunctions[] = 'efRawFile_Setup';
}
$wgHooks['LanguageGetMagic'][] = 'efRawFile_Magic';
$wgHooks['RawPageViewBeforeOutput'][] = 'fnRawFile_Strip';

</source>

===Setup function===
For the wiki parsing to create download links, the parser functions '''file''' and '''fileLink''' are equally treated, while '''fileAnchor''' will be simply left out. We also create a new tag '''file''' as explained [http://www.mediawiki.org/wiki/Manual:Tag_extensions here].
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Setup() {
global $wgParser;
$wgParser->setFunctionHook( 'file', 'efRawFile_Render' );
$wgParser->setFunctionHook( 'filelink', 'efRawFile_Render' );
$wgParser->setFunctionHook( 'fileanchor', 'efRawFile_Empty' );
$wgParser->setHook( 'file', 'efRawFile_FileTagRender' );
return true;
}

</source>

===Hook to initialize the magic words===
We add the magic words here: the first array element indicates if it is case sensitive, in this case it is not case sensitive. We could add extra elements to create synonyms for our parser function.
 Unless we return true, other parser functions extensions will not get loaded.
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Magic( &$magicWords, $langCode ) {
$magicWords['file'] = array( 0, 'file' );
$magicWords['filelink'] = array( 0, 'filelink' );
$magicWords['fileanchor'] = array( 0, 'fileanchor' );
return true;
}

</source>

===Parser functions of the magic words===
The transformation rule to replace link shortcuts to actual links for download, handling an optional local wiki page title if present.
 The input parameters are wikitext with templates expanded, the output should be wikitext too
 '''TODO''': what error to send out if there is no filename given?
 '''EDIT''': It seems that [http://svn.wikimedia.org/viewvc/mediawiki?view=rev&revision=27667 commit 27667] (1.11 -> 1.12) changed the default parser, which breaks the recursive parsing. Thanks to Tim Starling for helping me to get around the problem!
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Render( &$parser, $filename = '', $titleText = '') {
if( $titleText == '' )
$title = $parser->mTitle;
else
$title = Title::newFromText( $titleText );
//Don't expand templates or we'll lose our anchors {{#...}}
return $title->getFullURL( 'action=raw&anchor='.urlencode( $filename ) );
}

</source>
And the other one, just removing the anchors from the rendered wiki page.
 Curiously enough if the function doesn't exist at all the effect is exactly the same, MW doesn't throw any error.
 But let's keep things clean...
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Empty( &$parser, $filename = '') {
return '';
}

</source>

===Parser functions of the new tag <tt><file></tt>===
The transformation rule to replace <code><file></code> tag to actual links for download. The same parser function is used for both '''anchors''' and '''anchor-links'''. Since the link text may contain wiki text, we generate the link as wiki text that we ask the parser to parse again.
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_FileTagRender( $input, $args, $parser, $frame ) {
if( $args['title'] == '' )
$title = $parser->mTitle;
else
$title = Title::newFromText($parser->recursiveTagParse( $args['title'], $frame ));

//We expand templates, so <file> tag cannot be mixed with {{#fileanchor}} anchors
$link=$title->getFullURL( 'action=raw&templates=expand' );
if( $args['name'] != '' )
$link.='&name='.urlencode( $parser->recursiveTagParse( $args['name'], $frame ) );
if( $args['anchor'] != '' )
$link.='&anchor='.urlencode( $parser->recursiveTagParse( $args['anchor'], $frame ) );
if( $args['tag'] != '' )
$link.='&tag='.urlencode( $parser->recursiveTagParse( $args['tag'], $frame ) );

return $parser->recursiveTagParse( "[$link $input]", $frame );
}

</source>

===Hook to intercept the raw output===
This part of the code doesn't look that nice because we've to parse the raw wiki page ourselves to retrieve the code sections we want.

First we define a helper function that we will use to report error messages. This is simply done by replacing the content of the downloaded file with the error message and when necessary a copy of the raw text relevant to the error.
 '''TODO''': Cancel the file download header and return a proper error page
{{#fileanchor: RawFile.php}}
<source lang=php>
function fnRawFile_Strip_Error($msg,$out,&$text) {
$text=$msg;
if($out != '')
$text.="\nCandidate match: $out";
return true;
}

</source>

Next let's see if <code>?action=raw</code> was used in the context of this extension: in that case we receive the filename as GET parameter, otherwise we simply return from our extension with return value=true which means we authorize the raw display (originally the hook was created to add an authentication point)
{{#fileanchor: RawFile.php}}
<source lang=php>
function fnRawFile_Strip(&$rawPage, &$text) {
$filename=$_GET['name'];
$anchor=$_GET['anchor'];
// for backward compatibility, accept also URLs with parameter 'file'
if( $anchor=='' )
$anchor=$_GET['file'];
$tag=$_GET['tag'];
// Either anchor or name must be specified
if( $filename=='' )
$filename=$anchor;
if ( $filename=='' )
return true;
</source>
By default the downloadable file will still be handled by the ob_gzhandler session made by Mediawiki. To avoid output buffering and gzipping, one can uncomment the following line:
{{#fileanchor: RawFile.php}}
<source lang=php>
// Uncomment the following line to avoid output buffering and gzipping:
// wfResetOutputBuffers();
</source>
Raw action already set the headers with some client cache pragmas and is supposed to be displayed in the browser but in our case we want to make this "page" a downloadable file so we overwrite the headers which were defined and we add a few more, to ensure there is no caching on the client (it's very hard for the client to force a refresh on a file download, contrary to a web page) and to provide the adequate filename.
{{#fileanchor: RawFile.php}}
<source lang=php>
header("Content-disposition: attachment;filename={$filename}");
header("Content-type: application/octet-stream");
header("Content-Transfer-Encoding: binary");
header("Expires: 0");
header("Pragma: no-cache");
header("Cache-Control: no-store");
</source>
Then we'll strip the output, first we've to locate the anchors but there are anchors that could be protected in literal blocks like <code>nowiki</code>.
 So we'll mask the literal blocks before searching for the anchors (we mask with the same string length because we'll retrieve an offset that we will use on the initial string and offsets must match). This is done with the scary regex below:
* we use <code>!</code> instead of <code>/</code> as pattern indicator so that the pattern string is self-matching. This is necessary since we will apply the extension on this page as well.
* we use option <code>s</code> (multiline) and <code>e</code> (evaluate replace expression)
* Evaluated expression replaces all characters in the matched string with X's. However if there are single quote (<code>'</code>) in the matched string, they will be escaped with <code>\</code>. So we need to search for <code>\'|.</code>. The many back-slashes is because the expression is evaluated several times.
'''TODO''': should we care also of source, js, css, pre,... blocks?
{{#fileanchor: RawFile.php}}
<source lang=php>
$maskedtext=preg_replace_callback('!<nowiki>.*?</nowiki>!s',
function($m) { return ereg_replace(".","X",$m[0]); },
$text);
</source>
Now we can search for the anchors:
* If an anchor name is specified, we looked for '''all''' magic words <code><nowiki>{{#fileanchor:...}}</nowiki></code> or blocks with attribute <code>class="[someclass ]anchorname"</code>
* Otherwise we look for the first magic word <code><nowiki>{{#file:...}}</nowiki></code> with specified file name,
* And finally for the first <code><file></code> tag with the specified file name (no multiple blocks support)
And we free the memory used for the masked version
{{#fileanchor: RawFile.php}}
<source lang=php>
if (($anchor!='') && preg_match_all('/({{#fileanchor: *'.$anchor.' *}})|(<[^>]+ class *= *"([^"]*\w)?'.$anchor.'(\w[^"]*)?"[^>]*>)/i', $maskedtext, $matches, PREG_OFFSET_CAPTURE))
$offsets=$matches[0];
else if (preg_match_all('/{{#file: *'.$anchor.' *}}/i', $maskedtext, $matches, PREG_OFFSET_CAPTURE))
$offsets=array($matches[0][0]);
else if (preg_match_all('/<file( [^>]*)? name *= *"'.$filename.'"[^>]*>/i', $maskedtext, $matches, PREG_OFFSET_CAPTURE))
$offsets=array($matches[0][0]);
else {
// We didn't find our anchor
return fnRawFile_Strip_Error("ERROR - RawFile: anchor not found (anchor=$anchor, name=$filename, tag=$tag)","",$text);
}
unset($maskedtext);
</source>
$text is both input & output so we copy it and start with an empty output.
{{#fileanchor: RawFile.php}}
<source lang=php>
$textorig=$text;
$text='';
</source>
For each anchor found we've to isolate the content of the next block.
{{#fileanchor: RawFile.php}}
<source lang=php>
foreach ($offsets as $offset) {
</source>
We start from the position of the current anchor. If the tag name of the block attached to the anchor is not specified, we look for the first block that follows the anchor, excluding <code> </code> and <code><file></code> block. The search can be easily done with a regular expression, using the ''lookahead negative assertion'' <code>(?!br\b|file\b)</code> to exclude the tags to ignore. Note that we need to ignore the anchor-link block <code><file></code> since the anchor starts right before that tag, and so the regular expression would match the anchor-link block it that tag is not specifically excluded.
{{#fileanchor: RawFile.php}}
<source lang=php>
$out = substr($textorig, $offset[1]);
// If no tag specified, we take the first one
if ($tag == '')
{
// With a regex assertion, we can easily ignore 'br' and 'file' tags
if (!preg_match('/<((?!br\b|file\b)\w+\b)/', $out, $matches))
return fnRawFile_Strip_Error ("ERROR - RawFile: Can't find opening tag after anchor '$offset[0]' (anchor=$anchor, name=$filename, tag=$tag)",$out,$text);
$tag=$matches[1];
}
</source>
Now, we know the tag name of the block to download, either because it was already specified as a GET attribute in the URL, or because we've found it in the search above. Again, using a regular expression, we look for the first block matching the specified tag name that follows the current anchor, and extract the content of the blocks. Note the use of the regex option <code>/.../s</code> to tell the regex engine that the matched text can span on multiple lines (with that option, <code>.</code> does match any character or a newline character). Also, we skip the first carriage return after the opening tag, if any (with <code>\n?</code>).
{{#fileanchor: RawFile.php}}
<source lang=php>
// Find the first tag matching $tag, and return enclosed text
if (!preg_match('/<'.$tag.'( [^>]*)?>\n?(.*?)<\/'.$tag.'>/s', $out, $matches))
return fnRawFile_Strip_Error ("ERROR - RawFile: no closing '$tag' found after anchor '$offset[0]' (anchor=$anchor, name=$filename, tag=$tag)",$out,$text);
$text .= $matches[2];
}
</source>
No need to deal with a Content-Length header because Mediawiki will do it for us, moreover more properly than we could if the output is sent gzipped, which is the default.
 So that's it, $text contains our file!
{{#fileanchor: RawFile.php}}
<source lang=php>
return true;
}

</source>

===Credits===
There is an official way to register the extension in a Mediawiki installation, so that it will be visible on the [[Special:Version]] page.
 Let's say the extension is in the category of parser hooks even if there is also a hook on Raw action.
{{#fileanchor: RawFile.php}}
<source lang=php>
$wgExtensionCredits['parserhook'][] = array('name' => 'RawFile',
'version' => '0.5.1',
'author' => 'Philippe Teuwen, Michael Peeters',
'url' => 'http://www.mediawiki.org/wiki/Extension:RawFile',
// 'url' => 'http://wiki.yobi.be/wiki/Mediawiki_RawFile',
'description' => 'Downloads a RAW copy of <nowiki><tag>data</tag></nowiki> in a file '.
'Useful e.g. to download a script or a patch '.
'It also allows what is called [http://en.wikipedia.org/wiki/Literate_programming Literate Programming]');
}

?>
</source>
And finally registration of the extension at the Mediawiki website according to the [http://www.mediawiki.org/wiki/Manual:Extensions Extensions Manual].

So this extension has now [http://www.mediawiki.org/wiki/Extension:RawFile its own page on the official Mediawiki site].

==Installation==
Download [{{#filelink: RawFile.php}} RawFile.php] and save it under the MediaWiki directory as extensions/RawFile/RawFile.php

Add at the end of LocalSettings.php:
<source lang=php>
require_once("$IP/extensions/RawFile/RawFile.php");
</source>
==Status==
If you use the extension properly the code is fully functional but it's rather raw on error handling.
==ChangeLog==
'''0.5.1'''
* Integrate patch from Jani Uusitalo to recursively parse tag attribute values in case they contain [https://www.mediawiki.org/wiki/Help:Magic_words magic words] such as wiki templates or parameters. This is useful when using <code><file>...</file></code> in wiki templates.
* Prepare fix for <code>anchor not found</code> bug when using short notation <code><file name="filename" [tag="''tagname''"]>link text</file></code> in wiki templates. This still doesn't work because of the way MediaWiki (v1.22.1) expands templates in raw output (see [https://bugzilla.wikimedia.org/show_bug.cgi?id=61341 bugzilla 61341]).
'''0.5'''
* Fix since PHP 5.5.0: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead. Thanks to Stephen Kent for reporting.
* '''WARNING''' you should upgrade ASAP as the previous versions 0.4 and 0.4.1 are vulnerable to remote PHP code injection!!! See [[Talk:Mediawiki_RawFile]] for more info.
* Fix for MediaWiki 1.22.1.
'''0.4.1'''
* Fix octet-stream MIME type bug which was affecting Epiphany & Opera 11. Thanks to Jani Uusitalo for reporting & [[User:Sicvolo|Sicvolo]] for finding the solution
'''0.4'''
* '''Anchors''' can be specified using html '''class''' attribute
* New syntax for '''Links''' and '''Anchor-links''':
:<code><nowiki><file [name="..."] [anchor="..."] [tag="..."] [title="..."] >Link text</file></nowiki></code>
* Support multiple files on the same page with same name (differentiated by their anchor name) or even common blocks in multiple files.
* Can specify the tag name of the block to download (to skip some irrelevant blocks when using an '''anchor-link''').
* Ignore '''<code> </code>''' tag.
* Some error reporting.
'''0.3'''
* Added optional parameter to <code>#fileLink</code> to indicate that the file is on another local wiki page
'''0.2'''
* Fix problem with Content-Length mismatch when transport is gzipped (default for Mediawiki if client supports it)
'''0.1'''
* Initial version

==Known bugs==
==Questions and feedback==
If you've any trouble, questions or suggestions, you can [[User:PhilippeTeuwen|contact me]].
==Known sites using the extension==



* [http://wiki.hpc.ufl.edu University of Florida Research Computing Wiki]
* [http://wiki.scribus.net Scribus wiki]
* [http://nfc-tools.org/ NFC-Tools]
* [https://smkent.net/wiki/ smkent.net], website of Stephen Kent
* [http://mummila.net/wiki/ mummila.net], website of Jani Uusitalo
* [http://www.richud.com/wiki/ richud.com], website of Richard Moore


See also wikiapiary to follow usage of extensions [https://wikiapiary.com/wiki/Extension:RawFile RawFile] and [https://wikiapiary.com/wiki/Extension:Fram Fram]

Mediawiki RawFile

2014-02-17T10:08:28Z

Fuujuhi: /* Example using templates */ Remove redundant |}

==Very short introduction==
Just have a look to the 2 [[Mediawiki_RawFile#Short_example|examples]] to see how to use the extension
 and to the [[Mediawiki_RawFile#Installation|Installation]] section to see how to install the extension in your [[MediaWiki]] server
==Introduction==
Originally the idea was to be able to download directly a portion of code as a file.
 I've numerous code examples in my wiki and I wanted an easy way to download them, easier than a copy/paste!
 But from there it was rather easy to get something very close to [http://en.wikipedia.org/wiki/Literate_programming literate programming] just by allowing multiple blocks referring to the same file, which will be concatenated together at download time.

* It must work with pre, nowiki, js, css, code, source, so let's make it general: take the tag that comes after the parser function we'll create and select data up to the closing tag.
* There are two distinct functionalities provided by the extension:
** the parser that will convert a magic word into a link to the download URL
** an extended ?action=raw that will strip the raw output to keep the desired code

==Syntax==
The extension introduces 3 elements:
;Anchor
: Used to flag that the next code block in the wiki text belongs to a specific file. The code block can be any wiki block (such as '''<code><pre></code>''', '''<code><code></code>''', '''<code><tt></code>''', '''<code><source></code>'''...). '''<code> </code>''' tags are ignored. Note that anchors are invisible in the wiki display.
;Link
: They are transformed by the extension into links that allows for downloading all blocks attached to a given anchor name.
;Anchor-link
: A shortcut notation mixing both an anchor and download link, handy for regular use, when a single code block is used and when the download link can be at the same position as the anchor.

The syntax is as follows. The syntax using tag <code><file></code> and tag attribute <code>class</code> is new since v0.4. Note that elements of both syntaxes can be mixed in a same page.
{| border="2" cellspacing="4" cellpadding="3" style="margin: 1em 1em 1em 0; background: #f9f9f9; border: 1px #aaaaaa solid; border-collapse: collapse; empty-cells:show;"
!width="80em" style="background: #8da7d6;"|Element!!style="background: #8da7d6;"|Syntax and description
|-
|'''Anchor'''
|<pre><nowiki>
{{#fileAnchor: anchorname}}
<pre class='anchorname'>...</pre>
<code class="anchorname">...</code>
<code class="cssclass anchorname">...</code>
...
</nowiki></pre>
Indicates that the next wiki block is attached to an anchor ''anchorname''. The content of that block will be downloaded (possibly appended with other blocks if there are several blocks attached to the same ''anchorname'') when a file link is clicked on. 
'''(since v0.4)''' To attach an anchor ''anchorname'' to a wiki block, simply add an attribute <code>class="anchorname"</code> to it. The extension supports multi-class specification, meaning that a same block can be associated to different files, and that the <code>class</code> attribute can still be used to specify custom CSS properties as in standard wiki text.
; ''anchorname''
; class="''anchorname''"
: The name of the anchor to which the wiki block is attached
|-
|'''Link'''
|<pre><nowiki>
[{{#fileLink: anchorname}} link text]
[{{#fileLink: anchorname|pagetitle}} link text]
<file anchor="anchorname" [name="filename"] [title="pagetitle"]>link text</file>
</nowiki></pre>
Creates a link to download all blocks that are attached to an anchor ''anchorname''.
;''anchorname''
;anchor="''anchorname''"
: The name of the anchor to look for. All blocks attached to an anchor ''anchorname'' will be downloaded.
;name="''filename''"
:''Optional'' - Specifies the name of the file to download. If absent, ''anchorname'' is then used as the name of the downloaded file.
; ''pagetitle''
;title="''pagetitle''"
: ''Optional'' - Indicates that the blocks to download are on the wiki page titled ''pagetitle''. If absent, blocks are looked for on the current page.
; ''link text''
: The text of the link to display.
|-
|'''Anchor-link'''
|<pre><nowiki>
[{{#file: filename}} link text]
<file name="filename" [tag="''tagname''"]>link text</file>
</nowiki></pre>
Creates a link to download the next wiki block as a file named ''filename''. 
'''(since v0.4)''' The attribute <code>tag</code> can be used to specify the ''tagname'' of the block to download. 
; ''filename''
; name="''filename''"
: The name of the file to download.
;tag="''tagname''"
:''Optional'' - When set, the extension only looks for blocks whose name matches the given ''tagname''. This attribute is particularly useful when there are some irrelevant blocks between the '''anchor-link''' and the block you want to download. If absent, the first encountered block following the anchor is downloaded.
; ''link text''
: The text of the link to display.
|}

===Short example===
The extension works with any block such as pre, nowiki, js, css, code, source,...
 This example is using the syntax highlighting <nowiki><source></nowiki> tag provided by [http://www.mediawiki.org/wiki/Extension:SyntaxHighlight_GeSHi SyntaxHighlight extension] (using [http://qbnz.com/highlighter/ GeSHi Highlighter])
 If you didn't install that extension on your MediaWiki, you can try the example by using <nowiki><pre></nowiki> instead of <nowiki><source></nowiki>.

<pre><nowiki>Let's save the following code [{{#file: myscript.sh}} as myscript.sh]
<source lang=bash>
#!/bin/bash

echo 'Hello world!'
exit 0
</source></nowiki>
</pre>
will give:
----
Let's save the following code [{{#file: myscript.sh}} as myscript.sh]
<source lang=bash>
#!/bin/bash

echo 'Hello world!'
exit 0
</source>

===Complete example===
And a full example with anchors & link:

<pre><nowiki>
Let's start with the Bash usual header:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
#!/bin/bash
</source>
Then we'll display a welcome message:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
echo 'Welcome on Earth!'
</source>
And we finally exit cleanly:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
exit 0
</source>
[{{#filelink: myotherscript.sh}} myotherscript.sh is now available for download below the code]
</nowiki></pre>
will give:
----
Let's start with the Bash usual header:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
#!/bin/bash
</source>
Then we'll display a welcome message:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
echo 'Welcome on Earth!'
</source>
And we finally exit cleanly:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
exit 0
</source>
[{{#filelink: myotherscript.sh}} myotherscript.sh is now available for download below the code]

=== Example using templates ===

The new syntax <code><file name="..." [tag="..."]>...</file></code> allows for using the RawFile extension in templates as well.

An example of use is to create a template to avoid duplication between the file name in the text and as a parameter in the <code><file></code> tag. The example below uses the template [[Template:Rawfiledownloadexample]]. See the template source for instructions on how to create the template.

{| class=wikitable
|-
|
<pre>
{{Rawfiledownloadexample|name=myfile.txt|content=Once upon a time
There was a Tag
Tag was clickable
And clicked it was}}
</pre>
|
{{Rawfiledownloadexample|name=myfile.txt|content=Once upon a time
There was a Tag
Tag was clickable
And clicked it was}}
|}

==The code (the ultimate example)==
Which you can of course download just by following [{{#filelink: RawFile.php}} this link :-)]

So let's explain a bit the code in a Literate Programming way...
===Hooks===
First some hooks for our functions...

We will create:
* a [http://www.mediawiki.org/wiki/Manual:Parser_functions Parser Function] (see also [http://meta.wikimedia.org/wiki/Help:Parser_function here]), with help of
** [http://www.mediawiki.org/wiki/Manual:%24wgExtensionFunctions $wgExtensionFunctions] or [http://www.mediawiki.org/wiki/Manual:Hooks/ParserFirstCallInit ParserFirstCallInit global hook] to define the setup function
** [http://www.mediawiki.org/wiki/Manual:Magic_words Magic Words]
** [http://www.mediawiki.org/wiki/Manual:Tag_extensions Tag extensions]
** [http://www.mediawiki.org/wiki/Manual:Hooks/LanguageGetMagic LanguageGetMagic] hook to initialize the magic words
* a [http://www.mediawiki.org/wiki/Manual:Hooks/RawPageViewBeforeOutput RawPageViewBeforeOutput] hook to intercept the raw output

{{#fileanchor: RawFile.php}}
<source lang=php>
<?php

if (defined('MEDIAWIKI')) {

//Avoid unstubbing $wgParser on setHook() too early on modern (1.12+) MW versions, as per r35980
if ( defined( 'MW_SUPPORTS_PARSERFIRSTCALLINIT' ) ) {
$wgHooks['ParserFirstCallInit'][] = 'efRawFile_Setup';
} else { // Otherwise do things the old fashioned way
$wgExtensionFunctions[] = 'efRawFile_Setup';
}
$wgHooks['LanguageGetMagic'][] = 'efRawFile_Magic';
$wgHooks['RawPageViewBeforeOutput'][] = 'fnRawFile_Strip';

</source>

===Setup function===
For the wiki parsing to create download links, the parser functions '''file''' and '''fileLink''' are equally treated, while '''fileAnchor''' will be simply left out. We also create a new tag '''file''' as explained [http://www.mediawiki.org/wiki/Manual:Tag_extensions here].
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Setup() {
global $wgParser;
$wgParser->setFunctionHook( 'file', 'efRawFile_Render' );
$wgParser->setFunctionHook( 'filelink', 'efRawFile_Render' );
$wgParser->setFunctionHook( 'fileanchor', 'efRawFile_Empty' );
$wgParser->setHook( 'file', 'efRawFile_FileTagRender' );
return true;
}

</source>

===Hook to initialize the magic words===
We add the magic words here: the first array element indicates if it is case sensitive, in this case it is not case sensitive. We could add extra elements to create synonyms for our parser function.
 Unless we return true, other parser functions extensions will not get loaded.
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Magic( &$magicWords, $langCode ) {
$magicWords['file'] = array( 0, 'file' );
$magicWords['filelink'] = array( 0, 'filelink' );
$magicWords['fileanchor'] = array( 0, 'fileanchor' );
return true;
}

</source>

===Parser functions of the magic words===
The transformation rule to replace link shortcuts to actual links for download, handling an optional local wiki page title if present.
 The input parameters are wikitext with templates expanded, the output should be wikitext too
 '''TODO''': what error to send out if there is no filename given?
 '''EDIT''': It seems that [http://svn.wikimedia.org/viewvc/mediawiki?view=rev&revision=27667 commit 27667] (1.11 -> 1.12) changed the default parser, which breaks the recursive parsing. Thanks to Tim Starling for helping me to get around the problem!
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Render( &$parser, $filename = '', $titleText = '') {
if( $titleText == '' )
$title = $parser->mTitle;
else
$title = Title::newFromText( $titleText );
//Don't expand templates or we'll lose our anchors {{#...}}
return $title->getFullURL( 'action=raw&anchor='.urlencode( $filename ) );
}

</source>
And the other one, just removing the anchors from the rendered wiki page.
 Curiously enough if the function doesn't exist at all the effect is exactly the same, MW doesn't throw any error.
 But let's keep things clean...
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Empty( &$parser, $filename = '') {
return '';
}

</source>

===Parser functions of the new tag <tt><file></tt>===
The transformation rule to replace <code><file></code> tag to actual links for download. The same parser function is used for both '''anchors''' and '''anchor-links'''. Since the link text may contain wiki text, we generate the link as wiki text that we ask the parser to parse again.
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_FileTagRender( $input, $args, $parser, $frame ) {
if( $args['title'] == '' )
$title = $parser->mTitle;
else
$title = Title::newFromText($parser->recursiveTagParse( $args['title'], $frame ));

//We expand templates, so <file> tag cannot be mixed with {{#fileanchor}} anchors
$link=$title->getFullURL( 'action=raw&templates=expand' );
if( $args['name'] != '' )
$link.='&name='.urlencode( $parser->recursiveTagParse( $args['name'], $frame ) );
if( $args['anchor'] != '' )
$link.='&anchor='.urlencode( $parser->recursiveTagParse( $args['anchor'], $frame ) );
if( $args['tag'] != '' )
$link.='&tag='.urlencode( $parser->recursiveTagParse( $args['tag'], $frame ) );

return $parser->recursiveTagParse( "[$link $input]", $frame );
}

</source>

===Hook to intercept the raw output===
This part of the code doesn't look that nice because we've to parse the raw wiki page ourselves to retrieve the code sections we want.

First we define a helper function that we will use to report error messages. This is simply done by replacing the content of the downloaded file with the error message and when necessary a copy of the raw text relevant to the error.
 '''TODO''': Cancel the file download header and return a proper error page
{{#fileanchor: RawFile.php}}
<source lang=php>
function fnRawFile_Strip_Error($msg,$out,&$text) {
$text=$msg;
if($out != '')
$text.="\nCandidate match: $out";
return true;
}

</source>

Next let's see if <code>?action=raw</code> was used in the context of this extension: in that case we receive the filename as GET parameter, otherwise we simply return from our extension with return value=true which means we authorize the raw display (originally the hook was created to add an authentication point)
{{#fileanchor: RawFile.php}}
<source lang=php>
function fnRawFile_Strip(&$rawPage, &$text) {
$filename=$_GET['name'];
$anchor=$_GET['anchor'];
// for backward compatibility, accept also URLs with parameter 'file'
if( $anchor=='' )
$anchor=$_GET['file'];
$tag=$_GET['tag'];
// Either anchor or name must be specified
if( $filename=='' )
$filename=$anchor;
if ( $filename=='' )
return true;
</source>
By default the downloadable file will still be handled by the ob_gzhandler session made by Mediawiki. To avoid output buffering and gzipping, one can uncomment the following line:
{{#fileanchor: RawFile.php}}
<source lang=php>
// Uncomment the following line to avoid output buffering and gzipping:
// wfResetOutputBuffers();
</source>
Raw action already set the headers with some client cache pragmas and is supposed to be displayed in the browser but in our case we want to make this "page" a downloadable file so we overwrite the headers which were defined and we add a few more, to ensure there is no caching on the client (it's very hard for the client to force a refresh on a file download, contrary to a web page) and to provide the adequate filename.
{{#fileanchor: RawFile.php}}
<source lang=php>
header("Content-disposition: attachment;filename={$filename}");
header("Content-type: application/octet-stream");
header("Content-Transfer-Encoding: binary");
header("Expires: 0");
header("Pragma: no-cache");
header("Cache-Control: no-store");
</source>
Then we'll strip the output, first we've to locate the anchors but there are anchors that could be protected in literal blocks like <code>nowiki</code>.
 So we'll mask the literal blocks before searching for the anchors (we mask with the same string length because we'll retrieve an offset that we will use on the initial string and offsets must match). This is done with the scary regex below:
* we use <code>!</code> instead of <code>/</code> as pattern indicator so that the pattern string is self-matching. This is necessary since we will apply the extension on this page as well.
* we use option <code>s</code> (multiline) and <code>e</code> (evaluate replace expression)
* Evaluated expression replaces all characters in the matched string with X's. However if there are single quote (<code>'</code>) in the matched string, they will be escaped with <code>\</code>. So we need to search for <code>\'|.</code>. The many back-slashes is because the expression is evaluated several times.
'''TODO''': should we care also of source, js, css, pre,... blocks?
{{#fileanchor: RawFile.php}}
<source lang=php>
$maskedtext=preg_replace_callback('!<nowiki>.*?</nowiki>!s',
function($m) { return ereg_replace(".","X",$m[0]); },
$text);
</source>
Now we can search for the anchors:
* If an anchor name is specified, we looked for '''all''' magic words <code><nowiki>{{#fileanchor:...}}</nowiki></code> or blocks with attribute <code>class="[someclass ]anchorname"</code>
* Otherwise we look for the first magic word <code><nowiki>{{#file:...}}</nowiki></code> with specified file name,
* And finally for the first <code><file></code> tag with the specified file name (no multiple blocks support)
And we free the memory used for the masked version
{{#fileanchor: RawFile.php}}
<source lang=php>
if (($anchor!='') && preg_match_all('/({{#fileanchor: *'.$anchor.' *}})|(<[^>]+ class *= *"([^"]*\w)?'.$anchor.'(\w[^"]*)?"[^>]*>)/i', $maskedtext, $matches, PREG_OFFSET_CAPTURE))
$offsets=$matches[0];
else if (preg_match_all('/{{#file: *'.$anchor.' *}}/i', $maskedtext, $matches, PREG_OFFSET_CAPTURE))
$offsets=array($matches[0][0]);
else if (preg_match_all('/<file( [^>]*)? name *= *"'.$filename.'"[^>]*>/i', $maskedtext, $matches, PREG_OFFSET_CAPTURE))
$offsets=array($matches[0][0]);
else {
// We didn't find our anchor
return fnRawFile_Strip_Error("ERROR - RawFile: anchor not found (anchor=$anchor, name=$filename, tag=$tag)","",$text);
}
unset($maskedtext);
</source>
$text is both input & output so we copy it and start with an empty output.
{{#fileanchor: RawFile.php}}
<source lang=php>
$textorig=$text;
$text='';
</source>
For each anchor found we've to isolate the content of the next block.
{{#fileanchor: RawFile.php}}
<source lang=php>
foreach ($offsets as $offset) {
</source>
We start from the position of the current anchor. If the tag name of the block attached to the anchor is not specified, we look for the first block that follows the anchor, excluding <code> </code> and <code><file></code> block. The search can be easily done with a regular expression, using the ''lookahead negative assertion'' <code>(?!br\b|file\b)</code> to exclude the tags to ignore. Note that we need to ignore the anchor-link block <code><file></code> since the anchor starts right before that tag, and so the regular expression would match the anchor-link block it that tag is not specifically excluded.
{{#fileanchor: RawFile.php}}
<source lang=php>
$out = substr($textorig, $offset[1]);
// If no tag specified, we take the first one
if ($tag == '')
{
// With a regex assertion, we can easily ignore 'br' and 'file' tags
if (!preg_match('/<((?!br\b|file\b)\w+\b)/', $out, $matches))
return fnRawFile_Strip_Error ("ERROR - RawFile: Can't find opening tag after anchor '$offset[0]' (anchor=$anchor, name=$filename, tag=$tag)",$out,$text);
$tag=$matches[1];
}
</source>
Now, we know the tag name of the block to download, either because it was already specified as a GET attribute in the URL, or because we've found it in the search above. Again, using a regular expression, we look for the first block matching the specified tag name that follows the current anchor, and extract the content of the blocks. Note the use of the regex option <code>/.../s</code> to tell the regex engine that the matched text can span on multiple lines (with that option, <code>.</code> does match any character or a newline character). Also, we skip the first carriage return after the opening tag, if any (with <code>\n?</code>).
{{#fileanchor: RawFile.php}}
<source lang=php>
// Find the first tag matching $tag, and return enclosed text
if (!preg_match('/<'.$tag.'( [^>]*)?>\n?(.*?)<\/'.$tag.'>/s', $out, $matches))
return fnRawFile_Strip_Error ("ERROR - RawFile: no closing '$tag' found after anchor '$offset[0]' (anchor=$anchor, name=$filename, tag=$tag)",$out,$text);
$text .= $matches[2];
}
</source>
No need to deal with a Content-Length header because Mediawiki will do it for us, moreover more properly than we could if the output is sent gzipped, which is the default.
 So that's it, $text contains our file!
{{#fileanchor: RawFile.php}}
<source lang=php>
return true;
}

</source>

===Credits===
There is an official way to register the extension in a Mediawiki installation, so that it will be visible on the [[Special:Version]] page.
 Let's say the extension is in the category of parser hooks even if there is also a hook on Raw action.
{{#fileanchor: RawFile.php}}
<source lang=php>
$wgExtensionCredits['parserhook'][] = array('name' => 'RawFile',
'version' => '0.5.1',
'author' => 'Philippe Teuwen, Michael Peeters',
'url' => 'http://www.mediawiki.org/wiki/Extension:RawFile',
// 'url' => 'http://wiki.yobi.be/wiki/Mediawiki_RawFile',
'description' => 'Downloads a RAW copy of <nowiki><tag>data</tag></nowiki> in a file '.
'Useful e.g. to download a script or a patch '.
'It also allows what is called [http://en.wikipedia.org/wiki/Literate_programming Literate Programming]');
}

?>
</source>
And finally registration of the extension at the Mediawiki website according to the [http://www.mediawiki.org/wiki/Manual:Extensions Extensions Manual].

So this extension has now [http://www.mediawiki.org/wiki/Extension:RawFile its own page on the official Mediawiki site].

==Installation==
Download [{{#filelink: RawFile.php}} RawFile.php] and save it under the MediaWiki directory as extensions/RawFile/RawFile.php

Add at the end of LocalSettings.php:
<source lang=php>
require_once("$IP/extensions/RawFile/RawFile.php");
</source>
==Status==
If you use the extension properly the code is fully functional but it's rather raw on error handling.
==ChangeLog==
'''0.5.1'''
* Integrate patch from Jani Uusitalo to recursively parse tag attribute values in case they contain [https://www.mediawiki.org/wiki/Help:Magic_words magic words] such as wiki templates or parameters. This is useful when using <code><file>...</file></code> in wiki templates.
* Prepare fix for <code>anchor not found</code> bug when using short notation <code><file name="filename" [tag="''tagname''"]>link text</file></code> in wiki templates. This still doesn't work because of the way MediaWiki (v1.22.1) expands templates in raw output (see [https://bugzilla.wikimedia.org/show_bug.cgi?id=61341 bugzilla 61341]).
'''0.5'''
* Fix since PHP 5.5.0: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead. Thanks to Stephen Kent for reporting.
* '''WARNING''' you should upgrade ASAP as the previous versions 0.4 and 0.4.1 are vulnerable to remote PHP code injection!!! See [[Talk:Mediawiki_RawFile]] for more info.
* Fix for MediaWiki 1.22.1.
'''0.4.1'''
* Fix octet-stream MIME type bug which was affecting Epiphany & Opera 11. Thanks to Jani Uusitalo for reporting & [[User:Sicvolo|Sicvolo]] for finding the solution
'''0.4'''
* '''Anchors''' can be specified using html '''class''' attribute
* New syntax for '''Links''' and '''Anchor-links''':
:<code><nowiki><file [name="..."] [anchor="..."] [tag="..."] [title="..."] >Link text</file></nowiki></code>
* Support multiple files on the same page with same name (differentiated by their anchor name) or even common blocks in multiple files.
* Can specify the tag name of the block to download (to skip some irrelevant blocks when using an '''anchor-link''').
* Ignore '''<code> </code>''' tag.
* Some error reporting.
'''0.3'''
* Added optional parameter to <code>#fileLink</code> to indicate that the file is on another local wiki page
'''0.2'''
* Fix problem with Content-Length mismatch when transport is gzipped (default for Mediawiki if client supports it)
'''0.1'''
* Initial version

==Known bugs==
==Questions and feedback==
If you've any trouble, questions or suggestions, you can [[User:PhilippeTeuwen|contact me]].
==Known sites using the extension==



* [http://wiki.hpc.ufl.edu University of Florida Research Computing Wiki]
* [http://wiki.scribus.net Scribus wiki]
* [http://nfc-tools.org/ NFC-Tools]
* [https://smkent.net/wiki/ smkent.net], website of Stephen Kent
* [http://mummila.net/wiki/ mummila.net], website of Jani Uusitalo
* [http://www.richud.com/wiki/ richud.com], website of Richard Moore


See also wikiapiary to follow usage of extensions [https://wikiapiary.com/wiki/Extension:RawFile RawFile] and [https://wikiapiary.com/wiki/Extension:Fram Fram]

Mediawiki RawFile

2014-02-17T10:08:08Z

Fuujuhi: /* Example using templates */

==Very short introduction==
Just have a look to the 2 [[Mediawiki_RawFile#Short_example|examples]] to see how to use the extension
 and to the [[Mediawiki_RawFile#Installation|Installation]] section to see how to install the extension in your [[MediaWiki]] server
==Introduction==
Originally the idea was to be able to download directly a portion of code as a file.
 I've numerous code examples in my wiki and I wanted an easy way to download them, easier than a copy/paste!
 But from there it was rather easy to get something very close to [http://en.wikipedia.org/wiki/Literate_programming literate programming] just by allowing multiple blocks referring to the same file, which will be concatenated together at download time.

* It must work with pre, nowiki, js, css, code, source, so let's make it general: take the tag that comes after the parser function we'll create and select data up to the closing tag.
* There are two distinct functionalities provided by the extension:
** the parser that will convert a magic word into a link to the download URL
** an extended ?action=raw that will strip the raw output to keep the desired code

==Syntax==
The extension introduces 3 elements:
;Anchor
: Used to flag that the next code block in the wiki text belongs to a specific file. The code block can be any wiki block (such as '''<code><pre></code>''', '''<code><code></code>''', '''<code><tt></code>''', '''<code><source></code>'''...). '''<code> </code>''' tags are ignored. Note that anchors are invisible in the wiki display.
;Link
: They are transformed by the extension into links that allows for downloading all blocks attached to a given anchor name.
;Anchor-link
: A shortcut notation mixing both an anchor and download link, handy for regular use, when a single code block is used and when the download link can be at the same position as the anchor.

The syntax is as follows. The syntax using tag <code><file></code> and tag attribute <code>class</code> is new since v0.4. Note that elements of both syntaxes can be mixed in a same page.
{| border="2" cellspacing="4" cellpadding="3" style="margin: 1em 1em 1em 0; background: #f9f9f9; border: 1px #aaaaaa solid; border-collapse: collapse; empty-cells:show;"
!width="80em" style="background: #8da7d6;"|Element!!style="background: #8da7d6;"|Syntax and description
|-
|'''Anchor'''
|<pre><nowiki>
{{#fileAnchor: anchorname}}
<pre class='anchorname'>...</pre>
<code class="anchorname">...</code>
<code class="cssclass anchorname">...</code>
...
</nowiki></pre>
Indicates that the next wiki block is attached to an anchor ''anchorname''. The content of that block will be downloaded (possibly appended with other blocks if there are several blocks attached to the same ''anchorname'') when a file link is clicked on. 
'''(since v0.4)''' To attach an anchor ''anchorname'' to a wiki block, simply add an attribute <code>class="anchorname"</code> to it. The extension supports multi-class specification, meaning that a same block can be associated to different files, and that the <code>class</code> attribute can still be used to specify custom CSS properties as in standard wiki text.
; ''anchorname''
; class="''anchorname''"
: The name of the anchor to which the wiki block is attached
|-
|'''Link'''
|<pre><nowiki>
[{{#fileLink: anchorname}} link text]
[{{#fileLink: anchorname|pagetitle}} link text]
<file anchor="anchorname" [name="filename"] [title="pagetitle"]>link text</file>
</nowiki></pre>
Creates a link to download all blocks that are attached to an anchor ''anchorname''.
;''anchorname''
;anchor="''anchorname''"
: The name of the anchor to look for. All blocks attached to an anchor ''anchorname'' will be downloaded.
;name="''filename''"
:''Optional'' - Specifies the name of the file to download. If absent, ''anchorname'' is then used as the name of the downloaded file.
; ''pagetitle''
;title="''pagetitle''"
: ''Optional'' - Indicates that the blocks to download are on the wiki page titled ''pagetitle''. If absent, blocks are looked for on the current page.
; ''link text''
: The text of the link to display.
|-
|'''Anchor-link'''
|<pre><nowiki>
[{{#file: filename}} link text]
<file name="filename" [tag="''tagname''"]>link text</file>
</nowiki></pre>
Creates a link to download the next wiki block as a file named ''filename''. 
'''(since v0.4)''' The attribute <code>tag</code> can be used to specify the ''tagname'' of the block to download. 
; ''filename''
; name="''filename''"
: The name of the file to download.
;tag="''tagname''"
:''Optional'' - When set, the extension only looks for blocks whose name matches the given ''tagname''. This attribute is particularly useful when there are some irrelevant blocks between the '''anchor-link''' and the block you want to download. If absent, the first encountered block following the anchor is downloaded.
; ''link text''
: The text of the link to display.
|}

===Short example===
The extension works with any block such as pre, nowiki, js, css, code, source,...
 This example is using the syntax highlighting <nowiki><source></nowiki> tag provided by [http://www.mediawiki.org/wiki/Extension:SyntaxHighlight_GeSHi SyntaxHighlight extension] (using [http://qbnz.com/highlighter/ GeSHi Highlighter])
 If you didn't install that extension on your MediaWiki, you can try the example by using <nowiki><pre></nowiki> instead of <nowiki><source></nowiki>.

<pre><nowiki>Let's save the following code [{{#file: myscript.sh}} as myscript.sh]
<source lang=bash>
#!/bin/bash

echo 'Hello world!'
exit 0
</source></nowiki>
</pre>
will give:
----
Let's save the following code [{{#file: myscript.sh}} as myscript.sh]
<source lang=bash>
#!/bin/bash

echo 'Hello world!'
exit 0
</source>

===Complete example===
And a full example with anchors & link:

<pre><nowiki>
Let's start with the Bash usual header:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
#!/bin/bash
</source>
Then we'll display a welcome message:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
echo 'Welcome on Earth!'
</source>
And we finally exit cleanly:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
exit 0
</source>
[{{#filelink: myotherscript.sh}} myotherscript.sh is now available for download below the code]
</nowiki></pre>
will give:
----
Let's start with the Bash usual header:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
#!/bin/bash
</source>
Then we'll display a welcome message:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
echo 'Welcome on Earth!'
</source>
And we finally exit cleanly:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
exit 0
</source>
[{{#filelink: myotherscript.sh}} myotherscript.sh is now available for download below the code]

=== Example using templates ===

The new syntax <code><file name="..." [tag="..."]>...</file></code> allows for using the RawFile extension in templates as well.

An example of use is to create a template to avoid duplication between the file name in the text and as a parameter in the <code><file></code> tag. The example below uses the template [[Template:Rawfiledownloadexample]]. See the template source for instructions on how to create the template.

{| class=wikitable
|-
|
<pre>
{{Rawfiledownloadexample|name=myfile.txt|content=Once upon a time
There was a Tag
Tag was clickable
And clicked it was}}
</pre>
|
{{Rawfiledownloadexample|name=myfile.txt|content=Once upon a time
There was a Tag
Tag was clickable
And clicked it was}}
|}

|}

==The code (the ultimate example)==
Which you can of course download just by following [{{#filelink: RawFile.php}} this link :-)]

So let's explain a bit the code in a Literate Programming way...
===Hooks===
First some hooks for our functions...

We will create:
* a [http://www.mediawiki.org/wiki/Manual:Parser_functions Parser Function] (see also [http://meta.wikimedia.org/wiki/Help:Parser_function here]), with help of
** [http://www.mediawiki.org/wiki/Manual:%24wgExtensionFunctions $wgExtensionFunctions] or [http://www.mediawiki.org/wiki/Manual:Hooks/ParserFirstCallInit ParserFirstCallInit global hook] to define the setup function
** [http://www.mediawiki.org/wiki/Manual:Magic_words Magic Words]
** [http://www.mediawiki.org/wiki/Manual:Tag_extensions Tag extensions]
** [http://www.mediawiki.org/wiki/Manual:Hooks/LanguageGetMagic LanguageGetMagic] hook to initialize the magic words
* a [http://www.mediawiki.org/wiki/Manual:Hooks/RawPageViewBeforeOutput RawPageViewBeforeOutput] hook to intercept the raw output

{{#fileanchor: RawFile.php}}
<source lang=php>
<?php

if (defined('MEDIAWIKI')) {

//Avoid unstubbing $wgParser on setHook() too early on modern (1.12+) MW versions, as per r35980
if ( defined( 'MW_SUPPORTS_PARSERFIRSTCALLINIT' ) ) {
$wgHooks['ParserFirstCallInit'][] = 'efRawFile_Setup';
} else { // Otherwise do things the old fashioned way
$wgExtensionFunctions[] = 'efRawFile_Setup';
}
$wgHooks['LanguageGetMagic'][] = 'efRawFile_Magic';
$wgHooks['RawPageViewBeforeOutput'][] = 'fnRawFile_Strip';

</source>

===Setup function===
For the wiki parsing to create download links, the parser functions '''file''' and '''fileLink''' are equally treated, while '''fileAnchor''' will be simply left out. We also create a new tag '''file''' as explained [http://www.mediawiki.org/wiki/Manual:Tag_extensions here].
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Setup() {
global $wgParser;
$wgParser->setFunctionHook( 'file', 'efRawFile_Render' );
$wgParser->setFunctionHook( 'filelink', 'efRawFile_Render' );
$wgParser->setFunctionHook( 'fileanchor', 'efRawFile_Empty' );
$wgParser->setHook( 'file', 'efRawFile_FileTagRender' );
return true;
}

</source>

===Hook to initialize the magic words===
We add the magic words here: the first array element indicates if it is case sensitive, in this case it is not case sensitive. We could add extra elements to create synonyms for our parser function.
 Unless we return true, other parser functions extensions will not get loaded.
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Magic( &$magicWords, $langCode ) {
$magicWords['file'] = array( 0, 'file' );
$magicWords['filelink'] = array( 0, 'filelink' );
$magicWords['fileanchor'] = array( 0, 'fileanchor' );
return true;
}

</source>

===Parser functions of the magic words===
The transformation rule to replace link shortcuts to actual links for download, handling an optional local wiki page title if present.
 The input parameters are wikitext with templates expanded, the output should be wikitext too
 '''TODO''': what error to send out if there is no filename given?
 '''EDIT''': It seems that [http://svn.wikimedia.org/viewvc/mediawiki?view=rev&revision=27667 commit 27667] (1.11 -> 1.12) changed the default parser, which breaks the recursive parsing. Thanks to Tim Starling for helping me to get around the problem!
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Render( &$parser, $filename = '', $titleText = '') {
if( $titleText == '' )
$title = $parser->mTitle;
else
$title = Title::newFromText( $titleText );
//Don't expand templates or we'll lose our anchors {{#...}}
return $title->getFullURL( 'action=raw&anchor='.urlencode( $filename ) );
}

</source>
And the other one, just removing the anchors from the rendered wiki page.
 Curiously enough if the function doesn't exist at all the effect is exactly the same, MW doesn't throw any error.
 But let's keep things clean...
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Empty( &$parser, $filename = '') {
return '';
}

</source>

===Parser functions of the new tag <tt><file></tt>===
The transformation rule to replace <code><file></code> tag to actual links for download. The same parser function is used for both '''anchors''' and '''anchor-links'''. Since the link text may contain wiki text, we generate the link as wiki text that we ask the parser to parse again.
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_FileTagRender( $input, $args, $parser, $frame ) {
if( $args['title'] == '' )
$title = $parser->mTitle;
else
$title = Title::newFromText($parser->recursiveTagParse( $args['title'], $frame ));

//We expand templates, so <file> tag cannot be mixed with {{#fileanchor}} anchors
$link=$title->getFullURL( 'action=raw&templates=expand' );
if( $args['name'] != '' )
$link.='&name='.urlencode( $parser->recursiveTagParse( $args['name'], $frame ) );
if( $args['anchor'] != '' )
$link.='&anchor='.urlencode( $parser->recursiveTagParse( $args['anchor'], $frame ) );
if( $args['tag'] != '' )
$link.='&tag='.urlencode( $parser->recursiveTagParse( $args['tag'], $frame ) );

return $parser->recursiveTagParse( "[$link $input]", $frame );
}

</source>

===Hook to intercept the raw output===
This part of the code doesn't look that nice because we've to parse the raw wiki page ourselves to retrieve the code sections we want.

First we define a helper function that we will use to report error messages. This is simply done by replacing the content of the downloaded file with the error message and when necessary a copy of the raw text relevant to the error.
 '''TODO''': Cancel the file download header and return a proper error page
{{#fileanchor: RawFile.php}}
<source lang=php>
function fnRawFile_Strip_Error($msg,$out,&$text) {
$text=$msg;
if($out != '')
$text.="\nCandidate match: $out";
return true;
}

</source>

Next let's see if <code>?action=raw</code> was used in the context of this extension: in that case we receive the filename as GET parameter, otherwise we simply return from our extension with return value=true which means we authorize the raw display (originally the hook was created to add an authentication point)
{{#fileanchor: RawFile.php}}
<source lang=php>
function fnRawFile_Strip(&$rawPage, &$text) {
$filename=$_GET['name'];
$anchor=$_GET['anchor'];
// for backward compatibility, accept also URLs with parameter 'file'
if( $anchor=='' )
$anchor=$_GET['file'];
$tag=$_GET['tag'];
// Either anchor or name must be specified
if( $filename=='' )
$filename=$anchor;
if ( $filename=='' )
return true;
</source>
By default the downloadable file will still be handled by the ob_gzhandler session made by Mediawiki. To avoid output buffering and gzipping, one can uncomment the following line:
{{#fileanchor: RawFile.php}}
<source lang=php>
// Uncomment the following line to avoid output buffering and gzipping:
// wfResetOutputBuffers();
</source>
Raw action already set the headers with some client cache pragmas and is supposed to be displayed in the browser but in our case we want to make this "page" a downloadable file so we overwrite the headers which were defined and we add a few more, to ensure there is no caching on the client (it's very hard for the client to force a refresh on a file download, contrary to a web page) and to provide the adequate filename.
{{#fileanchor: RawFile.php}}
<source lang=php>
header("Content-disposition: attachment;filename={$filename}");
header("Content-type: application/octet-stream");
header("Content-Transfer-Encoding: binary");
header("Expires: 0");
header("Pragma: no-cache");
header("Cache-Control: no-store");
</source>
Then we'll strip the output, first we've to locate the anchors but there are anchors that could be protected in literal blocks like <code>nowiki</code>.
 So we'll mask the literal blocks before searching for the anchors (we mask with the same string length because we'll retrieve an offset that we will use on the initial string and offsets must match). This is done with the scary regex below:
* we use <code>!</code> instead of <code>/</code> as pattern indicator so that the pattern string is self-matching. This is necessary since we will apply the extension on this page as well.
* we use option <code>s</code> (multiline) and <code>e</code> (evaluate replace expression)
* Evaluated expression replaces all characters in the matched string with X's. However if there are single quote (<code>'</code>) in the matched string, they will be escaped with <code>\</code>. So we need to search for <code>\'|.</code>. The many back-slashes is because the expression is evaluated several times.
'''TODO''': should we care also of source, js, css, pre,... blocks?
{{#fileanchor: RawFile.php}}
<source lang=php>
$maskedtext=preg_replace_callback('!<nowiki>.*?</nowiki>!s',
function($m) { return ereg_replace(".","X",$m[0]); },
$text);
</source>
Now we can search for the anchors:
* If an anchor name is specified, we looked for '''all''' magic words <code><nowiki>{{#fileanchor:...}}</nowiki></code> or blocks with attribute <code>class="[someclass ]anchorname"</code>
* Otherwise we look for the first magic word <code><nowiki>{{#file:...}}</nowiki></code> with specified file name,
* And finally for the first <code><file></code> tag with the specified file name (no multiple blocks support)
And we free the memory used for the masked version
{{#fileanchor: RawFile.php}}
<source lang=php>
if (($anchor!='') && preg_match_all('/({{#fileanchor: *'.$anchor.' *}})|(<[^>]+ class *= *"([^"]*\w)?'.$anchor.'(\w[^"]*)?"[^>]*>)/i', $maskedtext, $matches, PREG_OFFSET_CAPTURE))
$offsets=$matches[0];
else if (preg_match_all('/{{#file: *'.$anchor.' *}}/i', $maskedtext, $matches, PREG_OFFSET_CAPTURE))
$offsets=array($matches[0][0]);
else if (preg_match_all('/<file( [^>]*)? name *= *"'.$filename.'"[^>]*>/i', $maskedtext, $matches, PREG_OFFSET_CAPTURE))
$offsets=array($matches[0][0]);
else {
// We didn't find our anchor
return fnRawFile_Strip_Error("ERROR - RawFile: anchor not found (anchor=$anchor, name=$filename, tag=$tag)","",$text);
}
unset($maskedtext);
</source>
$text is both input & output so we copy it and start with an empty output.
{{#fileanchor: RawFile.php}}
<source lang=php>
$textorig=$text;
$text='';
</source>
For each anchor found we've to isolate the content of the next block.
{{#fileanchor: RawFile.php}}
<source lang=php>
foreach ($offsets as $offset) {
</source>
We start from the position of the current anchor. If the tag name of the block attached to the anchor is not specified, we look for the first block that follows the anchor, excluding <code> </code> and <code><file></code> block. The search can be easily done with a regular expression, using the ''lookahead negative assertion'' <code>(?!br\b|file\b)</code> to exclude the tags to ignore. Note that we need to ignore the anchor-link block <code><file></code> since the anchor starts right before that tag, and so the regular expression would match the anchor-link block it that tag is not specifically excluded.
{{#fileanchor: RawFile.php}}
<source lang=php>
$out = substr($textorig, $offset[1]);
// If no tag specified, we take the first one
if ($tag == '')
{
// With a regex assertion, we can easily ignore 'br' and 'file' tags
if (!preg_match('/<((?!br\b|file\b)\w+\b)/', $out, $matches))
return fnRawFile_Strip_Error ("ERROR - RawFile: Can't find opening tag after anchor '$offset[0]' (anchor=$anchor, name=$filename, tag=$tag)",$out,$text);
$tag=$matches[1];
}
</source>
Now, we know the tag name of the block to download, either because it was already specified as a GET attribute in the URL, or because we've found it in the search above. Again, using a regular expression, we look for the first block matching the specified tag name that follows the current anchor, and extract the content of the blocks. Note the use of the regex option <code>/.../s</code> to tell the regex engine that the matched text can span on multiple lines (with that option, <code>.</code> does match any character or a newline character). Also, we skip the first carriage return after the opening tag, if any (with <code>\n?</code>).
{{#fileanchor: RawFile.php}}
<source lang=php>
// Find the first tag matching $tag, and return enclosed text
if (!preg_match('/<'.$tag.'( [^>]*)?>\n?(.*?)<\/'.$tag.'>/s', $out, $matches))
return fnRawFile_Strip_Error ("ERROR - RawFile: no closing '$tag' found after anchor '$offset[0]' (anchor=$anchor, name=$filename, tag=$tag)",$out,$text);
$text .= $matches[2];
}
</source>
No need to deal with a Content-Length header because Mediawiki will do it for us, moreover more properly than we could if the output is sent gzipped, which is the default.
 So that's it, $text contains our file!
{{#fileanchor: RawFile.php}}
<source lang=php>
return true;
}

</source>

===Credits===
There is an official way to register the extension in a Mediawiki installation, so that it will be visible on the [[Special:Version]] page.
 Let's say the extension is in the category of parser hooks even if there is also a hook on Raw action.
{{#fileanchor: RawFile.php}}
<source lang=php>
$wgExtensionCredits['parserhook'][] = array('name' => 'RawFile',
'version' => '0.5.1',
'author' => 'Philippe Teuwen, Michael Peeters',
'url' => 'http://www.mediawiki.org/wiki/Extension:RawFile',
// 'url' => 'http://wiki.yobi.be/wiki/Mediawiki_RawFile',
'description' => 'Downloads a RAW copy of <nowiki><tag>data</tag></nowiki> in a file '.
'Useful e.g. to download a script or a patch '.
'It also allows what is called [http://en.wikipedia.org/wiki/Literate_programming Literate Programming]');
}

?>
</source>
And finally registration of the extension at the Mediawiki website according to the [http://www.mediawiki.org/wiki/Manual:Extensions Extensions Manual].

So this extension has now [http://www.mediawiki.org/wiki/Extension:RawFile its own page on the official Mediawiki site].

==Installation==
Download [{{#filelink: RawFile.php}} RawFile.php] and save it under the MediaWiki directory as extensions/RawFile/RawFile.php

Add at the end of LocalSettings.php:
<source lang=php>
require_once("$IP/extensions/RawFile/RawFile.php");
</source>
==Status==
If you use the extension properly the code is fully functional but it's rather raw on error handling.
==ChangeLog==
'''0.5.1'''
* Integrate patch from Jani Uusitalo to recursively parse tag attribute values in case they contain [https://www.mediawiki.org/wiki/Help:Magic_words magic words] such as wiki templates or parameters. This is useful when using <code><file>...</file></code> in wiki templates.
* Prepare fix for <code>anchor not found</code> bug when using short notation <code><file name="filename" [tag="''tagname''"]>link text</file></code> in wiki templates. This still doesn't work because of the way MediaWiki (v1.22.1) expands templates in raw output (see [https://bugzilla.wikimedia.org/show_bug.cgi?id=61341 bugzilla 61341]).
'''0.5'''
* Fix since PHP 5.5.0: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead. Thanks to Stephen Kent for reporting.
* '''WARNING''' you should upgrade ASAP as the previous versions 0.4 and 0.4.1 are vulnerable to remote PHP code injection!!! See [[Talk:Mediawiki_RawFile]] for more info.
* Fix for MediaWiki 1.22.1.
'''0.4.1'''
* Fix octet-stream MIME type bug which was affecting Epiphany & Opera 11. Thanks to Jani Uusitalo for reporting & [[User:Sicvolo|Sicvolo]] for finding the solution
'''0.4'''
* '''Anchors''' can be specified using html '''class''' attribute
* New syntax for '''Links''' and '''Anchor-links''':
:<code><nowiki><file [name="..."] [anchor="..."] [tag="..."] [title="..."] >Link text</file></nowiki></code>
* Support multiple files on the same page with same name (differentiated by their anchor name) or even common blocks in multiple files.
* Can specify the tag name of the block to download (to skip some irrelevant blocks when using an '''anchor-link''').
* Ignore '''<code> </code>''' tag.
* Some error reporting.
'''0.3'''
* Added optional parameter to <code>#fileLink</code> to indicate that the file is on another local wiki page
'''0.2'''
* Fix problem with Content-Length mismatch when transport is gzipped (default for Mediawiki if client supports it)
'''0.1'''
* Initial version

==Known bugs==
==Questions and feedback==
If you've any trouble, questions or suggestions, you can [[User:PhilippeTeuwen|contact me]].
==Known sites using the extension==



* [http://wiki.hpc.ufl.edu University of Florida Research Computing Wiki]
* [http://wiki.scribus.net Scribus wiki]
* [http://nfc-tools.org/ NFC-Tools]
* [https://smkent.net/wiki/ smkent.net], website of Stephen Kent
* [http://mummila.net/wiki/ mummila.net], website of Jani Uusitalo
* [http://www.richud.com/wiki/ richud.com], website of Richard Moore


See also wikiapiary to follow usage of extensions [https://wikiapiary.com/wiki/Extension:RawFile RawFile] and [https://wikiapiary.com/wiki/Extension:Fram Fram]

Template:Rawfiledownloadexample

2014-02-17T10:06:23Z

Fuujuhi: Use name for the link as well

You can download file "{{{name}}}" below, just click this link: {{#tag:file|{{{name}}}|name={{{name}}}|tag=pre}}.
{{#tag:pre|{{{content}}}}}
<noinclude>
<hr/>
This is an example template to demonstrate how to use the RawFile extension tags in a template.

The special construct <code><nowiki>{{#tag:file|...}}</nowiki></code> is a work-around to a bug in mediawiki template expansion (see [https://bugzilla.wikimedia.org/show_bug.cgi?id=61341 bugzilla 61341]).
See [[Mediawiki RawFile]] for more information.

Example of use:

{| class=wikitable
|-
|
<pre>
{{Rawfiledownloadexample|name=myfile.txt|content=Once upon a time
There was a Tag
Tag was clickable
And clicked it was}}
</pre>
|
{{Rawfiledownloadexample|name=myfile.txt|content=Once upon a time
There was a Tag
Tag was clickable
And clicked it was}}
|}

</noinclude>

Template:Rawfiledownloadexample

2014-02-17T09:53:42Z

Fuujuhi:

You can download file "{{{name}}}" below, just click this link: {{#tag:file|download|name={{{name}}}|tag=pre}}.
{{#tag:pre|{{{content}}}}}
<noinclude>
<hr/>
This is an example template to demonstrate how to use the RawFile extension tags in a template.

The special construct <code><nowiki>{{#tag:file|...}}</nowiki></code> is a work-around to a bug in mediawiki template expansion (see [https://bugzilla.wikimedia.org/show_bug.cgi?id=61341 bugzilla 61341]).
See [[Mediawiki RawFile]] for more information.

Example of use:

{| class=wikitable
|-
|
<pre>
{{Rawfiledownloadexample|name=myfile.txt|content=Once upon a time
There was a Tag
Tag was clickable
And clicked it was}}
</pre>
|
{{Rawfiledownloadexample|name=myfile.txt|content=Once upon a time
There was a Tag
Tag was clickable
And clicked it was}}
|}

</noinclude>

Template:Rawfiledownloadexample

2014-02-17T09:51:04Z

Fuujuhi: First version

You can download file "{{{name}}}" below, just click this link: {{#tag:file|download|name={{{name}}}|tag=pre}}.
<pre>
{{{content}}}
</pre>
<noinclude>
<hr/>
This is an example template to demonstrate how to use the RawFile extension tags in a template.

The special construct <code><nowiki>{{#tag:file|...}}</nowiki></code> is a work-around to a bug in mediawiki template expansion (see [https://bugzilla.wikimedia.org/show_bug.cgi?id=61341 bugzilla 61341]).
See [[Mediawiki RawFile]] for more information.

Example of use:

{| class=wikitable
|-
|
<pre>
{{Rawfiledownloadexample|name=myfile.txt|content=Once upon a time
There was a Tag
Tag was clickable
And clicked it was}}
</pre>
|
{{Rawfiledownloadexample|name=myfile.txt|content=Once upon a time
There was a Tag
Tag was clickable
And clicked it was}}
|}

</noinclude>

Mediawiki RawFile

2014-02-14T08:11:06Z

Fuujuhi: /* ChangeLog */ Fix typos and font style

==Very short introduction==
Just have a look to the 2 [[Mediawiki_RawFile#Short_example|examples]] to see how to use the extension
 and to the [[Mediawiki_RawFile#Installation|Installation]] section to see how to install the extension in your [[MediaWiki]] server
==Introduction==
Originally the idea was to be able to download directly a portion of code as a file.
 I've numerous code examples in my wiki and I wanted an easy way to download them, easier than a copy/paste!
 But from there it was rather easy to get something very close to [http://en.wikipedia.org/wiki/Literate_programming literate programming] just by allowing multiple blocks referring to the same file, which will be concatenated together at download time.

* It must work with pre, nowiki, js, css, code, source, so let's make it general: take the tag that comes after the parser function we'll create and select data up to the closing tag.
* There are two distinct functionalities provided by the extension:
** the parser that will convert a magic word into a link to the download URL
** an extended ?action=raw that will strip the raw output to keep the desired code

==Syntax==
The extension introduces 3 elements:
;Anchor
: Used to flag that the next code block in the wiki text belongs to a specific file. The code block can be any wiki block (such as '''<code><pre></code>''', '''<code><code></code>''', '''<code><tt></code>''', '''<code><source></code>'''...). '''<code> </code>''' tags are ignored. Note that anchors are invisible in the wiki display.
;Link
: They are transformed by the extension into links that allows for downloading all blocks attached to a given anchor name.
;Anchor-link
: A shortcut notation mixing both an anchor and download link, handy for regular use, when a single code block is used and when the download link can be at the same position as the anchor.

The syntax is as follows. The syntax using tag <code><file></code> and tag attribute <code>class</code> is new since v0.4. Note that elements of both syntaxes can be mixed in a same page.
{| border="2" cellspacing="4" cellpadding="3" style="margin: 1em 1em 1em 0; background: #f9f9f9; border: 1px #aaaaaa solid; border-collapse: collapse; empty-cells:show;"
!width="80em" style="background: #8da7d6;"|Element!!style="background: #8da7d6;"|Syntax and description
|-
|'''Anchor'''
|<pre><nowiki>
{{#fileAnchor: anchorname}}
<pre class='anchorname'>...</pre>
<code class="anchorname">...</code>
<code class="cssclass anchorname">...</code>
...
</nowiki></pre>
Indicates that the next wiki block is attached to an anchor ''anchorname''. The content of that block will be downloaded (possibly appended with other blocks if there are several blocks attached to the same ''anchorname'') when a file link is clicked on. 
'''(since v0.4)''' To attach an anchor ''anchorname'' to a wiki block, simply add an attribute <code>class="anchorname"</code> to it. The extension supports multi-class specification, meaning that a same block can be associated to different files, and that the <code>class</code> attribute can still be used to specify custom CSS properties as in standard wiki text.
; ''anchorname''
; class="''anchorname''"
: The name of the anchor to which the wiki block is attached
|-
|'''Link'''
|<pre><nowiki>
[{{#fileLink: anchorname}} link text]
[{{#fileLink: anchorname|pagetitle}} link text]
<file anchor="anchorname" [name="filename"] [title="pagetitle"]>link text</file>
</nowiki></pre>
Creates a link to download all blocks that are attached to an anchor ''anchorname''.
;''anchorname''
;anchor="''anchorname''"
: The name of the anchor to look for. All blocks attached to an anchor ''anchorname'' will be downloaded.
;name="''filename''"
:''Optional'' - Specifies the name of the file to download. If absent, ''anchorname'' is then used as the name of the downloaded file.
; ''pagetitle''
;title="''pagetitle''"
: ''Optional'' - Indicates that the blocks to download are on the wiki page titled ''pagetitle''. If absent, blocks are looked for on the current page.
; ''link text''
: The text of the link to display.
|-
|'''Anchor-link'''
|<pre><nowiki>
[{{#file: filename}} link text]
<file name="filename" [tag="''tagname''"]>link text</file>
</nowiki></pre>
Creates a link to download the next wiki block as a file named ''filename''. 
'''(since v0.4)''' The attribute <code>tag</code> can be used to specify the ''tagname'' of the block to download. 
; ''filename''
; name="''filename''"
: The name of the file to download.
;tag="''tagname''"
:''Optional'' - When set, the extension only looks for blocks whose name matches the given ''tagname''. This attribute is particularly useful when there are some irrelevant blocks between the '''anchor-link''' and the block you want to download. If absent, the first encountered block following the anchor is downloaded.
; ''link text''
: The text of the link to display.
|}

===Short example===
The extension works with any block such as pre, nowiki, js, css, code, source,...
 This example is using the syntax highlighting <nowiki><source></nowiki> tag provided by [http://www.mediawiki.org/wiki/Extension:SyntaxHighlight_GeSHi SyntaxHighlight extension] (using [http://qbnz.com/highlighter/ GeSHi Highlighter])
 If you didn't install that extension on your MediaWiki, you can try the example by using <nowiki><pre></nowiki> instead of <nowiki><source></nowiki>.

<pre><nowiki>Let's save the following code [{{#file: myscript.sh}} as myscript.sh]
<source lang=bash>
#!/bin/bash

echo 'Hello world!'
exit 0
</source></nowiki>
</pre>
will give:
----
Let's save the following code [{{#file: myscript.sh}} as myscript.sh]
<source lang=bash>
#!/bin/bash

echo 'Hello world!'
exit 0
</source>

===Complete example===
And a full example with anchors & link:

<pre><nowiki>
Let's start with the Bash usual header:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
#!/bin/bash
</source>
Then we'll display a welcome message:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
echo 'Welcome on Earth!'
</source>
And we finally exit cleanly:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
exit 0
</source>
[{{#filelink: myotherscript.sh}} myotherscript.sh is now available for download below the code]
</nowiki></pre>
will give:
----
Let's start with the Bash usual header:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
#!/bin/bash
</source>
Then we'll display a welcome message:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
echo 'Welcome on Earth!'
</source>
And we finally exit cleanly:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
exit 0
</source>
[{{#filelink: myotherscript.sh}} myotherscript.sh is now available for download below the code]

==The code (the ultimate example)==
Which you can of course download just by following [{{#filelink: RawFile.php}} this link :-)]

So let's explain a bit the code in a Literate Programming way...
===Hooks===
First some hooks for our functions...

We will create:
* a [http://www.mediawiki.org/wiki/Manual:Parser_functions Parser Function] (see also [http://meta.wikimedia.org/wiki/Help:Parser_function here]), with help of
** [http://www.mediawiki.org/wiki/Manual:%24wgExtensionFunctions $wgExtensionFunctions] or [http://www.mediawiki.org/wiki/Manual:Hooks/ParserFirstCallInit ParserFirstCallInit global hook] to define the setup function
** [http://www.mediawiki.org/wiki/Manual:Magic_words Magic Words]
** [http://www.mediawiki.org/wiki/Manual:Tag_extensions Tag extensions]
** [http://www.mediawiki.org/wiki/Manual:Hooks/LanguageGetMagic LanguageGetMagic] hook to initialize the magic words
* a [http://www.mediawiki.org/wiki/Manual:Hooks/RawPageViewBeforeOutput RawPageViewBeforeOutput] hook to intercept the raw output

{{#fileanchor: RawFile.php}}
<source lang=php>
<?php

if (defined('MEDIAWIKI')) {

//Avoid unstubbing $wgParser on setHook() too early on modern (1.12+) MW versions, as per r35980
if ( defined( 'MW_SUPPORTS_PARSERFIRSTCALLINIT' ) ) {
$wgHooks['ParserFirstCallInit'][] = 'efRawFile_Setup';
} else { // Otherwise do things the old fashioned way
$wgExtensionFunctions[] = 'efRawFile_Setup';
}
$wgHooks['LanguageGetMagic'][] = 'efRawFile_Magic';
$wgHooks['RawPageViewBeforeOutput'][] = 'fnRawFile_Strip';

</source>

===Setup function===
For the wiki parsing to create download links, the parser functions '''file''' and '''fileLink''' are equally treated, while '''fileAnchor''' will be simply left out. We also create a new tag '''file''' as explained [http://www.mediawiki.org/wiki/Manual:Tag_extensions here].
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Setup() {
global $wgParser;
$wgParser->setFunctionHook( 'file', 'efRawFile_Render' );
$wgParser->setFunctionHook( 'filelink', 'efRawFile_Render' );
$wgParser->setFunctionHook( 'fileanchor', 'efRawFile_Empty' );
$wgParser->setHook( 'file', 'efRawFile_FileTagRender' );
return true;
}

</source>

===Hook to initialize the magic words===
We add the magic words here: the first array element indicates if it is case sensitive, in this case it is not case sensitive. We could add extra elements to create synonyms for our parser function.
 Unless we return true, other parser functions extensions will not get loaded.
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Magic( &$magicWords, $langCode ) {
$magicWords['file'] = array( 0, 'file' );
$magicWords['filelink'] = array( 0, 'filelink' );
$magicWords['fileanchor'] = array( 0, 'fileanchor' );
return true;
}

</source>

===Parser functions of the magic words===
The transformation rule to replace link shortcuts to actual links for download, handling an optional local wiki page title if present.
 The input parameters are wikitext with templates expanded, the output should be wikitext too
 '''TODO''': what error to send out if there is no filename given?
 '''EDIT''': It seems that [http://svn.wikimedia.org/viewvc/mediawiki?view=rev&revision=27667 commit 27667] (1.11 -> 1.12) changed the default parser, which breaks the recursive parsing. Thanks to Tim Starling for helping me to get around the problem!
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Render( &$parser, $filename = '', $titleText = '') {
if( $titleText == '' )
$title = $parser->mTitle;
else
$title = Title::newFromText( $titleText );
//Don't expand templates or we'll lose our anchors {{#...}}
return $title->getFullURL( 'action=raw&anchor='.urlencode( $filename ) );
}

</source>
And the other one, just removing the anchors from the rendered wiki page.
 Curiously enough if the function doesn't exist at all the effect is exactly the same, MW doesn't throw any error.
 But let's keep things clean...
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Empty( &$parser, $filename = '') {
return '';
}

</source>

===Parser functions of the new tag <tt><file></tt>===
The transformation rule to replace <code><file></code> tag to actual links for download. The same parser function is used for both '''anchors''' and '''anchor-links'''. Since the link text may contain wiki text, we generate the link as wiki text that we ask the parser to parse again.
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_FileTagRender( $input, $args, $parser, $frame ) {
if( $args['title'] == '' )
$title = $parser->mTitle;
else
$title = Title::newFromText($parser->recursiveTagParse( $args['title'], $frame ));

//We expand templates, so <file> tag cannot be mixed with {{#fileanchor}} anchors
$link=$title->getFullURL( 'action=raw&templates=expand' );
if( $args['name'] != '' )
$link.='&name='.urlencode( $parser->recursiveTagParse( $args['name'], $frame ) );
if( $args['anchor'] != '' )
$link.='&anchor='.urlencode( $parser->recursiveTagParse( $args['anchor'], $frame ) );
if( $args['tag'] != '' )
$link.='&tag='.urlencode( $parser->recursiveTagParse( $args['tag'], $frame ) );

return $parser->recursiveTagParse( "[$link $input]", $frame );
}

</source>

===Hook to intercept the raw output===
This part of the code doesn't look that nice because we've to parse the raw wiki page ourselves to retrieve the code sections we want.

First we define a helper function that we will use to report error messages. This is simply done by replacing the content of the downloaded file with the error message and when necessary a copy of the raw text relevant to the error.
 '''TODO''': Cancel the file download header and return a proper error page
{{#fileanchor: RawFile.php}}
<source lang=php>
function fnRawFile_Strip_Error($msg,$out,&$text) {
$text=$msg;
if($out != '')
$text.="\nCandidate match: $out";
return true;
}

</source>

Next let's see if <code>?action=raw</code> was used in the context of this extension: in that case we receive the filename as GET parameter, otherwise we simply return from our extension with return value=true which means we authorize the raw display (originally the hook was created to add an authentication point)
{{#fileanchor: RawFile.php}}
<source lang=php>
function fnRawFile_Strip(&$rawPage, &$text) {
$filename=$_GET['name'];
$anchor=$_GET['anchor'];
// for backward compatibility, accept also URLs with parameter 'file'
if( $anchor=='' )
$anchor=$_GET['file'];
$tag=$_GET['tag'];
// Either anchor or name must be specified
if( $filename=='' )
$filename=$anchor;
if ( $filename=='' )
return true;
</source>
By default the downloadable file will still be handled by the ob_gzhandler session made by Mediawiki. To avoid output buffering and gzipping, one can uncomment the following line:
{{#fileanchor: RawFile.php}}
<source lang=php>
// Uncomment the following line to avoid output buffering and gzipping:
// wfResetOutputBuffers();
</source>
Raw action already set the headers with some client cache pragmas and is supposed to be displayed in the browser but in our case we want to make this "page" a downloadable file so we overwrite the headers which were defined and we add a few more, to ensure there is no caching on the client (it's very hard for the client to force a refresh on a file download, contrary to a web page) and to provide the adequate filename.
{{#fileanchor: RawFile.php}}
<source lang=php>
header("Content-disposition: attachment;filename={$filename}");
header("Content-type: application/octet-stream");
header("Content-Transfer-Encoding: binary");
header("Expires: 0");
header("Pragma: no-cache");
header("Cache-Control: no-store");
</source>
Then we'll strip the output, first we've to locate the anchors but there are anchors that could be protected in literal blocks like <code>nowiki</code>.
 So we'll mask the literal blocks before searching for the anchors (we mask with the same string length because we'll retrieve an offset that we will use on the initial string and offsets must match). This is done with the scary regex below:
* we use <code>!</code> instead of <code>/</code> as pattern indicator so that the pattern string is self-matching. This is necessary since we will apply the extension on this page as well.
* we use option <code>s</code> (multiline) and <code>e</code> (evaluate replace expression)
* Evaluated expression replaces all characters in the matched string with X's. However if there are single quote (<code>'</code>) in the matched string, they will be escaped with <code>\</code>. So we need to search for <code>\'|.</code>. The many back-slashes is because the expression is evaluated several times.
'''TODO''': should we care also of source, js, css, pre,... blocks?
{{#fileanchor: RawFile.php}}
<source lang=php>
$maskedtext=preg_replace_callback('!<nowiki>.*?</nowiki>!s',
function($m) { return ereg_replace(".","X",$m[0]); },
$text);
</source>
Now we can search for the anchors:
* If an anchor name is specified, we looked for '''all''' magic words <code><nowiki>{{#fileanchor:...}}</nowiki></code> or blocks with attribute <code>class="[someclass ]anchorname"</code>
* Otherwise we look for the first magic word <code><nowiki>{{#file:...}}</nowiki></code> with specified file name,
* And finally for the first <code><file></code> tag with the specified file name (no multiple blocks support)
And we free the memory used for the masked version
{{#fileanchor: RawFile.php}}
<source lang=php>
if (($anchor!='') && preg_match_all('/({{#fileanchor: *'.$anchor.' *}})|(<[^>]+ class *= *"([^"]*\w)?'.$anchor.'(\w[^"]*)?"[^>]*>)/i', $maskedtext, $matches, PREG_OFFSET_CAPTURE))
$offsets=$matches[0];
else if (preg_match_all('/{{#file: *'.$anchor.' *}}/i', $maskedtext, $matches, PREG_OFFSET_CAPTURE))
$offsets=array($matches[0][0]);
else if (preg_match_all('/<file( [^>]*)? name *= *"'.$filename.'"[^>]*>/i', $maskedtext, $matches, PREG_OFFSET_CAPTURE))
$offsets=array($matches[0][0]);
else {
// We didn't find our anchor
return fnRawFile_Strip_Error("ERROR - RawFile: anchor not found (anchor=$anchor, name=$filename, tag=$tag)","",$text);
}
unset($maskedtext);
</source>
$text is both input & output so we copy it and start with an empty output.
{{#fileanchor: RawFile.php}}
<source lang=php>
$textorig=$text;
$text='';
</source>
For each anchor found we've to isolate the content of the next block.
{{#fileanchor: RawFile.php}}
<source lang=php>
foreach ($offsets as $offset) {
</source>
We start from the position of the current anchor. If the tag name of the block attached to the anchor is not specified, we look for the first block that follows the anchor, excluding <code> </code> and <code><file></code> block. The search can be easily done with a regular expression, using the ''lookahead negative assertion'' <code>(?!br\b|file\b)</code> to exclude the tags to ignore. Note that we need to ignore the anchor-link block <code><file></code> since the anchor starts right before that tag, and so the regular expression would match the anchor-link block it that tag is not specifically excluded.
{{#fileanchor: RawFile.php}}
<source lang=php>
$out = substr($textorig, $offset[1]);
// If no tag specified, we take the first one
if ($tag == '')
{
// With a regex assertion, we can easily ignore 'br' and 'file' tags
if (!preg_match('/<((?!br\b|file\b)\w+\b)/', $out, $matches))
return fnRawFile_Strip_Error ("ERROR - RawFile: Can't find opening tag after anchor '$offset[0]' (anchor=$anchor, name=$filename, tag=$tag)",$out,$text);
$tag=$matches[1];
}
</source>
Now, we know the tag name of the block to download, either because it was already specified as a GET attribute in the URL, or because we've found it in the search above. Again, using a regular expression, we look for the first block matching the specified tag name that follows the current anchor, and extract the content of the blocks. Note the use of the regex option <code>/.../s</code> to tell the regex engine that the matched text can span on multiple lines (with that option, <code>.</code> does match any character or a newline character). Also, we skip the first carriage return after the opening tag, if any (with <code>\n?</code>).
{{#fileanchor: RawFile.php}}
<source lang=php>
// Find the first tag matching $tag, and return enclosed text
if (!preg_match('/<'.$tag.'( [^>]*)?>\n?(.*?)<\/'.$tag.'>/s', $out, $matches))
return fnRawFile_Strip_Error ("ERROR - RawFile: no closing '$tag' found after anchor '$offset[0]' (anchor=$anchor, name=$filename, tag=$tag)",$out,$text);
$text .= $matches[2];
}
</source>
No need to deal with a Content-Length header because Mediawiki will do it for us, moreover more properly than we could if the output is sent gzipped, which is the default.
 So that's it, $text contains our file!
{{#fileanchor: RawFile.php}}
<source lang=php>
return true;
}

</source>

===Credits===
There is an official way to register the extension in a Mediawiki installation, so that it will be visible on the [[Special:Version]] page.
 Let's say the extension is in the category of parser hooks even if there is also a hook on Raw action.
{{#fileanchor: RawFile.php}}
<source lang=php>
$wgExtensionCredits['parserhook'][] = array('name' => 'RawFile',
'version' => '0.5.1',
'author' => 'Philippe Teuwen, Michael Peeters',
'url' => 'http://www.mediawiki.org/wiki/Extension:RawFile',
// 'url' => 'http://wiki.yobi.be/wiki/Mediawiki_RawFile',
'description' => 'Downloads a RAW copy of <nowiki><tag>data</tag></nowiki> in a file '.
'Useful e.g. to download a script or a patch '.
'It also allows what is called [http://en.wikipedia.org/wiki/Literate_programming Literate Programming]');
}

?>
</source>
And finally registration of the extension at the Mediawiki website according to the [http://www.mediawiki.org/wiki/Manual:Extensions Extensions Manual].

So this extension has now [http://www.mediawiki.org/wiki/Extension:RawFile its own page on the official Mediawiki site].

==Installation==
Download [{{#filelink: RawFile.php}} RawFile.php] and save it under the MediaWiki directory as extensions/RawFile/RawFile.php

Add at the end of LocalSettings.php:
<source lang=php>
require_once("$IP/extensions/RawFile/RawFile.php");
</source>
==Status==
If you use the extension properly the code is fully functional but it's rather raw on error handling.
==ChangeLog==
'''0.5.1'''
* Integrate patch from Jani Uusitalo to recursively parse tag attribute values in case they contain [https://www.mediawiki.org/wiki/Help:Magic_words magic words] such as wiki templates or parameters. This is useful when using <code><file>...</file></code> in wiki templates.
* Prepare fix for <code>anchor not found</code> bug when using short notation <code><file name="filename" [tag="''tagname''"]>link text</file></code> in wiki templates. This still doesn't work because of the way MediaWiki (v1.22.1) expands templates in raw output (see [https://bugzilla.wikimedia.org/show_bug.cgi?id=61341 bugzilla 61341]).
'''0.5'''
* Fix since PHP 5.5.0: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead. Thanks to Stephen Kent for reporting.
* '''WARNING''' you should upgrade ASAP as the previous versions 0.4 and 0.4.1 are vulnerable to remote PHP code injection!!! See [[Talk:Mediawiki_RawFile]] for more info.
* Fix for MediaWiki 1.22.1.
'''0.4.1'''
* Fix octet-stream MIME type bug which was affecting Epiphany & Opera 11. Thanks to Jani Uusitalo for reporting & [[User:Sicvolo|Sicvolo]] for finding the solution
'''0.4'''
* '''Anchors''' can be specified using html '''class''' attribute
* New syntax for '''Links''' and '''Anchor-links''':
:<code><nowiki><file [name="..."] [anchor="..."] [tag="..."] [title="..."] >Link text</file></nowiki></code>
* Support multiple files on the same page with same name (differentiated by their anchor name) or even common blocks in multiple files.
* Can specify the tag name of the block to download (to skip some irrelevant blocks when using an '''anchor-link''').
* Ignore '''<code> </code>''' tag.
* Some error reporting.
'''0.3'''
* Added optional parameter to <code>#fileLink</code> to indicate that the file is on another local wiki page
'''0.2'''
* Fix problem with Content-Length mismatch when transport is gzipped (default for Mediawiki if client supports it)
'''0.1'''
* Initial version

==Known bugs==
==Questions and feedback==
If you've any trouble, questions or suggestions, you can [[User:PhilippeTeuwen|contact me]].
==Known sites using the extension==



* [http://wiki.hpc.ufl.edu University of Florida Research Computing Wiki]
* [http://wiki.scribus.net Scribus wiki]
* [http://nfc-tools.org/ NFC-Tools]
* [https://smkent.net/wiki/ smkent.net], website of Stephen Kent
* [http://mummila.net/wiki/ mummila.net], website of Jani Uusitalo
* [http://www.richud.com/wiki/ richud.com], website of Richard Moore


See also wikiapiary to follow usage of extensions [https://wikiapiary.com/wiki/Extension:RawFile RawFile] and [https://wikiapiary.com/wiki/Extension:Fram Fram]

Mediawiki RawFile

2014-02-14T08:09:47Z

Fuujuhi: /* ChangeLog */ Fix broken link to mediawiki

==Very short introduction==
Just have a look to the 2 [[Mediawiki_RawFile#Short_example|examples]] to see how to use the extension
 and to the [[Mediawiki_RawFile#Installation|Installation]] section to see how to install the extension in your [[MediaWiki]] server
==Introduction==
Originally the idea was to be able to download directly a portion of code as a file.
 I've numerous code examples in my wiki and I wanted an easy way to download them, easier than a copy/paste!
 But from there it was rather easy to get something very close to [http://en.wikipedia.org/wiki/Literate_programming literate programming] just by allowing multiple blocks referring to the same file, which will be concatenated together at download time.

* It must work with pre, nowiki, js, css, code, source, so let's make it general: take the tag that comes after the parser function we'll create and select data up to the closing tag.
* There are two distinct functionalities provided by the extension:
** the parser that will convert a magic word into a link to the download URL
** an extended ?action=raw that will strip the raw output to keep the desired code

==Syntax==
The extension introduces 3 elements:
;Anchor
: Used to flag that the next code block in the wiki text belongs to a specific file. The code block can be any wiki block (such as '''<code><pre></code>''', '''<code><code></code>''', '''<code><tt></code>''', '''<code><source></code>'''...). '''<code> </code>''' tags are ignored. Note that anchors are invisible in the wiki display.
;Link
: They are transformed by the extension into links that allows for downloading all blocks attached to a given anchor name.
;Anchor-link
: A shortcut notation mixing both an anchor and download link, handy for regular use, when a single code block is used and when the download link can be at the same position as the anchor.

The syntax is as follows. The syntax using tag <code><file></code> and tag attribute <code>class</code> is new since v0.4. Note that elements of both syntaxes can be mixed in a same page.
{| border="2" cellspacing="4" cellpadding="3" style="margin: 1em 1em 1em 0; background: #f9f9f9; border: 1px #aaaaaa solid; border-collapse: collapse; empty-cells:show;"
!width="80em" style="background: #8da7d6;"|Element!!style="background: #8da7d6;"|Syntax and description
|-
|'''Anchor'''
|<pre><nowiki>
{{#fileAnchor: anchorname}}
<pre class='anchorname'>...</pre>
<code class="anchorname">...</code>
<code class="cssclass anchorname">...</code>
...
</nowiki></pre>
Indicates that the next wiki block is attached to an anchor ''anchorname''. The content of that block will be downloaded (possibly appended with other blocks if there are several blocks attached to the same ''anchorname'') when a file link is clicked on. 
'''(since v0.4)''' To attach an anchor ''anchorname'' to a wiki block, simply add an attribute <code>class="anchorname"</code> to it. The extension supports multi-class specification, meaning that a same block can be associated to different files, and that the <code>class</code> attribute can still be used to specify custom CSS properties as in standard wiki text.
; ''anchorname''
; class="''anchorname''"
: The name of the anchor to which the wiki block is attached
|-
|'''Link'''
|<pre><nowiki>
[{{#fileLink: anchorname}} link text]
[{{#fileLink: anchorname|pagetitle}} link text]
<file anchor="anchorname" [name="filename"] [title="pagetitle"]>link text</file>
</nowiki></pre>
Creates a link to download all blocks that are attached to an anchor ''anchorname''.
;''anchorname''
;anchor="''anchorname''"
: The name of the anchor to look for. All blocks attached to an anchor ''anchorname'' will be downloaded.
;name="''filename''"
:''Optional'' - Specifies the name of the file to download. If absent, ''anchorname'' is then used as the name of the downloaded file.
; ''pagetitle''
;title="''pagetitle''"
: ''Optional'' - Indicates that the blocks to download are on the wiki page titled ''pagetitle''. If absent, blocks are looked for on the current page.
; ''link text''
: The text of the link to display.
|-
|'''Anchor-link'''
|<pre><nowiki>
[{{#file: filename}} link text]
<file name="filename" [tag="''tagname''"]>link text</file>
</nowiki></pre>
Creates a link to download the next wiki block as a file named ''filename''. 
'''(since v0.4)''' The attribute <code>tag</code> can be used to specify the ''tagname'' of the block to download. 
; ''filename''
; name="''filename''"
: The name of the file to download.
;tag="''tagname''"
:''Optional'' - When set, the extension only looks for blocks whose name matches the given ''tagname''. This attribute is particularly useful when there are some irrelevant blocks between the '''anchor-link''' and the block you want to download. If absent, the first encountered block following the anchor is downloaded.
; ''link text''
: The text of the link to display.
|}

===Short example===
The extension works with any block such as pre, nowiki, js, css, code, source,...
 This example is using the syntax highlighting <nowiki><source></nowiki> tag provided by [http://www.mediawiki.org/wiki/Extension:SyntaxHighlight_GeSHi SyntaxHighlight extension] (using [http://qbnz.com/highlighter/ GeSHi Highlighter])
 If you didn't install that extension on your MediaWiki, you can try the example by using <nowiki><pre></nowiki> instead of <nowiki><source></nowiki>.

<pre><nowiki>Let's save the following code [{{#file: myscript.sh}} as myscript.sh]
<source lang=bash>
#!/bin/bash

echo 'Hello world!'
exit 0
</source></nowiki>
</pre>
will give:
----
Let's save the following code [{{#file: myscript.sh}} as myscript.sh]
<source lang=bash>
#!/bin/bash

echo 'Hello world!'
exit 0
</source>

===Complete example===
And a full example with anchors & link:

<pre><nowiki>
Let's start with the Bash usual header:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
#!/bin/bash
</source>
Then we'll display a welcome message:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
echo 'Welcome on Earth!'
</source>
And we finally exit cleanly:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
exit 0
</source>
[{{#filelink: myotherscript.sh}} myotherscript.sh is now available for download below the code]
</nowiki></pre>
will give:
----
Let's start with the Bash usual header:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
#!/bin/bash
</source>
Then we'll display a welcome message:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
echo 'Welcome on Earth!'
</source>
And we finally exit cleanly:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
exit 0
</source>
[{{#filelink: myotherscript.sh}} myotherscript.sh is now available for download below the code]

==The code (the ultimate example)==
Which you can of course download just by following [{{#filelink: RawFile.php}} this link :-)]

So let's explain a bit the code in a Literate Programming way...
===Hooks===
First some hooks for our functions...

We will create:
* a [http://www.mediawiki.org/wiki/Manual:Parser_functions Parser Function] (see also [http://meta.wikimedia.org/wiki/Help:Parser_function here]), with help of
** [http://www.mediawiki.org/wiki/Manual:%24wgExtensionFunctions $wgExtensionFunctions] or [http://www.mediawiki.org/wiki/Manual:Hooks/ParserFirstCallInit ParserFirstCallInit global hook] to define the setup function
** [http://www.mediawiki.org/wiki/Manual:Magic_words Magic Words]
** [http://www.mediawiki.org/wiki/Manual:Tag_extensions Tag extensions]
** [http://www.mediawiki.org/wiki/Manual:Hooks/LanguageGetMagic LanguageGetMagic] hook to initialize the magic words
* a [http://www.mediawiki.org/wiki/Manual:Hooks/RawPageViewBeforeOutput RawPageViewBeforeOutput] hook to intercept the raw output

{{#fileanchor: RawFile.php}}
<source lang=php>
<?php

if (defined('MEDIAWIKI')) {

//Avoid unstubbing $wgParser on setHook() too early on modern (1.12+) MW versions, as per r35980
if ( defined( 'MW_SUPPORTS_PARSERFIRSTCALLINIT' ) ) {
$wgHooks['ParserFirstCallInit'][] = 'efRawFile_Setup';
} else { // Otherwise do things the old fashioned way
$wgExtensionFunctions[] = 'efRawFile_Setup';
}
$wgHooks['LanguageGetMagic'][] = 'efRawFile_Magic';
$wgHooks['RawPageViewBeforeOutput'][] = 'fnRawFile_Strip';

</source>

===Setup function===
For the wiki parsing to create download links, the parser functions '''file''' and '''fileLink''' are equally treated, while '''fileAnchor''' will be simply left out. We also create a new tag '''file''' as explained [http://www.mediawiki.org/wiki/Manual:Tag_extensions here].
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Setup() {
global $wgParser;
$wgParser->setFunctionHook( 'file', 'efRawFile_Render' );
$wgParser->setFunctionHook( 'filelink', 'efRawFile_Render' );
$wgParser->setFunctionHook( 'fileanchor', 'efRawFile_Empty' );
$wgParser->setHook( 'file', 'efRawFile_FileTagRender' );
return true;
}

</source>

===Hook to initialize the magic words===
We add the magic words here: the first array element indicates if it is case sensitive, in this case it is not case sensitive. We could add extra elements to create synonyms for our parser function.
 Unless we return true, other parser functions extensions will not get loaded.
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Magic( &$magicWords, $langCode ) {
$magicWords['file'] = array( 0, 'file' );
$magicWords['filelink'] = array( 0, 'filelink' );
$magicWords['fileanchor'] = array( 0, 'fileanchor' );
return true;
}

</source>

===Parser functions of the magic words===
The transformation rule to replace link shortcuts to actual links for download, handling an optional local wiki page title if present.
 The input parameters are wikitext with templates expanded, the output should be wikitext too
 '''TODO''': what error to send out if there is no filename given?
 '''EDIT''': It seems that [http://svn.wikimedia.org/viewvc/mediawiki?view=rev&revision=27667 commit 27667] (1.11 -> 1.12) changed the default parser, which breaks the recursive parsing. Thanks to Tim Starling for helping me to get around the problem!
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Render( &$parser, $filename = '', $titleText = '') {
if( $titleText == '' )
$title = $parser->mTitle;
else
$title = Title::newFromText( $titleText );
//Don't expand templates or we'll lose our anchors {{#...}}
return $title->getFullURL( 'action=raw&anchor='.urlencode( $filename ) );
}

</source>
And the other one, just removing the anchors from the rendered wiki page.
 Curiously enough if the function doesn't exist at all the effect is exactly the same, MW doesn't throw any error.
 But let's keep things clean...
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Empty( &$parser, $filename = '') {
return '';
}

</source>

===Parser functions of the new tag <tt><file></tt>===
The transformation rule to replace <code><file></code> tag to actual links for download. The same parser function is used for both '''anchors''' and '''anchor-links'''. Since the link text may contain wiki text, we generate the link as wiki text that we ask the parser to parse again.
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_FileTagRender( $input, $args, $parser, $frame ) {
if( $args['title'] == '' )
$title = $parser->mTitle;
else
$title = Title::newFromText($parser->recursiveTagParse( $args['title'], $frame ));

//We expand templates, so <file> tag cannot be mixed with {{#fileanchor}} anchors
$link=$title->getFullURL( 'action=raw&templates=expand' );
if( $args['name'] != '' )
$link.='&name='.urlencode( $parser->recursiveTagParse( $args['name'], $frame ) );
if( $args['anchor'] != '' )
$link.='&anchor='.urlencode( $parser->recursiveTagParse( $args['anchor'], $frame ) );
if( $args['tag'] != '' )
$link.='&tag='.urlencode( $parser->recursiveTagParse( $args['tag'], $frame ) );

return $parser->recursiveTagParse( "[$link $input]", $frame );
}

</source>

===Hook to intercept the raw output===
This part of the code doesn't look that nice because we've to parse the raw wiki page ourselves to retrieve the code sections we want.

First we define a helper function that we will use to report error messages. This is simply done by replacing the content of the downloaded file with the error message and when necessary a copy of the raw text relevant to the error.
 '''TODO''': Cancel the file download header and return a proper error page
{{#fileanchor: RawFile.php}}
<source lang=php>
function fnRawFile_Strip_Error($msg,$out,&$text) {
$text=$msg;
if($out != '')
$text.="\nCandidate match: $out";
return true;
}

</source>

Next let's see if <code>?action=raw</code> was used in the context of this extension: in that case we receive the filename as GET parameter, otherwise we simply return from our extension with return value=true which means we authorize the raw display (originally the hook was created to add an authentication point)
{{#fileanchor: RawFile.php}}
<source lang=php>
function fnRawFile_Strip(&$rawPage, &$text) {
$filename=$_GET['name'];
$anchor=$_GET['anchor'];
// for backward compatibility, accept also URLs with parameter 'file'
if( $anchor=='' )
$anchor=$_GET['file'];
$tag=$_GET['tag'];
// Either anchor or name must be specified
if( $filename=='' )
$filename=$anchor;
if ( $filename=='' )
return true;
</source>
By default the downloadable file will still be handled by the ob_gzhandler session made by Mediawiki. To avoid output buffering and gzipping, one can uncomment the following line:
{{#fileanchor: RawFile.php}}
<source lang=php>
// Uncomment the following line to avoid output buffering and gzipping:
// wfResetOutputBuffers();
</source>
Raw action already set the headers with some client cache pragmas and is supposed to be displayed in the browser but in our case we want to make this "page" a downloadable file so we overwrite the headers which were defined and we add a few more, to ensure there is no caching on the client (it's very hard for the client to force a refresh on a file download, contrary to a web page) and to provide the adequate filename.
{{#fileanchor: RawFile.php}}
<source lang=php>
header("Content-disposition: attachment;filename={$filename}");
header("Content-type: application/octet-stream");
header("Content-Transfer-Encoding: binary");
header("Expires: 0");
header("Pragma: no-cache");
header("Cache-Control: no-store");
</source>
Then we'll strip the output, first we've to locate the anchors but there are anchors that could be protected in literal blocks like <code>nowiki</code>.
 So we'll mask the literal blocks before searching for the anchors (we mask with the same string length because we'll retrieve an offset that we will use on the initial string and offsets must match). This is done with the scary regex below:
* we use <code>!</code> instead of <code>/</code> as pattern indicator so that the pattern string is self-matching. This is necessary since we will apply the extension on this page as well.
* we use option <code>s</code> (multiline) and <code>e</code> (evaluate replace expression)
* Evaluated expression replaces all characters in the matched string with X's. However if there are single quote (<code>'</code>) in the matched string, they will be escaped with <code>\</code>. So we need to search for <code>\'|.</code>. The many back-slashes is because the expression is evaluated several times.
'''TODO''': should we care also of source, js, css, pre,... blocks?
{{#fileanchor: RawFile.php}}
<source lang=php>
$maskedtext=preg_replace_callback('!<nowiki>.*?</nowiki>!s',
function($m) { return ereg_replace(".","X",$m[0]); },
$text);
</source>
Now we can search for the anchors:
* If an anchor name is specified, we looked for '''all''' magic words <code><nowiki>{{#fileanchor:...}}</nowiki></code> or blocks with attribute <code>class="[someclass ]anchorname"</code>
* Otherwise we look for the first magic word <code><nowiki>{{#file:...}}</nowiki></code> with specified file name,
* And finally for the first <code><file></code> tag with the specified file name (no multiple blocks support)
And we free the memory used for the masked version
{{#fileanchor: RawFile.php}}
<source lang=php>
if (($anchor!='') && preg_match_all('/({{#fileanchor: *'.$anchor.' *}})|(<[^>]+ class *= *"([^"]*\w)?'.$anchor.'(\w[^"]*)?"[^>]*>)/i', $maskedtext, $matches, PREG_OFFSET_CAPTURE))
$offsets=$matches[0];
else if (preg_match_all('/{{#file: *'.$anchor.' *}}/i', $maskedtext, $matches, PREG_OFFSET_CAPTURE))
$offsets=array($matches[0][0]);
else if (preg_match_all('/<file( [^>]*)? name *= *"'.$filename.'"[^>]*>/i', $maskedtext, $matches, PREG_OFFSET_CAPTURE))
$offsets=array($matches[0][0]);
else {
// We didn't find our anchor
return fnRawFile_Strip_Error("ERROR - RawFile: anchor not found (anchor=$anchor, name=$filename, tag=$tag)","",$text);
}
unset($maskedtext);
</source>
$text is both input & output so we copy it and start with an empty output.
{{#fileanchor: RawFile.php}}
<source lang=php>
$textorig=$text;
$text='';
</source>
For each anchor found we've to isolate the content of the next block.
{{#fileanchor: RawFile.php}}
<source lang=php>
foreach ($offsets as $offset) {
</source>
We start from the position of the current anchor. If the tag name of the block attached to the anchor is not specified, we look for the first block that follows the anchor, excluding <code> </code> and <code><file></code> block. The search can be easily done with a regular expression, using the ''lookahead negative assertion'' <code>(?!br\b|file\b)</code> to exclude the tags to ignore. Note that we need to ignore the anchor-link block <code><file></code> since the anchor starts right before that tag, and so the regular expression would match the anchor-link block it that tag is not specifically excluded.
{{#fileanchor: RawFile.php}}
<source lang=php>
$out = substr($textorig, $offset[1]);
// If no tag specified, we take the first one
if ($tag == '')
{
// With a regex assertion, we can easily ignore 'br' and 'file' tags
if (!preg_match('/<((?!br\b|file\b)\w+\b)/', $out, $matches))
return fnRawFile_Strip_Error ("ERROR - RawFile: Can't find opening tag after anchor '$offset[0]' (anchor=$anchor, name=$filename, tag=$tag)",$out,$text);
$tag=$matches[1];
}
</source>
Now, we know the tag name of the block to download, either because it was already specified as a GET attribute in the URL, or because we've found it in the search above. Again, using a regular expression, we look for the first block matching the specified tag name that follows the current anchor, and extract the content of the blocks. Note the use of the regex option <code>/.../s</code> to tell the regex engine that the matched text can span on multiple lines (with that option, <code>.</code> does match any character or a newline character). Also, we skip the first carriage return after the opening tag, if any (with <code>\n?</code>).
{{#fileanchor: RawFile.php}}
<source lang=php>
// Find the first tag matching $tag, and return enclosed text
if (!preg_match('/<'.$tag.'( [^>]*)?>\n?(.*?)<\/'.$tag.'>/s', $out, $matches))
return fnRawFile_Strip_Error ("ERROR - RawFile: no closing '$tag' found after anchor '$offset[0]' (anchor=$anchor, name=$filename, tag=$tag)",$out,$text);
$text .= $matches[2];
}
</source>
No need to deal with a Content-Length header because Mediawiki will do it for us, moreover more properly than we could if the output is sent gzipped, which is the default.
 So that's it, $text contains our file!
{{#fileanchor: RawFile.php}}
<source lang=php>
return true;
}

</source>

===Credits===
There is an official way to register the extension in a Mediawiki installation, so that it will be visible on the [[Special:Version]] page.
 Let's say the extension is in the category of parser hooks even if there is also a hook on Raw action.
{{#fileanchor: RawFile.php}}
<source lang=php>
$wgExtensionCredits['parserhook'][] = array('name' => 'RawFile',
'version' => '0.5.1',
'author' => 'Philippe Teuwen, Michael Peeters',
'url' => 'http://www.mediawiki.org/wiki/Extension:RawFile',
// 'url' => 'http://wiki.yobi.be/wiki/Mediawiki_RawFile',
'description' => 'Downloads a RAW copy of <nowiki><tag>data</tag></nowiki> in a file '.
'Useful e.g. to download a script or a patch '.
'It also allows what is called [http://en.wikipedia.org/wiki/Literate_programming Literate Programming]');
}

?>
</source>
And finally registration of the extension at the Mediawiki website according to the [http://www.mediawiki.org/wiki/Manual:Extensions Extensions Manual].

So this extension has now [http://www.mediawiki.org/wiki/Extension:RawFile its own page on the official Mediawiki site].

==Installation==
Download [{{#filelink: RawFile.php}} RawFile.php] and save it under the MediaWiki directory as extensions/RawFile/RawFile.php

Add at the end of LocalSettings.php:
<source lang=php>
require_once("$IP/extensions/RawFile/RawFile.php");
</source>
==Status==
If you use the extension properly the code is fully functional but it's rather raw on error handling.
==ChangeLog==
'''0.5.1'''
* Integrate patch from Jani Uusitalo to recursively parse tag attribute values in case they contain [https://www.mediawiki.org/wiki/Help:Magic_words magic words] such as wiki templates or parameters. This is useful when using <code><file>...</file></code> in wiki templates.
* Prepare fix for <tt>anchor not found</tt> bug when using short notation <code><file name="filename" [tag="''tagname''"]>link text</file></code> in wiki templates. This still doesn't work because of the may MediaWiki (v1.22.1) expands templates in raw output (see [https://bugzilla.wikimedia.org/show_bug.cgi?id=61341 bugzilla 61341]).
'''0.5'''
* Fix since PHP 5.5.0: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead. Thanks to Stephen Kent for reporting.
* '''WARNING''' you should upgrade ASAP as the previous versions 0.4 and 0.4.1 are vulnerable to remote PHP code injection!!! See [[Talk:Mediawiki_RawFile]] for more info.
* Fix for MediaWiki 1.22.1.
'''0.4.1'''
* Fix octet-stream MIME type bug which was affecting Epiphany & Opera 11. Thanks to Jani Uusitalo for reporting & [[User:Sicvolo|Sicvolo]] for finding the solution
'''0.4'''
* '''Anchors''' can be specified using html '''class''' attribute
* New syntax for '''Links''' and '''Anchor-links''':
:<code><nowiki><file [name="..."] [anchor="..."] [tag="..."] [title="..."] >Link text</file></nowiki></code>
* Support multiple files on the same page with same name (differentiated by their anchor name) or even common blocks in multiple files.
* Can specify the tag name of the block to download (to skip some irrelevant blocks when using an '''anchor-link''').
* Ignore '''<code> </code>''' tag.
* Some error reporting.
'''0.3'''
* Added optional parameter to <code>#fileLink</code> to indicate that the file is on another local wiki page
'''0.2'''
* Fix problem with Content-Length mismatch when transport is gzipped (default for Mediawiki if client supports it)
'''0.1'''
* Initial version

==Known bugs==
==Questions and feedback==
If you've any trouble, questions or suggestions, you can [[User:PhilippeTeuwen|contact me]].
==Known sites using the extension==



* [http://wiki.hpc.ufl.edu University of Florida Research Computing Wiki]
* [http://wiki.scribus.net Scribus wiki]
* [http://nfc-tools.org/ NFC-Tools]
* [https://smkent.net/wiki/ smkent.net], website of Stephen Kent
* [http://mummila.net/wiki/ mummila.net], website of Jani Uusitalo
* [http://www.richud.com/wiki/ richud.com], website of Richard Moore


See also wikiapiary to follow usage of extensions [https://wikiapiary.com/wiki/Extension:RawFile RawFile] and [https://wikiapiary.com/wiki/Extension:Fram Fram]

Mediawiki RawFile

2014-02-14T08:05:52Z

Fuujuhi: v0.5.1 - Patch from Jani + prepare fix for 'anchor not found'

==Very short introduction==
Just have a look to the 2 [[Mediawiki_RawFile#Short_example|examples]] to see how to use the extension
 and to the [[Mediawiki_RawFile#Installation|Installation]] section to see how to install the extension in your [[MediaWiki]] server
==Introduction==
Originally the idea was to be able to download directly a portion of code as a file.
 I've numerous code examples in my wiki and I wanted an easy way to download them, easier than a copy/paste!
 But from there it was rather easy to get something very close to [http://en.wikipedia.org/wiki/Literate_programming literate programming] just by allowing multiple blocks referring to the same file, which will be concatenated together at download time.

* It must work with pre, nowiki, js, css, code, source, so let's make it general: take the tag that comes after the parser function we'll create and select data up to the closing tag.
* There are two distinct functionalities provided by the extension:
** the parser that will convert a magic word into a link to the download URL
** an extended ?action=raw that will strip the raw output to keep the desired code

==Syntax==
The extension introduces 3 elements:
;Anchor
: Used to flag that the next code block in the wiki text belongs to a specific file. The code block can be any wiki block (such as '''<code><pre></code>''', '''<code><code></code>''', '''<code><tt></code>''', '''<code><source></code>'''...). '''<code> </code>''' tags are ignored. Note that anchors are invisible in the wiki display.
;Link
: They are transformed by the extension into links that allows for downloading all blocks attached to a given anchor name.
;Anchor-link
: A shortcut notation mixing both an anchor and download link, handy for regular use, when a single code block is used and when the download link can be at the same position as the anchor.

The syntax is as follows. The syntax using tag <code><file></code> and tag attribute <code>class</code> is new since v0.4. Note that elements of both syntaxes can be mixed in a same page.
{| border="2" cellspacing="4" cellpadding="3" style="margin: 1em 1em 1em 0; background: #f9f9f9; border: 1px #aaaaaa solid; border-collapse: collapse; empty-cells:show;"
!width="80em" style="background: #8da7d6;"|Element!!style="background: #8da7d6;"|Syntax and description
|-
|'''Anchor'''
|<pre><nowiki>
{{#fileAnchor: anchorname}}
<pre class='anchorname'>...</pre>
<code class="anchorname">...</code>
<code class="cssclass anchorname">...</code>
...
</nowiki></pre>
Indicates that the next wiki block is attached to an anchor ''anchorname''. The content of that block will be downloaded (possibly appended with other blocks if there are several blocks attached to the same ''anchorname'') when a file link is clicked on. 
'''(since v0.4)''' To attach an anchor ''anchorname'' to a wiki block, simply add an attribute <code>class="anchorname"</code> to it. The extension supports multi-class specification, meaning that a same block can be associated to different files, and that the <code>class</code> attribute can still be used to specify custom CSS properties as in standard wiki text.
; ''anchorname''
; class="''anchorname''"
: The name of the anchor to which the wiki block is attached
|-
|'''Link'''
|<pre><nowiki>
[{{#fileLink: anchorname}} link text]
[{{#fileLink: anchorname|pagetitle}} link text]
<file anchor="anchorname" [name="filename"] [title="pagetitle"]>link text</file>
</nowiki></pre>
Creates a link to download all blocks that are attached to an anchor ''anchorname''.
;''anchorname''
;anchor="''anchorname''"
: The name of the anchor to look for. All blocks attached to an anchor ''anchorname'' will be downloaded.
;name="''filename''"
:''Optional'' - Specifies the name of the file to download. If absent, ''anchorname'' is then used as the name of the downloaded file.
; ''pagetitle''
;title="''pagetitle''"
: ''Optional'' - Indicates that the blocks to download are on the wiki page titled ''pagetitle''. If absent, blocks are looked for on the current page.
; ''link text''
: The text of the link to display.
|-
|'''Anchor-link'''
|<pre><nowiki>
[{{#file: filename}} link text]
<file name="filename" [tag="''tagname''"]>link text</file>
</nowiki></pre>
Creates a link to download the next wiki block as a file named ''filename''. 
'''(since v0.4)''' The attribute <code>tag</code> can be used to specify the ''tagname'' of the block to download. 
; ''filename''
; name="''filename''"
: The name of the file to download.
;tag="''tagname''"
:''Optional'' - When set, the extension only looks for blocks whose name matches the given ''tagname''. This attribute is particularly useful when there are some irrelevant blocks between the '''anchor-link''' and the block you want to download. If absent, the first encountered block following the anchor is downloaded.
; ''link text''
: The text of the link to display.
|}

===Short example===
The extension works with any block such as pre, nowiki, js, css, code, source,...
 This example is using the syntax highlighting <nowiki><source></nowiki> tag provided by [http://www.mediawiki.org/wiki/Extension:SyntaxHighlight_GeSHi SyntaxHighlight extension] (using [http://qbnz.com/highlighter/ GeSHi Highlighter])
 If you didn't install that extension on your MediaWiki, you can try the example by using <nowiki><pre></nowiki> instead of <nowiki><source></nowiki>.

<pre><nowiki>Let's save the following code [{{#file: myscript.sh}} as myscript.sh]
<source lang=bash>
#!/bin/bash

echo 'Hello world!'
exit 0
</source></nowiki>
</pre>
will give:
----
Let's save the following code [{{#file: myscript.sh}} as myscript.sh]
<source lang=bash>
#!/bin/bash

echo 'Hello world!'
exit 0
</source>

===Complete example===
And a full example with anchors & link:

<pre><nowiki>
Let's start with the Bash usual header:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
#!/bin/bash
</source>
Then we'll display a welcome message:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
echo 'Welcome on Earth!'
</source>
And we finally exit cleanly:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
exit 0
</source>
[{{#filelink: myotherscript.sh}} myotherscript.sh is now available for download below the code]
</nowiki></pre>
will give:
----
Let's start with the Bash usual header:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
#!/bin/bash
</source>
Then we'll display a welcome message:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
echo 'Welcome on Earth!'
</source>
And we finally exit cleanly:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
exit 0
</source>
[{{#filelink: myotherscript.sh}} myotherscript.sh is now available for download below the code]

==The code (the ultimate example)==
Which you can of course download just by following [{{#filelink: RawFile.php}} this link :-)]

So let's explain a bit the code in a Literate Programming way...
===Hooks===
First some hooks for our functions...

We will create:
* a [http://www.mediawiki.org/wiki/Manual:Parser_functions Parser Function] (see also [http://meta.wikimedia.org/wiki/Help:Parser_function here]), with help of
** [http://www.mediawiki.org/wiki/Manual:%24wgExtensionFunctions $wgExtensionFunctions] or [http://www.mediawiki.org/wiki/Manual:Hooks/ParserFirstCallInit ParserFirstCallInit global hook] to define the setup function
** [http://www.mediawiki.org/wiki/Manual:Magic_words Magic Words]
** [http://www.mediawiki.org/wiki/Manual:Tag_extensions Tag extensions]
** [http://www.mediawiki.org/wiki/Manual:Hooks/LanguageGetMagic LanguageGetMagic] hook to initialize the magic words
* a [http://www.mediawiki.org/wiki/Manual:Hooks/RawPageViewBeforeOutput RawPageViewBeforeOutput] hook to intercept the raw output

{{#fileanchor: RawFile.php}}
<source lang=php>
<?php

if (defined('MEDIAWIKI')) {

//Avoid unstubbing $wgParser on setHook() too early on modern (1.12+) MW versions, as per r35980
if ( defined( 'MW_SUPPORTS_PARSERFIRSTCALLINIT' ) ) {
$wgHooks['ParserFirstCallInit'][] = 'efRawFile_Setup';
} else { // Otherwise do things the old fashioned way
$wgExtensionFunctions[] = 'efRawFile_Setup';
}
$wgHooks['LanguageGetMagic'][] = 'efRawFile_Magic';
$wgHooks['RawPageViewBeforeOutput'][] = 'fnRawFile_Strip';

</source>

===Setup function===
For the wiki parsing to create download links, the parser functions '''file''' and '''fileLink''' are equally treated, while '''fileAnchor''' will be simply left out. We also create a new tag '''file''' as explained [http://www.mediawiki.org/wiki/Manual:Tag_extensions here].
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Setup() {
global $wgParser;
$wgParser->setFunctionHook( 'file', 'efRawFile_Render' );
$wgParser->setFunctionHook( 'filelink', 'efRawFile_Render' );
$wgParser->setFunctionHook( 'fileanchor', 'efRawFile_Empty' );
$wgParser->setHook( 'file', 'efRawFile_FileTagRender' );
return true;
}

</source>

===Hook to initialize the magic words===
We add the magic words here: the first array element indicates if it is case sensitive, in this case it is not case sensitive. We could add extra elements to create synonyms for our parser function.
 Unless we return true, other parser functions extensions will not get loaded.
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Magic( &$magicWords, $langCode ) {
$magicWords['file'] = array( 0, 'file' );
$magicWords['filelink'] = array( 0, 'filelink' );
$magicWords['fileanchor'] = array( 0, 'fileanchor' );
return true;
}

</source>

===Parser functions of the magic words===
The transformation rule to replace link shortcuts to actual links for download, handling an optional local wiki page title if present.
 The input parameters are wikitext with templates expanded, the output should be wikitext too
 '''TODO''': what error to send out if there is no filename given?
 '''EDIT''': It seems that [http://svn.wikimedia.org/viewvc/mediawiki?view=rev&revision=27667 commit 27667] (1.11 -> 1.12) changed the default parser, which breaks the recursive parsing. Thanks to Tim Starling for helping me to get around the problem!
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Render( &$parser, $filename = '', $titleText = '') {
if( $titleText == '' )
$title = $parser->mTitle;
else
$title = Title::newFromText( $titleText );
//Don't expand templates or we'll lose our anchors {{#...}}
return $title->getFullURL( 'action=raw&anchor='.urlencode( $filename ) );
}

</source>
And the other one, just removing the anchors from the rendered wiki page.
 Curiously enough if the function doesn't exist at all the effect is exactly the same, MW doesn't throw any error.
 But let's keep things clean...
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Empty( &$parser, $filename = '') {
return '';
}

</source>

===Parser functions of the new tag <tt><file></tt>===
The transformation rule to replace <code><file></code> tag to actual links for download. The same parser function is used for both '''anchors''' and '''anchor-links'''. Since the link text may contain wiki text, we generate the link as wiki text that we ask the parser to parse again.
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_FileTagRender( $input, $args, $parser, $frame ) {
if( $args['title'] == '' )
$title = $parser->mTitle;
else
$title = Title::newFromText($parser->recursiveTagParse( $args['title'], $frame ));

//We expand templates, so <file> tag cannot be mixed with {{#fileanchor}} anchors
$link=$title->getFullURL( 'action=raw&templates=expand' );
if( $args['name'] != '' )
$link.='&name='.urlencode( $parser->recursiveTagParse( $args['name'], $frame ) );
if( $args['anchor'] != '' )
$link.='&anchor='.urlencode( $parser->recursiveTagParse( $args['anchor'], $frame ) );
if( $args['tag'] != '' )
$link.='&tag='.urlencode( $parser->recursiveTagParse( $args['tag'], $frame ) );

return $parser->recursiveTagParse( "[$link $input]", $frame );
}

</source>

===Hook to intercept the raw output===
This part of the code doesn't look that nice because we've to parse the raw wiki page ourselves to retrieve the code sections we want.

First we define a helper function that we will use to report error messages. This is simply done by replacing the content of the downloaded file with the error message and when necessary a copy of the raw text relevant to the error.
 '''TODO''': Cancel the file download header and return a proper error page
{{#fileanchor: RawFile.php}}
<source lang=php>
function fnRawFile_Strip_Error($msg,$out,&$text) {
$text=$msg;
if($out != '')
$text.="\nCandidate match: $out";
return true;
}

</source>

Next let's see if <code>?action=raw</code> was used in the context of this extension: in that case we receive the filename as GET parameter, otherwise we simply return from our extension with return value=true which means we authorize the raw display (originally the hook was created to add an authentication point)
{{#fileanchor: RawFile.php}}
<source lang=php>
function fnRawFile_Strip(&$rawPage, &$text) {
$filename=$_GET['name'];
$anchor=$_GET['anchor'];
// for backward compatibility, accept also URLs with parameter 'file'
if( $anchor=='' )
$anchor=$_GET['file'];
$tag=$_GET['tag'];
// Either anchor or name must be specified
if( $filename=='' )
$filename=$anchor;
if ( $filename=='' )
return true;
</source>
By default the downloadable file will still be handled by the ob_gzhandler session made by Mediawiki. To avoid output buffering and gzipping, one can uncomment the following line:
{{#fileanchor: RawFile.php}}
<source lang=php>
// Uncomment the following line to avoid output buffering and gzipping:
// wfResetOutputBuffers();
</source>
Raw action already set the headers with some client cache pragmas and is supposed to be displayed in the browser but in our case we want to make this "page" a downloadable file so we overwrite the headers which were defined and we add a few more, to ensure there is no caching on the client (it's very hard for the client to force a refresh on a file download, contrary to a web page) and to provide the adequate filename.
{{#fileanchor: RawFile.php}}
<source lang=php>
header("Content-disposition: attachment;filename={$filename}");
header("Content-type: application/octet-stream");
header("Content-Transfer-Encoding: binary");
header("Expires: 0");
header("Pragma: no-cache");
header("Cache-Control: no-store");
</source>
Then we'll strip the output, first we've to locate the anchors but there are anchors that could be protected in literal blocks like <code>nowiki</code>.
 So we'll mask the literal blocks before searching for the anchors (we mask with the same string length because we'll retrieve an offset that we will use on the initial string and offsets must match). This is done with the scary regex below:
* we use <code>!</code> instead of <code>/</code> as pattern indicator so that the pattern string is self-matching. This is necessary since we will apply the extension on this page as well.
* we use option <code>s</code> (multiline) and <code>e</code> (evaluate replace expression)
* Evaluated expression replaces all characters in the matched string with X's. However if there are single quote (<code>'</code>) in the matched string, they will be escaped with <code>\</code>. So we need to search for <code>\'|.</code>. The many back-slashes is because the expression is evaluated several times.
'''TODO''': should we care also of source, js, css, pre,... blocks?
{{#fileanchor: RawFile.php}}
<source lang=php>
$maskedtext=preg_replace_callback('!<nowiki>.*?</nowiki>!s',
function($m) { return ereg_replace(".","X",$m[0]); },
$text);
</source>
Now we can search for the anchors:
* If an anchor name is specified, we looked for '''all''' magic words <code><nowiki>{{#fileanchor:...}}</nowiki></code> or blocks with attribute <code>class="[someclass ]anchorname"</code>
* Otherwise we look for the first magic word <code><nowiki>{{#file:...}}</nowiki></code> with specified file name,
* And finally for the first <code><file></code> tag with the specified file name (no multiple blocks support)
And we free the memory used for the masked version
{{#fileanchor: RawFile.php}}
<source lang=php>
if (($anchor!='') && preg_match_all('/({{#fileanchor: *'.$anchor.' *}})|(<[^>]+ class *= *"([^"]*\w)?'.$anchor.'(\w[^"]*)?"[^>]*>)/i', $maskedtext, $matches, PREG_OFFSET_CAPTURE))
$offsets=$matches[0];
else if (preg_match_all('/{{#file: *'.$anchor.' *}}/i', $maskedtext, $matches, PREG_OFFSET_CAPTURE))
$offsets=array($matches[0][0]);
else if (preg_match_all('/<file( [^>]*)? name *= *"'.$filename.'"[^>]*>/i', $maskedtext, $matches, PREG_OFFSET_CAPTURE))
$offsets=array($matches[0][0]);
else {
// We didn't find our anchor
return fnRawFile_Strip_Error("ERROR - RawFile: anchor not found (anchor=$anchor, name=$filename, tag=$tag)","",$text);
}
unset($maskedtext);
</source>
$text is both input & output so we copy it and start with an empty output.
{{#fileanchor: RawFile.php}}
<source lang=php>
$textorig=$text;
$text='';
</source>
For each anchor found we've to isolate the content of the next block.
{{#fileanchor: RawFile.php}}
<source lang=php>
foreach ($offsets as $offset) {
</source>
We start from the position of the current anchor. If the tag name of the block attached to the anchor is not specified, we look for the first block that follows the anchor, excluding <code> </code> and <code><file></code> block. The search can be easily done with a regular expression, using the ''lookahead negative assertion'' <code>(?!br\b|file\b)</code> to exclude the tags to ignore. Note that we need to ignore the anchor-link block <code><file></code> since the anchor starts right before that tag, and so the regular expression would match the anchor-link block it that tag is not specifically excluded.
{{#fileanchor: RawFile.php}}
<source lang=php>
$out = substr($textorig, $offset[1]);
// If no tag specified, we take the first one
if ($tag == '')
{
// With a regex assertion, we can easily ignore 'br' and 'file' tags
if (!preg_match('/<((?!br\b|file\b)\w+\b)/', $out, $matches))
return fnRawFile_Strip_Error ("ERROR - RawFile: Can't find opening tag after anchor '$offset[0]' (anchor=$anchor, name=$filename, tag=$tag)",$out,$text);
$tag=$matches[1];
}
</source>
Now, we know the tag name of the block to download, either because it was already specified as a GET attribute in the URL, or because we've found it in the search above. Again, using a regular expression, we look for the first block matching the specified tag name that follows the current anchor, and extract the content of the blocks. Note the use of the regex option <code>/.../s</code> to tell the regex engine that the matched text can span on multiple lines (with that option, <code>.</code> does match any character or a newline character). Also, we skip the first carriage return after the opening tag, if any (with <code>\n?</code>).
{{#fileanchor: RawFile.php}}
<source lang=php>
// Find the first tag matching $tag, and return enclosed text
if (!preg_match('/<'.$tag.'( [^>]*)?>\n?(.*?)<\/'.$tag.'>/s', $out, $matches))
return fnRawFile_Strip_Error ("ERROR - RawFile: no closing '$tag' found after anchor '$offset[0]' (anchor=$anchor, name=$filename, tag=$tag)",$out,$text);
$text .= $matches[2];
}
</source>
No need to deal with a Content-Length header because Mediawiki will do it for us, moreover more properly than we could if the output is sent gzipped, which is the default.
 So that's it, $text contains our file!
{{#fileanchor: RawFile.php}}
<source lang=php>
return true;
}

</source>

===Credits===
There is an official way to register the extension in a Mediawiki installation, so that it will be visible on the [[Special:Version]] page.
 Let's say the extension is in the category of parser hooks even if there is also a hook on Raw action.
{{#fileanchor: RawFile.php}}
<source lang=php>
$wgExtensionCredits['parserhook'][] = array('name' => 'RawFile',
'version' => '0.5.1',
'author' => 'Philippe Teuwen, Michael Peeters',
'url' => 'http://www.mediawiki.org/wiki/Extension:RawFile',
// 'url' => 'http://wiki.yobi.be/wiki/Mediawiki_RawFile',
'description' => 'Downloads a RAW copy of <nowiki><tag>data</tag></nowiki> in a file '.
'Useful e.g. to download a script or a patch '.
'It also allows what is called [http://en.wikipedia.org/wiki/Literate_programming Literate Programming]');
}

?>
</source>
And finally registration of the extension at the Mediawiki website according to the [http://www.mediawiki.org/wiki/Manual:Extensions Extensions Manual].

So this extension has now [http://www.mediawiki.org/wiki/Extension:RawFile its own page on the official Mediawiki site].

==Installation==
Download [{{#filelink: RawFile.php}} RawFile.php] and save it under the MediaWiki directory as extensions/RawFile/RawFile.php

Add at the end of LocalSettings.php:
<source lang=php>
require_once("$IP/extensions/RawFile/RawFile.php");
</source>
==Status==
If you use the extension properly the code is fully functional but it's rather raw on error handling.
==ChangeLog==
'''0.5.1'''
* Integrate patch from Jani Uusitalo to recursively parse tag attribute values in case they contain [[mediawikiwiki:Help:Magic_words|magic words]] such as wiki templates or parameters. This is useful when using <code><file>...</file></code> in wiki templates.
* Prepare fix for <tt>anchor not found</tt> bug when using short notation <code><file name="filename" [tag="''tagname''"]>link text</file></code> in wiki templates. This still doesn't work because of the may MediaWiki (v1.22.1) expands templates in raw output (see [https://bugzilla.wikimedia.org/show_bug.cgi?id=61341 bugzilla 61341]).
'''0.5'''
* Fix since PHP 5.5.0: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead. Thanks to Stephen Kent for reporting.
* '''WARNING''' you should upgrade ASAP as the previous versions 0.4 and 0.4.1 are vulnerable to remote PHP code injection!!! See [[Talk:Mediawiki_RawFile]] for more info.
* Fix for MediaWiki 1.22.1.
'''0.4.1'''
* Fix octet-stream MIME type bug which was affecting Epiphany & Opera 11. Thanks to Jani Uusitalo for reporting & [[User:Sicvolo|Sicvolo]] for finding the solution
'''0.4'''
* '''Anchors''' can be specified using html '''class''' attribute
* New syntax for '''Links''' and '''Anchor-links''':
:<code><nowiki><file [name="..."] [anchor="..."] [tag="..."] [title="..."] >Link text</file></nowiki></code>
* Support multiple files on the same page with same name (differentiated by their anchor name) or even common blocks in multiple files.
* Can specify the tag name of the block to download (to skip some irrelevant blocks when using an '''anchor-link''').
* Ignore '''<code> </code>''' tag.
* Some error reporting.
'''0.3'''
* Added optional parameter to <code>#fileLink</code> to indicate that the file is on another local wiki page
'''0.2'''
* Fix problem with Content-Length mismatch when transport is gzipped (default for Mediawiki if client supports it)
'''0.1'''
* Initial version

==Known bugs==
==Questions and feedback==
If you've any trouble, questions or suggestions, you can [[User:PhilippeTeuwen|contact me]].
==Known sites using the extension==



* [http://wiki.hpc.ufl.edu University of Florida Research Computing Wiki]
* [http://wiki.scribus.net Scribus wiki]
* [http://nfc-tools.org/ NFC-Tools]
* [https://smkent.net/wiki/ smkent.net], website of Stephen Kent
* [http://mummila.net/wiki/ mummila.net], website of Jani Uusitalo
* [http://www.richud.com/wiki/ richud.com], website of Richard Moore


See also wikiapiary to follow usage of extensions [https://wikiapiary.com/wiki/Extension:RawFile RawFile] and [https://wikiapiary.com/wiki/Extension:Fram Fram]

Mediawiki RawFile

2014-02-13T15:31:40Z

Fuujuhi: /* ChangeLog */ Fix for MediaWiki 1.22.1 -- Official release 0.5

==Very short introduction==
Just have a look to the 2 [[Mediawiki_RawFile#Short_example|examples]] to see how to use the extension
 and to the [[Mediawiki_RawFile#Installation|Installation]] section to see how to install the extension in your [[MediaWiki]] server
==Introduction==
Originally the idea was to be able to download directly a portion of code as a file.
 I've numerous code examples in my wiki and I wanted an easy way to download them, easier than a copy/paste!
 But from there it was rather easy to get something very close to [http://en.wikipedia.org/wiki/Literate_programming literate programming] just by allowing multiple blocks referring to the same file, which will be concatenated together at download time.

* It must work with pre, nowiki, js, css, code, source, so let's make it general: take the tag that comes after the parser function we'll create and select data up to the closing tag.
* There are two distinct functionalities provided by the extension:
** the parser that will convert a magic word into a link to the download URL
** an extended ?action=raw that will strip the raw output to keep the desired code

==Syntax==
The extension introduces 3 elements:
;Anchor
: Used to flag that the next code block in the wiki text belongs to a specific file. The code block can be any wiki block (such as '''<code><pre></code>''', '''<code><code></code>''', '''<code><tt></code>''', '''<code><source></code>'''...). '''<code> </code>''' tags are ignored. Note that anchors are invisible in the wiki display.
;Link
: They are transformed by the extension into links that allows for downloading all blocks attached to a given anchor name.
;Anchor-link
: A shortcut notation mixing both an anchor and download link, handy for regular use, when a single code block is used and when the download link can be at the same position as the anchor.

The syntax is as follows. The syntax using tag <code><file></code> and tag attribute <code>class</code> is new since v0.4. Note that elements of both syntaxes can be mixed in a same page.
{| border="2" cellspacing="4" cellpadding="3" style="margin: 1em 1em 1em 0; background: #f9f9f9; border: 1px #aaaaaa solid; border-collapse: collapse; empty-cells:show;"
!width="80em" style="background: #8da7d6;"|Element!!style="background: #8da7d6;"|Syntax and description
|-
|'''Anchor'''
|<pre><nowiki>
{{#fileAnchor: anchorname}}
<pre class='anchorname'>...</pre>
<code class="anchorname">...</code>
<code class="cssclass anchorname">...</code>
...
</nowiki></pre>
Indicates that the next wiki block is attached to an anchor ''anchorname''. The content of that block will be downloaded (possibly appended with other blocks if there are several blocks attached to the same ''anchorname'') when a file link is clicked on. 
'''(since v0.4)''' To attach an anchor ''anchorname'' to a wiki block, simply add an attribute <code>class="anchorname"</code> to it. The extension supports multi-class specification, meaning that a same block can be associated to different files, and that the <code>class</code> attribute can still be used to specify custom CSS properties as in standard wiki text.
; ''anchorname''
; class="''anchorname''"
: The name of the anchor to which the wiki block is attached
|-
|'''Link'''
|<pre><nowiki>
[{{#fileLink: anchorname}} link text]
[{{#fileLink: anchorname|pagetitle}} link text]
<file anchor="anchorname" [name="filename"] [title="pagetitle"]>link text</file>
</nowiki></pre>
Creates a link to download all blocks that are attached to an anchor ''anchorname''.
;''anchorname''
;anchor="''anchorname''"
: The name of the anchor to look for. All blocks attached to an anchor ''anchorname'' will be downloaded.
;name="''filename''"
:''Optional'' - Specifies the name of the file to download. If absent, ''anchorname'' is then used as the name of the downloaded file.
; ''pagetitle''
;title="''pagetitle''"
: ''Optional'' - Indicates that the blocks to download are on the wiki page titled ''pagetitle''. If absent, blocks are looked for on the current page.
; ''link text''
: The text of the link to display.
|-
|'''Anchor-link'''
|<pre><nowiki>
[{{#file: filename}} link text]
<file name="filename" [tag="''tagname''"]>link text</file>
</nowiki></pre>
Creates a link to download the next wiki block as a file named ''filename''. 
'''(since v0.4)''' The attribute <code>tag</code> can be used to specify the ''tagname'' of the block to download. 
; ''filename''
; name="''filename''"
: The name of the file to download.
;tag="''tagname''"
:''Optional'' - When set, the extension only looks for blocks whose name matches the given ''tagname''. This attribute is particularly useful when there are some irrelevant blocks between the '''anchor-link''' and the block you want to download. If absent, the first encountered block following the anchor is downloaded.
; ''link text''
: The text of the link to display.
|}

===Short example===
The extension works with any block such as pre, nowiki, js, css, code, source,...
 This example is using the syntax highlighting <nowiki><source></nowiki> tag provided by [http://www.mediawiki.org/wiki/Extension:SyntaxHighlight_GeSHi SyntaxHighlight extension] (using [http://qbnz.com/highlighter/ GeSHi Highlighter])
 If you didn't install that extension on your MediaWiki, you can try the example by using <nowiki><pre></nowiki> instead of <nowiki><source></nowiki>.

<pre><nowiki>Let's save the following code [{{#file: myscript.sh}} as myscript.sh]
<source lang=bash>
#!/bin/bash

echo 'Hello world!'
exit 0
</source></nowiki>
</pre>
will give:
----
Let's save the following code [{{#file: myscript.sh}} as myscript.sh]
<source lang=bash>
#!/bin/bash

echo 'Hello world!'
exit 0
</source>

===Complete example===
And a full example with anchors & link:

<pre><nowiki>
Let's start with the Bash usual header:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
#!/bin/bash
</source>
Then we'll display a welcome message:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
echo 'Welcome on Earth!'
</source>
And we finally exit cleanly:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
exit 0
</source>
[{{#filelink: myotherscript.sh}} myotherscript.sh is now available for download below the code]
</nowiki></pre>
will give:
----
Let's start with the Bash usual header:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
#!/bin/bash
</source>
Then we'll display a welcome message:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
echo 'Welcome on Earth!'
</source>
And we finally exit cleanly:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
exit 0
</source>
[{{#filelink: myotherscript.sh}} myotherscript.sh is now available for download below the code]

==The code (the ultimate example)==
Which you can of course download just by following [{{#filelink: RawFile.php}} this link :-)]

So let's explain a bit the code in a Literate Programming way...
===Hooks===
First some hooks for our functions...

We will create:
* a [http://www.mediawiki.org/wiki/Manual:Parser_functions Parser Function] (see also [http://meta.wikimedia.org/wiki/Help:Parser_function here]), with help of
** [http://www.mediawiki.org/wiki/Manual:%24wgExtensionFunctions $wgExtensionFunctions] or [http://www.mediawiki.org/wiki/Manual:Hooks/ParserFirstCallInit ParserFirstCallInit global hook] to define the setup function
** [http://www.mediawiki.org/wiki/Manual:Magic_words Magic Words]
** [http://www.mediawiki.org/wiki/Manual:Tag_extensions Tag extensions]
** [http://www.mediawiki.org/wiki/Manual:Hooks/LanguageGetMagic LanguageGetMagic] hook to initialize the magic words
* a [http://www.mediawiki.org/wiki/Manual:Hooks/RawPageViewBeforeOutput RawPageViewBeforeOutput] hook to intercept the raw output

{{#fileanchor: RawFile.php}}
<source lang=php>
<?php

if (defined('MEDIAWIKI')) {

//Avoid unstubbing $wgParser on setHook() too early on modern (1.12+) MW versions, as per r35980
if ( defined( 'MW_SUPPORTS_PARSERFIRSTCALLINIT' ) ) {
$wgHooks['ParserFirstCallInit'][] = 'efRawFile_Setup';
} else { // Otherwise do things the old fashioned way
$wgExtensionFunctions[] = 'efRawFile_Setup';
}
$wgHooks['LanguageGetMagic'][] = 'efRawFile_Magic';
$wgHooks['RawPageViewBeforeOutput'][] = 'fnRawFile_Strip';

</source>

===Setup function===
For the wiki parsing to create download links, the parser functions '''file''' and '''fileLink''' are equally treated, while '''fileAnchor''' will be simply left out. We also create a new tag '''file''' as explained [http://www.mediawiki.org/wiki/Manual:Tag_extensions here].
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Setup() {
global $wgParser;
$wgParser->setFunctionHook( 'file', 'efRawFile_Render' );
$wgParser->setFunctionHook( 'filelink', 'efRawFile_Render' );
$wgParser->setFunctionHook( 'fileanchor', 'efRawFile_Empty' );
$wgParser->setHook( 'file', 'efRawFile_FileTagRender' );
return true;
}

</source>

===Hook to initialize the magic words===
We add the magic words here: the first array element indicates if it is case sensitive, in this case it is not case sensitive. We could add extra elements to create synonyms for our parser function.
 Unless we return true, other parser functions extensions will not get loaded.
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Magic( &$magicWords, $langCode ) {
$magicWords['file'] = array( 0, 'file' );
$magicWords['filelink'] = array( 0, 'filelink' );
$magicWords['fileanchor'] = array( 0, 'fileanchor' );
return true;
}

</source>

===Parser functions of the magic words===
The transformation rule to replace link shortcuts to actual links for download, handling an optional local wiki page title if present.
 The input parameters are wikitext with templates expanded, the output should be wikitext too
 '''TODO''': what error to send out if there is no filename given?
 '''EDIT''': It seems that [http://svn.wikimedia.org/viewvc/mediawiki?view=rev&revision=27667 commit 27667] (1.11 -> 1.12) changed the default parser, which breaks the recursive parsing. Thanks to Tim Starling for helping me to get around the problem!
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Render( &$parser, $filename = '', $titleText = '') {
if( $titleText == '' )
$title = $parser->mTitle;
else
$title = Title::newFromText( $titleText );
return $title->getFullURL( 'action=raw&anchor='.urlencode( $filename ) );
}

</source>
And the other one, just removing the anchors from the rendered wiki page.
 Curiously enough if the function doesn't exist at all the effect is exactly the same, MW doesn't throw any error.
 But let's keep things clean...
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Empty( &$parser, $filename = '') {
return '';
}

</source>

===Parser functions of the new tag <tt><file></tt>===
The transformation rule to replace <code><file></code> tag to actual links for download. The same parser function is used for both '''anchors''' and '''anchor-links'''. Since the link text may contain wiki text, we generate the link as wiki text that we ask the parser to parse again.
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_FileTagRender( $input, $args, $parser ) {
if( $args['title'] == '' )
$title = $parser->mTitle;
else
$title = Title::newFromText( $args['title'] );
$link=$title->getFullURL( 'action=raw' );
if( $args['name'] != '' )
$link.='&name='.urlencode( $args['name'] );
if( $args['anchor'] != '' )
$link.='&anchor='.urlencode( $args['anchor'] );
if( $args['tag'] != '' )
$link.='&tag='.urlencode( $args['tag'] );
return $parser->recursiveTagParse( "[$link $input]" );
}

</source>

===Hook to intercept the raw output===
This part of the code doesn't look that nice because we've to parse the raw wiki page ourselves to retrieve the code sections we want.

First we define a helper function that we will use to report error messages. This is simply done by replacing the content of the downloaded file with the error message and when necessary a copy of the raw text relevant to the error.
 '''TODO''': Cancel the file download header and return a proper error page
{{#fileanchor: RawFile.php}}
<source lang=php>
function fnRawFile_Strip_Error($msg,$out,&$text) {
$text=$msg;
if($out != '')
$text.="\nCandidate match: $out";
return true;
}

</source>

Next let's see if <code>?action=raw</code> was used in the context of this extension: in that case we receive the filename as GET parameter, otherwise we simply return from our extension with return value=true which means we authorize the raw display (originally the hook was created to add an authentication point)
{{#fileanchor: RawFile.php}}
<source lang=php>
function fnRawFile_Strip(&$rawPage, &$text) {
$filename=$_GET['name'];
$anchor=$_GET['anchor'];
// for backward compatibility, accept also URLs with parameter 'file'
if( $anchor=='' )
$anchor=$_GET['file'];
$tag=$_GET['tag'];
// Either anchor or name must be specified
if( $filename=='' )
$filename=$anchor;
if ( $filename=='' )
return true;
</source>
By default the downloadable file will still be handled by the ob_gzhandler session made by Mediawiki. To avoid output buffering and gzipping, one can uncomment the following line:
{{#fileanchor: RawFile.php}}
<source lang=php>
// Uncomment the following line to avoid output buffering and gzipping:
// wfResetOutputBuffers();
</source>
Raw action already set the headers with some client cache pragmas and is supposed to be displayed in the browser but in our case we want to make this "page" a downloadable file so we overwrite the headers which were defined and we add a few more, to ensure there is no caching on the client (it's very hard for the client to force a refresh on a file download, contrary to a web page) and to provide the adequate filename.
{{#fileanchor: RawFile.php}}
<source lang=php>
header("Content-disposition: attachment;filename={$filename}");
header("Content-type: application/octet-stream");
header("Content-Transfer-Encoding: binary");
header("Expires: 0");
header("Pragma: no-cache");
header("Cache-Control: no-store");
</source>
Then we'll strip the output, first we've to locate the anchors but there are anchors that could be protected in literal blocks like <code>nowiki</code>.
 So we'll mask the literal blocks before searching for the anchors (we mask with the same string length because we'll retrieve an offset that we will use on the initial string and offsets must match). This is done with the scary regex below:
* we use <code>!</code> instead of <code>/</code> as pattern indicator so that the pattern string is self-matching. This is necessary since we will apply the extension on this page as well.
* we use option <code>s</code> (multiline) and <code>e</code> (evaluate replace expression)
* Evaluated expression replaces all characters in the matched string with X's. However if there are single quote (<code>'</code>) in the matched string, they will be escaped with <code>\</code>. So we need to search for <code>\'|.</code>. The many back-slashes is because the expression is evaluated several times.
'''TODO''': should we care also of source, js, css, pre,... blocks?
{{#fileanchor: RawFile.php}}
<source lang=php>
$maskedtext=preg_replace_callback('!<nowiki>.*?</nowiki>!s',
function($m) { return ereg_replace(".","X",$m[0]); },
$text);
</source>
Now we can search for the anchors:
* If an anchor name is specified, we looked for '''all''' magic words <code><nowiki>{{#fileanchor:...}}</nowiki></code> or blocks with attribute <code>class="[someclass ]anchorname"</code>
* Otherwise we look for the first magic word <code><nowiki>{{#file:...}}</nowiki></code> with specified file name,
* And finally for the first <code><file></code> tag with the specified file name (no multiple blocks support)
And we free the memory used for the masked version
{{#fileanchor: RawFile.php}}
<source lang=php>
if (($anchor!='') && preg_match_all('/({{#fileanchor: *'.$anchor.' *}})|(<[^>]+ class *= *"([^"]*\w)?'.$anchor.'(\w[^"]*)?"[^>]*>)/i', $maskedtext, $matches, PREG_OFFSET_CAPTURE))
$offsets=$matches[0];
else if (preg_match_all('/{{#file: *'.$anchor.' *}}/i', $maskedtext, $matches, PREG_OFFSET_CAPTURE))
$offsets=array($matches[0][0]);
else if (preg_match_all('/<file( [^>]*)? name *= *"'.$filename.'"[^>]*>/i', $maskedtext, $matches, PREG_OFFSET_CAPTURE))
$offsets=array($matches[0][0]);
else {
// We didn't find our anchor
return fnRawFile_Strip_Error("ERROR - RawFile: anchor not found (anchor=$anchor, name=$filename, tag=$tag)","",$text);
}
unset($maskedtext);
</source>
$text is both input & output so we copy it and start with an empty output.
{{#fileanchor: RawFile.php}}
<source lang=php>
$textorig=$text;
$text='';
</source>
For each anchor found we've to isolate the content of the next block.
{{#fileanchor: RawFile.php}}
<source lang=php>
foreach ($offsets as $offset) {
</source>
We start from the position of the current anchor. If the tag name of the block attached to the anchor is not specified, we look for the first block that follows the anchor, excluding <code> </code> and <code><file></code> block. The search can be easily done with a regular expression, using the ''lookahead negative assertion'' <code>(?!br\b|file\b)</code> to exclude the tags to ignore. Note that we need to ignore the anchor-link block <code><file></code> since the anchor starts right before that tag, and so the regular expression would match the anchor-link block it that tag is not specifically excluded.
{{#fileanchor: RawFile.php}}
<source lang=php>
$out = substr($textorig, $offset[1]);
// If no tag specified, we take the first one
if ($tag == '')
{
// With a regex assertion, we can easily ignore 'br' and 'file' tags
if (!preg_match('/<((?!br\b|file\b)\w+\b)/', $out, $matches))
return fnRawFile_Strip_Error ("ERROR - RawFile: Can't find opening tag after anchor '$offset[0]' (anchor=$anchor, name=$filename, tag=$tag)",$out,$text);
$tag=$matches[1];
}
</source>
Now, we know the tag name of the block to download, either because it was already specified as a GET attribute in the URL, or because we've found it in the search above. Again, using a regular expression, we look for the first block matching the specified tag name that follows the current anchor, and extract the content of the blocks. Note the use of the regex option <code>/.../s</code> to tell the regex engine that the matched text can span on multiple lines (with that option, <code>.</code> does match any character or a newline character). Also, we skip the first carriage return after the opening tag, if any (with <code>\n?</code>).
{{#fileanchor: RawFile.php}}
<source lang=php>
// Find the first tag matching $tag, and return enclosed text
if (!preg_match('/<'.$tag.'( [^>]*)?>\n?(.*?)<\/'.$tag.'>/s', $out, $matches))
return fnRawFile_Strip_Error ("ERROR - RawFile: no closing '$tag' found after anchor '$offset[0]' (anchor=$anchor, name=$filename, tag=$tag)",$out,$text);
$text .= $matches[2];
}
</source>
No need to deal with a Content-Length header because Mediawiki will do it for us, moreover more properly than we could if the output is sent gzipped, which is the default.
 So that's it, $text contains our file!
{{#fileanchor: RawFile.php}}
<source lang=php>
return true;
}

</source>

===Credits===
There is an official way to register the extension in a Mediawiki installation, so that it will be visible on the [[Special:Version]] page.
 Let's say the extension is in the category of parser hooks even if there is also a hook on Raw action.
{{#fileanchor: RawFile.php}}
<source lang=php>
$wgExtensionCredits['parserhook'][] = array('name' => 'RawFile',
'version' => '0.5',
'author' => 'Philippe Teuwen, Michael Peeters',
'url' => 'http://www.mediawiki.org/wiki/Extension:RawFile',
// 'url' => 'http://wiki.yobi.be/wiki/Mediawiki_RawFile',
'description' => 'Downloads a RAW copy of <nowiki><tag>data</tag></nowiki> in a file '.
'Useful e.g. to download a script or a patch '.
'It also allows what is called [http://en.wikipedia.org/wiki/Literate_programming Literate Programming]');
}

?>
</source>
And finally registration of the extension at the Mediawiki website according to the [http://www.mediawiki.org/wiki/Manual:Extensions Extensions Manual].

So this extension has now [http://www.mediawiki.org/wiki/Extension:RawFile its own page on the official Mediawiki site].

==Installation==
Download [{{#filelink: RawFile.php}} RawFile.php] and save it under the MediaWiki directory as extensions/RawFile/RawFile.php

Add at the end of LocalSettings.php:
<source lang=php>
require_once("$IP/extensions/RawFile/RawFile.php");
</source>
==Status==
If you use the extension properly the code is fully functional but it's rather raw on error handling.
==ChangeLog==
'''0.5'''
* Fix since PHP 5.5.0: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead. Thanks to Stephen Kent for reporting.
* '''WARNING''' you should upgrade ASAP as the previous versions are vulnerable to remote PHP code injection!!! See [[Talk:Mediawiki_RawFile]] for more info.
* Fix for MediaWiki 1.22.1.
'''0.4.1'''
* Fix octet-stream MIME type bug which was affecting Epiphany & Opera 11. Thanks to Jani Uusitalo for reporting & [[User:Sicvolo|Sicvolo]] for finding the solution
'''0.4'''
* '''Anchors''' can be specified using html '''class''' attribute
* New syntax for '''Links''' and '''Anchor-links''':
:<code><nowiki><file [name="..."] [anchor="..."] [tag="..."] [title="..."] >Link text</file></nowiki></code>
* Support multiple files on the same page with same name (differentiated by their anchor name) or even common blocks in multiple files.
* Can specify the tag name of the block to download (to skip some irrelevant blocks when using an '''anchor-link''').
* Ignore '''<code> </code>''' tag.
* Some error reporting.
'''0.3'''
* Added optional parameter to <code>#fileLink</code> to indicate that the file is on another local wiki page
'''0.2'''
* Fix problem with Content-Length mismatch when transport is gzipped (default for Mediawiki if client supports it)
'''0.1'''
* Initial version

==Known bugs==
==Questions and feedback==
If you've any trouble, questions or suggestions, you can [[User:PhilippeTeuwen|contact me]].
==Known sites using the extension==
* [http://tech.ivkin.net/wiki/Main_Page Tech Knowledge Base Wiki] by [[User:Sicvolo|Sicvolo]]
* [http://far.no/fram/index.php?title=Fram Far.no/Fram]
* Well, this site of course! where I update the actual file on the server with a simple <code>wget -N --content-disposition "http://wiki.yobi.be/index.php?title=Mediawiki_RawFile&action=raw&file=RawFile.php"</code>

Mediawiki RawFile

2014-02-13T15:30:26Z

Fuujuhi: /* Parser functions of the new tag <file> */ Fix for mw1.22.1 (UNIQxxxQINU tags appearing instead of file name)

==Very short introduction==
Just have a look to the 2 [[Mediawiki_RawFile#Short_example|examples]] to see how to use the extension
 and to the [[Mediawiki_RawFile#Installation|Installation]] section to see how to install the extension in your [[MediaWiki]] server
==Introduction==
Originally the idea was to be able to download directly a portion of code as a file.
 I've numerous code examples in my wiki and I wanted an easy way to download them, easier than a copy/paste!
 But from there it was rather easy to get something very close to [http://en.wikipedia.org/wiki/Literate_programming literate programming] just by allowing multiple blocks referring to the same file, which will be concatenated together at download time.

* It must work with pre, nowiki, js, css, code, source, so let's make it general: take the tag that comes after the parser function we'll create and select data up to the closing tag.
* There are two distinct functionalities provided by the extension:
** the parser that will convert a magic word into a link to the download URL
** an extended ?action=raw that will strip the raw output to keep the desired code

==Syntax==
The extension introduces 3 elements:
;Anchor
: Used to flag that the next code block in the wiki text belongs to a specific file. The code block can be any wiki block (such as '''<code><pre></code>''', '''<code><code></code>''', '''<code><tt></code>''', '''<code><source></code>'''...). '''<code> </code>''' tags are ignored. Note that anchors are invisible in the wiki display.
;Link
: They are transformed by the extension into links that allows for downloading all blocks attached to a given anchor name.
;Anchor-link
: A shortcut notation mixing both an anchor and download link, handy for regular use, when a single code block is used and when the download link can be at the same position as the anchor.

The syntax is as follows. The syntax using tag <code><file></code> and tag attribute <code>class</code> is new since v0.4. Note that elements of both syntaxes can be mixed in a same page.
{| border="2" cellspacing="4" cellpadding="3" style="margin: 1em 1em 1em 0; background: #f9f9f9; border: 1px #aaaaaa solid; border-collapse: collapse; empty-cells:show;"
!width="80em" style="background: #8da7d6;"|Element!!style="background: #8da7d6;"|Syntax and description
|-
|'''Anchor'''
|<pre><nowiki>
{{#fileAnchor: anchorname}}
<pre class='anchorname'>...</pre>
<code class="anchorname">...</code>
<code class="cssclass anchorname">...</code>
...
</nowiki></pre>
Indicates that the next wiki block is attached to an anchor ''anchorname''. The content of that block will be downloaded (possibly appended with other blocks if there are several blocks attached to the same ''anchorname'') when a file link is clicked on. 
'''(since v0.4)''' To attach an anchor ''anchorname'' to a wiki block, simply add an attribute <code>class="anchorname"</code> to it. The extension supports multi-class specification, meaning that a same block can be associated to different files, and that the <code>class</code> attribute can still be used to specify custom CSS properties as in standard wiki text.
; ''anchorname''
; class="''anchorname''"
: The name of the anchor to which the wiki block is attached
|-
|'''Link'''
|<pre><nowiki>
[{{#fileLink: anchorname}} link text]
[{{#fileLink: anchorname|pagetitle}} link text]
<file anchor="anchorname" [name="filename"] [title="pagetitle"]>link text</file>
</nowiki></pre>
Creates a link to download all blocks that are attached to an anchor ''anchorname''.
;''anchorname''
;anchor="''anchorname''"
: The name of the anchor to look for. All blocks attached to an anchor ''anchorname'' will be downloaded.
;name="''filename''"
:''Optional'' - Specifies the name of the file to download. If absent, ''anchorname'' is then used as the name of the downloaded file.
; ''pagetitle''
;title="''pagetitle''"
: ''Optional'' - Indicates that the blocks to download are on the wiki page titled ''pagetitle''. If absent, blocks are looked for on the current page.
; ''link text''
: The text of the link to display.
|-
|'''Anchor-link'''
|<pre><nowiki>
[{{#file: filename}} link text]
<file name="filename" [tag="''tagname''"]>link text</file>
</nowiki></pre>
Creates a link to download the next wiki block as a file named ''filename''. 
'''(since v0.4)''' The attribute <code>tag</code> can be used to specify the ''tagname'' of the block to download. 
; ''filename''
; name="''filename''"
: The name of the file to download.
;tag="''tagname''"
:''Optional'' - When set, the extension only looks for blocks whose name matches the given ''tagname''. This attribute is particularly useful when there are some irrelevant blocks between the '''anchor-link''' and the block you want to download. If absent, the first encountered block following the anchor is downloaded.
; ''link text''
: The text of the link to display.
|}

===Short example===
The extension works with any block such as pre, nowiki, js, css, code, source,...
 This example is using the syntax highlighting <nowiki><source></nowiki> tag provided by [http://www.mediawiki.org/wiki/Extension:SyntaxHighlight_GeSHi SyntaxHighlight extension] (using [http://qbnz.com/highlighter/ GeSHi Highlighter])
 If you didn't install that extension on your MediaWiki, you can try the example by using <nowiki><pre></nowiki> instead of <nowiki><source></nowiki>.

<pre><nowiki>Let's save the following code [{{#file: myscript.sh}} as myscript.sh]
<source lang=bash>
#!/bin/bash

echo 'Hello world!'
exit 0
</source></nowiki>
</pre>
will give:
----
Let's save the following code [{{#file: myscript.sh}} as myscript.sh]
<source lang=bash>
#!/bin/bash

echo 'Hello world!'
exit 0
</source>

===Complete example===
And a full example with anchors & link:

<pre><nowiki>
Let's start with the Bash usual header:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
#!/bin/bash
</source>
Then we'll display a welcome message:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
echo 'Welcome on Earth!'
</source>
And we finally exit cleanly:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
exit 0
</source>
[{{#filelink: myotherscript.sh}} myotherscript.sh is now available for download below the code]
</nowiki></pre>
will give:
----
Let's start with the Bash usual header:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
#!/bin/bash
</source>
Then we'll display a welcome message:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
echo 'Welcome on Earth!'
</source>
And we finally exit cleanly:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
exit 0
</source>
[{{#filelink: myotherscript.sh}} myotherscript.sh is now available for download below the code]

==The code (the ultimate example)==
Which you can of course download just by following [{{#filelink: RawFile.php}} this link :-)]

So let's explain a bit the code in a Literate Programming way...
===Hooks===
First some hooks for our functions...

We will create:
* a [http://www.mediawiki.org/wiki/Manual:Parser_functions Parser Function] (see also [http://meta.wikimedia.org/wiki/Help:Parser_function here]), with help of
** [http://www.mediawiki.org/wiki/Manual:%24wgExtensionFunctions $wgExtensionFunctions] or [http://www.mediawiki.org/wiki/Manual:Hooks/ParserFirstCallInit ParserFirstCallInit global hook] to define the setup function
** [http://www.mediawiki.org/wiki/Manual:Magic_words Magic Words]
** [http://www.mediawiki.org/wiki/Manual:Tag_extensions Tag extensions]
** [http://www.mediawiki.org/wiki/Manual:Hooks/LanguageGetMagic LanguageGetMagic] hook to initialize the magic words
* a [http://www.mediawiki.org/wiki/Manual:Hooks/RawPageViewBeforeOutput RawPageViewBeforeOutput] hook to intercept the raw output

{{#fileanchor: RawFile.php}}
<source lang=php>
<?php

if (defined('MEDIAWIKI')) {

//Avoid unstubbing $wgParser on setHook() too early on modern (1.12+) MW versions, as per r35980
if ( defined( 'MW_SUPPORTS_PARSERFIRSTCALLINIT' ) ) {
$wgHooks['ParserFirstCallInit'][] = 'efRawFile_Setup';
} else { // Otherwise do things the old fashioned way
$wgExtensionFunctions[] = 'efRawFile_Setup';
}
$wgHooks['LanguageGetMagic'][] = 'efRawFile_Magic';
$wgHooks['RawPageViewBeforeOutput'][] = 'fnRawFile_Strip';

</source>

===Setup function===
For the wiki parsing to create download links, the parser functions '''file''' and '''fileLink''' are equally treated, while '''fileAnchor''' will be simply left out. We also create a new tag '''file''' as explained [http://www.mediawiki.org/wiki/Manual:Tag_extensions here].
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Setup() {
global $wgParser;
$wgParser->setFunctionHook( 'file', 'efRawFile_Render' );
$wgParser->setFunctionHook( 'filelink', 'efRawFile_Render' );
$wgParser->setFunctionHook( 'fileanchor', 'efRawFile_Empty' );
$wgParser->setHook( 'file', 'efRawFile_FileTagRender' );
return true;
}

</source>

===Hook to initialize the magic words===
We add the magic words here: the first array element indicates if it is case sensitive, in this case it is not case sensitive. We could add extra elements to create synonyms for our parser function.
 Unless we return true, other parser functions extensions will not get loaded.
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Magic( &$magicWords, $langCode ) {
$magicWords['file'] = array( 0, 'file' );
$magicWords['filelink'] = array( 0, 'filelink' );
$magicWords['fileanchor'] = array( 0, 'fileanchor' );
return true;
}

</source>

===Parser functions of the magic words===
The transformation rule to replace link shortcuts to actual links for download, handling an optional local wiki page title if present.
 The input parameters are wikitext with templates expanded, the output should be wikitext too
 '''TODO''': what error to send out if there is no filename given?
 '''EDIT''': It seems that [http://svn.wikimedia.org/viewvc/mediawiki?view=rev&revision=27667 commit 27667] (1.11 -> 1.12) changed the default parser, which breaks the recursive parsing. Thanks to Tim Starling for helping me to get around the problem!
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Render( &$parser, $filename = '', $titleText = '') {
if( $titleText == '' )
$title = $parser->mTitle;
else
$title = Title::newFromText( $titleText );
return $title->getFullURL( 'action=raw&anchor='.urlencode( $filename ) );
}

</source>
And the other one, just removing the anchors from the rendered wiki page.
 Curiously enough if the function doesn't exist at all the effect is exactly the same, MW doesn't throw any error.
 But let's keep things clean...
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Empty( &$parser, $filename = '') {
return '';
}

</source>

===Parser functions of the new tag <tt><file></tt>===
The transformation rule to replace <code><file></code> tag to actual links for download. The same parser function is used for both '''anchors''' and '''anchor-links'''. Since the link text may contain wiki text, we generate the link as wiki text that we ask the parser to parse again.
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_FileTagRender( $input, $args, $parser ) {
if( $args['title'] == '' )
$title = $parser->mTitle;
else
$title = Title::newFromText( $args['title'] );
$link=$title->getFullURL( 'action=raw' );
if( $args['name'] != '' )
$link.='&name='.urlencode( $args['name'] );
if( $args['anchor'] != '' )
$link.='&anchor='.urlencode( $args['anchor'] );
if( $args['tag'] != '' )
$link.='&tag='.urlencode( $args['tag'] );
return $parser->recursiveTagParse( "[$link $input]" );
}

</source>

===Hook to intercept the raw output===
This part of the code doesn't look that nice because we've to parse the raw wiki page ourselves to retrieve the code sections we want.

First we define a helper function that we will use to report error messages. This is simply done by replacing the content of the downloaded file with the error message and when necessary a copy of the raw text relevant to the error.
 '''TODO''': Cancel the file download header and return a proper error page
{{#fileanchor: RawFile.php}}
<source lang=php>
function fnRawFile_Strip_Error($msg,$out,&$text) {
$text=$msg;
if($out != '')
$text.="\nCandidate match: $out";
return true;
}

</source>

Next let's see if <code>?action=raw</code> was used in the context of this extension: in that case we receive the filename as GET parameter, otherwise we simply return from our extension with return value=true which means we authorize the raw display (originally the hook was created to add an authentication point)
{{#fileanchor: RawFile.php}}
<source lang=php>
function fnRawFile_Strip(&$rawPage, &$text) {
$filename=$_GET['name'];
$anchor=$_GET['anchor'];
// for backward compatibility, accept also URLs with parameter 'file'
if( $anchor=='' )
$anchor=$_GET['file'];
$tag=$_GET['tag'];
// Either anchor or name must be specified
if( $filename=='' )
$filename=$anchor;
if ( $filename=='' )
return true;
</source>
By default the downloadable file will still be handled by the ob_gzhandler session made by Mediawiki. To avoid output buffering and gzipping, one can uncomment the following line:
{{#fileanchor: RawFile.php}}
<source lang=php>
// Uncomment the following line to avoid output buffering and gzipping:
// wfResetOutputBuffers();
</source>
Raw action already set the headers with some client cache pragmas and is supposed to be displayed in the browser but in our case we want to make this "page" a downloadable file so we overwrite the headers which were defined and we add a few more, to ensure there is no caching on the client (it's very hard for the client to force a refresh on a file download, contrary to a web page) and to provide the adequate filename.
{{#fileanchor: RawFile.php}}
<source lang=php>
header("Content-disposition: attachment;filename={$filename}");
header("Content-type: application/octet-stream");
header("Content-Transfer-Encoding: binary");
header("Expires: 0");
header("Pragma: no-cache");
header("Cache-Control: no-store");
</source>
Then we'll strip the output, first we've to locate the anchors but there are anchors that could be protected in literal blocks like <code>nowiki</code>.
 So we'll mask the literal blocks before searching for the anchors (we mask with the same string length because we'll retrieve an offset that we will use on the initial string and offsets must match). This is done with the scary regex below:
* we use <code>!</code> instead of <code>/</code> as pattern indicator so that the pattern string is self-matching. This is necessary since we will apply the extension on this page as well.
* we use option <code>s</code> (multiline) and <code>e</code> (evaluate replace expression)
* Evaluated expression replaces all characters in the matched string with X's. However if there are single quote (<code>'</code>) in the matched string, they will be escaped with <code>\</code>. So we need to search for <code>\'|.</code>. The many back-slashes is because the expression is evaluated several times.
'''TODO''': should we care also of source, js, css, pre,... blocks?
{{#fileanchor: RawFile.php}}
<source lang=php>
$maskedtext=preg_replace_callback('!<nowiki>.*?</nowiki>!s',
function($m) { return ereg_replace(".","X",$m[0]); },
$text);
</source>
Now we can search for the anchors:
* If an anchor name is specified, we looked for '''all''' magic words <code><nowiki>{{#fileanchor:...}}</nowiki></code> or blocks with attribute <code>class="[someclass ]anchorname"</code>
* Otherwise we look for the first magic word <code><nowiki>{{#file:...}}</nowiki></code> with specified file name,
* And finally for the first <code><file></code> tag with the specified file name (no multiple blocks support)
And we free the memory used for the masked version
{{#fileanchor: RawFile.php}}
<source lang=php>
if (($anchor!='') && preg_match_all('/({{#fileanchor: *'.$anchor.' *}})|(<[^>]+ class *= *"([^"]*\w)?'.$anchor.'(\w[^"]*)?"[^>]*>)/i', $maskedtext, $matches, PREG_OFFSET_CAPTURE))
$offsets=$matches[0];
else if (preg_match_all('/{{#file: *'.$anchor.' *}}/i', $maskedtext, $matches, PREG_OFFSET_CAPTURE))
$offsets=array($matches[0][0]);
else if (preg_match_all('/<file( [^>]*)? name *= *"'.$filename.'"[^>]*>/i', $maskedtext, $matches, PREG_OFFSET_CAPTURE))
$offsets=array($matches[0][0]);
else {
// We didn't find our anchor
return fnRawFile_Strip_Error("ERROR - RawFile: anchor not found (anchor=$anchor, name=$filename, tag=$tag)","",$text);
}
unset($maskedtext);
</source>
$text is both input & output so we copy it and start with an empty output.
{{#fileanchor: RawFile.php}}
<source lang=php>
$textorig=$text;
$text='';
</source>
For each anchor found we've to isolate the content of the next block.
{{#fileanchor: RawFile.php}}
<source lang=php>
foreach ($offsets as $offset) {
</source>
We start from the position of the current anchor. If the tag name of the block attached to the anchor is not specified, we look for the first block that follows the anchor, excluding <code> </code> and <code><file></code> block. The search can be easily done with a regular expression, using the ''lookahead negative assertion'' <code>(?!br\b|file\b)</code> to exclude the tags to ignore. Note that we need to ignore the anchor-link block <code><file></code> since the anchor starts right before that tag, and so the regular expression would match the anchor-link block it that tag is not specifically excluded.
{{#fileanchor: RawFile.php}}
<source lang=php>
$out = substr($textorig, $offset[1]);
// If no tag specified, we take the first one
if ($tag == '')
{
// With a regex assertion, we can easily ignore 'br' and 'file' tags
if (!preg_match('/<((?!br\b|file\b)\w+\b)/', $out, $matches))
return fnRawFile_Strip_Error ("ERROR - RawFile: Can't find opening tag after anchor '$offset[0]' (anchor=$anchor, name=$filename, tag=$tag)",$out,$text);
$tag=$matches[1];
}
</source>
Now, we know the tag name of the block to download, either because it was already specified as a GET attribute in the URL, or because we've found it in the search above. Again, using a regular expression, we look for the first block matching the specified tag name that follows the current anchor, and extract the content of the blocks. Note the use of the regex option <code>/.../s</code> to tell the regex engine that the matched text can span on multiple lines (with that option, <code>.</code> does match any character or a newline character). Also, we skip the first carriage return after the opening tag, if any (with <code>\n?</code>).
{{#fileanchor: RawFile.php}}
<source lang=php>
// Find the first tag matching $tag, and return enclosed text
if (!preg_match('/<'.$tag.'( [^>]*)?>\n?(.*?)<\/'.$tag.'>/s', $out, $matches))
return fnRawFile_Strip_Error ("ERROR - RawFile: no closing '$tag' found after anchor '$offset[0]' (anchor=$anchor, name=$filename, tag=$tag)",$out,$text);
$text .= $matches[2];
}
</source>
No need to deal with a Content-Length header because Mediawiki will do it for us, moreover more properly than we could if the output is sent gzipped, which is the default.
 So that's it, $text contains our file!
{{#fileanchor: RawFile.php}}
<source lang=php>
return true;
}

</source>

===Credits===
There is an official way to register the extension in a Mediawiki installation, so that it will be visible on the [[Special:Version]] page.
 Let's say the extension is in the category of parser hooks even if there is also a hook on Raw action.
{{#fileanchor: RawFile.php}}
<source lang=php>
$wgExtensionCredits['parserhook'][] = array('name' => 'RawFile',
'version' => '0.5',
'author' => 'Philippe Teuwen, Michael Peeters',
'url' => 'http://www.mediawiki.org/wiki/Extension:RawFile',
// 'url' => 'http://wiki.yobi.be/wiki/Mediawiki_RawFile',
'description' => 'Downloads a RAW copy of <nowiki><tag>data</tag></nowiki> in a file '.
'Useful e.g. to download a script or a patch '.
'It also allows what is called [http://en.wikipedia.org/wiki/Literate_programming Literate Programming]');
}

?>
</source>
And finally registration of the extension at the Mediawiki website according to the [http://www.mediawiki.org/wiki/Manual:Extensions Extensions Manual].

So this extension has now [http://www.mediawiki.org/wiki/Extension:RawFile its own page on the official Mediawiki site].

==Installation==
Download [{{#filelink: RawFile.php}} RawFile.php] and save it under the MediaWiki directory as extensions/RawFile/RawFile.php

Add at the end of LocalSettings.php:
<source lang=php>
require_once("$IP/extensions/RawFile/RawFile.php");
</source>
==Status==
If you use the extension properly the code is fully functional but it's rather raw on error handling.
==ChangeLog==
'''0.5'''
* Fix since PHP 5.5.0: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead. Thanks to Stephen Kent for reporting.
* '''WARNING''' you should upgrade ASAP as the previous versions are vulnerable to remote PHP code injection!!! See [[Talk:Mediawiki_RawFile]] for more info.
'''0.4.1'''
* Fix octet-stream MIME type bug which was affecting Epiphany & Opera 11. Thanks to Jani Uusitalo for reporting & [[User:Sicvolo|Sicvolo]] for finding the solution
'''0.4'''
* '''Anchors''' can be specified using html '''class''' attribute
* New syntax for '''Links''' and '''Anchor-links''':
:<code><nowiki><file [name="..."] [anchor="..."] [tag="..."] [title="..."] >Link text</file></nowiki></code>
* Support multiple files on the same page with same name (differentiated by their anchor name) or even common blocks in multiple files.
* Can specify the tag name of the block to download (to skip some irrelevant blocks when using an '''anchor-link''').
* Ignore '''<code> </code>''' tag.
* Some error reporting.
'''0.3'''
* Added optional parameter to <code>#fileLink</code> to indicate that the file is on another local wiki page
'''0.2'''
* Fix problem with Content-Length mismatch when transport is gzipped (default for Mediawiki if client supports it)
'''0.1'''
* Initial version

==Known bugs==
==Questions and feedback==
If you've any trouble, questions or suggestions, you can [[User:PhilippeTeuwen|contact me]].
==Known sites using the extension==
* [http://tech.ivkin.net/wiki/Main_Page Tech Knowledge Base Wiki] by [[User:Sicvolo|Sicvolo]]
* [http://far.no/fram/index.php?title=Fram Far.no/Fram]
* Well, this site of course! where I update the actual file on the server with a simple <code>wget -N --content-disposition "http://wiki.yobi.be/index.php?title=Mediawiki_RawFile&action=raw&file=RawFile.php"</code>

Mediawiki RawFile

2014-02-13T15:29:14Z

Fuujuhi: /* Credits */ Bump version

==Very short introduction==
Just have a look to the 2 [[Mediawiki_RawFile#Short_example|examples]] to see how to use the extension
 and to the [[Mediawiki_RawFile#Installation|Installation]] section to see how to install the extension in your [[MediaWiki]] server
==Introduction==
Originally the idea was to be able to download directly a portion of code as a file.
 I've numerous code examples in my wiki and I wanted an easy way to download them, easier than a copy/paste!
 But from there it was rather easy to get something very close to [http://en.wikipedia.org/wiki/Literate_programming literate programming] just by allowing multiple blocks referring to the same file, which will be concatenated together at download time.

* It must work with pre, nowiki, js, css, code, source, so let's make it general: take the tag that comes after the parser function we'll create and select data up to the closing tag.
* There are two distinct functionalities provided by the extension:
** the parser that will convert a magic word into a link to the download URL
** an extended ?action=raw that will strip the raw output to keep the desired code

==Syntax==
The extension introduces 3 elements:
;Anchor
: Used to flag that the next code block in the wiki text belongs to a specific file. The code block can be any wiki block (such as '''<code><pre></code>''', '''<code><code></code>''', '''<code><tt></code>''', '''<code><source></code>'''...). '''<code> </code>''' tags are ignored. Note that anchors are invisible in the wiki display.
;Link
: They are transformed by the extension into links that allows for downloading all blocks attached to a given anchor name.
;Anchor-link
: A shortcut notation mixing both an anchor and download link, handy for regular use, when a single code block is used and when the download link can be at the same position as the anchor.

The syntax is as follows. The syntax using tag <code><file></code> and tag attribute <code>class</code> is new since v0.4. Note that elements of both syntaxes can be mixed in a same page.
{| border="2" cellspacing="4" cellpadding="3" style="margin: 1em 1em 1em 0; background: #f9f9f9; border: 1px #aaaaaa solid; border-collapse: collapse; empty-cells:show;"
!width="80em" style="background: #8da7d6;"|Element!!style="background: #8da7d6;"|Syntax and description
|-
|'''Anchor'''
|<pre><nowiki>
{{#fileAnchor: anchorname}}
<pre class='anchorname'>...</pre>
<code class="anchorname">...</code>
<code class="cssclass anchorname">...</code>
...
</nowiki></pre>
Indicates that the next wiki block is attached to an anchor ''anchorname''. The content of that block will be downloaded (possibly appended with other blocks if there are several blocks attached to the same ''anchorname'') when a file link is clicked on. 
'''(since v0.4)''' To attach an anchor ''anchorname'' to a wiki block, simply add an attribute <code>class="anchorname"</code> to it. The extension supports multi-class specification, meaning that a same block can be associated to different files, and that the <code>class</code> attribute can still be used to specify custom CSS properties as in standard wiki text.
; ''anchorname''
; class="''anchorname''"
: The name of the anchor to which the wiki block is attached
|-
|'''Link'''
|<pre><nowiki>
[{{#fileLink: anchorname}} link text]
[{{#fileLink: anchorname|pagetitle}} link text]
<file anchor="anchorname" [name="filename"] [title="pagetitle"]>link text</file>
</nowiki></pre>
Creates a link to download all blocks that are attached to an anchor ''anchorname''.
;''anchorname''
;anchor="''anchorname''"
: The name of the anchor to look for. All blocks attached to an anchor ''anchorname'' will be downloaded.
;name="''filename''"
:''Optional'' - Specifies the name of the file to download. If absent, ''anchorname'' is then used as the name of the downloaded file.
; ''pagetitle''
;title="''pagetitle''"
: ''Optional'' - Indicates that the blocks to download are on the wiki page titled ''pagetitle''. If absent, blocks are looked for on the current page.
; ''link text''
: The text of the link to display.
|-
|'''Anchor-link'''
|<pre><nowiki>
[{{#file: filename}} link text]
<file name="filename" [tag="''tagname''"]>link text</file>
</nowiki></pre>
Creates a link to download the next wiki block as a file named ''filename''. 
'''(since v0.4)''' The attribute <code>tag</code> can be used to specify the ''tagname'' of the block to download. 
; ''filename''
; name="''filename''"
: The name of the file to download.
;tag="''tagname''"
:''Optional'' - When set, the extension only looks for blocks whose name matches the given ''tagname''. This attribute is particularly useful when there are some irrelevant blocks between the '''anchor-link''' and the block you want to download. If absent, the first encountered block following the anchor is downloaded.
; ''link text''
: The text of the link to display.
|}

===Short example===
The extension works with any block such as pre, nowiki, js, css, code, source,...
 This example is using the syntax highlighting <nowiki><source></nowiki> tag provided by [http://www.mediawiki.org/wiki/Extension:SyntaxHighlight_GeSHi SyntaxHighlight extension] (using [http://qbnz.com/highlighter/ GeSHi Highlighter])
 If you didn't install that extension on your MediaWiki, you can try the example by using <nowiki><pre></nowiki> instead of <nowiki><source></nowiki>.

<pre><nowiki>Let's save the following code [{{#file: myscript.sh}} as myscript.sh]
<source lang=bash>
#!/bin/bash

echo 'Hello world!'
exit 0
</source></nowiki>
</pre>
will give:
----
Let's save the following code [{{#file: myscript.sh}} as myscript.sh]
<source lang=bash>
#!/bin/bash

echo 'Hello world!'
exit 0
</source>

===Complete example===
And a full example with anchors & link:

<pre><nowiki>
Let's start with the Bash usual header:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
#!/bin/bash
</source>
Then we'll display a welcome message:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
echo 'Welcome on Earth!'
</source>
And we finally exit cleanly:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
exit 0
</source>
[{{#filelink: myotherscript.sh}} myotherscript.sh is now available for download below the code]
</nowiki></pre>
will give:
----
Let's start with the Bash usual header:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
#!/bin/bash
</source>
Then we'll display a welcome message:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
echo 'Welcome on Earth!'
</source>
And we finally exit cleanly:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
exit 0
</source>
[{{#filelink: myotherscript.sh}} myotherscript.sh is now available for download below the code]

==The code (the ultimate example)==
Which you can of course download just by following [{{#filelink: RawFile.php}} this link :-)]

So let's explain a bit the code in a Literate Programming way...
===Hooks===
First some hooks for our functions...

We will create:
* a [http://www.mediawiki.org/wiki/Manual:Parser_functions Parser Function] (see also [http://meta.wikimedia.org/wiki/Help:Parser_function here]), with help of
** [http://www.mediawiki.org/wiki/Manual:%24wgExtensionFunctions $wgExtensionFunctions] or [http://www.mediawiki.org/wiki/Manual:Hooks/ParserFirstCallInit ParserFirstCallInit global hook] to define the setup function
** [http://www.mediawiki.org/wiki/Manual:Magic_words Magic Words]
** [http://www.mediawiki.org/wiki/Manual:Tag_extensions Tag extensions]
** [http://www.mediawiki.org/wiki/Manual:Hooks/LanguageGetMagic LanguageGetMagic] hook to initialize the magic words
* a [http://www.mediawiki.org/wiki/Manual:Hooks/RawPageViewBeforeOutput RawPageViewBeforeOutput] hook to intercept the raw output

{{#fileanchor: RawFile.php}}
<source lang=php>
<?php

if (defined('MEDIAWIKI')) {

//Avoid unstubbing $wgParser on setHook() too early on modern (1.12+) MW versions, as per r35980
if ( defined( 'MW_SUPPORTS_PARSERFIRSTCALLINIT' ) ) {
$wgHooks['ParserFirstCallInit'][] = 'efRawFile_Setup';
} else { // Otherwise do things the old fashioned way
$wgExtensionFunctions[] = 'efRawFile_Setup';
}
$wgHooks['LanguageGetMagic'][] = 'efRawFile_Magic';
$wgHooks['RawPageViewBeforeOutput'][] = 'fnRawFile_Strip';

</source>

===Setup function===
For the wiki parsing to create download links, the parser functions '''file''' and '''fileLink''' are equally treated, while '''fileAnchor''' will be simply left out. We also create a new tag '''file''' as explained [http://www.mediawiki.org/wiki/Manual:Tag_extensions here].
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Setup() {
global $wgParser;
$wgParser->setFunctionHook( 'file', 'efRawFile_Render' );
$wgParser->setFunctionHook( 'filelink', 'efRawFile_Render' );
$wgParser->setFunctionHook( 'fileanchor', 'efRawFile_Empty' );
$wgParser->setHook( 'file', 'efRawFile_FileTagRender' );
return true;
}

</source>

===Hook to initialize the magic words===
We add the magic words here: the first array element indicates if it is case sensitive, in this case it is not case sensitive. We could add extra elements to create synonyms for our parser function.
 Unless we return true, other parser functions extensions will not get loaded.
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Magic( &$magicWords, $langCode ) {
$magicWords['file'] = array( 0, 'file' );
$magicWords['filelink'] = array( 0, 'filelink' );
$magicWords['fileanchor'] = array( 0, 'fileanchor' );
return true;
}

</source>

===Parser functions of the magic words===
The transformation rule to replace link shortcuts to actual links for download, handling an optional local wiki page title if present.
 The input parameters are wikitext with templates expanded, the output should be wikitext too
 '''TODO''': what error to send out if there is no filename given?
 '''EDIT''': It seems that [http://svn.wikimedia.org/viewvc/mediawiki?view=rev&revision=27667 commit 27667] (1.11 -> 1.12) changed the default parser, which breaks the recursive parsing. Thanks to Tim Starling for helping me to get around the problem!
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Render( &$parser, $filename = '', $titleText = '') {
if( $titleText == '' )
$title = $parser->mTitle;
else
$title = Title::newFromText( $titleText );
return $title->getFullURL( 'action=raw&anchor='.urlencode( $filename ) );
}

</source>
And the other one, just removing the anchors from the rendered wiki page.
 Curiously enough if the function doesn't exist at all the effect is exactly the same, MW doesn't throw any error.
 But let's keep things clean...
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Empty( &$parser, $filename = '') {
return '';
}

</source>

===Parser functions of the new tag <tt><file></tt>===
The transformation rule to replace <code><file></code> tag to actual links for download. The same parser function is used for both '''anchors''' and '''anchor-links'''. Since the link text may contain wiki text, we generate the link as wiki text that we ask the parser to parse again.
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_FileTagRender( $input, $args, &$parser ) {
if( $args['title'] == '' )
$title = $parser->mTitle;
else
$title = Title::newFromText( $args['title'] );
$link=$title->getFullURL( 'action=raw' );
if( $args['name'] != '' )
$link.='&name='.urlencode( $args['name'] );
if( $args['anchor'] != '' )
$link.='&anchor='.urlencode( $args['anchor'] );
if( $args['tag'] != '' )
$link.='&tag='.urlencode( $args['tag'] );
return $parser->recursiveTagParse( "[$link $input]" );
}

</source>

===Hook to intercept the raw output===
This part of the code doesn't look that nice because we've to parse the raw wiki page ourselves to retrieve the code sections we want.

First we define a helper function that we will use to report error messages. This is simply done by replacing the content of the downloaded file with the error message and when necessary a copy of the raw text relevant to the error.
 '''TODO''': Cancel the file download header and return a proper error page
{{#fileanchor: RawFile.php}}
<source lang=php>
function fnRawFile_Strip_Error($msg,$out,&$text) {
$text=$msg;
if($out != '')
$text.="\nCandidate match: $out";
return true;
}

</source>

Next let's see if <code>?action=raw</code> was used in the context of this extension: in that case we receive the filename as GET parameter, otherwise we simply return from our extension with return value=true which means we authorize the raw display (originally the hook was created to add an authentication point)
{{#fileanchor: RawFile.php}}
<source lang=php>
function fnRawFile_Strip(&$rawPage, &$text) {
$filename=$_GET['name'];
$anchor=$_GET['anchor'];
// for backward compatibility, accept also URLs with parameter 'file'
if( $anchor=='' )
$anchor=$_GET['file'];
$tag=$_GET['tag'];
// Either anchor or name must be specified
if( $filename=='' )
$filename=$anchor;
if ( $filename=='' )
return true;
</source>
By default the downloadable file will still be handled by the ob_gzhandler session made by Mediawiki. To avoid output buffering and gzipping, one can uncomment the following line:
{{#fileanchor: RawFile.php}}
<source lang=php>
// Uncomment the following line to avoid output buffering and gzipping:
// wfResetOutputBuffers();
</source>
Raw action already set the headers with some client cache pragmas and is supposed to be displayed in the browser but in our case we want to make this "page" a downloadable file so we overwrite the headers which were defined and we add a few more, to ensure there is no caching on the client (it's very hard for the client to force a refresh on a file download, contrary to a web page) and to provide the adequate filename.
{{#fileanchor: RawFile.php}}
<source lang=php>
header("Content-disposition: attachment;filename={$filename}");
header("Content-type: application/octet-stream");
header("Content-Transfer-Encoding: binary");
header("Expires: 0");
header("Pragma: no-cache");
header("Cache-Control: no-store");
</source>
Then we'll strip the output, first we've to locate the anchors but there are anchors that could be protected in literal blocks like <code>nowiki</code>.
 So we'll mask the literal blocks before searching for the anchors (we mask with the same string length because we'll retrieve an offset that we will use on the initial string and offsets must match). This is done with the scary regex below:
* we use <code>!</code> instead of <code>/</code> as pattern indicator so that the pattern string is self-matching. This is necessary since we will apply the extension on this page as well.
* we use option <code>s</code> (multiline) and <code>e</code> (evaluate replace expression)
* Evaluated expression replaces all characters in the matched string with X's. However if there are single quote (<code>'</code>) in the matched string, they will be escaped with <code>\</code>. So we need to search for <code>\'|.</code>. The many back-slashes is because the expression is evaluated several times.
'''TODO''': should we care also of source, js, css, pre,... blocks?
{{#fileanchor: RawFile.php}}
<source lang=php>
$maskedtext=preg_replace_callback('!<nowiki>.*?</nowiki>!s',
function($m) { return ereg_replace(".","X",$m[0]); },
$text);
</source>
Now we can search for the anchors:
* If an anchor name is specified, we looked for '''all''' magic words <code><nowiki>{{#fileanchor:...}}</nowiki></code> or blocks with attribute <code>class="[someclass ]anchorname"</code>
* Otherwise we look for the first magic word <code><nowiki>{{#file:...}}</nowiki></code> with specified file name,
* And finally for the first <code><file></code> tag with the specified file name (no multiple blocks support)
And we free the memory used for the masked version
{{#fileanchor: RawFile.php}}
<source lang=php>
if (($anchor!='') && preg_match_all('/({{#fileanchor: *'.$anchor.' *}})|(<[^>]+ class *= *"([^"]*\w)?'.$anchor.'(\w[^"]*)?"[^>]*>)/i', $maskedtext, $matches, PREG_OFFSET_CAPTURE))
$offsets=$matches[0];
else if (preg_match_all('/{{#file: *'.$anchor.' *}}/i', $maskedtext, $matches, PREG_OFFSET_CAPTURE))
$offsets=array($matches[0][0]);
else if (preg_match_all('/<file( [^>]*)? name *= *"'.$filename.'"[^>]*>/i', $maskedtext, $matches, PREG_OFFSET_CAPTURE))
$offsets=array($matches[0][0]);
else {
// We didn't find our anchor
return fnRawFile_Strip_Error("ERROR - RawFile: anchor not found (anchor=$anchor, name=$filename, tag=$tag)","",$text);
}
unset($maskedtext);
</source>
$text is both input & output so we copy it and start with an empty output.
{{#fileanchor: RawFile.php}}
<source lang=php>
$textorig=$text;
$text='';
</source>
For each anchor found we've to isolate the content of the next block.
{{#fileanchor: RawFile.php}}
<source lang=php>
foreach ($offsets as $offset) {
</source>
We start from the position of the current anchor. If the tag name of the block attached to the anchor is not specified, we look for the first block that follows the anchor, excluding <code> </code> and <code><file></code> block. The search can be easily done with a regular expression, using the ''lookahead negative assertion'' <code>(?!br\b|file\b)</code> to exclude the tags to ignore. Note that we need to ignore the anchor-link block <code><file></code> since the anchor starts right before that tag, and so the regular expression would match the anchor-link block it that tag is not specifically excluded.
{{#fileanchor: RawFile.php}}
<source lang=php>
$out = substr($textorig, $offset[1]);
// If no tag specified, we take the first one
if ($tag == '')
{
// With a regex assertion, we can easily ignore 'br' and 'file' tags
if (!preg_match('/<((?!br\b|file\b)\w+\b)/', $out, $matches))
return fnRawFile_Strip_Error ("ERROR - RawFile: Can't find opening tag after anchor '$offset[0]' (anchor=$anchor, name=$filename, tag=$tag)",$out,$text);
$tag=$matches[1];
}
</source>
Now, we know the tag name of the block to download, either because it was already specified as a GET attribute in the URL, or because we've found it in the search above. Again, using a regular expression, we look for the first block matching the specified tag name that follows the current anchor, and extract the content of the blocks. Note the use of the regex option <code>/.../s</code> to tell the regex engine that the matched text can span on multiple lines (with that option, <code>.</code> does match any character or a newline character). Also, we skip the first carriage return after the opening tag, if any (with <code>\n?</code>).
{{#fileanchor: RawFile.php}}
<source lang=php>
// Find the first tag matching $tag, and return enclosed text
if (!preg_match('/<'.$tag.'( [^>]*)?>\n?(.*?)<\/'.$tag.'>/s', $out, $matches))
return fnRawFile_Strip_Error ("ERROR - RawFile: no closing '$tag' found after anchor '$offset[0]' (anchor=$anchor, name=$filename, tag=$tag)",$out,$text);
$text .= $matches[2];
}
</source>
No need to deal with a Content-Length header because Mediawiki will do it for us, moreover more properly than we could if the output is sent gzipped, which is the default.
 So that's it, $text contains our file!
{{#fileanchor: RawFile.php}}
<source lang=php>
return true;
}

</source>

===Credits===
There is an official way to register the extension in a Mediawiki installation, so that it will be visible on the [[Special:Version]] page.
 Let's say the extension is in the category of parser hooks even if there is also a hook on Raw action.
{{#fileanchor: RawFile.php}}
<source lang=php>
$wgExtensionCredits['parserhook'][] = array('name' => 'RawFile',
'version' => '0.5',
'author' => 'Philippe Teuwen, Michael Peeters',
'url' => 'http://www.mediawiki.org/wiki/Extension:RawFile',
// 'url' => 'http://wiki.yobi.be/wiki/Mediawiki_RawFile',
'description' => 'Downloads a RAW copy of <nowiki><tag>data</tag></nowiki> in a file '.
'Useful e.g. to download a script or a patch '.
'It also allows what is called [http://en.wikipedia.org/wiki/Literate_programming Literate Programming]');
}

?>
</source>
And finally registration of the extension at the Mediawiki website according to the [http://www.mediawiki.org/wiki/Manual:Extensions Extensions Manual].

So this extension has now [http://www.mediawiki.org/wiki/Extension:RawFile its own page on the official Mediawiki site].

==Installation==
Download [{{#filelink: RawFile.php}} RawFile.php] and save it under the MediaWiki directory as extensions/RawFile/RawFile.php

Add at the end of LocalSettings.php:
<source lang=php>
require_once("$IP/extensions/RawFile/RawFile.php");
</source>
==Status==
If you use the extension properly the code is fully functional but it's rather raw on error handling.
==ChangeLog==
'''0.5'''
* Fix since PHP 5.5.0: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead. Thanks to Stephen Kent for reporting.
* '''WARNING''' you should upgrade ASAP as the previous versions are vulnerable to remote PHP code injection!!! See [[Talk:Mediawiki_RawFile]] for more info.
'''0.4.1'''
* Fix octet-stream MIME type bug which was affecting Epiphany & Opera 11. Thanks to Jani Uusitalo for reporting & [[User:Sicvolo|Sicvolo]] for finding the solution
'''0.4'''
* '''Anchors''' can be specified using html '''class''' attribute
* New syntax for '''Links''' and '''Anchor-links''':
:<code><nowiki><file [name="..."] [anchor="..."] [tag="..."] [title="..."] >Link text</file></nowiki></code>
* Support multiple files on the same page with same name (differentiated by their anchor name) or even common blocks in multiple files.
* Can specify the tag name of the block to download (to skip some irrelevant blocks when using an '''anchor-link''').
* Ignore '''<code> </code>''' tag.
* Some error reporting.
'''0.3'''
* Added optional parameter to <code>#fileLink</code> to indicate that the file is on another local wiki page
'''0.2'''
* Fix problem with Content-Length mismatch when transport is gzipped (default for Mediawiki if client supports it)
'''0.1'''
* Initial version

==Known bugs==
==Questions and feedback==
If you've any trouble, questions or suggestions, you can [[User:PhilippeTeuwen|contact me]].
==Known sites using the extension==
* [http://tech.ivkin.net/wiki/Main_Page Tech Knowledge Base Wiki] by [[User:Sicvolo|Sicvolo]]
* [http://far.no/fram/index.php?title=Fram Far.no/Fram]
* Well, this site of course! where I update the actual file on the server with a simple <code>wget -N --content-disposition "http://wiki.yobi.be/index.php?title=Mediawiki_RawFile&action=raw&file=RawFile.php"</code>

Talk:Mediawiki RawFile

2014-02-13T15:03:34Z

Fuujuhi:

Testing PHP code injection on Rawfile < 0.5 :
 Adding this code to a wikipage:
<pre>
<nowiki><nowiki>{${phpinfo()}}</nowiki></nowiki>
[{{#file: myscript.sh}} as myscript.sh]
<source lang=bash>
#!/bin/bash

echo 'Hello world!'
exit 0
</source>
</pre>
and trying to download myscript.sh will execute phpinfo() on the server.

Mediawiki RawFile

2009-11-30T21:34:35Z

Fuujuhi: /* Syntax */

==Very short introduction==
Just have a look to the 2 [[Mediawiki_RawFile#Short_example|examples]] to see how to use the extension
 and to the [[Mediawiki_RawFile#Installation|Installation]] section to see how to install the extension in your [[MediaWiki]] server
==Introduction==
Originally the idea was to be able to download directly a portion of code as a file.
 I've numerous code examples in my wiki and I wanted an easy way to download them, easier than a copy/paste!
 But from there it was rather easy to get something very close to [http://en.wikipedia.org/wiki/Literate_programming literate programming] just by allowing multiple blocks referring to the same file, which will be concatenated together at download time.

* It must work with pre, nowiki, js, css, code, source, so let's make it general: take the tag that comes after the parser function we'll create and select data up to the closing tag.
* There are two distinct functionalities provided by the extension:
** the parser that will convert a magic word into a link to the download URL
** an extended ?action=raw that will strip the raw output to keep the desired code

==Syntax==
The extension introduces 3 elements:
;Anchor
: Used to flag that the next code block in the wiki text belongs to a specific file. The code block can be any wiki block (such as '''<code><pre></code>''', '''<code><code></code>''', '''<code><tt></code>''', '''<code><source></code>'''...). '''<code> </code>''' tags are ignored. Note that anchors are invisible in the wiki display.
;Link
: They are transformed by the extension into links that allows for downloading all blocks attached to a given anchor name.
;Anchor-link
: A shortcut notation mixing both an anchor and download link, handy for regular use, when a single code block is used and when the download link can be at the same position as the anchor.

The syntax is as follows. The syntax using tag <code><file></code> and tag attribute <code>class</code> is new since v0.4. Note that elements of both syntaxes can be mixed in a same page.
{| border="2" cellspacing="4" cellpadding="3" style="margin: 1em 1em 1em 0; background: #f9f9f9; border: 1px #aaaaaa solid; border-collapse: collapse; empty-cells:show;"
!width="80em" style="background: #8da7d6;"|Element!!style="background: #8da7d6;"|Syntax and description
|-
|'''Anchor'''
|<pre><nowiki>
{{#fileAnchor: anchorname}}
<pre class='anchorname'>...</pre>
<code class="anchorname">...</code>
<code class="cssclass anchorname">...</code>
...
</nowiki></pre>
Indicates that the next wiki block is attached to an anchor ''anchorname''. The content of that block will be downloaded (possibly appended with other blocks if there are several blocks attached to the same ''anchorname'') when a file link is clicked on. 
'''(since v0.4)''' To attach an anchor ''anchorname'' to a wiki block, simply add an attribute <code>class="anchorname"</code> to it. The extension supports multi-class specification, meaning that a same block can be associated to different files, and that the <code>class</code> attribute can still be used to specify custom CSS properties as in standard wiki text.
; ''anchorname''
; class="''anchorname''"
: The name of the anchor to which the wiki block is attached
|-
|'''Link'''
|<pre><nowiki>
[{{#fileLink: anchorname}} link text]
[{{#fileLink: anchorname|pagetitle}} link text]
<file anchor="anchorname" [name="filename"] [title="pagetitle"]>link text</file>
</nowiki></pre>
Creates a link to download all blocks that are attached to an anchor ''anchorname''.
;''anchorname''
;anchor="''anchorname''"
: The name of the anchor to look for. All blocks attached to an anchor ''anchorname'' will be downloaded.
;name="''filename''"
:''Optional'' - Specifies the name of the file to download. If absent, ''anchorname'' is then used as the name of the downloaded file.
; ''pagetitle''
;title="''pagetitle''"
: ''Optional'' - Indicates that the blocks to download are on the wiki page titled ''pagetitle''. If absent, blocks are looked for on the current page.
; ''link text''
: The text of the link to display.
|-
|'''Anchor-link'''
|<pre><nowiki>
[{{#file: filename}} link text]
<file name="filename" [tag="''tagname''"]>link text</file>
</nowiki></pre>
Creates a link to download the next wiki block as a file named ''filename''. 
'''(since v0.4)''' The attribute <code>tag</code> can be used to specify the ''tagname'' of the block to download. 
; ''filename''
; name="''filename''"
: The name of the file to download.
;tag="''tagname''"
:''Optional'' - When set, the extension only looks for blocks whose name matches the given ''tagname''. This attribute is particularly useful when there are some irrelevant blocks between the '''anchor-link''' and the block you want to download. If absent, the first encountered block following the anchor is downloaded.
; ''link text''
: The text of the link to display.
|}

===Short example===
The extension works with any block such as pre, nowiki, js, css, code, source,...
 This example is using the syntax highlighting <nowiki><source></nowiki> tag provided by [http://www.mediawiki.org/wiki/Extension:SyntaxHighlight_GeSHi SyntaxHighlight extension] (using [http://qbnz.com/highlighter/ GeSHi Highlighter])
 If you didn't install that extension on your MediaWiki, you can try the example by using <nowiki><pre></nowiki> instead of <nowiki><source></nowiki>.

<pre>Let's save the following code [{{#file: myscript.sh}} as myscript.sh]
<source lang=bash>
#!/bin/bash

echo 'Hello world!'
exit 0
</source>
</pre>
will give:
----
Let's save the following code [{{#file: myscript.sh}} as myscript.sh]
<source lang=bash>
#!/bin/bash

echo 'Hello world!'
exit 0
</source>

===Complete example===
And a full example with anchors & link:

<pre>
Let's start with the Bash usual header:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
#!/bin/bash
</source>
Then we'll display a welcome message:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
echo 'Welcome on earth!'
</source>
And we finally exit cleanly:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
exit 0
</source>
[{{#filelink: myotherscript.sh}} myotherscript.sh is now available for download below the code]
</pre>
will give:
----
Let's start with the Bash usual header:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
#!/bin/bash
</source>
Then we'll display a welcome message:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
echo 'Welcome on earth!'
</source>
And we finally exit cleanly:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
exit 0
</source>
[{{#filelink: myotherscript.sh}} myotherscript.sh is now available for download below the code]

==The code (the ultimate example)==
Which you can of course download just by following [{{#filelink: RawFile.php}} this link :-)]

So let's explain a bit the code in a Literate Programming way...
===Hooks===
First some hooks for our functions...

We will create:
* a [http://www.mediawiki.org/wiki/Manual:Parser_functions Parser Function] (see also [http://meta.wikimedia.org/wiki/Help:Parser_function here]), with help of
** [http://www.mediawiki.org/wiki/Manual:%24wgExtensionFunctions $wgExtensionFunctions] or [http://www.mediawiki.org/wiki/Manual:Hooks/ParserFirstCallInit ParserFirstCallInit global hook] to define the setup function
** [http://www.mediawiki.org/wiki/Manual:Magic_words Magic Words]
** [http://www.mediawiki.org/wiki/Manual:Tag_extensions Tag extensions]
** [http://www.mediawiki.org/wiki/Manual:Hooks/LanguageGetMagic LanguageGetMagic] hook to initialize the magic words
* a [http://www.mediawiki.org/wiki/Manual:Hooks/RawPageViewBeforeOutput RawPageViewBeforeOutput] hook to intercept the raw output

{{#fileanchor: RawFile.php}}
<source lang=php>
<?php

if (defined('MEDIAWIKI')) {

//Avoid unstubbing $wgParser on setHook() too early on modern (1.12+) MW versions, as per r35980
if ( defined( 'MW_SUPPORTS_PARSERFIRSTCALLINIT' ) ) {
$wgHooks['ParserFirstCallInit'][] = 'efRawFile_Setup';
} else { // Otherwise do things the old fashioned way
$wgExtensionFunctions[] = 'efRawFile_Setup';
}
$wgHooks['LanguageGetMagic'][] = 'efRawFile_Magic';
$wgHooks['RawPageViewBeforeOutput'][] = 'fnRawFile_Strip';

</source>

===Setup function===
For the wiki parsing to create download links, the parser functions '''file''' and '''fileLink''' are equally treated, while '''fileAnchor''' will be simply left out. We also create a new tag '''file''' as explained [http://www.mediawiki.org/wiki/Manual:Tag_extensions here].
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Setup() {
global $wgParser;
$wgParser->setFunctionHook( 'file', 'efRawFile_Render' );
$wgParser->setFunctionHook( 'filelink', 'efRawFile_Render' );
$wgParser->setFunctionHook( 'fileanchor', 'efRawFile_Empty' );
$wgParser->setHook( 'file', 'efRawFile_FileTagRender' );
return true;
}

</source>

===Hook to initialize the magic words===
We add the magic words here: the first array element indicates if it is case sensitive, in this case it is not case sensitive. We could add extra elements to create synonyms for our parser function.
 Unless we return true, other parser functions extensions will not get loaded.
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Magic( &$magicWords, $langCode ) {
$magicWords['file'] = array( 0, 'file' );
$magicWords['filelink'] = array( 0, 'filelink' );
$magicWords['fileanchor'] = array( 0, 'fileanchor' );
return true;
}

</source>

===Parser functions of the magic words===
The transformation rule to replace link shortcuts to actual links for download, handling an optional local wiki page title if present.
 The input parameters are wikitext with templates expanded, the output should be wikitext too
 '''TODO''': what error to send out if there is no filename given?
 '''EDIT''': It seems that [http://svn.wikimedia.org/viewvc/mediawiki?view=rev&revision=27667 commit 27667] (1.11 -> 1.12) changed the default parser, which breaks the recursive parsing. Thanks to Tim Starling for helping me to get around the problem!
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Render( &$parser, $filename = '', $titleText = '') {
if( $titleText == '' )
$title = $parser->mTitle;
else
$title = Title::newFromText( $titleText );
return $title->getFullURL( 'action=raw&anchor='.urlencode( $filename ) );
}

</source>
And the other one, just removing the anchors from the rendered wiki page.
 Curiously enough if the function doesn't exist at all the effect is exactly the same, MW doesn't throw any error.
 But let's keep things clean...
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Empty( &$parser, $filename = '') {
return '';
}

</source>

===Parser functions of the new tag <tt><file></tt>===
The transformation rule to replace <code><file></code> tag to actual links for download. The same parser function is used for both '''anchors''' and '''anchor-links'''. Since the link text may contain wiki text, we generate the link as wiki text that we ask the parser to parse again.
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_FileTagRender( $input, $args, &$parser ) {
if( $args['title'] == '' )
$title = $parser->mTitle;
else
$title = Title::newFromText( $args['title'] );
$link=$title->getFullURL( 'action=raw' );
if( $args['name'] != '' )
$link.='&name='.urlencode( $args['name'] );
if( $args['anchor'] != '' )
$link.='&anchor='.urlencode( $args['anchor'] );
if( $args['tag'] != '' )
$link.='&tag='.urlencode( $args['tag'] );
return $parser->recursiveTagParse( "[$link $input]" );
}

</source>

===Hook to intercept the raw output===
This part of the code doesn't look that nice because we've to parse the raw wiki page ourselves to retrieve the code sections we want.

First we define a helper function that we will use to report error messages. This is simply done by replacing the content of the downloaded file with the error message and when necessary a copy of the raw text relevant to the error.
 '''TODO''': Cancel the file download header and return a proper error page
{{#fileanchor: RawFile.php}}
<source lang=php>
function fnRawFile_Strip_Error($msg,$out,&$text) {
$text=$msg;
if($out != '')
$text.="\nCandidate match: $out";
return true;
}

</source>

Next let's see if <code>?action=raw</code> was used in the context of this extension: in that case we receive the filename as GET parameter, otherwise we simply return from our extension with return value=true which means we authorize the raw display (originally the hook was created to add an authentication point)
{{#fileanchor: RawFile.php}}
<source lang=php>
function fnRawFile_Strip(&$rawPage, &$text) {
$filename=$_GET['name'];
$anchor=$_GET['anchor'];
$tag=$_GET['tag'];
// Either anchor or name must be specified
if( $filename=='' )
$filename=$anchor;
if ( $filename=='' )
return true;
</source>
By default the downloadable file will still be handled by the ob_gzhandler session made by Mediawiki. To avoid output buffering and gzipping, one can uncomment the following line:
{{#fileanchor: RawFile.php}}
<source lang=php>
// Uncomment the following line to avoid output buffering and gzipping:
// wfResetOutputBuffers();
</source>
Raw action already set the headers with some client cache pragmas and is supposed to be displayed in the browser but in our case we want to make this "page" a downloadable file so we overwrite the headers which were defined and we add a few more, to ensure there is no caching on the client (it's very hard for the client to force a refresh on a file download, contrary to a web page) and to provide the adequate filename.
{{#fileanchor: RawFile.php}}
<source lang=php>
header("Content-disposition: attachment;filename={$filename}");
header("Content-type: application/octetstream");
header("Content-Transfer-Encoding: binary");
header("Expires: 0");
header("Pragma: no-cache");
header("Cache-Control: no-store");
</source>
Then we'll strip the output, first we've to locate the anchors but there are anchors that could be protected in literal blocks like <code>nowiki</code>.
 So we'll mask the literal blocks before searching for the anchors (we mask with the same string length because we'll retrieve an offset that we will use on the initial string and offsets must match). This is done with the scary regex below:
* we use <code>!</code> instead of <code>/</code> as pattern indicator so that the pattern string is self-matching. This is necessary since we will apply the extension on this page as well.
* we use option <code>s</code> (multiline) and <code>e</code> (evaluate replace expression)
* Evaluated expression replaces all characters in the matched string with X's. However if there are single quote (<code>'</code>) in the matched string, they will be escaped with <code>\</code>. So we need to search for <code>\'|.</code>. The many back-slashes is because the expression is evaluated several times.
'''TODO''': should we care also of source, js, css, pre,... blocks?
{{#fileanchor: RawFile.php}}
<source lang=php>
$maskedtext=preg_replace('!<nowiki>.*?</nowiki>!se',
'preg_replace("/\\\\\\\\\\\'|./","X","$0")',
$text);
</source>
Now we can search for the anchors:
* If an anchor name is specified, we looked for '''all''' magic words <code><nowiki>{{#fileanchor:...}}</nowiki></code> or blocks with attribute <code>class="[someclass ]anchorname"</code>
* Otherwise we look for the first magic word <code><nowiki>{{#file:...}}</nowiki></code> with specified file name,
* And finally for the first <code><file></code> tag with the specified file name (no multiple blocks support)
And we free the memory used for the masked version
{{#fileanchor: RawFile.php}}
<source lang=php>
if (($anchor!='') && preg_match_all('/({{#fileanchor: *'.$anchor.' *}})|(<[^>]+ class *= *"([^"]*\w)?'.$anchor.'(\w[^"]*)?"[^>]*>)/i', $maskedtext, $matches, PREG_OFFSET_CAPTURE))
$offsets=$matches[0];
else if (preg_match_all('/{{#file: *'.$anchor.' *}}/i', $maskedtext, $matches, PREG_OFFSET_CAPTURE))
$offsets=array($matches[0][0]);
else if (preg_match_all('/<file( [^>]*)? name *= *"'.$filename.'"[^>]*>/i', $maskedtext, $matches, PREG_OFFSET_CAPTURE))
$offsets=array($matches[0][0]);
else {
// We didn't find our anchor
return fnRawFile_Strip_Error("ERROR - RawFile: anchor not found (anchor=$anchor, name=$filename, tag=$tag)","",$text);
}
unset($maskedtext);
</source>
$text is both input & output so we copy it and start with an empty output.
{{#fileanchor: RawFile.php}}
<source lang=php>
$textorig=$text;
$text='';
</source>
For each anchor found we've to isolate the content of the next block.
{{#fileanchor: RawFile.php}}
<source lang=php>
foreach ($offsets as $offset) {
</source>
We start from the position of the current anchor. If the tag name of the block attached to the anchor is not specified, we look for the first block that follows the anchor, excluding <code> </code> and <code><file></code> block. The search can be easily done with a regular expression, using the ''lookahead negative assertion'' <code>(?!br\b|file\b)</code> to exclude the tags to ignore. Note that we need to ignore the anchor-link block <code><file></code> since the anchor starts right before that tag, and so the regular expression would match the anchor-link block it that tag is not specifically excluded.
{{#fileanchor: RawFile.php}}
<source lang=php>
$out = substr($textorig, $offset[1]);
// If no tag specified, we take the first one
if ($tag == '')
{
// With a regex assertion, we can easily ignore 'br' and 'file' tags
if (!preg_match('/<((?!br\b|file\b)\w+\b)/', $out, $matches))
return fnRawFile_Strip_Error ("ERROR - RawFile: Can't find opening tag after anchor '$offset[0]' (anchor=$anchor, name=$filename, tag=$tag)",$out,$text);
$tag=$matches[1];
}
</source>
Now, we know the tag name of the block to download, either because it was already specified as a GET attribute in the URL, or because we've found it in the search above. Again, using a regular expression, we look for the first block matching the specified tag name that follows the current anchor, and extract the content of the blocks. Note the use of the regex option <code>/.../s</code> to tell the regex engine that the matched text can span on multiple lines (with that option, <code>.</code> does match any character or a newline character). Also, we skip the first carriage return after the opening tag, if any (with <code>\n?</code>).
{{#fileanchor: RawFile.php}}
<source lang=php>
// Find the first tag matching $tag, and return enclosed text
if (!preg_match('/<'.$tag.'( [^>]*)?>\n?(.*?)<\/'.$tag.'>/s', $out, $matches))
return fnRawFile_Strip_Error ("ERROR - RawFile: no closing '$tag' found after anchor '$offset[0]' (anchor=$anchor, name=$filename, tag=$tag)",$out,$text);
$text .= $matches[2];
}
</source>
No need to deal with a Content-Length header because Mediawiki will do it for us, moreover more properly than we could if the output is sent gzipped, which is the default.
 So that's it, $text contains our file!
{{#fileanchor: RawFile.php}}
<source lang=php>
return true;
}

</source>

===Credits===
There is an official way to register the extension in a Mediawiki installation, so that it will be visible on the [[Special:Version]] page.
 Let's say the extension is in the category of parser hooks even if there is also a hook on Raw action.
{{#fileanchor: RawFile.php}}
<source lang=php>
$wgExtensionCredits['parserhook'][] = array('name' => 'RawFile',
'version' => '0.4',
'author' => 'Philippe Teuwen, Michael Peeters',
'url' => 'http://www.mediawiki.org/wiki/Extension:RawFile',
// 'url' => 'http://wiki.yobi.be/wiki/Mediawiki_RawFile',
'description' => 'Downloads a RAW copy of <nowiki><tag>data</tag></nowiki> in a file '.
'Useful e.g. to download a script or a patch '.
'It also allows what is called [http://en.wikipedia.org/wiki/Literate_programming Literate Programming]');
}

?>
</source>
And finally registration of the extension at the Mediawiki website according to the [http://www.mediawiki.org/wiki/Manual:Extensions Extensions Manual].

So this extension has now [http://www.mediawiki.org/wiki/Extension:RawFile its own page on the official Mediawiki site].

==Installation==
Download [{{#filelink: RawFile.php}} RawFile.php] and save it under the MediaWiki directory as extensions/RawFile/RawFile.php

Add at the end of LocalSettings.php:
<source lang=php>
require_once("$IP/extensions/RawFile/RawFile.php");
</source>
==Status==
If you use the extension properly the code is fully functional but it's rather raw on error handling.
==ChangeLog==
'''0.4'''
* '''Anchors''' can be specified using html '''class''' attribute
* New syntax for '''Links''' and '''Anchor-links''':
:<code><nowiki><file [name="..."] [anchor="..."] [tag="..."] [title="..."] >Link text</file></nowiki></code>
* Support multiple files on the same page with same name.
* Can specify the tag name of the block to download (to skip some irrelevant blocks when using an '''anchor-link''').
* Ignore '''<code> </code>''' tag.
* Some error reporting.
'''0.3'''
* Added optional parameter to <code>#fileLink</code> to indicate that the file is on another local wiki page
'''0.2'''
* Fix problem with Content-Length mismatch when transport is gzipped (default for Mediawiki if client supports it)
'''0.1'''
* Initial version

==Known bugs==
Jani Uusitalo reported the following issue:
 For some reason, if you use Epiphany's 'Save as' instead of a direct left-click, the downloaded file is a single byte. In Firefox the links work just fine, so this is probably an Epiphany bug. Uncommenting the <code>// wfResetOutputBuffers();</code> line didn't help. If anyone knows a workaround to help Epiphany users without breaking the support of the other browsers, please speak up :-)

==Questions and feedback==
If you've any trouble, questions or suggestions, you can [[User:PhilippeTeuwen|contact me]].

Mediawiki RawFile

2009-11-28T03:09:25Z

Fuujuhi: Version 0.4 - new tag <file>, anchor using class attribute, error reporting, ignore tag...

==Very short introduction==
Just have a look to the 2 [[Mediawiki_RawFile#Short_example|examples]] to see how to use the extension
 and to the [[Mediawiki_RawFile#Installation|Installation]] section to see how to install the extension in your [[MediaWiki]] server
==Introduction==
Originally the idea was to be able to download directly a portion of code as a file.
 I've numerous code examples in my wiki and I wanted an easy way to download them, easier than a copy/paste!
 But from there it was rather easy to get something very close to [http://en.wikipedia.org/wiki/Literate_programming literate programming] just by allowing multiple blocks referring to the same file, which will be concatenated together at download time.

* It must work with pre, nowiki, js, css, code, source, so let's make it general: take the tag that comes after the parser function we'll create and select data up to the closing tag.
* There are two distinct functionalities provided by the extension:
** the parser that will convert a magic word into a link to the download URL
** an extended ?action=raw that will strip the raw output to keep the desired code

==Syntax==
The extension introduces 3 elements:
;Anchor
: Used to flag that the next code block in the wiki text belongs to a specific file. The code block can be any wiki block (such as '''<code><pre></code>''', '''<code><code></code>''', '''<code><tt></code>''', '''<code><source></code>'''...). '''<code> </code>''' tags are ignored. Note that anchors are invisible in the wiki display.
;Link
: They are transformed by the extension into links that allows for downloading all blocks attached to a given anchor name.
;Anchor-link
: A shortcut notation mixing both an anchor and download link, handy for regular use, when a single code block is used and when the download link can be at the same position as the anchor.

The syntax is as follows. The syntax using tag <code><file></code> and tag attribute <code>class</code> is new since v0.4. Note that elements of both syntaxes can be mixed in a same page.
{| border="2" cellspacing="4" cellpadding="3" style="margin: 1em 1em 1em 0; background: #f9f9f9; border: 1px #aaaaaa solid; border-collapse: collapse; empty-cells:show;"
!width="80em" style="background: #8da7d6;"|Element!!style="background: #8da7d6;"|Syntax and description
|-
|'''Anchor'''
|<pre><nowiki>
{{#fileAnchor: anchorname}}
<pre class='anchorname'>...</pre>
<code class="anchorname">...</code>
<code class="cssclass anchorname">...</code>
...
</nowiki></pre>
Indicates that the next wiki block is attached to an anchor ''anchorname''. The content of that block will be downloaded (possibly appended with other blocks if there are several blocks attached to the same ''anchorname'') when a file link is clicked on. 
'''(since v0.4)''' To attach an anchor ''anchorname'' to a wiki block, simply add an attribute <code>class="anchorname"</code> to it. The extension supports multi-class specification, meaning that a same block can be associated to different files, and that the <code>class</code> attribute can still be used to specify custom CSS properties as in standard wiki text.
; ''anchorname''
; class="''anchorname''"
: The name of the anchor to which the wiki block is attached
|-
|'''Link'''
|<pre><nowiki>
[{{#fileLink: anchorname}} link text]
[{{#fileLink: anchorname|pagetitle}} link text]
<file anchor="anchorname" [name="filename"] [title="pagetitle"]>link text</file>
</nowiki></pre>
Creates a link to download all blocks that are attached to an anchor ''anchorname''.
;''anchorname''
;anchor="''anchorname''"
: The name of the anchor to look for. All blocks attached to an anchor ''anchorname'' will be downloaded.
;name="''filename''"
:''Optional'' - Specifies the name of the file to download. If absent, ''anchorname'' is then used as the name of the downloaded file.
; ''pagetitle''
;tag="''pagetitle''"
: ''Optional'' - Indicates that the blocks to download are on the wiki page titled ''pagetitle''. If absent, blocks are looked for on the current page.
; ''link text''
: The text of the link to display.
|-
|'''Anchor-link'''
|<pre><nowiki>
[{{#file: filename}} link text]
<file name="filename" [tag="''tagname''"]>link text</file>
</nowiki></pre>
Creates a link to download the next wiki block as a file named ''filename''. 
'''(since v0.4)''' The attribute <code>tag</code> can be used to specify the ''tagname'' of the block to download. 
; ''filename''
; name="''filename''"
: The name of the file to download.
;tag="''tagname''"
:''Optional'' - When set, the extension only looks for blocks whose name matches the given ''tagname''. This attribute is particularly useful when there are some irrelevant blocks between the '''anchor-link''' and the block you want to download. If absent, the first encountered block following the anchor is downloaded.
; ''link text''
: The text of the link to display.
|}

===Short example===
The extension works with any block such as pre, nowiki, js, css, code, source,...
 This example is using the syntax highlighting <nowiki><source></nowiki> tag provided by [http://www.mediawiki.org/wiki/Extension:SyntaxHighlight_GeSHi SyntaxHighlight extension] (using [http://qbnz.com/highlighter/ GeSHi Highlighter])
 If you didn't install that extension on your MediaWiki, you can try the example by using <nowiki><pre></nowiki> instead of <nowiki><source></nowiki>.

<pre>Let's save the following code [{{#file: myscript.sh}} as myscript.sh]
<source lang=bash>
#!/bin/bash

echo 'Hello world!'
exit 0
</source>
</pre>
will give:
----
Let's save the following code [{{#file: myscript.sh}} as myscript.sh]
<source lang=bash>
#!/bin/bash

echo 'Hello world!'
exit 0
</source>

===Complete example===
And a full example with anchors & link:

<pre>
Let's start with the Bash usual header:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
#!/bin/bash
</source>
Then we'll display a welcome message:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
echo 'Welcome on earth!'
</source>
And we finally exit cleanly:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
exit 0
</source>
[{{#filelink: myotherscript.sh}} myotherscript.sh is now available for download below the code]
</pre>
will give:
----
Let's start with the Bash usual header:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
#!/bin/bash
</source>
Then we'll display a welcome message:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
echo 'Welcome on earth!'
</source>
And we finally exit cleanly:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
exit 0
</source>
[{{#filelink: myotherscript.sh}} myotherscript.sh is now available for download below the code]

==The code (the ultimate example)==
Which you can of course download just by following [{{#filelink: RawFile.php}} this link :-)]

So let's explain a bit the code in a Literate Programming way...
===Hooks===
First some hooks for our functions...

We will create:
* a [http://www.mediawiki.org/wiki/Manual:Parser_functions Parser Function] (see also [http://meta.wikimedia.org/wiki/Help:Parser_function here]), with help of
** [http://www.mediawiki.org/wiki/Manual:%24wgExtensionFunctions $wgExtensionFunctions] or [http://www.mediawiki.org/wiki/Manual:Hooks/ParserFirstCallInit ParserFirstCallInit global hook] to define the setup function
** [http://www.mediawiki.org/wiki/Manual:Magic_words Magic Words]
** [http://www.mediawiki.org/wiki/Manual:Tag_extensions Tag extensions]
** [http://www.mediawiki.org/wiki/Manual:Hooks/LanguageGetMagic LanguageGetMagic] hook to initialize the magic words
* a [http://www.mediawiki.org/wiki/Manual:Hooks/RawPageViewBeforeOutput RawPageViewBeforeOutput] hook to intercept the raw output

{{#fileanchor: RawFile.php}}
<source lang=php>
<?php

if (defined('MEDIAWIKI')) {

//Avoid unstubbing $wgParser on setHook() too early on modern (1.12+) MW versions, as per r35980
if ( defined( 'MW_SUPPORTS_PARSERFIRSTCALLINIT' ) ) {
$wgHooks['ParserFirstCallInit'][] = 'efRawFile_Setup';
} else { // Otherwise do things the old fashioned way
$wgExtensionFunctions[] = 'efRawFile_Setup';
}
$wgHooks['LanguageGetMagic'][] = 'efRawFile_Magic';
$wgHooks['RawPageViewBeforeOutput'][] = 'fnRawFile_Strip';

</source>

===Setup function===
For the wiki parsing to create download links, the parser functions '''file''' and '''fileLink''' are equally treated, while '''fileAnchor''' will be simply left out. We also create a new tag '''file''' as explained [http://www.mediawiki.org/wiki/Manual:Tag_extensions here].
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Setup() {
global $wgParser;
$wgParser->setFunctionHook( 'file', 'efRawFile_Render' );
$wgParser->setFunctionHook( 'filelink', 'efRawFile_Render' );
$wgParser->setFunctionHook( 'fileanchor', 'efRawFile_Empty' );
$wgParser->setHook( 'file', 'efRawFile_FileTagRender' );
return true;
}

</source>

===Hook to initialize the magic words===
We add the magic words here: the first array element indicates if it is case sensitive, in this case it is not case sensitive. We could add extra elements to create synonyms for our parser function.
 Unless we return true, other parser functions extensions will not get loaded.
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Magic( &$magicWords, $langCode ) {
$magicWords['file'] = array( 0, 'file' );
$magicWords['filelink'] = array( 0, 'filelink' );
$magicWords['fileanchor'] = array( 0, 'fileanchor' );
return true;
}

</source>

===Parser functions of the magic words===
The transformation rule to replace link shortcuts to actual links for download, handling an optional local wiki page title if present.
 The input parameters are wikitext with templates expanded, the output should be wikitext too
 '''TODO''': what error to send out if there is no filename given?
 '''EDIT''': It seems that [http://svn.wikimedia.org/viewvc/mediawiki?view=rev&revision=27667 commit 27667] (1.11 -> 1.12) changed the default parser, which breaks the recursive parsing. Thanks to Tim Starling for helping me to get around the problem!
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Render( &$parser, $filename = '', $titleText = '') {
if( $titleText == '' )
$title = $parser->mTitle;
else
$title = Title::newFromText( $titleText );
return $title->getFullURL( 'action=raw&anchor='.urlencode( $filename ) );
}

</source>
And the other one, just removing the anchors from the rendered wiki page.
 Curiously enough if the function doesn't exist at all the effect is exactly the same, MW doesn't throw any error.
 But let's keep things clean...
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Empty( &$parser, $filename = '') {
return '';
}

</source>

===Parser functions of the new tag <tt><file></tt>===
The transformation rule to replace <code><file></code> tag to actual links for download. The same parser function is used for both '''anchors''' and '''anchor-links'''. Since the link text may contain wiki text, we generate the link as wiki text that we ask the parser to parse again.
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_FileTagRender( $input, $args, &$parser ) {
if( $args['title'] == '' )
$title = $parser->mTitle;
else
$title = Title::newFromText( $args['title'] );
$link=$title->getFullURL( 'action=raw' );
if( $args['name'] != '' )
$link.='&name='.urlencode( $args['name'] );
if( $args['anchor'] != '' )
$link.='&anchor='.urlencode( $args['anchor'] );
if( $args['tag'] != '' )
$link.='&tag='.urlencode( $args['tag'] );
return $parser->recursiveTagParse( "[$link $input]" );
}

</source>

===Hook to intercept the raw output===
This part of the code doesn't look that nice because we've to parse the raw wiki page ourselves to retrieve the code sections we want.

First we define a helper function that we will use to report error messages. This is simply done by replacing the content of the downloaded file with the error message and when necessary a copy of the raw text relevant to the error.
 '''TODO''': Cancel the file download header and return a proper error page
{{#fileanchor: RawFile.php}}
<source lang=php>
function fnRawFile_Strip_Error($msg,$out,&$text) {
$text=$msg;
if($out != '')
$text.="\nCandidate match: $out";
return true;
}

</source>

Next let's see if <code>?action=raw</code> was used in the context of this extension: in that case we receive the filename as GET parameter, otherwise we simply return from our extension with return value=true which means we authorize the raw display (originally the hook was created to add an authentication point)
{{#fileanchor: RawFile.php}}
<source lang=php>
function fnRawFile_Strip(&$rawPage, &$text) {
$filename=$_GET['name'];
$anchor=$_GET['anchor'];
$tag=$_GET['tag'];
// Either anchor or name must be specified
if( $filename=='' )
$filename=$anchor;
if ( $filename=='' )
return true;
</source>
By default the downloadable file will still be handled by the ob_gzhandler session made by Mediawiki. To avoid output buffering and gzipping, one can uncomment the following line:
{{#fileanchor: RawFile.php}}
<source lang=php>
// Uncomment the following line to avoid output buffering and gzipping:
// wfResetOutputBuffers();
</source>
Raw action already set the headers with some client cache pragmas and is supposed to be displayed in the browser but in our case we want to make this "page" a downloadable file so we overwrite the headers which were defined and we add a few more, to ensure there is no caching on the client (it's very hard for the client to force a refresh on a file download, contrary to a web page) and to provide the adequate filename.
{{#fileanchor: RawFile.php}}
<source lang=php>
header("Content-disposition: attachment;filename={$filename}");
header("Content-type: application/octetstream");
header("Content-Transfer-Encoding: binary");
header("Expires: 0");
header("Pragma: no-cache");
header("Cache-Control: no-store");
</source>
Then we'll strip the output, first we've to locate the anchors but there are anchors that could be protected in literal blocks like <code>nowiki</code>.
 So we'll mask the literal blocks before searching for the anchors (we mask with the same string length because we'll retrieve an offset that we will use on the initial string and offsets must match). This is done with the scary regex below:
* we use <code>!</code> instead of <code>/</code> as pattern indicator so that the pattern string is self-matching. This is necessary since we will apply the extension on this page as well.
* we use option <code>s</code> (multiline) and <code>e</code> (evaluate replace expression)
* Evaluated expression replaces all characters in the matched string with X's. However if there are single quote (<code>'</code>) in the matched string, they will be escaped with <code>\</code>. So we need to search for <code>\'|.</code>. The many back-slashes is because the expression is evaluated several times.
'''TODO''': should we care also of source, js, css, pre,... blocks?
{{#fileanchor: RawFile.php}}
<source lang=php>
$maskedtext=preg_replace('!<nowiki>.*?</nowiki>!se',
'preg_replace("/\\\\\\\\\\\'|./","X","$0")',
$text);
</source>
Now we can search for the anchors:
* If an anchor name is specified, we looked for '''all''' magic words <code><nowiki>{{#fileanchor:...}}</nowiki></code> or blocks with attribute <code>class="[someclass ]anchorname"</code>
* Otherwise we look for the first magic word <code><nowiki>{{#file:...}}</nowiki></code> with specified file name,
* And finally for the first <code><file></code> tag with the specified file name (no multiple blocks support)
And we free the memory used for the masked version
{{#fileanchor: RawFile.php}}
<source lang=php>
if (($anchor!='') && preg_match_all('/({{#fileanchor: *'.$anchor.' *}})|(<[^>]+ class *= *"([^"]*\w)?'.$anchor.'(\w[^"]*)?"[^>]*>)/i', $maskedtext, $matches, PREG_OFFSET_CAPTURE))
$offsets=$matches[0];
else if (preg_match_all('/{{#file: *'.$anchor.' *}}/i', $maskedtext, $matches, PREG_OFFSET_CAPTURE))
$offsets=array($matches[0][0]);
else if (preg_match_all('/<file( [^>]*)? name *= *"'.$filename.'"[^>]*>/i', $maskedtext, $matches, PREG_OFFSET_CAPTURE))
$offsets=array($matches[0][0]);
else {
// We didn't find our anchor
return fnRawFile_Strip_Error("ERROR - RawFile: anchor not found (anchor=$anchor, name=$filename, tag=$tag)","",$text);
}
unset($maskedtext);
</source>
$text is both input & output so we copy it and start with an empty output.
{{#fileanchor: RawFile.php}}
<source lang=php>
$textorig=$text;
$text='';
</source>
For each anchor found we've to isolate the content of the next block.
{{#fileanchor: RawFile.php}}
<source lang=php>
foreach ($offsets as $offset) {
</source>
We start from the position of the current anchor. If the tag name of the block attached to the anchor is not specified, we look for the first block that follows the anchor, excluding <code> </code> and <code><file></code> block. The search can be easily done with a regular expression, using the ''lookahead negative assertion'' <code>(?!br\b|file\b)</code> to exclude the tags to ignore. Note that we need to ignore the anchor-link block <code><file></code> since the anchor starts right before that tag, and so the regular expression would match the anchor-link block it that tag is not specifically excluded.
{{#fileanchor: RawFile.php}}
<source lang=php>
$out = substr($textorig, $offset[1]);
// If no tag specified, we take the first one
if ($tag == '')
{
// With a regex assertion, we can easily ignore 'br' and 'file' tags
if (!preg_match('/<((?!br\b|file\b)\w+\b)/', $out, $matches))
return fnRawFile_Strip_Error ("ERROR - RawFile: Can't find opening tag after anchor '$offset[0]' (anchor=$anchor, name=$filename, tag=$tag)",$out,$text);
$tag=$matches[1];
}
</source>
Now, we know the tag name of the block to download, either because it was already specified as a GET attribute in the URL, or because we've found it in the search above. Again, using a regular expression, we look for the first block matching the specified tag name that follows the current anchor, and extract the content of the blocks. Note the use of the regex option <code>/.../s</code> to tell the regex engine that the matched text can span on multiple lines (with that option, <code>.</code> does match any character or a newline character). Also, we skip the first carriage return after the opening tag, if any (with <code>\n?</code>).
{{#fileanchor: RawFile.php}}
<source lang=php>
// Find the first tag matching $tag, and return enclosed text
if (!preg_match('/<'.$tag.'( [^>]*)?>\n?(.*?)<\/'.$tag.'>/s', $out, $matches))
return fnRawFile_Strip_Error ("ERROR - RawFile: no closing '$tag' found after anchor '$offset[0]' (anchor=$anchor, name=$filename, tag=$tag)",$out,$text);
$text .= $matches[2];
}
</source>
No need to deal with a Content-Length header because Mediawiki will do it for us, moreover more properly than we could if the output is sent gzipped, which is the default.
 So that's it, $text contains our file!
{{#fileanchor: RawFile.php}}
<source lang=php>
return true;
}

</source>

===Credits===
There is an official way to register the extension in a Mediawiki installation, so that it will be visible on the [[Special:Version]] page.
 Let's say the extension is in the category of parser hooks even if there is also a hook on Raw action.
{{#fileanchor: RawFile.php}}
<source lang=php>
$wgExtensionCredits['parserhook'][] = array('name' => 'RawFile',
'version' => '0.4',
'author' => 'Philippe Teuwen, Michael Peeters',
'url' => 'http://www.mediawiki.org/wiki/Extension:RawFile',
// 'url' => 'http://wiki.yobi.be/wiki/Mediawiki_RawFile',
'description' => 'Downloads a RAW copy of <nowiki><tag>data</tag></nowiki> in a file '.
'Useful e.g. to download a script or a patch '.
'It also allows what is called [http://en.wikipedia.org/wiki/Literate_programming Literate Programming]');
}

?>
</source>
And finally registration of the extension at the Mediawiki website according to the [http://www.mediawiki.org/wiki/Manual:Extensions Extensions Manual].

So this extension has now [http://www.mediawiki.org/wiki/Extension:RawFile its own page on the official Mediawiki site].

==Installation==
Download [{{#filelink: RawFile.php}} RawFile.php] and save it under the MediaWiki directory as extensions/RawFile/RawFile.php

Add at the end of LocalSettings.php:
<source lang=php>
require_once("$IP/extensions/RawFile/RawFile.php");
</source>
==Status==
If you use the extension properly the code is fully functional but it's rather raw on error handling.
==ChangeLog==
'''0.4'''
* '''Anchors''' can be specified using html '''class''' attribute
* New syntax for '''Links''' and '''Anchor-links''':
:<code><nowiki><file [name="..."] [anchor="..."] [tag="..."] [title="..."] >Link text</file></nowiki></code>
* Support multiple files on the same page with same name.
* Can specify the tag name of the block to download (to skip some irrelevant blocks when using an '''anchor-link''').
* Ignore '''<code> </code>''' tag.
* Some error reporting.
'''0.3'''
* Added optional parameter to <code>#fileLink</code> to indicate that the file is on another local wiki page
'''0.2'''
* Fix problem with Content-Length mismatch when transport is gzipped (default for Mediawiki if client supports it)
'''0.1'''
* Initial version

==Known bugs==
Jani Uusitalo reported the following issue:
 For some reason, if you use Epiphany's 'Save as' instead of a direct left-click, the downloaded file is a single byte. In Firefox the links work just fine, so this is probably an Epiphany bug. Uncommenting the <code>// wfResetOutputBuffers();</code> line didn't help. If anyone knows a workaround to help Epiphany users without breaking the support of the other browsers, please speak up :-)

==Questions and feedback==
If you've any trouble, questions or suggestions, you can [[User:PhilippeTeuwen|contact me]].

Mediawiki RawFile

2009-10-13T10:04:52Z

Fuujuhi: /* Syntax */ There are actually 3 elements - merged the 3rd one in the list for clarity

Mediawiki RawFile

2009-09-21T22:13:45Z

Fuujuhi: Adding optional parameter to #fileLink to indicate that the file is on a different local wiki page

MiscCrypto

2009-03-10T08:34:11Z

Fuujuhi: /* To unlock password-protected pdfs */

==Under Windows==
===[http://www.cryptool.com/ CryptTool]===
* Demonstration and Reference Program for Cryptography + making AES executables
* freeware
===[http://axcrypt.sourceforge.net/ AxCrypt]===
* AES-128 File Encryption, Compression and double-click Edit/View for secure document storage on Windows 95/98/ME/NT/2K/XP, locally or remote.
* GPL
===[http://www.chilkatsoft.com/ChilkatSfx.asp Zip 2 Secure EXE]===
* AES-128/192/256 bits File Encryption, Create self-extracting EXE files from zip files
* freeware
==[[LoopCrypt]]==
==[[Encfs]]==
==[[TrueCrypt]]==
==[[LUKS]]==

==Misc==
===To unlock password-protected pdfs===
* PDF documents may have up to 2 passwords:
** '''User password''': protects read access to the document, and when no ''Owner password'' is specified, also protects permission settings.
** '''Owner password''': protects full access to the document, and protects permission settings (printing, copying...)

* Use <tt>pdftops</tt> to convert a protected PDF to Postscript, and possibly combine it with <tt>ps2pdf</tt> to obtain an unrestricted pdf. Knowing the ''Owner password'' is the easiest since that allows removing all restrictions. If not, it also works but in that case the permission <tt>allow Printing</tt> must be set. If it is not the case, then ''Owner password'' is necessary. Also if ''User password'' is set (ie. pdf reader asks for a password at open), then it must be provided (providing ''Owner password'' is also ok of course).

<source lang="bash">
apt-get install gs-common xpdf-utils
# on Cygwin: install package xpdf
pdftops <encrypted>.pdf <decrypted>.ps # Owner pwd set but unknown and 'allow printing' set
pdftops -upw <userpwd> <encrypted>.pdf <decrypted>.ps # Same as above but user password also set
pdftops -opw <ownerpwd> <encrypted>.pdf <decrypted>.ps # 'allow printing' not set -> owner pwd is necessary

pdftops -upw <password> <encrypted>.pdf -|ps2pdf - <decrypted>.pdf # To generate a new PDF immediately
</source>

* An other solution is to use ''PDF Toolkit'' <tt>pdftk</tt> (<tt>apt-get install pdftk</tt>). However unlike ''pdftops'' this method doesn't work when only ''user password'' is known but both passwords are set.

<source lang="bash">pdftk <encrypted>.pdf input_pw <password> output <decrypted>.pdf</source>

* <tt>pdftk</tt> can also be used to generate protected documents, and specifying permissions.

<source lang="bash">
pdftk <unprotected>.pdf output <protected>.pdf allow Printing owner_pw <ownerpwd> user_pw <userpwd> #Allow HQ printing
</source>

===pwsafe===
To use 'secure ram' (thereby locking the data in memory and preventing it being flushed to disk) you may like to run it setuid root:
$ sudo dpkg-statoverride --update --add --force root root 4755 /usr/bin/pwsafe

AndLinux

2009-01-21T16:47:59Z

Fuujuhi: /* Troubleshooting */

==Intro==
[http://www.andlinux.org/ andLinux] is a bundle of several pieces enabling a linux distribution to run with its kernel as a Windows service.
* [http://www.ubuntu.com/ Ubuntu]
* [http://www.colinux.org/ coLinux], a port of the Linux kernel to Windows
* [http://www.straightrunning.com/XmingNotes/ Xming], a X server for Windows
* [http://www.pulseaudio.org/ PulseAudio], the sound server
==Tips==
===Adding a shared drive===
Add sth like this to andLinux\settings.txt
cofs1=D:\
And in the /etc/fstab of the Ubuntu:
1 /mnt/d cofs defaults 0 0
===Running kgpg===
The problem is that once kgpg is iconified, I've no idea where this icon goes and I cannot maximize the window anymore (it's also true for e.g. klipper)
 The trick is to use a dcop command to raise the window, you can play with kdcop to experiment
 So I added to andLinux\Launcher\menu.txt the following:
Kgpg;kgpg.ico;pgrep kgpg > /dev/null || kgpg;dcop kgpg KeyInterface showKeyManager
The icon was converted from /usr/share/icons/hicolor/32x32/apps/kgpg.png with The Gimp
===Windows Explorer integration===
In the Windows Explorer there is now on right-click on a file a possibility to open it with Kate.
 And right-click on a folder allows it to be opened in Konqueror or Konsole.
 To do that it is creating the path file:///mnt/win/... so be sure that /mnt/win points to your data partition.

If you want or need to hack into that part, have a look here:
* [http://www.andlinux.org/faq.php last point of the FAQ: Changing the Launcher Port]
* [http://andlinux.org/forum/viewtopic.php?p=939#939 a patched launcher.pl] to get it working with andCmd.exe too, as with [http://www.andlinux.org/forum/viewtopic.php?t=55 this example]

And here is a patch by Thierry Walrant to allow more than one share and to fix the andCmd.exe bug as well, you can [{{#file:launcher.patch}} get the patch here]
<source lang=diff>
--- launcher.pl 2008-07-03 05:26:40.000000000 -0400
+++ launcher.pl 2008-07-03 05:52:45.000000000 -0400
@@ -1,27 +1,44 @@
#!/usr/bin/perl

use IO::Socket;
+use strict;

+# Default settings.
+our $localPort = 81;
+our %pathPrefixes = (
+ '/mnt/win/' => "C:\\"
+ );

-$windowsPathPrefix = "C:\\";
-$linuxPathPrefix = "/mnt/win/";
+# Load config, if any.
+my $conf = '/etc/launcher-conf.pl';
+require $conf if -f $conf;

-$sock = new IO::Socket::INET(LocalPort => 81, Reuse => 1, Listen => 20) or die("Error creating local socket: $@\n");
+my $sock = new IO::Socket::INET(LocalPort => $localPort, Reuse => 1, Listen => 20) or die("Error creating local socket: $@\n");

-while($client = $sock->accept())
+while(my $client = $sock->accept())
{
- $request = <$client>;
+ my $request = <$client>;
chomp $request;
if($request =~ m/^cmd=(.+)&file=(.*)$/)
{
- $cmd = $1;
- $file = $2;
+ my $cmd = $1;
+ my $file = $2;

- $escapedWindowsPathPrefix = $windowsPathPrefix;
- $escapedWindowsPathPrefix =~ s/\\/\\\\/g;
- $file =~ s/$escapedWindowsPathPrefix/$linuxPathPrefix/g;
+ # Apply all mappings
+ foreach my $linuxPathPrefix ( keys %pathPrefixes ) {
+ my $windowsPathPrefix = $pathPrefixes{$linuxPathPrefix};
+ $windowsPathPrefix =~ s/\\/\\\\/g;
+ $file =~ s/$windowsPathPrefix/$linuxPathPrefix/g;
+
+ # Apply mapping to 'cmd' as well (circumvent issue with
+ # andCmd.exe which passes all arguments in 'cmd').
+ $cmd =~ s/$windowsPathPrefix/$linuxPathPrefix/g;
+ }
+
+ # Finalize with back-slashes replacement.
$file =~ s/\\/\//g;
+ $cmd =~ s/\\/\//g;

system("$cmd $file &");
</source>
To use it you can use the hardcoded mapping as before or you can define your mappings in a config file /etc/launcher-conf.pl such as:
<source lang=perl>
# Configuration file for the launcher: /usr/local/sbin/launcher.pl

%pathPrefixes = (
'/mnt/winC/' => 'C:\\',
'/mnt/winD/' => 'D:\\',
'/mnt/winZ/' => 'Z:\\'
);

# A Perl library MUST return 1
1;
</source>

Please note this configuration assumes that the mount points /mnt/winC, /mnt/winD and /mnt/winZ are mounted on the root of the windows drives C:, D: and Z:. This allows you to launch KDE applications on any of your windows files and folders.

===Starting Xming our own way===
By default Xming is started at boot time (cf folder Startup)
 If you don't want that and start Xming only when you need it, here is a replacement of andLinux/srvstart.bat
<source lang=dos>
@echo off
setlocal ENABLEEXTENSIONS

rem Do we need to start xming?
set TASKNAME=xming.exe
tasklist /FI "IMAGENAME eq %TASKNAME%" 2> \nul | find /i "%TASKNAME%" > \nul

IF ERRORLEVEL 1 CALL :STARTXMING

net start andLinux
goto :EOF

:STARTXMING

echo Starting XMing...
start /D"C:\Program Files\andLinux" C:\Progra~1\andLinux\Xming\Xming.exe :0 -dpi 85 -ac -clipboard -notrayicon -c -multiwindow -reset -terminate -unixkill -logfile Xming.log

goto :EOF
</source>
===Port forwarding===
To forward a port of the host to andLinux so it can run servers available to everybody: (instructions extracted from [http://colinux.wikia.com/wiki/Network#Forwarding_Ports_over_Slirp here])
 Edit your settings.txt and append some port forwarding infos to eth0:
#eth0=slirp,<MAC>,tcp:<from Hosting OS port>:<to coLinux OS port>/...
eth0=slirp,,tcp:22:22/tcp:80:80

==Troubleshooting==
===Startup delays===
* '''Interaction with Anti-Virus software''' - AV software may interact with the startup of AndLinux, typically by maintaining a lock on the 4GB disk image file <tt>base.drv</tt> while it is scanned.
** '''Solution''': To prevent this, one can exclude the andlinux directory from the AV on-access scanner. In McAfee, this is done by ''opening the '''VirusScan console''' → '''double-ckick''' On-Access Scanner → then select '''All Processes''' → open the pane '''Detection''' and click the button '''Exclusions...''' → click '''Add...''' to add a new exclusion → '''Browse''' to AndLinux directory and select '''Also exclude subfolders'''''.
* '''Stalling when mounting /''' - You experience a huge delay during the startup process, and when launching the ''FLTK'' console, you observe that andlinux is stalling on the message below.
<div style="padding-left:2em;"><pre>ReiserFS: cobd0: warning: sh-2021: reiserfs_fill_super: can not find reiserfs on cobd0
kjournald starting. Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
Modules already installed.
Closing /
kjournald starting. Commit interval 5 seconds
</pre></div>
:* '''Solution''': The fix proposed in [http://www.andlinux.org/forum/viewtopic.php?t=291&sid=7ef2f7b1234865dd3fabe645c1b48a25 this post] is to actually change the extension of the disk image file from <tt>.drv</tt> to something like <tt>.drive</tt> (ie. change <tt>./Drives/base.drv</tt> to <tt>./Drives/base.drive</tt> and <tt>./Drives/swap.drv</tt> to <tt>./Drives/swap.drive</tt>) and adapt the configuration files <tt>./settings.txt</tt> accordingly.

AndLinux

2009-01-21T16:46:16Z

Fuujuhi: Troubleshooting

==Intro==
[http://www.andlinux.org/ andLinux] is a bundle of several pieces enabling a linux distribution to run with its kernel as a Windows service.
* [http://www.ubuntu.com/ Ubuntu]
* [http://www.colinux.org/ coLinux], a port of the Linux kernel to Windows
* [http://www.straightrunning.com/XmingNotes/ Xming], a X server for Windows
* [http://www.pulseaudio.org/ PulseAudio], the sound server
==Tips==
===Adding a shared drive===
Add sth like this to andLinux\settings.txt
cofs1=D:\
And in the /etc/fstab of the Ubuntu:
1 /mnt/d cofs defaults 0 0
===Running kgpg===
The problem is that once kgpg is iconified, I've no idea where this icon goes and I cannot maximize the window anymore (it's also true for e.g. klipper)
 The trick is to use a dcop command to raise the window, you can play with kdcop to experiment
 So I added to andLinux\Launcher\menu.txt the following:
Kgpg;kgpg.ico;pgrep kgpg > /dev/null || kgpg;dcop kgpg KeyInterface showKeyManager
The icon was converted from /usr/share/icons/hicolor/32x32/apps/kgpg.png with The Gimp
===Windows Explorer integration===
In the Windows Explorer there is now on right-click on a file a possibility to open it with Kate.
 And right-click on a folder allows it to be opened in Konqueror or Konsole.
 To do that it is creating the path file:///mnt/win/... so be sure that /mnt/win points to your data partition.

If you want or need to hack into that part, have a look here:
* [http://www.andlinux.org/faq.php last point of the FAQ: Changing the Launcher Port]
* [http://andlinux.org/forum/viewtopic.php?p=939#939 a patched launcher.pl] to get it working with andCmd.exe too, as with [http://www.andlinux.org/forum/viewtopic.php?t=55 this example]

And here is a patch by Thierry Walrant to allow more than one share and to fix the andCmd.exe bug as well, you can [{{#file:launcher.patch}} get the patch here]
<source lang=diff>
--- launcher.pl 2008-07-03 05:26:40.000000000 -0400
+++ launcher.pl 2008-07-03 05:52:45.000000000 -0400
@@ -1,27 +1,44 @@
#!/usr/bin/perl

use IO::Socket;
+use strict;

+# Default settings.
+our $localPort = 81;
+our %pathPrefixes = (
+ '/mnt/win/' => "C:\\"
+ );

-$windowsPathPrefix = "C:\\";
-$linuxPathPrefix = "/mnt/win/";
+# Load config, if any.
+my $conf = '/etc/launcher-conf.pl';
+require $conf if -f $conf;

-$sock = new IO::Socket::INET(LocalPort => 81, Reuse => 1, Listen => 20) or die("Error creating local socket: $@\n");
+my $sock = new IO::Socket::INET(LocalPort => $localPort, Reuse => 1, Listen => 20) or die("Error creating local socket: $@\n");

-while($client = $sock->accept())
+while(my $client = $sock->accept())
{
- $request = <$client>;
+ my $request = <$client>;
chomp $request;
if($request =~ m/^cmd=(.+)&file=(.*)$/)
{
- $cmd = $1;
- $file = $2;
+ my $cmd = $1;
+ my $file = $2;

- $escapedWindowsPathPrefix = $windowsPathPrefix;
- $escapedWindowsPathPrefix =~ s/\\/\\\\/g;
- $file =~ s/$escapedWindowsPathPrefix/$linuxPathPrefix/g;
+ # Apply all mappings
+ foreach my $linuxPathPrefix ( keys %pathPrefixes ) {
+ my $windowsPathPrefix = $pathPrefixes{$linuxPathPrefix};
+ $windowsPathPrefix =~ s/\\/\\\\/g;
+ $file =~ s/$windowsPathPrefix/$linuxPathPrefix/g;
+
+ # Apply mapping to 'cmd' as well (circumvent issue with
+ # andCmd.exe which passes all arguments in 'cmd').
+ $cmd =~ s/$windowsPathPrefix/$linuxPathPrefix/g;
+ }
+
+ # Finalize with back-slashes replacement.
$file =~ s/\\/\//g;
+ $cmd =~ s/\\/\//g;

system("$cmd $file &");
</source>
To use it you can use the hardcoded mapping as before or you can define your mappings in a config file /etc/launcher-conf.pl such as:
<source lang=perl>
# Configuration file for the launcher: /usr/local/sbin/launcher.pl

%pathPrefixes = (
'/mnt/winC/' => 'C:\\',
'/mnt/winD/' => 'D:\\',
'/mnt/winZ/' => 'Z:\\'
);

# A Perl library MUST return 1
1;
</source>

Please note this configuration assumes that the mount points /mnt/winC, /mnt/winD and /mnt/winZ are mounted on the root of the windows drives C:, D: and Z:. This allows you to launch KDE applications on any of your windows files and folders.

===Starting Xming our own way===
By default Xming is started at boot time (cf folder Startup)
 If you don't want that and start Xming only when you need it, here is a replacement of andLinux/srvstart.bat
<source lang=dos>
@echo off
setlocal ENABLEEXTENSIONS

rem Do we need to start xming?
set TASKNAME=xming.exe
tasklist /FI "IMAGENAME eq %TASKNAME%" 2> \nul | find /i "%TASKNAME%" > \nul

IF ERRORLEVEL 1 CALL :STARTXMING

net start andLinux
goto :EOF

:STARTXMING

echo Starting XMing...
start /D"C:\Program Files\andLinux" C:\Progra~1\andLinux\Xming\Xming.exe :0 -dpi 85 -ac -clipboard -notrayicon -c -multiwindow -reset -terminate -unixkill -logfile Xming.log

goto :EOF
</source>
===Port forwarding===
To forward a port of the host to andLinux so it can run servers available to everybody: (instructions extracted from [http://colinux.wikia.com/wiki/Network#Forwarding_Ports_over_Slirp here])
 Edit your settings.txt and append some port forwarding infos to eth0:
#eth0=slirp,<MAC>,tcp:<from Hosting OS port>:<to coLinux OS port>/...
eth0=slirp,,tcp:22:22/tcp:80:80

==Troubleshooting==
===Startup delays===
* '''Interaction with Anti-Virus software''' - AV software may interact with the startup of AndLinux, typically by maintaining a lock on the 4GB disk image file <tt>base.drv</tt> while it is scanned.
** '''Solution''': To prevent this, one can exclude the andlinux directory from the AV on-access scanner. In McAfee, this is done by ''opening the '''VirusScan console''' → '''double-ckick''' On-Access Scanner → then select '''All Processes''' → open the pane '''Detection''' and click the button '''Exclusions...''' → click '''Add...''' to add a new exclusion → '''Browse''' to AndLinux directory and select '''Also exclude subfolders'''''.
* '''Stalling when mounting /''' - You experience a huge delay during the startup process, and when launching the ''FLTK'' console, you observe that andlinux is stalling on the message below.
<div style="padding-left:2em;"><pre>ReiserFS: codb0: warning sh-2021: reiserfs_fill_super: can not find reiserfs on codb0\nkjournald starting. Commit 5 seconds</pre></div>
:* '''Solution''': The fix proposed in [http://www.andlinux.org/forum/viewtopic.php?t=291&sid=7ef2f7b1234865dd3fabe645c1b48a25 this post] is to actually change the extension of the disk image file from <tt>.drv</tt> to something like <tt>.drive</tt> (ie. change <tt>./Drives/base.drv</tt> to <tt>./Drives/base.drive</tt> and <tt>./Drives/swap.drv</tt> to <tt>./Drives/swap.drive</tt>) and adapt the configuration files <tt>./settings.txt</tt> accordingly.

Cygwin-multitee

2008-11-25T02:33:18Z

Fuujuhi: /* The patch */

'''multitee''' sends multiple inputs to multiple outputs. Check this [http://code.dogmap.org/fdtools/multitee/ page].
Original is [http://cr.yp.to/software/multitee-3.0.shar.gz here], or can also be found here on [http://ftp.de.debian.org/debian/pool/main/m/multitee/multitee_3.0.orig.tar.gz Debian].

Below you'll find a patch to build ''multitee'' on cygwin (and possibly any other POSIX distribution). To build, just untar the ''multitee'' archive, and then in a shell:

<source lang="bash">
$ patch -Np1 <../multitee-3.0-cygwin.patch
$ make
$ cp multitee.1 /usr/local/man/man1
$ cp multitee.exe /usr/local/bin
</source>

Example of use:
<source lang="bash">
$ multitee 0-1,4,5 4> foo 5> bar # same as tee foo bar with better blocking behavior
$ multitee 0:1 3:1 4:1,2 6:7 # various merge and copy
$ tcpclient server smtp multitee 0:7 6:1e0 # e0 tell multitee to quit as soon connection close
$ multitee 0:3 3:1 # same as 'socat - FD:3'
</source>

== The patch ==
The patch ports calls to BSD ''signal.h'' API (<tt>struct sigvec</tt>, <tt>sigvec()</tt>, <tt>sigmask()</tt>, <tt>sigblock()</tt>...) to the POSIX API (<tt>struct sigaction</tt>, <tt>sigaction()</tt>, <tt>sigprocmask()</tt>, <tt>sigfillset()</tt>...). Here the [{{#file: multitee-3.0-cygwin.patch}} patch]:

<source lang=diff>
--- multitee-3.0/Makefile 1970-01-01 01:00:00.000000000 +0100
+++ multitee-3.0-cygwin/Makefile 2008-11-24 03:35:43.875000000 +0100
@@ -0,0 +1,9 @@
+
+all: multitee tee
+
+multitee: multitee.o sigdfl.o sigsched.o ralloc.o getopt.o fmt.o scan.o
+tee: tee.o getopt.o ralloc.o fmt.o
+
+clean:
+ rm -f tee multitee *.o *.a *.0 *~ core
+
--- multitee-3.0/multitee.c 1998-02-17 03:25:43.000000000 +0100
+++ multitee-3.0-cygwin/multitee.c 2008-11-24 03:38:51.296875000 +0100
@@ -180,7 +180,7 @@
int in;
int w;

- ioscp = (ioscstack) p;
+ ioscp = (struct iosc *) p;
in = ioscp->in;

if (ioscp->dead)
--- multitee-3.0/ralloc.c 1998-02-17 03:25:43.000000000 +0100
+++ multitee-3.0-cygwin/ralloc.c 2008-11-24 03:41:24.093750000 +0100
@@ -40,7 +40,7 @@

#include "ralloc.h"
#include "sod.h"
-extern char *malloc(); /*XXXX*/
+extern void *malloc(); /*XXXX*/
extern void free();

typedef int (*foo)();
--- multitee-3.0/sigdfl.c 1998-02-17 03:25:44.000000000 +0100
+++ multitee-3.0-cygwin/sigdfl.c 2008-11-24 03:39:57.734375000 +0100
@@ -2,6 +2,7 @@
Daniel J. Bernstein, brnstnd@nyu.edu.
No dependencies.
Requires BSD signal syscalls.
+24/11/08: Michael Peeters - Ported to POSIX signal.h (tested under Cygwin)
7/18/91: Baseline. sigdfl 1.0, public domain.
No known patent problems.

@@ -18,18 +19,20 @@

static int cont = 0;

-static sigcont() /* XXX: should declare with right signal type */
+static void sigcont(int sig) /* XXX: should declare with right signal type */
{
+ (void)sig;
cont = 1;
}

int sigdfl(sig)
int sig;
{
- int oldmask;
- struct sigvec oldvec;
- struct sigvec vec;
- struct sigvec contvec;
+ sigset_t oldmask;
+ sigset_t workmask;
+ struct sigaction oldvec;
+ struct sigaction vec;
+ struct sigaction contvec;

if (sig == SIGCONT)
return 0; /* strategy below simply cannot work for CONT */
@@ -43,19 +46,20 @@
/* killing the process and without stopping the process so that it'll */
/* receive CONT later, then we will enter an infinite loop. [sigh] */
/* XXX: put maximum time wastage on this? */
- oldmask = sigblock(0);
- sigblock(~0);
+ sigprocmask(SIG_BLOCK,NULL,&oldmask);
+ sigfillset(&workmask);
+ sigprocmask(SIG_BLOCK,&workmask,NULL);
/* now we won't receive any signals */
- vec.sv_handler = SIG_DFL;
- vec.sv_mask = ~0;
- vec.sv_flags = 0;
- if (sigvec(sig,&vec,&oldvec) == -1)
+ vec.sa_handler = SIG_DFL;
+ vec.sa_mask = ~0;
+ vec.sa_flags = 0;
+ if (sigaction(sig,&vec,&oldvec) == -1)
if ((sig != SIGSTOP) && (sig != SIGKILL))
return -1;
- vec.sv_handler = sigcont;
- vec.sv_mask = ~0;
- vec.sv_flags = 0;
- if (sigvec(SIGCONT,&vec,&contvec) == -1)
+ vec.sa_handler = sigcont;
+ vec.sa_mask = ~0;
+ vec.sa_flags = 0;
+ if (sigaction(SIGCONT,&vec,&contvec) == -1)
return -1;
cont = 0;
if (kill(getpid(),sig) == -1)
@@ -66,16 +70,20 @@
reached this point, sigcont() might already have been run. that's
why cont must be set to 0 before the kill(). */
/* after this next bit we may receive sig and/or CONT */
- sigsetmask(~(sigmask(sig) | sigmask(SIGCONT)));
+ sigfillset(&workmask);
+ sigdelset(&workmask,sig);
+ sigdelset(&workmask,SIGCONT);
+ sigprocmask(SIG_SETMASK,&workmask,NULL);
/* in the near future, sig will in fact be received */
while (!cont) /* dead loop until we receive CONT */
; /* XXX: there should be a syscall so we don't have to loop here */
- sigblock(~0);
+ sigfillset(&workmask);
+ sigprocmask(SIG_BLOCK,&workmask,NULL);
/* now we won't receive any signals */
- (void) sigvec(sig,&oldvec,&vec); /* we don't care if it fails */
- (void) sigvec(SIGCONT,&contvec,&vec);
+ (void) sigaction(sig,&oldvec,&vec); /* we don't care if it fails */
+ (void) sigaction(SIGCONT,&contvec,&vec);
/* now signal handlers are back to normal */
- (void) sigsetmask(oldmask);
+ (void) sigprocmask(SIG_SETMASK,&oldmask,NULL);
return 0;
}

--- multitee-3.0/sigsched.c 1998-02-17 03:25:44.000000000 +0100
+++ multitee-3.0-cygwin/sigsched.c 2008-11-24 03:35:43.906250000 +0100
@@ -2,6 +2,7 @@
Daniel J. Bernstein, brnstnd@nyu.edu.
Depends on ralloc.h, sod.h, config/fdsettrouble.h.
Requires BSDish environment: reliable signals, sig{vec,block,setmask}, select.
+24/11/08: Michael Peeters - Ported to POSIX signal.h (tested under Cygwin)
9/1/91: Added worst-case fdset, FD_ZERO, etc. definitions.
8/25/91: sigsched 1.1, public domain.
8/25/91: Fixed bug that sigs[sched->blah].r didn't force instant timeout.
@@ -31,17 +32,6 @@

/* XXX: should restore signal set exactly after ss_exec returns */

-typedef int sigc_set; /*XXX */
-
-#define sigc_ismember(x,i) (*(x) & (1 << ((i) - 1)))
-#define sigc_addset(x,i) (*(x) |= (1 << ((i) - 1)))
-#define sigc_emptyset(x) (*(x) = 0)
-
-/* sigprocmask(SIG_UNBLOCK,xxxx,(sigc_set *) 0); */
-#define sigc_unblock(x) (sigsetmask(sigblock(0) & ~*(x)))
-/* sigprocmask(SIG_BLOCK,xxxx,(sigc_set *) 0); */
-#define sigc_block(x) (sigblock(*(x)))
-
#ifndef NSIG
#define NSIG 64 /* it's not as if any sane system has more than 32 */
#endif
@@ -333,8 +323,8 @@
sigs[i].r = 1;
}

-static sigc_set sigstorage;
-static sigc_set *xxxx = 0;
+static sigset_t sigstorage;
+static sigset_t *xxxx = 0;

int ss_addsig(i)
int i;
@@ -344,9 +334,9 @@
if (!xxxx)
{
xxxx = &sigstorage;
- sigc_emptyset(xxxx);
+ sigemptyset(xxxx);
}
- sigc_addset(xxxx,i);
+ sigaddset(xxxx,i);
return 0;
}

@@ -362,7 +352,7 @@
int ss_exec()
{
int i;
- struct sigvec sv;
+ struct sigaction sv;
recvlist recvhead;
recvlist temp;
schedlist sch;
@@ -371,18 +361,18 @@

if (xxxx)
{
- sigc_block(xxxx);
+ sigprocmask(SIG_BLOCK,xxxx,NULL);

- sv.sv_handler = handle;
- sv.sv_mask = *xxxx; /* so handle won't interrupt itself */
- sv.sv_flags = 0;
+ sv.sa_handler = handle;
+ sv.sa_mask = *xxxx; /* so handle won't interrupt itself */
+ sv.sa_flags = 0;

/* XXX: Does anyone but me find it absolutely idiotic that POSIX
doesn't provide a way to get each member of a signal set in turn? */
for (i = 0;i < NUMSIGS;i++)
{
- if (sigc_ismember(xxxx,i))
- if (sigvec(i,&sv,(struct sigvec *) 0) == -1) /*XXX: really trash orig? */
+ if (sigismember(xxxx,i))
+ if (sigaction(i,&sv,(struct sigaction *) 0) == -1) /*XXX: really trash orig? */
; /* not our problem */
}
}
@@ -498,7 +488,7 @@
}

if (xxxx)
- sigc_unblock(xxxx);
+ sigprocmask(SIG_UNBLOCK,xxxx,NULL);
/* This is the only section where handle() can be called. */
/* XXX: If maxfd == -1, this select functions as a pause. */
/* XXX: If maxfd == -1 and timeout is instant, should skip select. */
@@ -510,7 +500,7 @@
r = select(maxfd + 1,&rfds,&wfds,&efds,&timeout);
/* XXX: does this necessarily prevent timeout race conditions? */
if (xxxx)
- sigc_block(xxxx);
+ sigprocmask(SIG_BLOCK,xxxx,NULL);

if (r == -1)
{
@@ -529,7 +519,7 @@

for (sp = schedhead;sp;sp = SODnext(sp))
{
- switch(SODdata(sp).sig->type)
+ switch(SODdata(sp).sig->type)
{
case JUNK:
break;
@@ -595,7 +585,7 @@
}
}
if (xxxx)
- sigc_unblock(xxxx);
+ sigprocmask(SIG_UNBLOCK,xxxx,NULL);
/* XXX: should put this at other returns as well */
return 0;
}
</source>

Cygwin-multitee

2008-11-25T02:30:11Z

Fuujuhi: New page: '''multitee''' sends multiple inputs to multiple outputs. Check this [http://code.dogmap.org/fdtools/multitee/ page]. Original is [http://cr.yp.to/software/multitee-3.0.shar.gz here], or c...

'''multitee''' sends multiple inputs to multiple outputs. Check this [http://code.dogmap.org/fdtools/multitee/ page].
Original is [http://cr.yp.to/software/multitee-3.0.shar.gz here], or can also be found here on [http://ftp.de.debian.org/debian/pool/main/m/multitee/multitee_3.0.orig.tar.gz Debian].

Below you'll find a patch to build ''multitee'' on cygwin (and possibly any other POSIX distribution). To build, just untar the ''multitee'' archive, and then in a shell:

<source lang="bash">
$ patch -Np1 <../multitee-3.0-cygwin.patch
$ make
$ cp multitee.1 /usr/local/man/man1
$ cp multitee.exe /usr/local/bin
</source>

Example of use:
<source lang="bash">
$ multitee 0-1,4,5 4> foo 5> bar # same as tee foo bar with better blocking behavior
$ multitee 0:1 3:1 4:1,2 6:7 # various merge and copy
$ tcpclient server smtp multitee 0:7 6:1e0 # e0 tell multitee to quit as soon connection close
$ multitee 0:3 3:1 # same as 'socat - FD:3'
</source>

== The patch ==
Here the [{{#file: multitee-3.0-cygwin.patch}} patch] to build ''multitee'' on ''Cygwin''. The patch ports calls to BSD ''signal.h'' API (<tt>struct sigvec</tt>, <tt>sigvec()</tt>, <tt>sigmask()</tt>, <tt>sigblock()</tt>...) to the POSIX API (<tt>struct sigaction</tt>, <tt>sigaction()</tt>, <tt>sigprocmask()</tt>, <tt>sigfillset()</tt>...).

<source lang=diff>
--- multitee-3.0/Makefile 1970-01-01 01:00:00.000000000 +0100
+++ multitee-3.0-cygwin/Makefile 2008-11-24 03:35:43.875000000 +0100
@@ -0,0 +1,9 @@
+
+all: multitee tee
+
+multitee: multitee.o sigdfl.o sigsched.o ralloc.o getopt.o fmt.o scan.o
+tee: tee.o getopt.o ralloc.o fmt.o
+
+clean:
+ rm -f tee multitee *.o *.a *.0 *~ core
+
--- multitee-3.0/multitee.c 1998-02-17 03:25:43.000000000 +0100
+++ multitee-3.0-cygwin/multitee.c 2008-11-24 03:38:51.296875000 +0100
@@ -180,7 +180,7 @@
int in;
int w;

- ioscp = (ioscstack) p;
+ ioscp = (struct iosc *) p;
in = ioscp->in;

if (ioscp->dead)
--- multitee-3.0/ralloc.c 1998-02-17 03:25:43.000000000 +0100
+++ multitee-3.0-cygwin/ralloc.c 2008-11-24 03:41:24.093750000 +0100
@@ -40,7 +40,7 @@

#include "ralloc.h"
#include "sod.h"
-extern char *malloc(); /*XXXX*/
+extern void *malloc(); /*XXXX*/
extern void free();

typedef int (*foo)();
--- multitee-3.0/sigdfl.c 1998-02-17 03:25:44.000000000 +0100
+++ multitee-3.0-cygwin/sigdfl.c 2008-11-24 03:39:57.734375000 +0100
@@ -2,6 +2,7 @@
Daniel J. Bernstein, brnstnd@nyu.edu.
No dependencies.
Requires BSD signal syscalls.
+24/11/08: Michael Peeters - Ported to POSIX signal.h (tested under Cygwin)
7/18/91: Baseline. sigdfl 1.0, public domain.
No known patent problems.

@@ -18,18 +19,20 @@

static int cont = 0;

-static sigcont() /* XXX: should declare with right signal type */
+static void sigcont(int sig) /* XXX: should declare with right signal type */
{
+ (void)sig;
cont = 1;
}

int sigdfl(sig)
int sig;
{
- int oldmask;
- struct sigvec oldvec;
- struct sigvec vec;
- struct sigvec contvec;
+ sigset_t oldmask;
+ sigset_t workmask;
+ struct sigaction oldvec;
+ struct sigaction vec;
+ struct sigaction contvec;

if (sig == SIGCONT)
return 0; /* strategy below simply cannot work for CONT */
@@ -43,19 +46,20 @@
/* killing the process and without stopping the process so that it'll */
/* receive CONT later, then we will enter an infinite loop. [sigh] */
/* XXX: put maximum time wastage on this? */
- oldmask = sigblock(0);
- sigblock(~0);
+ sigprocmask(SIG_BLOCK,NULL,&oldmask);
+ sigfillset(&workmask);
+ sigprocmask(SIG_BLOCK,&workmask,NULL);
/* now we won't receive any signals */
- vec.sv_handler = SIG_DFL;
- vec.sv_mask = ~0;
- vec.sv_flags = 0;
- if (sigvec(sig,&vec,&oldvec) == -1)
+ vec.sa_handler = SIG_DFL;
+ vec.sa_mask = ~0;
+ vec.sa_flags = 0;
+ if (sigaction(sig,&vec,&oldvec) == -1)
if ((sig != SIGSTOP) && (sig != SIGKILL))
return -1;
- vec.sv_handler = sigcont;
- vec.sv_mask = ~0;
- vec.sv_flags = 0;
- if (sigvec(SIGCONT,&vec,&contvec) == -1)
+ vec.sa_handler = sigcont;
+ vec.sa_mask = ~0;
+ vec.sa_flags = 0;
+ if (sigaction(SIGCONT,&vec,&contvec) == -1)
return -1;
cont = 0;
if (kill(getpid(),sig) == -1)
@@ -66,16 +70,20 @@
reached this point, sigcont() might already have been run. that's
why cont must be set to 0 before the kill(). */
/* after this next bit we may receive sig and/or CONT */
- sigsetmask(~(sigmask(sig) | sigmask(SIGCONT)));
+ sigfillset(&workmask);
+ sigdelset(&workmask,sig);
+ sigdelset(&workmask,SIGCONT);
+ sigprocmask(SIG_SETMASK,&workmask,NULL);
/* in the near future, sig will in fact be received */
while (!cont) /* dead loop until we receive CONT */
; /* XXX: there should be a syscall so we don't have to loop here */
- sigblock(~0);
+ sigfillset(&workmask);
+ sigprocmask(SIG_BLOCK,&workmask,NULL);
/* now we won't receive any signals */
- (void) sigvec(sig,&oldvec,&vec); /* we don't care if it fails */
- (void) sigvec(SIGCONT,&contvec,&vec);
+ (void) sigaction(sig,&oldvec,&vec); /* we don't care if it fails */
+ (void) sigaction(SIGCONT,&contvec,&vec);
/* now signal handlers are back to normal */
- (void) sigsetmask(oldmask);
+ (void) sigprocmask(SIG_SETMASK,&oldmask,NULL);
return 0;
}

--- multitee-3.0/sigsched.c 1998-02-17 03:25:44.000000000 +0100
+++ multitee-3.0-cygwin/sigsched.c 2008-11-24 03:35:43.906250000 +0100
@@ -2,6 +2,7 @@
Daniel J. Bernstein, brnstnd@nyu.edu.
Depends on ralloc.h, sod.h, config/fdsettrouble.h.
Requires BSDish environment: reliable signals, sig{vec,block,setmask}, select.
+24/11/08: Michael Peeters - Ported to POSIX signal.h (tested under Cygwin)
9/1/91: Added worst-case fdset, FD_ZERO, etc. definitions.
8/25/91: sigsched 1.1, public domain.
8/25/91: Fixed bug that sigs[sched->blah].r didn't force instant timeout.
@@ -31,17 +32,6 @@

/* XXX: should restore signal set exactly after ss_exec returns */

-typedef int sigc_set; /*XXX */
-
-#define sigc_ismember(x,i) (*(x) & (1 << ((i) - 1)))
-#define sigc_addset(x,i) (*(x) |= (1 << ((i) - 1)))
-#define sigc_emptyset(x) (*(x) = 0)
-
-/* sigprocmask(SIG_UNBLOCK,xxxx,(sigc_set *) 0); */
-#define sigc_unblock(x) (sigsetmask(sigblock(0) & ~*(x)))
-/* sigprocmask(SIG_BLOCK,xxxx,(sigc_set *) 0); */
-#define sigc_block(x) (sigblock(*(x)))
-
#ifndef NSIG
#define NSIG 64 /* it's not as if any sane system has more than 32 */
#endif
@@ -333,8 +323,8 @@
sigs[i].r = 1;
}

-static sigc_set sigstorage;
-static sigc_set *xxxx = 0;
+static sigset_t sigstorage;
+static sigset_t *xxxx = 0;

int ss_addsig(i)
int i;
@@ -344,9 +334,9 @@
if (!xxxx)
{
xxxx = &sigstorage;
- sigc_emptyset(xxxx);
+ sigemptyset(xxxx);
}
- sigc_addset(xxxx,i);
+ sigaddset(xxxx,i);
return 0;
}

@@ -362,7 +352,7 @@
int ss_exec()
{
int i;
- struct sigvec sv;
+ struct sigaction sv;
recvlist recvhead;
recvlist temp;
schedlist sch;
@@ -371,18 +361,18 @@

if (xxxx)
{
- sigc_block(xxxx);
+ sigprocmask(SIG_BLOCK,xxxx,NULL);

- sv.sv_handler = handle;
- sv.sv_mask = *xxxx; /* so handle won't interrupt itself */
- sv.sv_flags = 0;
+ sv.sa_handler = handle;
+ sv.sa_mask = *xxxx; /* so handle won't interrupt itself */
+ sv.sa_flags = 0;

/* XXX: Does anyone but me find it absolutely idiotic that POSIX
doesn't provide a way to get each member of a signal set in turn? */
for (i = 0;i < NUMSIGS;i++)
{
- if (sigc_ismember(xxxx,i))
- if (sigvec(i,&sv,(struct sigvec *) 0) == -1) /*XXX: really trash orig? */
+ if (sigismember(xxxx,i))
+ if (sigaction(i,&sv,(struct sigaction *) 0) == -1) /*XXX: really trash orig? */
; /* not our problem */
}
}
@@ -498,7 +488,7 @@
}

if (xxxx)
- sigc_unblock(xxxx);
+ sigprocmask(SIG_UNBLOCK,xxxx,NULL);
/* This is the only section where handle() can be called. */
/* XXX: If maxfd == -1, this select functions as a pause. */
/* XXX: If maxfd == -1 and timeout is instant, should skip select. */
@@ -510,7 +500,7 @@
r = select(maxfd + 1,&rfds,&wfds,&efds,&timeout);
/* XXX: does this necessarily prevent timeout race conditions? */
if (xxxx)
- sigc_block(xxxx);
+ sigprocmask(SIG_BLOCK,xxxx,NULL);

if (r == -1)
{
@@ -529,7 +519,7 @@

for (sp = schedhead;sp;sp = SODnext(sp))
{
- switch(SODdata(sp).sig->type)
+ switch(SODdata(sp).sig->type)
{
case JUNK:
break;
@@ -595,7 +585,7 @@
}
}
if (xxxx)
- sigc_unblock(xxxx);
+ sigprocmask(SIG_UNBLOCK,xxxx,NULL);
/* XXX: should put this at other returns as well */
return 0;
}
</source>

Courier-Cygwin

2008-11-07T15:14:09Z

Fuujuhi: Master was not happy... Let's roll back first to previous version

=Pre-requisite=

Courier-IMAP requires the installation of the [http://www.courier-mta.org/authlib/ Courier Authentication Library] and the following cygwin tools and packages

* patch, tar, make, gcc
* crypt
* libgdbm-devel
* libtool
* inetutils
* cygrunsrv

Run the [http://www.cygwin.com/setup.exe Cygwin setup] program. Locate, select and install the required packages.

=Courier Authentication Library=

Installation of Courier Authentication Library using the module userdb to manage mail accounts. All other modules are disabled.

==Compilation and installation==

Download and untar [http://www.courier-mta.org/download.php#authlib courier-authlib] in your favorite sandbox.
Version used: [http://prdownloads.sourceforge.net/courier/courier-authlib-0.61.0.tar.bz2 0.61.0]

Apply the [{{#file:courier-authlib-0.61.0-cygwin.patch}} patch] before configuration to correct following issues:

# incorrect usage of EXEEXT in ''makedat/Makefile''
# add libtool flag (-no-undefined) required for DLL creation
# add missing dependencies
# add PATH in start script to access new DLL (installed in a different directory)

Solution source: http://lists.cairographics.org/archives/cairo/2004-April/001125.html

<source lang=diff>
--- courier-authlib-0.61.0/makedat/Makefile.in 2008-05-24 16:21:09.000000000 +0200
+++ courier-authlib-0.61.0-cygwin/makedat/Makefile.in 2008-10-21 16:02:38.709166700 +0200
@@ -182,7 +182,7 @@
libexecdir = @libexecdir@
localedir = @localedir@
localstatedir = @localstatedir@
-makedatprog_target = @makedatprog_target@
+makedatprog_target = @makedatprog_target@$(EXEEXT)
makedatprogpath = @makedatprogpath@
mandir = @mandir@
mkdir_p = @mkdir_p@
@@ -198,7 +198,7 @@
target_alias = @target_alias@
top_builddir = @top_builddir@
top_srcdir = @top_srcdir@
-noinst_PROGRAMS = @makedatprog_target@
+noinst_PROGRAMS = @makedatprog_target@$(EXEEXT)
makedatprog_SOURCES = makedatprog.c
makedatprog_DEPENDENCIES = @dblibrary@
makedatprog_LDADD = @dblibrary@
--- courier-authlib-0.61.0/Makefile.in 2008-07-12 21:41:08.000000000 +0200
+++ courier-authlib-0.61.0-cygwin/Makefile.in 2008-10-23 22:59:03.843750000 +0200
@@ -213,9 +213,10 @@
LTCOMPILE = $(LIBTOOL) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) \
--mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) \
$(AM_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS)
-CCLD = $(CC)
+CCLD = $(CC) -no-undefined
+CCLDEXE = $(CC)
LINK = $(LIBTOOL) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) \
- --mode=link $(CCLD) $(AM_CFLAGS) $(CFLAGS) $(AM_LDFLAGS) \
+ --mode=link $(CCLDEXE) $(AM_CFLAGS) $(CFLAGS) $(AM_LDFLAGS) \
$(LDFLAGS) -o $@
SOURCES = $(libauthcustom_la_SOURCES) $(libauthldap_la_SOURCES) \
$(libauthmysql_la_SOURCES) $(libauthpam_la_SOURCES) \
@@ -452,9 +453,9 @@
README.authdebug.html

DISTCLEANFILES = dbobj.config README_authlib.html
-commonlibdep = libcourierauthcommon.la
+commonlibdep = libcourierauthcommon.la libcourierauth.la
commonldflags = -module -rpath $(pkglibdir) -export-symbols-regex 'courier_auth.*_init' -avoid-version
-commonlibadd = libcourierauthcommon.la
+commonlibadd = libcourierauthcommon.la libcourierauth.la
libcourierauthcommon_t = @CRYPTLIBS@
libcourierauthcommon_la_SOURCES = \
auth.h courierauth.h \
--- courier-authlib-0.61.0/authdaemond.in 2005-07-05 14:25:08.000000000 +0200
+++ courier-authlib-0.61.0-cygwin/authdaemond.in 2008-10-25 12:03:32.140625000 +0200
@@ -15,4 +15,10 @@
set -a
. @authdaemonrc@

+# Some shared libraries (DLL) are installed in @libdir@/bin
+# instead of @libdir@/@PACKAGE@
+# Setting LD_LIBRARY_PATH at runtime or LR_RUN_PATH at linktime doesn't
+# work in cygwin, only setting PATH works.
+export PATH=$PATH:@libdir@/bin
+
exec ${sbindir}/courierlogger -pid=@authdaemonvar@/pid $LOGGEROPTS -$1 @libexecdir@/courier-authlib/authdaemond
</source>

Install the patch

$ patch -Np1 < courier-authlib-0.61.0-cygwin.patch

Configure the package without most authentication modules, keeping only ''userdb''.

'''Note''': replace ''mailuser'' by an existing user name (in your /etc/passwd file). I used my own username.

$ ./configure --disable-root-check --with-waitfunc=wait --without-authpam --without-authldap --without-authpwd \
--without-authshadow --without-authcustom --without-authpipe --without-authmysql --without-authpgsql --with-mailuser=mailuser \
--with-mailgroup=mkgroup-l-d

Take a long pause... and when ready execute the following command.

$ make

Take again a long pause... then check the result and install the libraries

$ make check
$ make install

'''NOTES''': list of related topics found during investigation for compilation
* authldap: configure: error: -lresolve is needed for res_query... http://www.cygwin.com/ml/cygwin/2005-04/msg00419.html

==Configuration==

Create authdaemonrc file: /usr/local/etc/authlib/authdaemonrc

$ cp /usr/local/etc/authlib/authdaemonrc.dist /usr/local/etc/authlib/authdaemonrc

Change number of daemons (I use three) and enable DEBUG_LOGIN

<source lang=text>
# The number of daemon processes that are started.
daemons=3

# DEBUG_LOGIN=2 - turn on debugging + log passwords too
DEBUG_LOGIN=2
</source>

See http://www.courier-mta.org/authlib/README.authdebug.html for details.

==IMAP accounts==

User accounts and settings are managed by ''userdb''. See [http://linux.die.net/man/8/makeuserdb makeuserdb] man page for details, or used the freshly new install man page - man makeuserdb

Assuming ''username'' is your username, 1001 and 10545 are respectively the UID and GID of an existing user (part of your /etc/passwd and /etc/group). I use my own UID and GID. Create the mailuser home directory, if required, and create Maildir folder.

$ mkdir -p /home/username
$ cd /home/username
$ /usr/lib/courier-imap/bin/maildirmake Maildir

Create the userdb text file /usr/local/etc/authlib/userdb

$ touch /usr/local/etc/authlib/userdb
$ chmod 700 /usr/local/etc/authlib/userdb

Edit /usr/local/etc/authlib/userdb, one line per account

<source lang=text>
username[TAB]uid=1001|gid=10111|home=/home/username|mail=/home/username/Maildir|systempw=XXXXXX
</source>

where XXXXXX is the encrypted password created with

$ /usr/local/sbin/userdbpw.exe

Create the (binary) userdb

$ /usr/local/sbin/makeuserdb

You should now have the following files created:

* /usr/local/etc/authlib/userdb.dat
* /usr/local/etc/authlib/userdbshadow.dat

==Running==

Install and configure syslogd. Run the [http://www.cygwin.com/setup.exe Cygwin setup] program. Locate, select and install inetutils (contains syslogd). Configure syslogd with the following command. It will create the /etc/syslogd.conf file and install a windows service using cygrunsrv

$ syslogd-config
$ net start syslogd

Start the authlib daemon with the following command:

$ /usr/local/sbin/authdaemond start

Stop the daemon with the following command:

$ /usr/local/sbin/authdaemond stop

You'll find log messages in /var/log/messages (default target file as specified in /etc/syslogd.conf)

==Testing authlib==

$ /usr/local/sbin/authtest username CCCCCC
Authentication succeeded.

Authenticated: username (uid 1001, gid 10545)
Home Directory: /home/username
Maildir: /home/username/Maildir
Quota: (none)
Encrypted Password: XXXXXX
Cleartext Password: CCCCCC
Options: (none)

where username is your user name and XXXXXX is your encrypted password as provided in ''/usr/local/etc/authlib/userdb''. CCCCCC is your password in clear.

==Troubleshooting==

Should you have a doubt on the execution of a command (ie: no output) you can check if the application is able to load required DLL. This error is not reported. Using ''strace'', windows will complain about DLL not found, if any. To solve this problem you may need to add some path in your PATH. For example, in order to test authlib with authtest:

$ strace /usr/local/sbin/authtest
# This will open a dialog box to complain about DLL.
$ export PATH=$PATH:/usr/local/lib/bin
$ /usr/local/sbin/authtest ausername

==Courier Authentication Config==

$ /usr/local/bin/courierauthconfig.exe --version
$ /usr/local/bin/courierauthconfig.exe --ldflags
$ /usr/local/bin/courierauthconfig.exe --cppflags
$ /usr/local/bin/courierauthconfig.exe --configfiles

=Courier-IMAP=
==Compilation and installation==

Download and untar package [http://www.courier-mta.org/download.php#imap courier-imap] in your favorite sandbox.
Version used: [http://prdownloads.sourceforge.net/courier/courier-imap-4.4.1.tar.bz2 4.4.1.20080920]

Apply the [{{#file:courier-imap-4.4.1-cygwin.patch}} patch] before configuration to correct following issues:

# incorrect usage of EXEEXT in ''makedat/Makefile''
# incorrect usage of EXTEXT in main ''Makefile''
# remove usage of /usr/lib/env in start/stop script (/usr/lib/env does not seem to work under Cygwin)

<source lang=diff>
--- courier-imap-4.4.1.20080920/makedat/Makefile.in 2008-08-24 19:52:51.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/makedat/Makefile.in 2008-10-21 20:43:27.500000000 +0200
@@ -182,7 +182,7 @@
libexecdir = @libexecdir@
localedir = @localedir@
localstatedir = @localstatedir@
-makedatprog_target = @makedatprog_target@
+makedatprog_target = @makedatprog_target@$(EXEEXT)
makedatprogpath = @makedatprogpath@
mandir = @mandir@
mkdir_p = @mkdir_p@
@@ -198,7 +198,7 @@
target_alias = @target_alias@
top_builddir = @top_builddir@
top_srcdir = @top_srcdir@
-noinst_PROGRAMS = @makedatprog_target@
+noinst_PROGRAMS = @makedatprog_target@$(EXEEXT)
makedatprog_SOURCES = makedatprog.c
makedatprog_DEPENDENCIES = @dblibrary@
makedatprog_LDADD = @dblibrary@
--- courier-imap-4.4.1.20080920/Makefile.in 2008-09-20 14:48:46.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/Makefile.in 2008-10-21 20:59:45.156250000 +0200
@@ -247,9 +247,9 @@
CLEANFILES = $(databin_SCRIPTS) $(man_MANS) $(sysconf_DATA) $(sbin_SCRIPTS)
databindir = $(datadir)
databin_SCRIPTS = mkimapdcert mkpop3dcert
-binPROGRAMS = imapd pop3d maildirmake maildiracl deliverquota maildirkw
-sbinPROGRAMS = imaplogin pop3login
-libexecPROGRAMS = makedatprog couriertcpd
+binPROGRAMS = imapd$(EXEEXT) pop3d$(EXEEXT) maildirmake$(EXEEXT) maildiracl$(EXEEXT) deliverquota$(EXEEXT) maildirkw$(EXEEXT)
+sbinPROGRAMS = imaplogin$(EXEEXT) pop3login$(EXEEXT)
+libexecPROGRAMS = makedatprog$(EXEEXT) couriertcpd$(EXEEXT)
bin_PROGRAMS = @binPROGRAMS_exec@
sbin_PROGRAMS = @sbinPROGRAMS_exec@
libexec_PROGRAMS = @libexecPROGRAMS_exec@
--- courier-imap-4.4.1.20080920/imapd.rc.in 2008-05-04 15:12:47.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/imapd.rc.in 2008-10-26 10:17:51.359375000 +0100
@@ -22,6 +22,15 @@
exit 1
fi

+# Location for some shared libraries (cygcourierauth*.dll) from authlib
+AUTHLIBDIR=/usr/local
+if test ! -f $AUTHLIBDIR/lib/bin/cygcourierauth.dll
+then
+ echo "$AUTHLIBDIR/lib/bin/cygcourierauth.dll not found."
+ exit 1
+fi
+export PATH=$PATH:$AUTHLIBDIR/lib/bin
+
TLS_CACHEFILE=""
. @sysconfdir@/imapd-ssl
. @sysconfdir@/imapd
@@ -35,7 +44,7 @@

umask $IMAP_UMASK
@ULIMIT@ $IMAP_ULIMITD
- @SETENV@ -i @SHELL@ -c " set -a ;
+ @SHELL@ -c " set -a ;
prefix=@prefix@ ;
exec_prefix=@exec_prefix@ ;
bindir=@bindir@ ;
--- courier-imap-4.4.1.20080920/pop3d.rc.in 2008-05-04 15:12:47.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/pop3d.rc.in 2008-10-26 10:18:02.187500000 +0100
@@ -22,12 +22,21 @@
exit 1
fi

+# Location for some shared libraries (cygcourierauth*.dll) from authlib
+AUTHLIBDIR=/usr/local
+if test ! -f $AUTHLIBDIR/lib/bin/cygcourierauth.dll
+then
+ echo "$AUTHLIBDIR/lib/bin/cygcourierauth.dll not found."
+ exit 1
+fi
+export PATH=$PATH:$AUTHLIBDIR/lib/bin
+
. @sysconfdir@/pop3d-ssl
. @sysconfdir@/pop3d

case $1 in
start)
- @SETENV@ -i @SHELL@ -c " set -a ;
+ @SHELL@ -c " set -a ;
prefix=@prefix@ ;
exec_prefix=@exec_prefix@ ;
bindir=@bindir@ ;
</source>

Install the patch

$ patch -Np1 < courier-imap-4.4.1-cygwin.patch

Configure the package

$ ./configure --disable-root-check --with-waitfunc=wait

Take a walk... and when ready execute the following command to build the package.

$ make

Time to go for some coffee... and then install the package with the usual

$ make install
$ make install-configure

==Configuration==

Setup courier-imap: $prefix/etc/imapd (where $prefix=/usr/lib/courier-imap)

Set ADDRESS to a valid value (0 or 127.0.0.1) and change IMAP_ULIMITD to a value accepted by the command ulimit. I simply used the existing value returned by

$ ulimit -v

==Running==

Syslogd service and authdaemond should be started. See above.

Start the imap daemon with the following command:

$ /usr/lib/courier-imap/libexec/imapd.rc start

Stop the daemon with the following command:

$ /usr/lib/courier-imap/libexec/imapd.rc stop

You'll find log messages in /var/log/messages (default target file as specified in /etc/syslogd.conf)

==Testing==
When authdaemond and imapd are running you can do a simple login test:

$ telnet localhost 143

and type the login command

a login <username> <password>

the server should reply

a OK LOGIN Ok.

to which you can reply with a logout

a logout

the session is terminated by the server

* BYE Courier-IMAP server shutting down
a OK LOGOUT completed
Connection closed by foreign host.

'''Hint:''' if the connection is immediately closed by foreign host it probably means that ''imapd'' does not have access to all required DLLs. Check your PATH.

==All in one==
If everything works fine you can use the following [{{#file:imapd}} startup script] to launch or stop imap and authlib daemons. It can be saved in /usr/local/sbin.

<source lang=bash>
#! /bin/sh
#
# Courier imap daemon startup script.

authlib=/usr/local/sbin/authdaemond
imapd=/usr/lib/courier-imap/libexec/imapd.rc

test ! -f $authlib && echo "File not found: $authlib" && exit 1;
test ! -f $imapd && echo "File not found: $imapd" && exit 1;

case $1 in
start)
$authlib start
$imapd start
;;
stop)
$imapd stop
$authlib stop
;;
*)
echo "Usage: $0 <start|stop>"
;;
esac
</source>

==Cygwin service==
Create the [{{#file:imapd-srv.sh}} service script]

<source lang=bash>
#!/bin/sh
# File: imapd-service.sh
# Purpose: Courier imap daemon service script.
#
# NOTE: This script must be excutable by SYSTEM
username=username # <<<< Change me !!!
pidsleep=''

# Function to handle QUIT signal.
# Stop imapd daemon and kill background sleep.
function handleQuit
{
echo "STOPPING: /usr/local/sbin/imapd"
su $username -c "/usr/local/sbin/imapd stop"
test -n "$pidsleep" && kill -9 $pidsleep
exit
}

# Interruptible sleep
# Note: sleep is an external command (not builtin command), hence
# not interruptible by the QUIT signal.
function intsleep
{
sleep $1 &
pidsleep=$!
wait $pidsleep
}

echo "STARTING: /usr/local/sbin/imapd as $username at "`date`
su $username -c "/usr/local/sbin/imapd start"

trap "handleQuit" SIGQUIT

echo "WAITING: "$$
while true; do intsleep 1d; done
</source>

Change username and save it to your favorite location, says /usr/local/sbin. The service script must be executable by SYSTEM, ie: the user that run the windows services.

$ chmod +rx /usr/local/sbin/imapd-service.sh

Install the new service with the following command.

$ cygrunsrv --install imapd --desc "Courier IMAP daemon" --disp "CYGWIN imapd" \
--path /usr/local/sbin/imapd-service.sh --termsig QUIT --type auto --shutdown

Finally, just start the service

$ cygrunsrv --start imapd
or
$ net start imapd

You should now have couple of processes related to courier-imap

$ ps -a

'''NOTES''':
* If the service could not be started, it's probably because the service script is not executable by SYSTEM

$ net start imapd
The CYGWIN imapd service is starting.
The CYGWIN imapd service could not be started.

The service did not report an error.

More help is available by typing NET HELPMSG 3534.

* The combination of background sleep and builtin wait command is used to make the waiting statment interruptible. The original script found on the net, was simply using a 'sleep 1' which turned to consume ~10% of the CPU.

=Maildir=
'''maildirmake''' is part of the courier-imap package

$ /usr/lib/courier-imap/bin/maildirmake

=Mairix - index and search mail folders=
Run the [http://www.cygwin.com/setup.exe Cygwin setup] program. Locate, select and install the package mairix.

Create the configuration file in your home directory: ~/.mairixrc

base=/home/username
maildir=Maildir...
omit=Maildir/.Mairix**
mfolder=Maildir/.Mairix
database=/home/username/.mairix_database

The create the mairix index

$ mairix
and
$ mairix --fast-index # on daily base

See [[Mail_Tips]] for additional details.

=Offlineimap=
Requires python, so run the [http://www.cygwin.com/setup.exe Cygwin setup] program. Locate, select and install the package python.

Download offlineimap [http://software.complete.org/software/projects/list_files/offlineimap here]. I use [http://software.complete.org/software/versions/download/242?attachment_id=334 version 6.0.3]. Extract files in you favorite sandbox.

$ tar xvfz offlineimap_6.0.3.tar.gz
$ cd offlineimap

Apply the following [{{#file:offlineimap_6.0.3-cygwin.patch}} patch] to correct the following issues:

* [http://en.wikipedia.org/wiki/Maildir#Windows_software Suffix separator] character colon (:) is not allowed in Windows. Courier-imap for cygwin is using (!)
* While used by courier-imap for cygwin Maildir mail files should not use CRLF, rather only LF. The issue appears only in mail with attachment, that is the raw message is not interpreted and displayed as it in the mail body (under Thunderbird).

<source lang=diff>
--- offlineimap.conf 2008-08-13 20:56:04.000000000 +0200
+++ offlineimap-cygwin.conf 2008-11-06 10:44:31.066750700 +0100
@@ -215,6 +215,19 @@

restoreatime = no

+# Cygwin/Windows users may have issues with the Maildir mail filename
+# containing the suffix separator character (:). In this case, they
+# could change the suffix separator character to whatever required.
+# For example, Courier-IMAP for Cygwin is using (!)
+
+# suffixsep = !
+
+# Cygwin/Windows users may need to create Maildir mail filename in
+# binary mode. In this case, set the 'filemode' to 'binary'. Default is
+# 'text'. Courier-IMAP for Cygwin works better with binary mode.
+
+# filemode = binary
+
[Repository RemoteExample]

# And this is the remote repository. We only support IMAP or Gmail here.
--- offlineimap/folder/Maildir.py 2008-08-13 20:55:48.000000000 +0200
+++ offlineimap/folder/Maildir-cygwin.py 2008-11-06 10:44:05.344202700 +0100
@@ -23,7 +23,6 @@
import os.path, os, re, time, socket, md5

uidmatchre = re.compile(',U=(\d+)')
-flagmatchre = re.compile(':.*2,([A-Z]+)')

timeseq = 0
lasttime = long(0)
@@ -54,6 +53,12 @@
self.messagelist = None
self.repository = repository
self.accountname = accountname
+ self.filemode = {
+ 'binary': 'wb',
+ 'text': 'wt',
+ }[repository.getconf("filemode", 'text')]
+ self.suffixsep = repository.getconf("suffixsep",':')
+ self.flagmatchre = re.compile('%s.*2,([A-Z]+)' % self.suffixsep)
BaseFolder.__init__(self)

def getaccountname(self):
@@ -102,7 +107,7 @@
nouidcounter -= 1
else:
uid = long(uidmatch.group(1))
- flagmatch = flagmatchre.search(messagename)
+ flagmatch = self.flagmatchre.search(messagename)
flags = []
if flagmatch:
flags = [x for x in flagmatch.group(1)]
@@ -180,7 +185,7 @@
break
tmpmessagename = messagename.split(',')[0]
ui.debug('maildir', 'savemessage: using temporary name %s' % tmpmessagename)
- file = open(os.path.join(tmpdir, tmpmessagename), "wt")
+ file = open(os.path.join(tmpdir, tmpmessagename), self.filemode)
file.write(content)

# Make sure the data hits the disk
@@ -226,11 +231,11 @@
newpath = os.path.join(self.getfullname(), 'cur')
else:
newpath = os.path.join(self.getfullname(), 'new')
- infostr = ':'
- infomatch = re.search('(:.*)$', newname)
+ infostr = self.suffixsep
+ infomatch = re.search('(%s.*)$' % self.suffixsep, newname)
if infomatch: # If the info string is present..
infostr = infomatch.group(1)
- newname = newname.split(':')[0] # Strip off the info string.
+ newname = newname.split(self.suffixsep)[0] # Strip off the info string.
infostr = re.sub('2,[A-Z]*', '', infostr)
flags.sort()
infostr += '2,' + ''.join(flags)
</source>

$ patch -Np0 < offlineimap_6.0.3-cygwin.patch

Install offlineimap

$ python setup.py install

Configure offlineimap as described in [[Offlineimap]] and launch offlineimap. '''CAUTION:''' Add the NEW suffixsep and filemode options in the Maildir repository section of the configuration file.

$ offlineimap
or
$ offlineimap -1 -o -u Noninteractive.Basic
or
$ offlineimap -c /home/username/offlineimap.conf

Here is a simple [{{#file:offlineimap.conf}} offlineimap configuration file].

<source lang=text>
[general]
metadata = /home/username/.offlineimap
accounts = Test
maxsyncaccounts = 1
ui = Curses.Blinkenlights, TTY.TTYUI,
Noninteractive.Basic, Noninteractive.Quiet
ignore-readonly = no

[mbnames]
enabled = no

[ui.Curses.Blinkenlights]
statuschar = .

[Account Test]
localrepository = LocalExample
remoterepository = RemoteExample

[Repository LocalExample]
type = Maildir
localfolders = /home/username/Maildir
sep = .
restoreatime = no

suffixsep = ! # <<< NEW
filemode = binary # <<< NEW

[Repository RemoteExample]
type = IMAP
remotehost = imapserver
ssl = no
remoteport = 143
remoteuser = username
remotepass = XXXXXX
maxconnections = 1
holdconnectionopen = no
nametrans = lambda foldername: re.sub('^INBOX\.*', '.', foldername)
folderfilter = lambda foldername: not re.search('(SPAM$|Mairix)', foldername)
</source>

Courier-Cygwin

2008-11-07T10:40:45Z

Fuujuhi: /* Installing as a Windows Service */

=Prerequisites=

Courier-IMAP requires the installation of the [http://www.courier-mta.org/authlib/ Courier Authentication Library] and the following cygwin tools and packages

* <tt>patch</tt>, <tt>tar</tt>, <tt>make</tt>, <tt>gcc</tt>
* <tt>crypt</tt>
* <tt>libgdbm-devel</tt>
* <tt>libtool</tt>
* <tt>inetutils</tt>
* <tt>cygrunsrv</tt>

Run the [http://www.cygwin.com/setup.exe Cygwin setup] program. Locate, select and install the required packages.

=Building and Installing Courier Authentication Library=

Installation of ''Courier Authentication Library'' using the module '''userdb''' to manage mail accounts. All other modules are disabled.

Download and untar [http://www.courier-mta.org/download.php#authlib courier-authlib] in your favorite sandbox.
Version used: [http://prdownloads.sourceforge.net/courier/courier-authlib-0.61.0.tar.bz2 0.61.0]

<source lang="bash">
$ tar -xvjf courier-authlib-0.61.0.tar.bz2
$ cd courier-authlib-0.61.0
</source>

Apply the [{{#file:courier-authlib-0.61.0-cygwin.patch}} patch] before configuration to correct following issues:

# incorrect usage of EXEEXT in ''makedat/Makefile''
# add libtool flag (-no-undefined) required for DLL creation ([http://lists.cairographics.org/archives/cairo/2004-April/001125.html])
# add missing dependencies
# add PATH in start script to access new DLL (installed in a different directory)

<source lang=diff>
--- courier-authlib-0.61.0/makedat/Makefile.in 2008-05-24 16:21:09.000000000 +0200
+++ courier-authlib-0.61.0-cygwin/makedat/Makefile.in 2008-10-21 16:02:38.709166700 +0200
@@ -182,7 +182,7 @@
libexecdir = @libexecdir@
localedir = @localedir@
localstatedir = @localstatedir@
-makedatprog_target = @makedatprog_target@
+makedatprog_target = @makedatprog_target@$(EXEEXT)
makedatprogpath = @makedatprogpath@
mandir = @mandir@
mkdir_p = @mkdir_p@
@@ -198,7 +198,7 @@
target_alias = @target_alias@
top_builddir = @top_builddir@
top_srcdir = @top_srcdir@
-noinst_PROGRAMS = @makedatprog_target@
+noinst_PROGRAMS = @makedatprog_target@$(EXEEXT)
makedatprog_SOURCES = makedatprog.c
makedatprog_DEPENDENCIES = @dblibrary@
makedatprog_LDADD = @dblibrary@
--- courier-authlib-0.61.0/Makefile.in 2008-07-12 21:41:08.000000000 +0200
+++ courier-authlib-0.61.0-cygwin/Makefile.in 2008-10-23 22:59:03.843750000 +0200
@@ -213,9 +213,10 @@
LTCOMPILE = $(LIBTOOL) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) \
--mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) \
$(AM_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS)
-CCLD = $(CC)
+CCLD = $(CC) -no-undefined
+CCLDEXE = $(CC)
LINK = $(LIBTOOL) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) \
- --mode=link $(CCLD) $(AM_CFLAGS) $(CFLAGS) $(AM_LDFLAGS) \
+ --mode=link $(CCLDEXE) $(AM_CFLAGS) $(CFLAGS) $(AM_LDFLAGS) \
$(LDFLAGS) -o $@
SOURCES = $(libauthcustom_la_SOURCES) $(libauthldap_la_SOURCES) \
$(libauthmysql_la_SOURCES) $(libauthpam_la_SOURCES) \
@@ -452,9 +453,9 @@
README.authdebug.html

DISTCLEANFILES = dbobj.config README_authlib.html
-commonlibdep = libcourierauthcommon.la
+commonlibdep = libcourierauthcommon.la libcourierauth.la
commonldflags = -module -rpath $(pkglibdir) -export-symbols-regex 'courier_auth.*_init' -avoid-version
-commonlibadd = libcourierauthcommon.la
+commonlibadd = libcourierauthcommon.la libcourierauth.la
libcourierauthcommon_t = @CRYPTLIBS@
libcourierauthcommon_la_SOURCES = \
auth.h courierauth.h \
--- courier-authlib-0.61.0/authdaemond.in 2005-07-05 14:25:08.000000000 +0200
+++ courier-authlib-0.61.0-cygwin/authdaemond.in 2008-10-25 12:03:32.140625000 +0200
@@ -15,4 +15,10 @@
set -a
. @authdaemonrc@

+# Some shared libraries (DLL) are installed in @libdir@/bin
+# instead of @libdir@/@PACKAGE@
+# Setting LD_LIBRARY_PATH at runtime or LR_RUN_PATH at linktime doesn't
+# work in cygwin, only setting PATH works.
+export PATH=$PATH:@libdir@/bin
+
exec ${sbindir}/courierlogger -pid=@authdaemonvar@/pid $LOGGEROPTS -$1 @libexecdir@/courier-authlib/authdaemond
</source>

Install the patch

<source lang="bash">
$ patch -Np1 < courier-authlib-0.61.0-cygwin.patch
</source>

Configure the package without most authentication modules, keeping only '''userdb'''. Replace ''mailuser'' by an existing user name (in your /etc/passwd file). I used my own username.

<source lang="bash">
$ ./configure --disable-root-check --with-waitfunc=wait --without-authpam --without-authldap --without-authpwd \
--without-authshadow --without-authcustom --without-authpipe --without-authmysql --without-authpgsql --with-mailuser=mailuser \
--with-mailgroup=mkgroup-l-d
</source>

Take a long pause... and when ready build, check the result and install the libraries.

<source lang="bash">
$ make
$ make check
$ make install
</source>

'''NOTES''': list of related topics found during investigation for compilation
* authldap: configure: error: -lresolve is needed for res_query... http://www.cygwin.com/ml/cygwin/2005-04/msg00419.html

=Building and Installing Courier-IMAP=

Download and untar package [http://www.courier-mta.org/download.php#imap courier-imap] in your favorite sandbox.
Version used: [http://prdownloads.sourceforge.net/courier/courier-imap-4.4.1.tar.bz2 4.4.1.20080920]

<source lang="bash">
$ tar -xvjf courier-imap-4.4.1.tar.bz2
$ cd courier-imap-4.4.1
</source>

Apply the [{{#file:courier-imap-4.4.1-cygwin.patch}} patch] before configuration to correct following issues:

# incorrect usage of EXEEXT in ''makedat/Makefile''
# incorrect usage of EXTEXT in main ''Makefile''
# remove usage of /usr/lib/env in start/stop script (/usr/lib/env does not seem to work under Cygwin)

<source lang=diff>
--- courier-imap-4.4.1.20080920/makedat/Makefile.in 2008-08-24 19:52:51.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/makedat/Makefile.in 2008-10-21 20:43:27.500000000 +0200
@@ -182,7 +182,7 @@
libexecdir = @libexecdir@
localedir = @localedir@
localstatedir = @localstatedir@
-makedatprog_target = @makedatprog_target@
+makedatprog_target = @makedatprog_target@$(EXEEXT)
makedatprogpath = @makedatprogpath@
mandir = @mandir@
mkdir_p = @mkdir_p@
@@ -198,7 +198,7 @@
target_alias = @target_alias@
top_builddir = @top_builddir@
top_srcdir = @top_srcdir@
-noinst_PROGRAMS = @makedatprog_target@
+noinst_PROGRAMS = @makedatprog_target@$(EXEEXT)
makedatprog_SOURCES = makedatprog.c
makedatprog_DEPENDENCIES = @dblibrary@
makedatprog_LDADD = @dblibrary@
--- courier-imap-4.4.1.20080920/Makefile.in 2008-09-20 14:48:46.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/Makefile.in 2008-10-21 20:59:45.156250000 +0200
@@ -247,9 +247,9 @@
CLEANFILES = $(databin_SCRIPTS) $(man_MANS) $(sysconf_DATA) $(sbin_SCRIPTS)
databindir = $(datadir)
databin_SCRIPTS = mkimapdcert mkpop3dcert
-binPROGRAMS = imapd pop3d maildirmake maildiracl deliverquota maildirkw
-sbinPROGRAMS = imaplogin pop3login
-libexecPROGRAMS = makedatprog couriertcpd
+binPROGRAMS = imapd$(EXEEXT) pop3d$(EXEEXT) maildirmake$(EXEEXT) maildiracl$(EXEEXT) deliverquota$(EXEEXT) maildirkw$(EXEEXT)
+sbinPROGRAMS = imaplogin$(EXEEXT) pop3login$(EXEEXT)
+libexecPROGRAMS = makedatprog$(EXEEXT) couriertcpd$(EXEEXT)
bin_PROGRAMS = @binPROGRAMS_exec@
sbin_PROGRAMS = @sbinPROGRAMS_exec@
libexec_PROGRAMS = @libexecPROGRAMS_exec@
--- courier-imap-4.4.1.20080920/imapd.rc.in 2008-05-04 15:12:47.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/imapd.rc.in 2008-10-26 10:17:51.359375000 +0100
@@ -22,6 +22,15 @@
exit 1
fi

+# Location for some shared libraries (cygcourierauth*.dll) from authlib
+AUTHLIBDIR=/usr/local
+if test ! -f $AUTHLIBDIR/lib/bin/cygcourierauth.dll
+then
+ echo "$AUTHLIBDIR/lib/bin/cygcourierauth.dll not found."
+ exit 1
+fi
+export PATH=$PATH:$AUTHLIBDIR/lib/bin
+
TLS_CACHEFILE=""
. @sysconfdir@/imapd-ssl
. @sysconfdir@/imapd
@@ -35,7 +44,7 @@

umask $IMAP_UMASK
@ULIMIT@ $IMAP_ULIMITD
- @SETENV@ -i @SHELL@ -c " set -a ;
+ @SHELL@ -c " set -a ;
prefix=@prefix@ ;
exec_prefix=@exec_prefix@ ;
bindir=@bindir@ ;
--- courier-imap-4.4.1.20080920/pop3d.rc.in 2008-05-04 15:12:47.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/pop3d.rc.in 2008-10-26 10:18:02.187500000 +0100
@@ -22,12 +22,21 @@
exit 1
fi

+# Location for some shared libraries (cygcourierauth*.dll) from authlib
+AUTHLIBDIR=/usr/local
+if test ! -f $AUTHLIBDIR/lib/bin/cygcourierauth.dll
+then
+ echo "$AUTHLIBDIR/lib/bin/cygcourierauth.dll not found."
+ exit 1
+fi
+export PATH=$PATH:$AUTHLIBDIR/lib/bin
+
. @sysconfdir@/pop3d-ssl
. @sysconfdir@/pop3d

case $1 in
start)
- @SETENV@ -i @SHELL@ -c " set -a ;
+ @SHELL@ -c " set -a ;
prefix=@prefix@ ;
exec_prefix=@exec_prefix@ ;
bindir=@bindir@ ;
</source>

Install the patch

<source lang="bash">
$ patch -Np1 < courier-imap-4.4.1-cygwin.patch
</source>

Configure the package

<source lang="bash">
$ ./configure --disable-root-check --with-waitfunc=wait
</source>

Take a walk or some coffee... and when ready build the package and install it.

<source lang="bash">
$ make
$ make install
$ make install-configure
</source>

=Configuring Courier=

== Configuration Files and Scripts ==

If not present yet, add <tt>/usr/local/bin</tt> and <tt>/usr/local/sbin</tt> to your path (ideally in <tt>/etc/bash.bashrc</tt>):

<source lang="bash">
PATH=/usr/local/bin:/usr/local/sbin:$PATH
</source>

Create configuration file for '''authdaemond''' (<tt>/usr/local/etc/authlib/authdaemonrc</tt>)

$ cp /usr/local/etc/authlib/authdaemonrc.dist /usr/local/etc/authlib/authdaemonrc

And edit its content to change number of daemons (I use three):

<source lang=text>
# The number of daemon processes that are started.
daemons=3
</source>

Edit the file <tt>/usr/lib/courier-imap/etc/imapd</tt>, taking care to adapt '''IMAP_ULIMITD''' accordingly:
<source lang="bash">
ADDRESS=127.0.0.1 #Typ. use 0 or 127.0.0.1
...
IMAP_ULIMITD=2097152 #Set same value as returned by 'ulimit -v' or imapd.rc will complain
</source>

Create link to authlib and imap daemon in <tt>/etc/rc.d/init.d</tt>
<source lang="bash">
ln -s /usr/local/sbin/authdaemond /etc/rc.d/init.d/courier-authdaemon
ln -s /usr/lib/courier-imap/libexec/imapd.rc /etc/rc.d/init.d/courier-imap
</source>

== IMAP Accounts ==

User accounts and settings are managed by ''userdb''. <tt>[http://linux.die.net/man/8/makeuserdb man makeuserdb]</tt> for details.

First create user mailbox (replace ''username'' with user account name) with <tt>maildirmake</tt>. Repeat for all users:
<source lang="bash">
mkdir -p /home/username #if don't exist yet...
cd /home/username
for i in "" .Drafts .Sent .Trash .Templates; do /usr/lib/courier-imap/bin/maildirmake ~username/MailDir/$i; done
# Create also default Maildir for new users
for i in "" .Drafts .Sent .Trash .Templates; do /usr/lib/courier-imap/bin/maildirmake /etc/skel/MailDir/$i; done
</source>

Next create the user authentication database <tt>/usr/local/etc/authlib/userdb</tt>:
<source lang="bash">
touch /usr/local/etc/authlib/userdb
chmod 700 /usr/local/etc/authlib/userdb
#Repeat for each username:
pw2userdb | grep "^username">>/usr/local/etc/authlib/userdb # Fill userdb from /etc/passwd user entry
userdb username set mail=~username/MailDir # Specify user MailDir location
userdbpw | userdb username set systempw # Set user account password
...
makeuserdb # Create binary database
</source>

You should now have the following files created:

* /usr/local/etc/authlib/userdb.dat
* /usr/local/etc/authlib/userdbshadow.dat

==Configuration Summary==

$ /usr/local/bin/courierauthconfig.exe --version
$ /usr/local/bin/courierauthconfig.exe --ldflags
$ /usr/local/bin/courierauthconfig.exe --cppflags
$ /usr/local/bin/courierauthconfig.exe --configfiles

=Running=

Start the daemons with the following commands:

<source lang="bash">
/etc/rc.d/init.d/courier-authdaemon start
/etc/rc.d/init.d/courier-imap start
</source>

Stop the daemons with the following commands:

<source lang="bash">
/etc/rc.d/init.d/courier-imap stop
/etc/rc.d/init.d/courier-authdaemon stop
</source>

=Testing=

If not done yet, install and configure '''syslogd'''. Run the [http://www.cygwin.com/setup.exe Cygwin setup] program and install '''inetutils''' package (contains '''syslogd'''). Configure '''syslogd''' with the following command, which will create the file <tt>/etc/syslogd.conf</tt> file and install a windows service using '''cygrunsrv''':

<source lang="bash">
$ syslogd-config #if not configured yet
$ net start syslogd #if not started yet
</source>

Edit '''authdaemond''' configuration file <tt>/usr/local/etc/authlib/authdaemonrc</tt> to enable full debug messages:

<source lang="text">
# DEBUG_LOGIN=2 - turn on debugging + log passwords too
DEBUG_LOGIN=2
</source>

and restart '''courier-authdaemon''' to take changes into effect:

<source lang="bash">
$ /etc/rc.d/init.d/courier-authdaemon stop
$ /etc/rc.d/init.d/courier-authdaemon start
</source>

Testing ''Courier Authentication Library'' with '''authtest''':

<source lang="bash">
$ export PATH=$PATH:/usr/local/lib/bin #because authtest needs a .DLL library there (found with strace)
$ /usr/local/sbin/authtest username CCCCCC
</source>

This should produce something similar to the transcript below (where <tt>CCCCCC</tt> is your cleartext password, and <tt>XXXXXX</tt> is your encrypted password as provided in <tt>/usr/local/etc/authlib/userdb</tt>):
<source lang="text">
Authentication succeeded.

Authenticated: username (uid 1001, gid 10545)
Home Directory: /home/username
Maildir: /home/username/Maildir
Quota: (none)
Encrypted Password: XXXXXX
Cleartext Password: CCCCCC
Options: (none)
</source>

Check that everything is fine in <tt>/var/log/messages</tt> (see [http://www.courier-mta.org/authlib/README.authdebug.html] for details):

<source lang="bash">
$ cat /var/log/messages
</source>

Testing '''Courier-IMAP''' through '''telnet''':

<source lang="bash">
$ telnet localhost 143
</source

In the ''telnet'' session, type the following test scripts (replace ''username'' and ''password'' accordingly):

<source lang="text">
001 LOGIN username password
002 EXAMINE INBOX
003 LOGOUT
</source>

Server must reply with '''OK''' messages along with some more information (see also [http://www.courier-mta.org/authlib/README.authdebug.html] for details).

'''Hint:''' if the connection is immediately closed by foreign host it probably means that ''imapd'' does not have access to all required DLLs. Check your PATH.

When testing is done, don't forget to reset debugging in <tt>/usr/local/etc/authlib/authdaemonrc</tt>:
<pre>
# DEBUG_LOGIN=2 - turn on debugging + log passwords too
DEBUG_LOGIN=0
</pre>

==Troubleshooting==

Should you have a doubt on the execution of a command (ie: no output) you can check if the application is able to load required DLL. This error is not reported. Using '''strace''', windows will complain about DLL not found, if any. To solve this problem you may need to add some path in your PATH. For example, in order to test authlib with authtest:

<source lang="bash">
$ strace /usr/local/sbin/authtest # This will open a dialog box to complain about DLL.
$ export PATH=$PATH:/usr/local/lib/bin
$ /usr/local/sbin/authtest ausername
</source>

=Installing as a Windows Service=

If everything works fine you can install ''Courier'' as a Windows Service. Create the [{{#file:imapd-srv.sh}} service script]

<source lang=bash>
#!/bin/sh
# File: imapd-service.sh
# Purpose: Courier imap daemon service script.
#
# NOTE: This script must be excutable by SYSTEM
username=username # <<<< Change me !!!
pidsleep=''

# Function to handle QUIT signal.
# Stop imapd daemon and kill background sleep.
function handleQuit
{
echo "STOPPING: /etc/rc.d/init.d/courier-authdaemon"
su $username -c "/etc/rc.d/init.d/courier-authdaemon stop"
echo "STOPPING: /etc/rc.d/init.d/courier-imap"
su $username -c "/etc/rc.d/init.d/courier-imap stop"
test -n "$pidsleep" && kill -9 $pidsleep
exit
}

# Interruptible sleep
# Note: sleep is an external command (not builtin command), hence
# not interruptible by the QUIT signal.
function intsleep
{
sleep $1 &
pidsleep=$!
wait $pidsleep
}

echo "STARTING: /etc/rc.d/init.d/courier-authdaemon as $username at "`date`
su $username -c "/etc/rc.d/init.d/courier-authdaemon start"
echo "STARTING: /etc/rc.d/init.d/courier-imap as $username at "`date`
su $username -c "/etc/rc.d/init.d/courier-imap start"

trap "handleQuit" SIGQUIT

echo "WAITING: "$$
while true; do intsleep 1d; done
</source>

Change ''username'' and save it to your favorite location, say <tt>/usr/local/sbin</tt>. The service script must be executable by ''SYSTEM'', ie: the user that run the windows services.

<source lang="bash">
$ chmod +rx /usr/local/sbin/imapd-service.sh
</source>

Install the new service with the following command.

<source lang="bash">
$ cygrunsrv --install imapd --desc "Courier IMAP daemon" --disp "CYGWIN imapd" \
--path /usr/local/sbin/imapd-service.sh --termsig QUIT --type auto --shutdown
</source>

Finally, just start the service

<source lang="bash">
$ cygrunsrv --start imapd
# or
$ net start imapd
</source>

You should now have couple of processes related to courier-imap

<source lang="bash">
$ ps -a
</source>

'''NOTES''':
* If the service could not be started, it's probably because the service script is not executable by SYSTEM

<source lang="bash">
$ net start imapd
The CYGWIN imapd service is starting.
The CYGWIN imapd service could not be started.

The service did not report an error.

More help is available by typing NET HELPMSG 3534.
</source>

* The combination of background sleep and builtin wait command is used to make the waiting statment interruptible. The original script found on the net, was simply using a 'sleep 1' which turned to consume ~10% of the CPU.

=Other Interesting Packages=

==Maildir==

'''maildirmake''' is part of the courier-imap package

$ /usr/lib/courier-imap/bin/maildirmake

==Mairix - index and search mail folders==
Run the [http://www.cygwin.com/setup.exe Cygwin setup] program. Locate, select and install the package mairix.

Create the configuration file in your home directory: ~/.mairixrc

base=/home/username
maildir=Maildir...
omit=Maildir/.Mairix**
mfolder=Maildir/.Mairix
database=/home/username/.mairix_database

The create the mairix index

$ mairix
and
$ mairix --fast-index # on daily base

See [[Mail_Tips]] for additional details.

==Offlineimap==
Requires python, so run the [http://www.cygwin.com/setup.exe Cygwin setup] program. Locate, select and install the package python.

Download offlineimap [http://software.complete.org/software/projects/list_files/offlineimap here]. I use [http://software.complete.org/software/versions/download/242?attachment_id=334 version 6.0.3]. Extract files in you favorite sandbox.

$ tar xvfz offlineimap_6.0.3.tar.gz
$ cd offlineimap

Apply the following [{{#file:offlineimap_6.0.3-cygwin.patch}} patch] to correct the following issues:

* [http://en.wikipedia.org/wiki/Maildir#Windows_software Suffix separator] character colon (:) is not allowed in Windows. Courier-imap for cygwin is using (!)
* While used by courier-imap for cygwin Maildir mail files should not use CRLF, rather only LF. The issue appears only in mail with attachment, that is the raw message is not interpreted and displayed as it in the mail body (under Thunderbird).

<source lang=diff>
--- offlineimap.conf 2008-08-13 20:56:04.000000000 +0200
+++ offlineimap-cygwin.conf 2008-11-06 10:44:31.066750700 +0100
@@ -215,6 +215,19 @@

restoreatime = no

+# Cygwin/Windows users may have issues with the Maildir mail filename
+# containing the suffix separator character (:). In this case, they
+# could change the suffix separator character to whatever required.
+# For example, Courier-IMAP for Cygwin is using (!)
+
+# suffixsep = !
+
+# Cygwin/Windows users may need to create Maildir mail filename in
+# binary mode. In this case, set the 'filemode' to 'binary'. Default is
+# 'text'. Courier-IMAP for Cygwin works better with binary mode.
+
+# filemode = binary
+
[Repository RemoteExample]

# And this is the remote repository. We only support IMAP or Gmail here.
--- offlineimap/folder/Maildir.py 2008-08-13 20:55:48.000000000 +0200
+++ offlineimap/folder/Maildir-cygwin.py 2008-11-06 10:44:05.344202700 +0100
@@ -23,7 +23,6 @@
import os.path, os, re, time, socket, md5

uidmatchre = re.compile(',U=(\d+)')
-flagmatchre = re.compile(':.*2,([A-Z]+)')

timeseq = 0
lasttime = long(0)
@@ -54,6 +53,12 @@
self.messagelist = None
self.repository = repository
self.accountname = accountname
+ self.filemode = {
+ 'binary': 'wb',
+ 'text': 'wt',
+ }[repository.getconf("filemode", 'text')]
+ self.suffixsep = repository.getconf("suffixsep",':')
+ self.flagmatchre = re.compile('%s.*2,([A-Z]+)' % self.suffixsep)
BaseFolder.__init__(self)

def getaccountname(self):
@@ -102,7 +107,7 @@
nouidcounter -= 1
else:
uid = long(uidmatch.group(1))
- flagmatch = flagmatchre.search(messagename)
+ flagmatch = self.flagmatchre.search(messagename)
flags = []
if flagmatch:
flags = [x for x in flagmatch.group(1)]
@@ -180,7 +185,7 @@
break
tmpmessagename = messagename.split(',')[0]
ui.debug('maildir', 'savemessage: using temporary name %s' % tmpmessagename)
- file = open(os.path.join(tmpdir, tmpmessagename), "wt")
+ file = open(os.path.join(tmpdir, tmpmessagename), self.filemode)
file.write(content)

# Make sure the data hits the disk
@@ -226,11 +231,11 @@
newpath = os.path.join(self.getfullname(), 'cur')
else:
newpath = os.path.join(self.getfullname(), 'new')
- infostr = ':'
- infomatch = re.search('(:.*)$', newname)
+ infostr = self.suffixsep
+ infomatch = re.search('(%s.*)$' % self.suffixsep, newname)
if infomatch: # If the info string is present..
infostr = infomatch.group(1)
- newname = newname.split(':')[0] # Strip off the info string.
+ newname = newname.split(self.suffixsep)[0] # Strip off the info string.
infostr = re.sub('2,[A-Z]*', '', infostr)
flags.sort()
infostr += '2,' + ''.join(flags)
</source>

$ patch -Np0 < offlineimap_6.0.3-cygwin.patch

Install offlineimap

$ python setup.py install

Configure offlineimap as described in [[Offlineimap]] and launch offlineimap. '''CAUTION:''' Add the NEW suffixsep and filemode options in the Maildir repository section of the configuration file.

$ offlineimap
or
$ offlineimap -1 -o -u Noninteractive.Basic
or
$ offlineimap -c /home/username/offlineimap.conf

Here is a simple [{{#file:offlineimap.conf}} offlineimap configuration file].

<source lang=text>
[general]
metadata = /home/username/.offlineimap
accounts = Test
maxsyncaccounts = 1
ui = Curses.Blinkenlights, TTY.TTYUI,
Noninteractive.Basic, Noninteractive.Quiet
ignore-readonly = no

[mbnames]
enabled = no

[ui.Curses.Blinkenlights]
statuschar = .

[Account Test]
localrepository = LocalExample
remoterepository = RemoteExample

[Repository LocalExample]
type = Maildir
localfolders = /home/username/Maildir
sep = .
restoreatime = no

suffixsep = ! # <<< NEW
filemode = binary # <<< NEW

[Repository RemoteExample]
type = IMAP
remotehost = imapserver
ssl = no
remoteport = 143
remoteuser = username
remotepass = XXXXXX
maxconnections = 1
holdconnectionopen = no
nametrans = lambda foldername: re.sub('^INBOX\.*', '.', foldername)
folderfilter = lambda foldername: not re.search('(SPAM$|Mairix)', foldername)
</source>

Courier-Cygwin

2008-11-07T10:32:09Z

Fuujuhi: /* Testing */

=Prerequisites=

Courier-IMAP requires the installation of the [http://www.courier-mta.org/authlib/ Courier Authentication Library] and the following cygwin tools and packages

* <tt>patch</tt>, <tt>tar</tt>, <tt>make</tt>, <tt>gcc</tt>
* <tt>crypt</tt>
* <tt>libgdbm-devel</tt>
* <tt>libtool</tt>
* <tt>inetutils</tt>
* <tt>cygrunsrv</tt>

Run the [http://www.cygwin.com/setup.exe Cygwin setup] program. Locate, select and install the required packages.

=Building and Installing Courier Authentication Library=

Installation of ''Courier Authentication Library'' using the module '''userdb''' to manage mail accounts. All other modules are disabled.

Download and untar [http://www.courier-mta.org/download.php#authlib courier-authlib] in your favorite sandbox.
Version used: [http://prdownloads.sourceforge.net/courier/courier-authlib-0.61.0.tar.bz2 0.61.0]

<source lang="bash">
$ tar -xvjf courier-authlib-0.61.0.tar.bz2
$ cd courier-authlib-0.61.0
</source>

Apply the [{{#file:courier-authlib-0.61.0-cygwin.patch}} patch] before configuration to correct following issues:

# incorrect usage of EXEEXT in ''makedat/Makefile''
# add libtool flag (-no-undefined) required for DLL creation ([http://lists.cairographics.org/archives/cairo/2004-April/001125.html])
# add missing dependencies
# add PATH in start script to access new DLL (installed in a different directory)

<source lang=diff>
--- courier-authlib-0.61.0/makedat/Makefile.in 2008-05-24 16:21:09.000000000 +0200
+++ courier-authlib-0.61.0-cygwin/makedat/Makefile.in 2008-10-21 16:02:38.709166700 +0200
@@ -182,7 +182,7 @@
libexecdir = @libexecdir@
localedir = @localedir@
localstatedir = @localstatedir@
-makedatprog_target = @makedatprog_target@
+makedatprog_target = @makedatprog_target@$(EXEEXT)
makedatprogpath = @makedatprogpath@
mandir = @mandir@
mkdir_p = @mkdir_p@
@@ -198,7 +198,7 @@
target_alias = @target_alias@
top_builddir = @top_builddir@
top_srcdir = @top_srcdir@
-noinst_PROGRAMS = @makedatprog_target@
+noinst_PROGRAMS = @makedatprog_target@$(EXEEXT)
makedatprog_SOURCES = makedatprog.c
makedatprog_DEPENDENCIES = @dblibrary@
makedatprog_LDADD = @dblibrary@
--- courier-authlib-0.61.0/Makefile.in 2008-07-12 21:41:08.000000000 +0200
+++ courier-authlib-0.61.0-cygwin/Makefile.in 2008-10-23 22:59:03.843750000 +0200
@@ -213,9 +213,10 @@
LTCOMPILE = $(LIBTOOL) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) \
--mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) \
$(AM_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS)
-CCLD = $(CC)
+CCLD = $(CC) -no-undefined
+CCLDEXE = $(CC)
LINK = $(LIBTOOL) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) \
- --mode=link $(CCLD) $(AM_CFLAGS) $(CFLAGS) $(AM_LDFLAGS) \
+ --mode=link $(CCLDEXE) $(AM_CFLAGS) $(CFLAGS) $(AM_LDFLAGS) \
$(LDFLAGS) -o $@
SOURCES = $(libauthcustom_la_SOURCES) $(libauthldap_la_SOURCES) \
$(libauthmysql_la_SOURCES) $(libauthpam_la_SOURCES) \
@@ -452,9 +453,9 @@
README.authdebug.html

DISTCLEANFILES = dbobj.config README_authlib.html
-commonlibdep = libcourierauthcommon.la
+commonlibdep = libcourierauthcommon.la libcourierauth.la
commonldflags = -module -rpath $(pkglibdir) -export-symbols-regex 'courier_auth.*_init' -avoid-version
-commonlibadd = libcourierauthcommon.la
+commonlibadd = libcourierauthcommon.la libcourierauth.la
libcourierauthcommon_t = @CRYPTLIBS@
libcourierauthcommon_la_SOURCES = \
auth.h courierauth.h \
--- courier-authlib-0.61.0/authdaemond.in 2005-07-05 14:25:08.000000000 +0200
+++ courier-authlib-0.61.0-cygwin/authdaemond.in 2008-10-25 12:03:32.140625000 +0200
@@ -15,4 +15,10 @@
set -a
. @authdaemonrc@

+# Some shared libraries (DLL) are installed in @libdir@/bin
+# instead of @libdir@/@PACKAGE@
+# Setting LD_LIBRARY_PATH at runtime or LR_RUN_PATH at linktime doesn't
+# work in cygwin, only setting PATH works.
+export PATH=$PATH:@libdir@/bin
+
exec ${sbindir}/courierlogger -pid=@authdaemonvar@/pid $LOGGEROPTS -$1 @libexecdir@/courier-authlib/authdaemond
</source>

Install the patch

<source lang="bash">
$ patch -Np1 < courier-authlib-0.61.0-cygwin.patch
</source>

Configure the package without most authentication modules, keeping only '''userdb'''. Replace ''mailuser'' by an existing user name (in your /etc/passwd file). I used my own username.

<source lang="bash">
$ ./configure --disable-root-check --with-waitfunc=wait --without-authpam --without-authldap --without-authpwd \
--without-authshadow --without-authcustom --without-authpipe --without-authmysql --without-authpgsql --with-mailuser=mailuser \
--with-mailgroup=mkgroup-l-d
</source>

Take a long pause... and when ready build, check the result and install the libraries.

<source lang="bash">
$ make
$ make check
$ make install
</source>

'''NOTES''': list of related topics found during investigation for compilation
* authldap: configure: error: -lresolve is needed for res_query... http://www.cygwin.com/ml/cygwin/2005-04/msg00419.html

=Building and Installing Courier-IMAP=

Download and untar package [http://www.courier-mta.org/download.php#imap courier-imap] in your favorite sandbox.
Version used: [http://prdownloads.sourceforge.net/courier/courier-imap-4.4.1.tar.bz2 4.4.1.20080920]

<source lang="bash">
$ tar -xvjf courier-imap-4.4.1.tar.bz2
$ cd courier-imap-4.4.1
</source>

Apply the [{{#file:courier-imap-4.4.1-cygwin.patch}} patch] before configuration to correct following issues:

# incorrect usage of EXEEXT in ''makedat/Makefile''
# incorrect usage of EXTEXT in main ''Makefile''
# remove usage of /usr/lib/env in start/stop script (/usr/lib/env does not seem to work under Cygwin)

<source lang=diff>
--- courier-imap-4.4.1.20080920/makedat/Makefile.in 2008-08-24 19:52:51.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/makedat/Makefile.in 2008-10-21 20:43:27.500000000 +0200
@@ -182,7 +182,7 @@
libexecdir = @libexecdir@
localedir = @localedir@
localstatedir = @localstatedir@
-makedatprog_target = @makedatprog_target@
+makedatprog_target = @makedatprog_target@$(EXEEXT)
makedatprogpath = @makedatprogpath@
mandir = @mandir@
mkdir_p = @mkdir_p@
@@ -198,7 +198,7 @@
target_alias = @target_alias@
top_builddir = @top_builddir@
top_srcdir = @top_srcdir@
-noinst_PROGRAMS = @makedatprog_target@
+noinst_PROGRAMS = @makedatprog_target@$(EXEEXT)
makedatprog_SOURCES = makedatprog.c
makedatprog_DEPENDENCIES = @dblibrary@
makedatprog_LDADD = @dblibrary@
--- courier-imap-4.4.1.20080920/Makefile.in 2008-09-20 14:48:46.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/Makefile.in 2008-10-21 20:59:45.156250000 +0200
@@ -247,9 +247,9 @@
CLEANFILES = $(databin_SCRIPTS) $(man_MANS) $(sysconf_DATA) $(sbin_SCRIPTS)
databindir = $(datadir)
databin_SCRIPTS = mkimapdcert mkpop3dcert
-binPROGRAMS = imapd pop3d maildirmake maildiracl deliverquota maildirkw
-sbinPROGRAMS = imaplogin pop3login
-libexecPROGRAMS = makedatprog couriertcpd
+binPROGRAMS = imapd$(EXEEXT) pop3d$(EXEEXT) maildirmake$(EXEEXT) maildiracl$(EXEEXT) deliverquota$(EXEEXT) maildirkw$(EXEEXT)
+sbinPROGRAMS = imaplogin$(EXEEXT) pop3login$(EXEEXT)
+libexecPROGRAMS = makedatprog$(EXEEXT) couriertcpd$(EXEEXT)
bin_PROGRAMS = @binPROGRAMS_exec@
sbin_PROGRAMS = @sbinPROGRAMS_exec@
libexec_PROGRAMS = @libexecPROGRAMS_exec@
--- courier-imap-4.4.1.20080920/imapd.rc.in 2008-05-04 15:12:47.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/imapd.rc.in 2008-10-26 10:17:51.359375000 +0100
@@ -22,6 +22,15 @@
exit 1
fi

+# Location for some shared libraries (cygcourierauth*.dll) from authlib
+AUTHLIBDIR=/usr/local
+if test ! -f $AUTHLIBDIR/lib/bin/cygcourierauth.dll
+then
+ echo "$AUTHLIBDIR/lib/bin/cygcourierauth.dll not found."
+ exit 1
+fi
+export PATH=$PATH:$AUTHLIBDIR/lib/bin
+
TLS_CACHEFILE=""
. @sysconfdir@/imapd-ssl
. @sysconfdir@/imapd
@@ -35,7 +44,7 @@

umask $IMAP_UMASK
@ULIMIT@ $IMAP_ULIMITD
- @SETENV@ -i @SHELL@ -c " set -a ;
+ @SHELL@ -c " set -a ;
prefix=@prefix@ ;
exec_prefix=@exec_prefix@ ;
bindir=@bindir@ ;
--- courier-imap-4.4.1.20080920/pop3d.rc.in 2008-05-04 15:12:47.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/pop3d.rc.in 2008-10-26 10:18:02.187500000 +0100
@@ -22,12 +22,21 @@
exit 1
fi

+# Location for some shared libraries (cygcourierauth*.dll) from authlib
+AUTHLIBDIR=/usr/local
+if test ! -f $AUTHLIBDIR/lib/bin/cygcourierauth.dll
+then
+ echo "$AUTHLIBDIR/lib/bin/cygcourierauth.dll not found."
+ exit 1
+fi
+export PATH=$PATH:$AUTHLIBDIR/lib/bin
+
. @sysconfdir@/pop3d-ssl
. @sysconfdir@/pop3d

case $1 in
start)
- @SETENV@ -i @SHELL@ -c " set -a ;
+ @SHELL@ -c " set -a ;
prefix=@prefix@ ;
exec_prefix=@exec_prefix@ ;
bindir=@bindir@ ;
</source>

Install the patch

<source lang="bash">
$ patch -Np1 < courier-imap-4.4.1-cygwin.patch
</source>

Configure the package

<source lang="bash">
$ ./configure --disable-root-check --with-waitfunc=wait
</source>

Take a walk or some coffee... and when ready build the package and install it.

<source lang="bash">
$ make
$ make install
$ make install-configure
</source>

=Configuring Courier=

== Configuration Files and Scripts ==

If not present yet, add <tt>/usr/local/bin</tt> and <tt>/usr/local/sbin</tt> to your path (ideally in <tt>/etc/bash.bashrc</tt>):

<source lang="bash">
PATH=/usr/local/bin:/usr/local/sbin:$PATH
</source>

Create configuration file for '''authdaemond''' (<tt>/usr/local/etc/authlib/authdaemonrc</tt>)

$ cp /usr/local/etc/authlib/authdaemonrc.dist /usr/local/etc/authlib/authdaemonrc

And edit its content to change number of daemons (I use three):

<source lang=text>
# The number of daemon processes that are started.
daemons=3
</source>

Edit the file <tt>/usr/lib/courier-imap/etc/imapd</tt>, taking care to adapt '''IMAP_ULIMITD''' accordingly:
<source lang="bash">
ADDRESS=127.0.0.1 #Typ. use 0 or 127.0.0.1
...
IMAP_ULIMITD=2097152 #Set same value as returned by 'ulimit -v' or imapd.rc will complain
</source>

Create link to authlib and imap daemon in <tt>/etc/rc.d/init.d</tt>
<source lang="bash">
ln -s /usr/local/sbin/authdaemond /etc/rc.d/init.d/courier-authdaemon
ln -s /usr/lib/courier-imap/libexec/imapd.rc /etc/rc.d/init.d/courier-imap
</source>

== IMAP Accounts ==

User accounts and settings are managed by ''userdb''. <tt>[http://linux.die.net/man/8/makeuserdb man makeuserdb]</tt> for details.

First create user mailbox (replace ''username'' with user account name) with <tt>maildirmake</tt>. Repeat for all users:
<source lang="bash">
mkdir -p /home/username #if don't exist yet...
cd /home/username
for i in "" .Drafts .Sent .Trash .Templates; do /usr/lib/courier-imap/bin/maildirmake ~username/MailDir/$i; done
# Create also default Maildir for new users
for i in "" .Drafts .Sent .Trash .Templates; do /usr/lib/courier-imap/bin/maildirmake /etc/skel/MailDir/$i; done
</source>

Next create the user authentication database <tt>/usr/local/etc/authlib/userdb</tt>:
<source lang="bash">
touch /usr/local/etc/authlib/userdb
chmod 700 /usr/local/etc/authlib/userdb
#Repeat for each username:
pw2userdb | grep "^username">>/usr/local/etc/authlib/userdb # Fill userdb from /etc/passwd user entry
userdb username set mail=~username/MailDir # Specify user MailDir location
userdbpw | userdb username set systempw # Set user account password
...
makeuserdb # Create binary database
</source>

You should now have the following files created:

* /usr/local/etc/authlib/userdb.dat
* /usr/local/etc/authlib/userdbshadow.dat

==Configuration Summary==

$ /usr/local/bin/courierauthconfig.exe --version
$ /usr/local/bin/courierauthconfig.exe --ldflags
$ /usr/local/bin/courierauthconfig.exe --cppflags
$ /usr/local/bin/courierauthconfig.exe --configfiles

=Running=

Start the daemons with the following commands:

<source lang="bash">
/etc/rc.d/init.d/courier-authdaemon start
/etc/rc.d/init.d/courier-imap start
</source>

Stop the daemons with the following commands:

<source lang="bash">
/etc/rc.d/init.d/courier-imap stop
/etc/rc.d/init.d/courier-authdaemon stop
</source>

=Testing=

If not done yet, install and configure '''syslogd'''. Run the [http://www.cygwin.com/setup.exe Cygwin setup] program and install '''inetutils''' package (contains '''syslogd'''). Configure '''syslogd''' with the following command, which will create the file <tt>/etc/syslogd.conf</tt> file and install a windows service using '''cygrunsrv''':

<source lang="bash">
$ syslogd-config #if not configured yet
$ net start syslogd #if not started yet
</source>

Edit '''authdaemond''' configuration file <tt>/usr/local/etc/authlib/authdaemonrc</tt> to enable full debug messages:

<source lang="text">
# DEBUG_LOGIN=2 - turn on debugging + log passwords too
DEBUG_LOGIN=2
</source>

and restart '''courier-authdaemon''' to take changes into effect:

<source lang="bash">
$ /etc/rc.d/init.d/courier-authdaemon stop
$ /etc/rc.d/init.d/courier-authdaemon start
</source>

Testing ''Courier Authentication Library'' with '''authtest''':

<source lang="bash">
$ export PATH=$PATH:/usr/local/lib/bin #because authtest needs a .DLL library there (found with strace)
$ /usr/local/sbin/authtest username CCCCCC
</source>

This should produce something similar to the transcript below (where <tt>CCCCCC</tt> is your cleartext password, and <tt>XXXXXX</tt> is your encrypted password as provided in <tt>/usr/local/etc/authlib/userdb</tt>):
<source lang="text">
Authentication succeeded.

Authenticated: username (uid 1001, gid 10545)
Home Directory: /home/username
Maildir: /home/username/Maildir
Quota: (none)
Encrypted Password: XXXXXX
Cleartext Password: CCCCCC
Options: (none)
</source>

Check that everything is fine in <tt>/var/log/messages</tt> (see [http://www.courier-mta.org/authlib/README.authdebug.html] for details):

<source lang="bash">
$ cat /var/log/messages
</source>

Testing '''Courier-IMAP''' through '''telnet''':

<source lang="bash">
$ telnet localhost 143
</source

In the ''telnet'' session, type the following test scripts (replace ''username'' and ''password'' accordingly):

<source lang="text">
001 LOGIN username password
002 EXAMINE INBOX
003 LOGOUT
</source>

Server must reply with '''OK''' messages along with some more information (see also [http://www.courier-mta.org/authlib/README.authdebug.html] for details).

'''Hint:''' if the connection is immediately closed by foreign host it probably means that ''imapd'' does not have access to all required DLLs. Check your PATH.

When testing is done, don't forget to reset debugging in <tt>/usr/local/etc/authlib/authdaemonrc</tt>:
<pre>
# DEBUG_LOGIN=2 - turn on debugging + log passwords too
DEBUG_LOGIN=0
</pre>

==Troubleshooting==

Should you have a doubt on the execution of a command (ie: no output) you can check if the application is able to load required DLL. This error is not reported. Using '''strace''', windows will complain about DLL not found, if any. To solve this problem you may need to add some path in your PATH. For example, in order to test authlib with authtest:

<source lang="bash">
$ strace /usr/local/sbin/authtest # This will open a dialog box to complain about DLL.
$ export PATH=$PATH:/usr/local/lib/bin
$ /usr/local/sbin/authtest ausername
</source>

=Installing as a Windows Service=

==Startup Script==

If everything works fine you can use the following [{{#file:imapd}} startup script] to launch or stop imap and authlib daemons. It can be saved in /usr/local/sbin.

<source lang=bash>
#! /bin/sh
#
# Courier imap daemon startup script.

authlib=/usr/local/sbin/authdaemond
imapd=/usr/lib/courier-imap/libexec/imapd.rc

test ! -f $authlib && echo "File not found: $authlib" && exit 1;
test ! -f $imapd && echo "File not found: $imapd" && exit 1;

case $1 in
start)
$authlib start
$imapd start
;;
stop)
$imapd stop
$authlib stop
;;
*)
echo "Usage: $0 <start|stop>"
;;
esac
</source>

==Cygwin Service==

Create the [{{#file:imapd-srv.sh}} service script]

<source lang=bash>
#!/bin/sh
# File: imapd-service.sh
# Purpose: Courier imap daemon service script.
#
# NOTE: This script must be excutable by SYSTEM
username=username # <<<< Change me !!!
pidsleep=''

# Function to handle QUIT signal.
# Stop imapd daemon and kill background sleep.
function handleQuit
{
echo "STOPPING: /usr/local/sbin/imapd"
su $username -c "/usr/local/sbin/imapd stop"
test -n "$pidsleep" && kill -9 $pidsleep
exit
}

# Interruptible sleep
# Note: sleep is an external command (not builtin command), hence
# not interruptible by the QUIT signal.
function intsleep
{
sleep $1 &
pidsleep=$!
wait $pidsleep
}

echo "STARTING: /usr/local/sbin/imapd as $username at "`date`
su $username -c "/usr/local/sbin/imapd start"

trap "handleQuit" SIGQUIT

echo "WAITING: "$$
while true; do intsleep 1d; done
</source>

Change username and save it to your favorite location, says /usr/local/sbin. The service script must be executable by SYSTEM, ie: the user that run the windows services.

$ chmod +rx /usr/local/sbin/imapd-service.sh

Install the new service with the following command.

$ cygrunsrv --install imapd --desc "Courier IMAP daemon" --disp "CYGWIN imapd" \
--path /usr/local/sbin/imapd-service.sh --termsig QUIT --type auto --shutdown

Finally, just start the service

$ cygrunsrv --start imapd
or
$ net start imapd

You should now have couple of processes related to courier-imap

$ ps -a

'''NOTES''':
* If the service could not be started, it's probably because the service script is not executable by SYSTEM

$ net start imapd
The CYGWIN imapd service is starting.
The CYGWIN imapd service could not be started.

The service did not report an error.

More help is available by typing NET HELPMSG 3534.

* The combination of background sleep and builtin wait command is used to make the waiting statment interruptible. The original script found on the net, was simply using a 'sleep 1' which turned to consume ~10% of the CPU.

=Other Interesting Packages=

==Maildir==

'''maildirmake''' is part of the courier-imap package

$ /usr/lib/courier-imap/bin/maildirmake

==Mairix - index and search mail folders==
Run the [http://www.cygwin.com/setup.exe Cygwin setup] program. Locate, select and install the package mairix.

Create the configuration file in your home directory: ~/.mairixrc

base=/home/username
maildir=Maildir...
omit=Maildir/.Mairix**
mfolder=Maildir/.Mairix
database=/home/username/.mairix_database

The create the mairix index

$ mairix
and
$ mairix --fast-index # on daily base

See [[Mail_Tips]] for additional details.

==Offlineimap==
Requires python, so run the [http://www.cygwin.com/setup.exe Cygwin setup] program. Locate, select and install the package python.

Download offlineimap [http://software.complete.org/software/projects/list_files/offlineimap here]. I use [http://software.complete.org/software/versions/download/242?attachment_id=334 version 6.0.3]. Extract files in you favorite sandbox.

$ tar xvfz offlineimap_6.0.3.tar.gz
$ cd offlineimap

Apply the following [{{#file:offlineimap_6.0.3-cygwin.patch}} patch] to correct the following issues:

* [http://en.wikipedia.org/wiki/Maildir#Windows_software Suffix separator] character colon (:) is not allowed in Windows. Courier-imap for cygwin is using (!)
* While used by courier-imap for cygwin Maildir mail files should not use CRLF, rather only LF. The issue appears only in mail with attachment, that is the raw message is not interpreted and displayed as it in the mail body (under Thunderbird).

<source lang=diff>
--- offlineimap.conf 2008-08-13 20:56:04.000000000 +0200
+++ offlineimap-cygwin.conf 2008-11-06 10:44:31.066750700 +0100
@@ -215,6 +215,19 @@

restoreatime = no

+# Cygwin/Windows users may have issues with the Maildir mail filename
+# containing the suffix separator character (:). In this case, they
+# could change the suffix separator character to whatever required.
+# For example, Courier-IMAP for Cygwin is using (!)
+
+# suffixsep = !
+
+# Cygwin/Windows users may need to create Maildir mail filename in
+# binary mode. In this case, set the 'filemode' to 'binary'. Default is
+# 'text'. Courier-IMAP for Cygwin works better with binary mode.
+
+# filemode = binary
+
[Repository RemoteExample]

# And this is the remote repository. We only support IMAP or Gmail here.
--- offlineimap/folder/Maildir.py 2008-08-13 20:55:48.000000000 +0200
+++ offlineimap/folder/Maildir-cygwin.py 2008-11-06 10:44:05.344202700 +0100
@@ -23,7 +23,6 @@
import os.path, os, re, time, socket, md5

uidmatchre = re.compile(',U=(\d+)')
-flagmatchre = re.compile(':.*2,([A-Z]+)')

timeseq = 0
lasttime = long(0)
@@ -54,6 +53,12 @@
self.messagelist = None
self.repository = repository
self.accountname = accountname
+ self.filemode = {
+ 'binary': 'wb',
+ 'text': 'wt',
+ }[repository.getconf("filemode", 'text')]
+ self.suffixsep = repository.getconf("suffixsep",':')
+ self.flagmatchre = re.compile('%s.*2,([A-Z]+)' % self.suffixsep)
BaseFolder.__init__(self)

def getaccountname(self):
@@ -102,7 +107,7 @@
nouidcounter -= 1
else:
uid = long(uidmatch.group(1))
- flagmatch = flagmatchre.search(messagename)
+ flagmatch = self.flagmatchre.search(messagename)
flags = []
if flagmatch:
flags = [x for x in flagmatch.group(1)]
@@ -180,7 +185,7 @@
break
tmpmessagename = messagename.split(',')[0]
ui.debug('maildir', 'savemessage: using temporary name %s' % tmpmessagename)
- file = open(os.path.join(tmpdir, tmpmessagename), "wt")
+ file = open(os.path.join(tmpdir, tmpmessagename), self.filemode)
file.write(content)

# Make sure the data hits the disk
@@ -226,11 +231,11 @@
newpath = os.path.join(self.getfullname(), 'cur')
else:
newpath = os.path.join(self.getfullname(), 'new')
- infostr = ':'
- infomatch = re.search('(:.*)$', newname)
+ infostr = self.suffixsep
+ infomatch = re.search('(%s.*)$' % self.suffixsep, newname)
if infomatch: # If the info string is present..
infostr = infomatch.group(1)
- newname = newname.split(':')[0] # Strip off the info string.
+ newname = newname.split(self.suffixsep)[0] # Strip off the info string.
infostr = re.sub('2,[A-Z]*', '', infostr)
flags.sort()
infostr += '2,' + ''.join(flags)
</source>

$ patch -Np0 < offlineimap_6.0.3-cygwin.patch

Install offlineimap

$ python setup.py install

Configure offlineimap as described in [[Offlineimap]] and launch offlineimap. '''CAUTION:''' Add the NEW suffixsep and filemode options in the Maildir repository section of the configuration file.

$ offlineimap
or
$ offlineimap -1 -o -u Noninteractive.Basic
or
$ offlineimap -c /home/username/offlineimap.conf

Here is a simple [{{#file:offlineimap.conf}} offlineimap configuration file].

<source lang=text>
[general]
metadata = /home/username/.offlineimap
accounts = Test
maxsyncaccounts = 1
ui = Curses.Blinkenlights, TTY.TTYUI,
Noninteractive.Basic, Noninteractive.Quiet
ignore-readonly = no

[mbnames]
enabled = no

[ui.Curses.Blinkenlights]
statuschar = .

[Account Test]
localrepository = LocalExample
remoterepository = RemoteExample

[Repository LocalExample]
type = Maildir
localfolders = /home/username/Maildir
sep = .
restoreatime = no

suffixsep = ! # <<< NEW
filemode = binary # <<< NEW

[Repository RemoteExample]
type = IMAP
remotehost = imapserver
ssl = no
remoteport = 143
remoteuser = username
remotepass = XXXXXX
maxconnections = 1
holdconnectionopen = no
nametrans = lambda foldername: re.sub('^INBOX\.*', '.', foldername)
folderfilter = lambda foldername: not re.search('(SPAM$|Mairix)', foldername)
</source>

Courier-Cygwin

2008-11-07T10:07:05Z

Fuujuhi: /* Running */

=Prerequisites=

Courier-IMAP requires the installation of the [http://www.courier-mta.org/authlib/ Courier Authentication Library] and the following cygwin tools and packages

* <tt>patch</tt>, <tt>tar</tt>, <tt>make</tt>, <tt>gcc</tt>
* <tt>crypt</tt>
* <tt>libgdbm-devel</tt>
* <tt>libtool</tt>
* <tt>inetutils</tt>
* <tt>cygrunsrv</tt>

Run the [http://www.cygwin.com/setup.exe Cygwin setup] program. Locate, select and install the required packages.

=Building and Installing Courier Authentication Library=

Installation of ''Courier Authentication Library'' using the module '''userdb''' to manage mail accounts. All other modules are disabled.

Download and untar [http://www.courier-mta.org/download.php#authlib courier-authlib] in your favorite sandbox.
Version used: [http://prdownloads.sourceforge.net/courier/courier-authlib-0.61.0.tar.bz2 0.61.0]

<source lang="bash">
$ tar -xvjf courier-authlib-0.61.0.tar.bz2
$ cd courier-authlib-0.61.0
</source>

Apply the [{{#file:courier-authlib-0.61.0-cygwin.patch}} patch] before configuration to correct following issues:

# incorrect usage of EXEEXT in ''makedat/Makefile''
# add libtool flag (-no-undefined) required for DLL creation ([http://lists.cairographics.org/archives/cairo/2004-April/001125.html])
# add missing dependencies
# add PATH in start script to access new DLL (installed in a different directory)

<source lang=diff>
--- courier-authlib-0.61.0/makedat/Makefile.in 2008-05-24 16:21:09.000000000 +0200
+++ courier-authlib-0.61.0-cygwin/makedat/Makefile.in 2008-10-21 16:02:38.709166700 +0200
@@ -182,7 +182,7 @@
libexecdir = @libexecdir@
localedir = @localedir@
localstatedir = @localstatedir@
-makedatprog_target = @makedatprog_target@
+makedatprog_target = @makedatprog_target@$(EXEEXT)
makedatprogpath = @makedatprogpath@
mandir = @mandir@
mkdir_p = @mkdir_p@
@@ -198,7 +198,7 @@
target_alias = @target_alias@
top_builddir = @top_builddir@
top_srcdir = @top_srcdir@
-noinst_PROGRAMS = @makedatprog_target@
+noinst_PROGRAMS = @makedatprog_target@$(EXEEXT)
makedatprog_SOURCES = makedatprog.c
makedatprog_DEPENDENCIES = @dblibrary@
makedatprog_LDADD = @dblibrary@
--- courier-authlib-0.61.0/Makefile.in 2008-07-12 21:41:08.000000000 +0200
+++ courier-authlib-0.61.0-cygwin/Makefile.in 2008-10-23 22:59:03.843750000 +0200
@@ -213,9 +213,10 @@
LTCOMPILE = $(LIBTOOL) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) \
--mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) \
$(AM_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS)
-CCLD = $(CC)
+CCLD = $(CC) -no-undefined
+CCLDEXE = $(CC)
LINK = $(LIBTOOL) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) \
- --mode=link $(CCLD) $(AM_CFLAGS) $(CFLAGS) $(AM_LDFLAGS) \
+ --mode=link $(CCLDEXE) $(AM_CFLAGS) $(CFLAGS) $(AM_LDFLAGS) \
$(LDFLAGS) -o $@
SOURCES = $(libauthcustom_la_SOURCES) $(libauthldap_la_SOURCES) \
$(libauthmysql_la_SOURCES) $(libauthpam_la_SOURCES) \
@@ -452,9 +453,9 @@
README.authdebug.html

DISTCLEANFILES = dbobj.config README_authlib.html
-commonlibdep = libcourierauthcommon.la
+commonlibdep = libcourierauthcommon.la libcourierauth.la
commonldflags = -module -rpath $(pkglibdir) -export-symbols-regex 'courier_auth.*_init' -avoid-version
-commonlibadd = libcourierauthcommon.la
+commonlibadd = libcourierauthcommon.la libcourierauth.la
libcourierauthcommon_t = @CRYPTLIBS@
libcourierauthcommon_la_SOURCES = \
auth.h courierauth.h \
--- courier-authlib-0.61.0/authdaemond.in 2005-07-05 14:25:08.000000000 +0200
+++ courier-authlib-0.61.0-cygwin/authdaemond.in 2008-10-25 12:03:32.140625000 +0200
@@ -15,4 +15,10 @@
set -a
. @authdaemonrc@

+# Some shared libraries (DLL) are installed in @libdir@/bin
+# instead of @libdir@/@PACKAGE@
+# Setting LD_LIBRARY_PATH at runtime or LR_RUN_PATH at linktime doesn't
+# work in cygwin, only setting PATH works.
+export PATH=$PATH:@libdir@/bin
+
exec ${sbindir}/courierlogger -pid=@authdaemonvar@/pid $LOGGEROPTS -$1 @libexecdir@/courier-authlib/authdaemond
</source>

Install the patch

<source lang="bash">
$ patch -Np1 < courier-authlib-0.61.0-cygwin.patch
</source>

Configure the package without most authentication modules, keeping only '''userdb'''. Replace ''mailuser'' by an existing user name (in your /etc/passwd file). I used my own username.

<source lang="bash">
$ ./configure --disable-root-check --with-waitfunc=wait --without-authpam --without-authldap --without-authpwd \
--without-authshadow --without-authcustom --without-authpipe --without-authmysql --without-authpgsql --with-mailuser=mailuser \
--with-mailgroup=mkgroup-l-d
</source>

Take a long pause... and when ready build, check the result and install the libraries.

<source lang="bash">
$ make
$ make check
$ make install
</source>

'''NOTES''': list of related topics found during investigation for compilation
* authldap: configure: error: -lresolve is needed for res_query... http://www.cygwin.com/ml/cygwin/2005-04/msg00419.html

=Building and Installing Courier-IMAP=

Download and untar package [http://www.courier-mta.org/download.php#imap courier-imap] in your favorite sandbox.
Version used: [http://prdownloads.sourceforge.net/courier/courier-imap-4.4.1.tar.bz2 4.4.1.20080920]

<source lang="bash">
$ tar -xvjf courier-imap-4.4.1.tar.bz2
$ cd courier-imap-4.4.1
</source>

Apply the [{{#file:courier-imap-4.4.1-cygwin.patch}} patch] before configuration to correct following issues:

# incorrect usage of EXEEXT in ''makedat/Makefile''
# incorrect usage of EXTEXT in main ''Makefile''
# remove usage of /usr/lib/env in start/stop script (/usr/lib/env does not seem to work under Cygwin)

<source lang=diff>
--- courier-imap-4.4.1.20080920/makedat/Makefile.in 2008-08-24 19:52:51.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/makedat/Makefile.in 2008-10-21 20:43:27.500000000 +0200
@@ -182,7 +182,7 @@
libexecdir = @libexecdir@
localedir = @localedir@
localstatedir = @localstatedir@
-makedatprog_target = @makedatprog_target@
+makedatprog_target = @makedatprog_target@$(EXEEXT)
makedatprogpath = @makedatprogpath@
mandir = @mandir@
mkdir_p = @mkdir_p@
@@ -198,7 +198,7 @@
target_alias = @target_alias@
top_builddir = @top_builddir@
top_srcdir = @top_srcdir@
-noinst_PROGRAMS = @makedatprog_target@
+noinst_PROGRAMS = @makedatprog_target@$(EXEEXT)
makedatprog_SOURCES = makedatprog.c
makedatprog_DEPENDENCIES = @dblibrary@
makedatprog_LDADD = @dblibrary@
--- courier-imap-4.4.1.20080920/Makefile.in 2008-09-20 14:48:46.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/Makefile.in 2008-10-21 20:59:45.156250000 +0200
@@ -247,9 +247,9 @@
CLEANFILES = $(databin_SCRIPTS) $(man_MANS) $(sysconf_DATA) $(sbin_SCRIPTS)
databindir = $(datadir)
databin_SCRIPTS = mkimapdcert mkpop3dcert
-binPROGRAMS = imapd pop3d maildirmake maildiracl deliverquota maildirkw
-sbinPROGRAMS = imaplogin pop3login
-libexecPROGRAMS = makedatprog couriertcpd
+binPROGRAMS = imapd$(EXEEXT) pop3d$(EXEEXT) maildirmake$(EXEEXT) maildiracl$(EXEEXT) deliverquota$(EXEEXT) maildirkw$(EXEEXT)
+sbinPROGRAMS = imaplogin$(EXEEXT) pop3login$(EXEEXT)
+libexecPROGRAMS = makedatprog$(EXEEXT) couriertcpd$(EXEEXT)
bin_PROGRAMS = @binPROGRAMS_exec@
sbin_PROGRAMS = @sbinPROGRAMS_exec@
libexec_PROGRAMS = @libexecPROGRAMS_exec@
--- courier-imap-4.4.1.20080920/imapd.rc.in 2008-05-04 15:12:47.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/imapd.rc.in 2008-10-26 10:17:51.359375000 +0100
@@ -22,6 +22,15 @@
exit 1
fi

+# Location for some shared libraries (cygcourierauth*.dll) from authlib
+AUTHLIBDIR=/usr/local
+if test ! -f $AUTHLIBDIR/lib/bin/cygcourierauth.dll
+then
+ echo "$AUTHLIBDIR/lib/bin/cygcourierauth.dll not found."
+ exit 1
+fi
+export PATH=$PATH:$AUTHLIBDIR/lib/bin
+
TLS_CACHEFILE=""
. @sysconfdir@/imapd-ssl
. @sysconfdir@/imapd
@@ -35,7 +44,7 @@

umask $IMAP_UMASK
@ULIMIT@ $IMAP_ULIMITD
- @SETENV@ -i @SHELL@ -c " set -a ;
+ @SHELL@ -c " set -a ;
prefix=@prefix@ ;
exec_prefix=@exec_prefix@ ;
bindir=@bindir@ ;
--- courier-imap-4.4.1.20080920/pop3d.rc.in 2008-05-04 15:12:47.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/pop3d.rc.in 2008-10-26 10:18:02.187500000 +0100
@@ -22,12 +22,21 @@
exit 1
fi

+# Location for some shared libraries (cygcourierauth*.dll) from authlib
+AUTHLIBDIR=/usr/local
+if test ! -f $AUTHLIBDIR/lib/bin/cygcourierauth.dll
+then
+ echo "$AUTHLIBDIR/lib/bin/cygcourierauth.dll not found."
+ exit 1
+fi
+export PATH=$PATH:$AUTHLIBDIR/lib/bin
+
. @sysconfdir@/pop3d-ssl
. @sysconfdir@/pop3d

case $1 in
start)
- @SETENV@ -i @SHELL@ -c " set -a ;
+ @SHELL@ -c " set -a ;
prefix=@prefix@ ;
exec_prefix=@exec_prefix@ ;
bindir=@bindir@ ;
</source>

Install the patch

<source lang="bash">
$ patch -Np1 < courier-imap-4.4.1-cygwin.patch
</source>

Configure the package

<source lang="bash">
$ ./configure --disable-root-check --with-waitfunc=wait
</source>

Take a walk or some coffee... and when ready build the package and install it.

<source lang="bash">
$ make
$ make install
$ make install-configure
</source>

=Configuring Courier=

== Configuration Files and Scripts ==

If not present yet, add <tt>/usr/local/bin</tt> and <tt>/usr/local/sbin</tt> to your path (ideally in <tt>/etc/bash.bashrc</tt>):

<source lang="bash">
PATH=/usr/local/bin:/usr/local/sbin:$PATH
</source>

Create configuration file for '''authdaemond''' (<tt>/usr/local/etc/authlib/authdaemonrc</tt>)

$ cp /usr/local/etc/authlib/authdaemonrc.dist /usr/local/etc/authlib/authdaemonrc

And edit its content to change number of daemons (I use three):

<source lang=text>
# The number of daemon processes that are started.
daemons=3
</source>

Edit the file <tt>/usr/lib/courier-imap/etc/imapd</tt>, taking care to adapt '''IMAP_ULIMITD''' accordingly:
<source lang="bash">
ADDRESS=127.0.0.1 #Typ. use 0 or 127.0.0.1
...
IMAP_ULIMITD=2097152 #Set same value as returned by 'ulimit -v' or imapd.rc will complain
</source>

Create link to authlib and imap daemon in <tt>/etc/rc.d/init.d</tt>
<source lang="bash">
ln -s /usr/local/sbin/authdaemond /etc/rc.d/init.d/courier-authdaemon
ln -s /usr/lib/courier-imap/libexec/imapd.rc /etc/rc.d/init.d/courier-imap
</source>

== IMAP Accounts ==

User accounts and settings are managed by ''userdb''. <tt>[http://linux.die.net/man/8/makeuserdb man makeuserdb]</tt> for details.

First create user mailbox (replace ''username'' with user account name) with <tt>maildirmake</tt>. Repeat for all users:
<source lang="bash">
mkdir -p /home/username #if don't exist yet...
cd /home/username
for i in "" .Drafts .Sent .Trash .Templates; do /usr/lib/courier-imap/bin/maildirmake ~username/MailDir/$i; done
# Create also default Maildir for new users
for i in "" .Drafts .Sent .Trash .Templates; do /usr/lib/courier-imap/bin/maildirmake /etc/skel/MailDir/$i; done
</source>

Next create the user authentication database <tt>/usr/local/etc/authlib/userdb</tt>:
<source lang="bash">
touch /usr/local/etc/authlib/userdb
chmod 700 /usr/local/etc/authlib/userdb
#Repeat for each username:
pw2userdb | grep "^username">>/usr/local/etc/authlib/userdb # Fill userdb from /etc/passwd user entry
userdb username set mail=~username/MailDir # Specify user MailDir location
userdbpw | userdb username set systempw # Set user account password
...
makeuserdb # Create binary database
</source>

You should now have the following files created:

* /usr/local/etc/authlib/userdb.dat
* /usr/local/etc/authlib/userdbshadow.dat

==Configuration Summary==

$ /usr/local/bin/courierauthconfig.exe --version
$ /usr/local/bin/courierauthconfig.exe --ldflags
$ /usr/local/bin/courierauthconfig.exe --cppflags
$ /usr/local/bin/courierauthconfig.exe --configfiles

=Running=

Start the daemons with the following commands:

<source lang="bash">
/etc/rc.d/init.d/courier-authdaemon start
/etc/rc.d/init.d/courier-imap start
</source>

Stop the daemons with the following commands:

<source lang="bash">
/etc/rc.d/init.d/courier-imap stop
/etc/rc.d/init.d/courier-authdaemon stop
</source>

=Testing=

$ /usr/local/sbin/authtest username CCCCCC
Authentication succeeded.

Authenticated: username (uid 1001, gid 10545)
Home Directory: /home/username
Maildir: /home/username/Maildir
Quota: (none)
Encrypted Password: XXXXXX
Cleartext Password: CCCCCC
Options: (none)

where username is your user name and XXXXXX is your encrypted password as provided in ''/usr/local/etc/authlib/userdb''. CCCCCC is your password in clear.

When authdaemond and imapd are running you can do a simple login test:

$ telnet localhost 143

and type the login command

a login <username> <password>

the server should reply

a OK LOGIN Ok.

to which you can reply with a logout

a logout

the session is terminated by the server

* BYE Courier-IMAP server shutting down
a OK LOGOUT completed
Connection closed by foreign host.

'''Hint:''' if the connection is immediately closed by foreign host it probably means that ''imapd'' does not have access to all required DLLs. Check your PATH.

==Troubleshooting==

Should you have a doubt on the execution of a command (ie: no output) you can check if the application is able to load required DLL. This error is not reported. Using ''strace'', windows will complain about DLL not found, if any. To solve this problem you may need to add some path in your PATH. For example, in order to test authlib with authtest:

$ strace /usr/local/sbin/authtest
# This will open a dialog box to complain about DLL.
$ export PATH=$PATH:/usr/local/lib/bin
$ /usr/local/sbin/authtest ausername

=Installing as a Windows Service=

==Startup Script==

If everything works fine you can use the following [{{#file:imapd}} startup script] to launch or stop imap and authlib daemons. It can be saved in /usr/local/sbin.

<source lang=bash>
#! /bin/sh
#
# Courier imap daemon startup script.

authlib=/usr/local/sbin/authdaemond
imapd=/usr/lib/courier-imap/libexec/imapd.rc

test ! -f $authlib && echo "File not found: $authlib" && exit 1;
test ! -f $imapd && echo "File not found: $imapd" && exit 1;

case $1 in
start)
$authlib start
$imapd start
;;
stop)
$imapd stop
$authlib stop
;;
*)
echo "Usage: $0 <start|stop>"
;;
esac
</source>

==Cygwin Service==

Create the [{{#file:imapd-srv.sh}} service script]

<source lang=bash>
#!/bin/sh
# File: imapd-service.sh
# Purpose: Courier imap daemon service script.
#
# NOTE: This script must be excutable by SYSTEM
username=username # <<<< Change me !!!
pidsleep=''

# Function to handle QUIT signal.
# Stop imapd daemon and kill background sleep.
function handleQuit
{
echo "STOPPING: /usr/local/sbin/imapd"
su $username -c "/usr/local/sbin/imapd stop"
test -n "$pidsleep" && kill -9 $pidsleep
exit
}

# Interruptible sleep
# Note: sleep is an external command (not builtin command), hence
# not interruptible by the QUIT signal.
function intsleep
{
sleep $1 &
pidsleep=$!
wait $pidsleep
}

echo "STARTING: /usr/local/sbin/imapd as $username at "`date`
su $username -c "/usr/local/sbin/imapd start"

trap "handleQuit" SIGQUIT

echo "WAITING: "$$
while true; do intsleep 1d; done
</source>

Change username and save it to your favorite location, says /usr/local/sbin. The service script must be executable by SYSTEM, ie: the user that run the windows services.

$ chmod +rx /usr/local/sbin/imapd-service.sh

Install the new service with the following command.

$ cygrunsrv --install imapd --desc "Courier IMAP daemon" --disp "CYGWIN imapd" \
--path /usr/local/sbin/imapd-service.sh --termsig QUIT --type auto --shutdown

Finally, just start the service

$ cygrunsrv --start imapd
or
$ net start imapd

You should now have couple of processes related to courier-imap

$ ps -a

'''NOTES''':
* If the service could not be started, it's probably because the service script is not executable by SYSTEM

$ net start imapd
The CYGWIN imapd service is starting.
The CYGWIN imapd service could not be started.

The service did not report an error.

More help is available by typing NET HELPMSG 3534.

* The combination of background sleep and builtin wait command is used to make the waiting statment interruptible. The original script found on the net, was simply using a 'sleep 1' which turned to consume ~10% of the CPU.

=Other Interesting Packages=

==Maildir==

'''maildirmake''' is part of the courier-imap package

$ /usr/lib/courier-imap/bin/maildirmake

==Mairix - index and search mail folders==
Run the [http://www.cygwin.com/setup.exe Cygwin setup] program. Locate, select and install the package mairix.

Create the configuration file in your home directory: ~/.mairixrc

base=/home/username
maildir=Maildir...
omit=Maildir/.Mairix**
mfolder=Maildir/.Mairix
database=/home/username/.mairix_database

The create the mairix index

$ mairix
and
$ mairix --fast-index # on daily base

See [[Mail_Tips]] for additional details.

==Offlineimap==
Requires python, so run the [http://www.cygwin.com/setup.exe Cygwin setup] program. Locate, select and install the package python.

Download offlineimap [http://software.complete.org/software/projects/list_files/offlineimap here]. I use [http://software.complete.org/software/versions/download/242?attachment_id=334 version 6.0.3]. Extract files in you favorite sandbox.

$ tar xvfz offlineimap_6.0.3.tar.gz
$ cd offlineimap

Apply the following [{{#file:offlineimap_6.0.3-cygwin.patch}} patch] to correct the following issues:

* [http://en.wikipedia.org/wiki/Maildir#Windows_software Suffix separator] character colon (:) is not allowed in Windows. Courier-imap for cygwin is using (!)
* While used by courier-imap for cygwin Maildir mail files should not use CRLF, rather only LF. The issue appears only in mail with attachment, that is the raw message is not interpreted and displayed as it in the mail body (under Thunderbird).

<source lang=diff>
--- offlineimap.conf 2008-08-13 20:56:04.000000000 +0200
+++ offlineimap-cygwin.conf 2008-11-06 10:44:31.066750700 +0100
@@ -215,6 +215,19 @@

restoreatime = no

+# Cygwin/Windows users may have issues with the Maildir mail filename
+# containing the suffix separator character (:). In this case, they
+# could change the suffix separator character to whatever required.
+# For example, Courier-IMAP for Cygwin is using (!)
+
+# suffixsep = !
+
+# Cygwin/Windows users may need to create Maildir mail filename in
+# binary mode. In this case, set the 'filemode' to 'binary'. Default is
+# 'text'. Courier-IMAP for Cygwin works better with binary mode.
+
+# filemode = binary
+
[Repository RemoteExample]

# And this is the remote repository. We only support IMAP or Gmail here.
--- offlineimap/folder/Maildir.py 2008-08-13 20:55:48.000000000 +0200
+++ offlineimap/folder/Maildir-cygwin.py 2008-11-06 10:44:05.344202700 +0100
@@ -23,7 +23,6 @@
import os.path, os, re, time, socket, md5

uidmatchre = re.compile(',U=(\d+)')
-flagmatchre = re.compile(':.*2,([A-Z]+)')

timeseq = 0
lasttime = long(0)
@@ -54,6 +53,12 @@
self.messagelist = None
self.repository = repository
self.accountname = accountname
+ self.filemode = {
+ 'binary': 'wb',
+ 'text': 'wt',
+ }[repository.getconf("filemode", 'text')]
+ self.suffixsep = repository.getconf("suffixsep",':')
+ self.flagmatchre = re.compile('%s.*2,([A-Z]+)' % self.suffixsep)
BaseFolder.__init__(self)

def getaccountname(self):
@@ -102,7 +107,7 @@
nouidcounter -= 1
else:
uid = long(uidmatch.group(1))
- flagmatch = flagmatchre.search(messagename)
+ flagmatch = self.flagmatchre.search(messagename)
flags = []
if flagmatch:
flags = [x for x in flagmatch.group(1)]
@@ -180,7 +185,7 @@
break
tmpmessagename = messagename.split(',')[0]
ui.debug('maildir', 'savemessage: using temporary name %s' % tmpmessagename)
- file = open(os.path.join(tmpdir, tmpmessagename), "wt")
+ file = open(os.path.join(tmpdir, tmpmessagename), self.filemode)
file.write(content)

# Make sure the data hits the disk
@@ -226,11 +231,11 @@
newpath = os.path.join(self.getfullname(), 'cur')
else:
newpath = os.path.join(self.getfullname(), 'new')
- infostr = ':'
- infomatch = re.search('(:.*)$', newname)
+ infostr = self.suffixsep
+ infomatch = re.search('(%s.*)$' % self.suffixsep, newname)
if infomatch: # If the info string is present..
infostr = infomatch.group(1)
- newname = newname.split(':')[0] # Strip off the info string.
+ newname = newname.split(self.suffixsep)[0] # Strip off the info string.
infostr = re.sub('2,[A-Z]*', '', infostr)
flags.sort()
infostr += '2,' + ''.join(flags)
</source>

$ patch -Np0 < offlineimap_6.0.3-cygwin.patch

Install offlineimap

$ python setup.py install

Configure offlineimap as described in [[Offlineimap]] and launch offlineimap. '''CAUTION:''' Add the NEW suffixsep and filemode options in the Maildir repository section of the configuration file.

$ offlineimap
or
$ offlineimap -1 -o -u Noninteractive.Basic
or
$ offlineimap -c /home/username/offlineimap.conf

Here is a simple [{{#file:offlineimap.conf}} offlineimap configuration file].

<source lang=text>
[general]
metadata = /home/username/.offlineimap
accounts = Test
maxsyncaccounts = 1
ui = Curses.Blinkenlights, TTY.TTYUI,
Noninteractive.Basic, Noninteractive.Quiet
ignore-readonly = no

[mbnames]
enabled = no

[ui.Curses.Blinkenlights]
statuschar = .

[Account Test]
localrepository = LocalExample
remoterepository = RemoteExample

[Repository LocalExample]
type = Maildir
localfolders = /home/username/Maildir
sep = .
restoreatime = no

suffixsep = ! # <<< NEW
filemode = binary # <<< NEW

[Repository RemoteExample]
type = IMAP
remotehost = imapserver
ssl = no
remoteport = 143
remoteuser = username
remotepass = XXXXXX
maxconnections = 1
holdconnectionopen = no
nametrans = lambda foldername: re.sub('^INBOX\.*', '.', foldername)
folderfilter = lambda foldername: not re.search('(SPAM$|Mairix)', foldername)
</source>

Courier-Cygwin

2008-11-07T10:03:39Z

Fuujuhi: /* Configuring Courier */ Too much subtitles... why not a subtitle for each sentence ;-)

=Prerequisites=

Courier-IMAP requires the installation of the [http://www.courier-mta.org/authlib/ Courier Authentication Library] and the following cygwin tools and packages

* <tt>patch</tt>, <tt>tar</tt>, <tt>make</tt>, <tt>gcc</tt>
* <tt>crypt</tt>
* <tt>libgdbm-devel</tt>
* <tt>libtool</tt>
* <tt>inetutils</tt>
* <tt>cygrunsrv</tt>

Run the [http://www.cygwin.com/setup.exe Cygwin setup] program. Locate, select and install the required packages.

=Building and Installing Courier Authentication Library=

Installation of ''Courier Authentication Library'' using the module '''userdb''' to manage mail accounts. All other modules are disabled.

Download and untar [http://www.courier-mta.org/download.php#authlib courier-authlib] in your favorite sandbox.
Version used: [http://prdownloads.sourceforge.net/courier/courier-authlib-0.61.0.tar.bz2 0.61.0]

<source lang="bash">
$ tar -xvjf courier-authlib-0.61.0.tar.bz2
$ cd courier-authlib-0.61.0
</source>

Apply the [{{#file:courier-authlib-0.61.0-cygwin.patch}} patch] before configuration to correct following issues:

# incorrect usage of EXEEXT in ''makedat/Makefile''
# add libtool flag (-no-undefined) required for DLL creation ([http://lists.cairographics.org/archives/cairo/2004-April/001125.html])
# add missing dependencies
# add PATH in start script to access new DLL (installed in a different directory)

<source lang=diff>
--- courier-authlib-0.61.0/makedat/Makefile.in 2008-05-24 16:21:09.000000000 +0200
+++ courier-authlib-0.61.0-cygwin/makedat/Makefile.in 2008-10-21 16:02:38.709166700 +0200
@@ -182,7 +182,7 @@
libexecdir = @libexecdir@
localedir = @localedir@
localstatedir = @localstatedir@
-makedatprog_target = @makedatprog_target@
+makedatprog_target = @makedatprog_target@$(EXEEXT)
makedatprogpath = @makedatprogpath@
mandir = @mandir@
mkdir_p = @mkdir_p@
@@ -198,7 +198,7 @@
target_alias = @target_alias@
top_builddir = @top_builddir@
top_srcdir = @top_srcdir@
-noinst_PROGRAMS = @makedatprog_target@
+noinst_PROGRAMS = @makedatprog_target@$(EXEEXT)
makedatprog_SOURCES = makedatprog.c
makedatprog_DEPENDENCIES = @dblibrary@
makedatprog_LDADD = @dblibrary@
--- courier-authlib-0.61.0/Makefile.in 2008-07-12 21:41:08.000000000 +0200
+++ courier-authlib-0.61.0-cygwin/Makefile.in 2008-10-23 22:59:03.843750000 +0200
@@ -213,9 +213,10 @@
LTCOMPILE = $(LIBTOOL) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) \
--mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) \
$(AM_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS)
-CCLD = $(CC)
+CCLD = $(CC) -no-undefined
+CCLDEXE = $(CC)
LINK = $(LIBTOOL) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) \
- --mode=link $(CCLD) $(AM_CFLAGS) $(CFLAGS) $(AM_LDFLAGS) \
+ --mode=link $(CCLDEXE) $(AM_CFLAGS) $(CFLAGS) $(AM_LDFLAGS) \
$(LDFLAGS) -o $@
SOURCES = $(libauthcustom_la_SOURCES) $(libauthldap_la_SOURCES) \
$(libauthmysql_la_SOURCES) $(libauthpam_la_SOURCES) \
@@ -452,9 +453,9 @@
README.authdebug.html

DISTCLEANFILES = dbobj.config README_authlib.html
-commonlibdep = libcourierauthcommon.la
+commonlibdep = libcourierauthcommon.la libcourierauth.la
commonldflags = -module -rpath $(pkglibdir) -export-symbols-regex 'courier_auth.*_init' -avoid-version
-commonlibadd = libcourierauthcommon.la
+commonlibadd = libcourierauthcommon.la libcourierauth.la
libcourierauthcommon_t = @CRYPTLIBS@
libcourierauthcommon_la_SOURCES = \
auth.h courierauth.h \
--- courier-authlib-0.61.0/authdaemond.in 2005-07-05 14:25:08.000000000 +0200
+++ courier-authlib-0.61.0-cygwin/authdaemond.in 2008-10-25 12:03:32.140625000 +0200
@@ -15,4 +15,10 @@
set -a
. @authdaemonrc@

+# Some shared libraries (DLL) are installed in @libdir@/bin
+# instead of @libdir@/@PACKAGE@
+# Setting LD_LIBRARY_PATH at runtime or LR_RUN_PATH at linktime doesn't
+# work in cygwin, only setting PATH works.
+export PATH=$PATH:@libdir@/bin
+
exec ${sbindir}/courierlogger -pid=@authdaemonvar@/pid $LOGGEROPTS -$1 @libexecdir@/courier-authlib/authdaemond
</source>

Install the patch

<source lang="bash">
$ patch -Np1 < courier-authlib-0.61.0-cygwin.patch
</source>

Configure the package without most authentication modules, keeping only '''userdb'''. Replace ''mailuser'' by an existing user name (in your /etc/passwd file). I used my own username.

<source lang="bash">
$ ./configure --disable-root-check --with-waitfunc=wait --without-authpam --without-authldap --without-authpwd \
--without-authshadow --without-authcustom --without-authpipe --without-authmysql --without-authpgsql --with-mailuser=mailuser \
--with-mailgroup=mkgroup-l-d
</source>

Take a long pause... and when ready build, check the result and install the libraries.

<source lang="bash">
$ make
$ make check
$ make install
</source>

'''NOTES''': list of related topics found during investigation for compilation
* authldap: configure: error: -lresolve is needed for res_query... http://www.cygwin.com/ml/cygwin/2005-04/msg00419.html

=Building and Installing Courier-IMAP=

Download and untar package [http://www.courier-mta.org/download.php#imap courier-imap] in your favorite sandbox.
Version used: [http://prdownloads.sourceforge.net/courier/courier-imap-4.4.1.tar.bz2 4.4.1.20080920]

<source lang="bash">
$ tar -xvjf courier-imap-4.4.1.tar.bz2
$ cd courier-imap-4.4.1
</source>

Apply the [{{#file:courier-imap-4.4.1-cygwin.patch}} patch] before configuration to correct following issues:

# incorrect usage of EXEEXT in ''makedat/Makefile''
# incorrect usage of EXTEXT in main ''Makefile''
# remove usage of /usr/lib/env in start/stop script (/usr/lib/env does not seem to work under Cygwin)

<source lang=diff>
--- courier-imap-4.4.1.20080920/makedat/Makefile.in 2008-08-24 19:52:51.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/makedat/Makefile.in 2008-10-21 20:43:27.500000000 +0200
@@ -182,7 +182,7 @@
libexecdir = @libexecdir@
localedir = @localedir@
localstatedir = @localstatedir@
-makedatprog_target = @makedatprog_target@
+makedatprog_target = @makedatprog_target@$(EXEEXT)
makedatprogpath = @makedatprogpath@
mandir = @mandir@
mkdir_p = @mkdir_p@
@@ -198,7 +198,7 @@
target_alias = @target_alias@
top_builddir = @top_builddir@
top_srcdir = @top_srcdir@
-noinst_PROGRAMS = @makedatprog_target@
+noinst_PROGRAMS = @makedatprog_target@$(EXEEXT)
makedatprog_SOURCES = makedatprog.c
makedatprog_DEPENDENCIES = @dblibrary@
makedatprog_LDADD = @dblibrary@
--- courier-imap-4.4.1.20080920/Makefile.in 2008-09-20 14:48:46.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/Makefile.in 2008-10-21 20:59:45.156250000 +0200
@@ -247,9 +247,9 @@
CLEANFILES = $(databin_SCRIPTS) $(man_MANS) $(sysconf_DATA) $(sbin_SCRIPTS)
databindir = $(datadir)
databin_SCRIPTS = mkimapdcert mkpop3dcert
-binPROGRAMS = imapd pop3d maildirmake maildiracl deliverquota maildirkw
-sbinPROGRAMS = imaplogin pop3login
-libexecPROGRAMS = makedatprog couriertcpd
+binPROGRAMS = imapd$(EXEEXT) pop3d$(EXEEXT) maildirmake$(EXEEXT) maildiracl$(EXEEXT) deliverquota$(EXEEXT) maildirkw$(EXEEXT)
+sbinPROGRAMS = imaplogin$(EXEEXT) pop3login$(EXEEXT)
+libexecPROGRAMS = makedatprog$(EXEEXT) couriertcpd$(EXEEXT)
bin_PROGRAMS = @binPROGRAMS_exec@
sbin_PROGRAMS = @sbinPROGRAMS_exec@
libexec_PROGRAMS = @libexecPROGRAMS_exec@
--- courier-imap-4.4.1.20080920/imapd.rc.in 2008-05-04 15:12:47.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/imapd.rc.in 2008-10-26 10:17:51.359375000 +0100
@@ -22,6 +22,15 @@
exit 1
fi

+# Location for some shared libraries (cygcourierauth*.dll) from authlib
+AUTHLIBDIR=/usr/local
+if test ! -f $AUTHLIBDIR/lib/bin/cygcourierauth.dll
+then
+ echo "$AUTHLIBDIR/lib/bin/cygcourierauth.dll not found."
+ exit 1
+fi
+export PATH=$PATH:$AUTHLIBDIR/lib/bin
+
TLS_CACHEFILE=""
. @sysconfdir@/imapd-ssl
. @sysconfdir@/imapd
@@ -35,7 +44,7 @@

umask $IMAP_UMASK
@ULIMIT@ $IMAP_ULIMITD
- @SETENV@ -i @SHELL@ -c " set -a ;
+ @SHELL@ -c " set -a ;
prefix=@prefix@ ;
exec_prefix=@exec_prefix@ ;
bindir=@bindir@ ;
--- courier-imap-4.4.1.20080920/pop3d.rc.in 2008-05-04 15:12:47.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/pop3d.rc.in 2008-10-26 10:18:02.187500000 +0100
@@ -22,12 +22,21 @@
exit 1
fi

+# Location for some shared libraries (cygcourierauth*.dll) from authlib
+AUTHLIBDIR=/usr/local
+if test ! -f $AUTHLIBDIR/lib/bin/cygcourierauth.dll
+then
+ echo "$AUTHLIBDIR/lib/bin/cygcourierauth.dll not found."
+ exit 1
+fi
+export PATH=$PATH:$AUTHLIBDIR/lib/bin
+
. @sysconfdir@/pop3d-ssl
. @sysconfdir@/pop3d

case $1 in
start)
- @SETENV@ -i @SHELL@ -c " set -a ;
+ @SHELL@ -c " set -a ;
prefix=@prefix@ ;
exec_prefix=@exec_prefix@ ;
bindir=@bindir@ ;
</source>

Install the patch

<source lang="bash">
$ patch -Np1 < courier-imap-4.4.1-cygwin.patch
</source>

Configure the package

<source lang="bash">
$ ./configure --disable-root-check --with-waitfunc=wait
</source>

Take a walk or some coffee... and when ready build the package and install it.

<source lang="bash">
$ make
$ make install
$ make install-configure
</source>

=Configuring Courier=

== Configuration Files and Scripts ==

If not present yet, add <tt>/usr/local/bin</tt> and <tt>/usr/local/sbin</tt> to your path (ideally in <tt>/etc/bash.bashrc</tt>):

<source lang="bash">
PATH=/usr/local/bin:/usr/local/sbin:$PATH
</source>

Create configuration file for '''authdaemond''' (<tt>/usr/local/etc/authlib/authdaemonrc</tt>)

$ cp /usr/local/etc/authlib/authdaemonrc.dist /usr/local/etc/authlib/authdaemonrc

And edit its content to change number of daemons (I use three):

<source lang=text>
# The number of daemon processes that are started.
daemons=3
</source>

Edit the file <tt>/usr/lib/courier-imap/etc/imapd</tt>, taking care to adapt '''IMAP_ULIMITD''' accordingly:
<source lang="bash">
ADDRESS=127.0.0.1 #Typ. use 0 or 127.0.0.1
...
IMAP_ULIMITD=2097152 #Set same value as returned by 'ulimit -v' or imapd.rc will complain
</source>

Create link to authlib and imap daemon in <tt>/etc/rc.d/init.d</tt>
<source lang="bash">
ln -s /usr/local/sbin/authdaemond /etc/rc.d/init.d/courier-authdaemon
ln -s /usr/lib/courier-imap/libexec/imapd.rc /etc/rc.d/init.d/courier-imap
</source>

== IMAP Accounts ==

User accounts and settings are managed by ''userdb''. <tt>[http://linux.die.net/man/8/makeuserdb man makeuserdb]</tt> for details.

First create user mailbox (replace ''username'' with user account name) with <tt>maildirmake</tt>. Repeat for all users:
<source lang="bash">
mkdir -p /home/username #if don't exist yet...
cd /home/username
for i in "" .Drafts .Sent .Trash .Templates; do /usr/lib/courier-imap/bin/maildirmake ~username/MailDir/$i; done
# Create also default Maildir for new users
for i in "" .Drafts .Sent .Trash .Templates; do /usr/lib/courier-imap/bin/maildirmake /etc/skel/MailDir/$i; done
</source>

Next create the user authentication database <tt>/usr/local/etc/authlib/userdb</tt>:
<source lang="bash">
touch /usr/local/etc/authlib/userdb
chmod 700 /usr/local/etc/authlib/userdb
#Repeat for each username:
pw2userdb | grep "^username">>/usr/local/etc/authlib/userdb # Fill userdb from /etc/passwd user entry
userdb username set mail=~username/MailDir # Specify user MailDir location
userdbpw | userdb username set systempw # Set user account password
...
makeuserdb # Create binary database
</source>

You should now have the following files created:

* /usr/local/etc/authlib/userdb.dat
* /usr/local/etc/authlib/userdbshadow.dat

==Configuration Summary==

$ /usr/local/bin/courierauthconfig.exe --version
$ /usr/local/bin/courierauthconfig.exe --ldflags
$ /usr/local/bin/courierauthconfig.exe --cppflags
$ /usr/local/bin/courierauthconfig.exe --configfiles

=Running=

Install and configure syslogd. Run the [http://www.cygwin.com/setup.exe Cygwin setup] program. Locate, select and install inetutils (contains syslogd). Configure syslogd with the following command. It will create the /etc/syslogd.conf file and install a windows service using cygrunsrv

$ syslogd-config
$ net start syslogd

Start the authlib daemon with the following command:

$ /usr/local/sbin/authdaemond start

Stop the daemon with the following command:

$ /usr/local/sbin/authdaemond stop

You'll find log messages in /var/log/messages (default target file as specified in /etc/syslogd.conf)

Syslogd service and authdaemond should be started. See above.

Start the imap daemon with the following command:

$ /usr/lib/courier-imap/libexec/imapd.rc start

Stop the daemon with the following command:

$ /usr/lib/courier-imap/libexec/imapd.rc stop

You'll find log messages in /var/log/messages (default target file as specified in /etc/syslogd.conf)

=Testing=

$ /usr/local/sbin/authtest username CCCCCC
Authentication succeeded.

Authenticated: username (uid 1001, gid 10545)
Home Directory: /home/username
Maildir: /home/username/Maildir
Quota: (none)
Encrypted Password: XXXXXX
Cleartext Password: CCCCCC
Options: (none)

where username is your user name and XXXXXX is your encrypted password as provided in ''/usr/local/etc/authlib/userdb''. CCCCCC is your password in clear.

When authdaemond and imapd are running you can do a simple login test:

$ telnet localhost 143

and type the login command

a login <username> <password>

the server should reply

a OK LOGIN Ok.

to which you can reply with a logout

a logout

the session is terminated by the server

* BYE Courier-IMAP server shutting down
a OK LOGOUT completed
Connection closed by foreign host.

'''Hint:''' if the connection is immediately closed by foreign host it probably means that ''imapd'' does not have access to all required DLLs. Check your PATH.

==Troubleshooting==

Should you have a doubt on the execution of a command (ie: no output) you can check if the application is able to load required DLL. This error is not reported. Using ''strace'', windows will complain about DLL not found, if any. To solve this problem you may need to add some path in your PATH. For example, in order to test authlib with authtest:

$ strace /usr/local/sbin/authtest
# This will open a dialog box to complain about DLL.
$ export PATH=$PATH:/usr/local/lib/bin
$ /usr/local/sbin/authtest ausername

=Installing as a Windows Service=

==Startup Script==

If everything works fine you can use the following [{{#file:imapd}} startup script] to launch or stop imap and authlib daemons. It can be saved in /usr/local/sbin.

<source lang=bash>
#! /bin/sh
#
# Courier imap daemon startup script.

authlib=/usr/local/sbin/authdaemond
imapd=/usr/lib/courier-imap/libexec/imapd.rc

test ! -f $authlib && echo "File not found: $authlib" && exit 1;
test ! -f $imapd && echo "File not found: $imapd" && exit 1;

case $1 in
start)
$authlib start
$imapd start
;;
stop)
$imapd stop
$authlib stop
;;
*)
echo "Usage: $0 <start|stop>"
;;
esac
</source>

==Cygwin Service==

Create the [{{#file:imapd-srv.sh}} service script]

<source lang=bash>
#!/bin/sh
# File: imapd-service.sh
# Purpose: Courier imap daemon service script.
#
# NOTE: This script must be excutable by SYSTEM
username=username # <<<< Change me !!!
pidsleep=''

# Function to handle QUIT signal.
# Stop imapd daemon and kill background sleep.
function handleQuit
{
echo "STOPPING: /usr/local/sbin/imapd"
su $username -c "/usr/local/sbin/imapd stop"
test -n "$pidsleep" && kill -9 $pidsleep
exit
}

# Interruptible sleep
# Note: sleep is an external command (not builtin command), hence
# not interruptible by the QUIT signal.
function intsleep
{
sleep $1 &
pidsleep=$!
wait $pidsleep
}

echo "STARTING: /usr/local/sbin/imapd as $username at "`date`
su $username -c "/usr/local/sbin/imapd start"

trap "handleQuit" SIGQUIT

echo "WAITING: "$$
while true; do intsleep 1d; done
</source>

Change username and save it to your favorite location, says /usr/local/sbin. The service script must be executable by SYSTEM, ie: the user that run the windows services.

$ chmod +rx /usr/local/sbin/imapd-service.sh

Install the new service with the following command.

$ cygrunsrv --install imapd --desc "Courier IMAP daemon" --disp "CYGWIN imapd" \
--path /usr/local/sbin/imapd-service.sh --termsig QUIT --type auto --shutdown

Finally, just start the service

$ cygrunsrv --start imapd
or
$ net start imapd

You should now have couple of processes related to courier-imap

$ ps -a

'''NOTES''':
* If the service could not be started, it's probably because the service script is not executable by SYSTEM

$ net start imapd
The CYGWIN imapd service is starting.
The CYGWIN imapd service could not be started.

The service did not report an error.

More help is available by typing NET HELPMSG 3534.

* The combination of background sleep and builtin wait command is used to make the waiting statment interruptible. The original script found on the net, was simply using a 'sleep 1' which turned to consume ~10% of the CPU.

=Other Interesting Packages=

==Maildir==

'''maildirmake''' is part of the courier-imap package

$ /usr/lib/courier-imap/bin/maildirmake

==Mairix - index and search mail folders==
Run the [http://www.cygwin.com/setup.exe Cygwin setup] program. Locate, select and install the package mairix.

Create the configuration file in your home directory: ~/.mairixrc

base=/home/username
maildir=Maildir...
omit=Maildir/.Mairix**
mfolder=Maildir/.Mairix
database=/home/username/.mairix_database

The create the mairix index

$ mairix
and
$ mairix --fast-index # on daily base

See [[Mail_Tips]] for additional details.

==Offlineimap==
Requires python, so run the [http://www.cygwin.com/setup.exe Cygwin setup] program. Locate, select and install the package python.

Download offlineimap [http://software.complete.org/software/projects/list_files/offlineimap here]. I use [http://software.complete.org/software/versions/download/242?attachment_id=334 version 6.0.3]. Extract files in you favorite sandbox.

$ tar xvfz offlineimap_6.0.3.tar.gz
$ cd offlineimap

Apply the following [{{#file:offlineimap_6.0.3-cygwin.patch}} patch] to correct the following issues:

* [http://en.wikipedia.org/wiki/Maildir#Windows_software Suffix separator] character colon (:) is not allowed in Windows. Courier-imap for cygwin is using (!)
* While used by courier-imap for cygwin Maildir mail files should not use CRLF, rather only LF. The issue appears only in mail with attachment, that is the raw message is not interpreted and displayed as it in the mail body (under Thunderbird).

<source lang=diff>
--- offlineimap.conf 2008-08-13 20:56:04.000000000 +0200
+++ offlineimap-cygwin.conf 2008-11-06 10:44:31.066750700 +0100
@@ -215,6 +215,19 @@

restoreatime = no

+# Cygwin/Windows users may have issues with the Maildir mail filename
+# containing the suffix separator character (:). In this case, they
+# could change the suffix separator character to whatever required.
+# For example, Courier-IMAP for Cygwin is using (!)
+
+# suffixsep = !
+
+# Cygwin/Windows users may need to create Maildir mail filename in
+# binary mode. In this case, set the 'filemode' to 'binary'. Default is
+# 'text'. Courier-IMAP for Cygwin works better with binary mode.
+
+# filemode = binary
+
[Repository RemoteExample]

# And this is the remote repository. We only support IMAP or Gmail here.
--- offlineimap/folder/Maildir.py 2008-08-13 20:55:48.000000000 +0200
+++ offlineimap/folder/Maildir-cygwin.py 2008-11-06 10:44:05.344202700 +0100
@@ -23,7 +23,6 @@
import os.path, os, re, time, socket, md5

uidmatchre = re.compile(',U=(\d+)')
-flagmatchre = re.compile(':.*2,([A-Z]+)')

timeseq = 0
lasttime = long(0)
@@ -54,6 +53,12 @@
self.messagelist = None
self.repository = repository
self.accountname = accountname
+ self.filemode = {
+ 'binary': 'wb',
+ 'text': 'wt',
+ }[repository.getconf("filemode", 'text')]
+ self.suffixsep = repository.getconf("suffixsep",':')
+ self.flagmatchre = re.compile('%s.*2,([A-Z]+)' % self.suffixsep)
BaseFolder.__init__(self)

def getaccountname(self):
@@ -102,7 +107,7 @@
nouidcounter -= 1
else:
uid = long(uidmatch.group(1))
- flagmatch = flagmatchre.search(messagename)
+ flagmatch = self.flagmatchre.search(messagename)
flags = []
if flagmatch:
flags = [x for x in flagmatch.group(1)]
@@ -180,7 +185,7 @@
break
tmpmessagename = messagename.split(',')[0]
ui.debug('maildir', 'savemessage: using temporary name %s' % tmpmessagename)
- file = open(os.path.join(tmpdir, tmpmessagename), "wt")
+ file = open(os.path.join(tmpdir, tmpmessagename), self.filemode)
file.write(content)

# Make sure the data hits the disk
@@ -226,11 +231,11 @@
newpath = os.path.join(self.getfullname(), 'cur')
else:
newpath = os.path.join(self.getfullname(), 'new')
- infostr = ':'
- infomatch = re.search('(:.*)$', newname)
+ infostr = self.suffixsep
+ infomatch = re.search('(%s.*)$' % self.suffixsep, newname)
if infomatch: # If the info string is present..
infostr = infomatch.group(1)
- newname = newname.split(':')[0] # Strip off the info string.
+ newname = newname.split(self.suffixsep)[0] # Strip off the info string.
infostr = re.sub('2,[A-Z]*', '', infostr)
flags.sort()
infostr += '2,' + ''.join(flags)
</source>

$ patch -Np0 < offlineimap_6.0.3-cygwin.patch

Install offlineimap

$ python setup.py install

Configure offlineimap as described in [[Offlineimap]] and launch offlineimap. '''CAUTION:''' Add the NEW suffixsep and filemode options in the Maildir repository section of the configuration file.

$ offlineimap
or
$ offlineimap -1 -o -u Noninteractive.Basic
or
$ offlineimap -c /home/username/offlineimap.conf

Here is a simple [{{#file:offlineimap.conf}} offlineimap configuration file].

<source lang=text>
[general]
metadata = /home/username/.offlineimap
accounts = Test
maxsyncaccounts = 1
ui = Curses.Blinkenlights, TTY.TTYUI,
Noninteractive.Basic, Noninteractive.Quiet
ignore-readonly = no

[mbnames]
enabled = no

[ui.Curses.Blinkenlights]
statuschar = .

[Account Test]
localrepository = LocalExample
remoterepository = RemoteExample

[Repository LocalExample]
type = Maildir
localfolders = /home/username/Maildir
sep = .
restoreatime = no

suffixsep = ! # <<< NEW
filemode = binary # <<< NEW

[Repository RemoteExample]
type = IMAP
remotehost = imapserver
ssl = no
remoteport = 143
remoteuser = username
remotepass = XXXXXX
maxconnections = 1
holdconnectionopen = no
nametrans = lambda foldername: re.sub('^INBOX\.*', '.', foldername)
folderfilter = lambda foldername: not re.search('(SPAM$|Mairix)', foldername)
</source>

Courier-Cygwin

2008-11-07T09:58:08Z

Fuujuhi: /* Configuring Courier */

=Prerequisites=

Courier-IMAP requires the installation of the [http://www.courier-mta.org/authlib/ Courier Authentication Library] and the following cygwin tools and packages

* <tt>patch</tt>, <tt>tar</tt>, <tt>make</tt>, <tt>gcc</tt>
* <tt>crypt</tt>
* <tt>libgdbm-devel</tt>
* <tt>libtool</tt>
* <tt>inetutils</tt>
* <tt>cygrunsrv</tt>

Run the [http://www.cygwin.com/setup.exe Cygwin setup] program. Locate, select and install the required packages.

=Building and Installing Courier Authentication Library=

Installation of ''Courier Authentication Library'' using the module '''userdb''' to manage mail accounts. All other modules are disabled.

Download and untar [http://www.courier-mta.org/download.php#authlib courier-authlib] in your favorite sandbox.
Version used: [http://prdownloads.sourceforge.net/courier/courier-authlib-0.61.0.tar.bz2 0.61.0]

<source lang="bash">
$ tar -xvjf courier-authlib-0.61.0.tar.bz2
$ cd courier-authlib-0.61.0
</source>

Apply the [{{#file:courier-authlib-0.61.0-cygwin.patch}} patch] before configuration to correct following issues:

# incorrect usage of EXEEXT in ''makedat/Makefile''
# add libtool flag (-no-undefined) required for DLL creation ([http://lists.cairographics.org/archives/cairo/2004-April/001125.html])
# add missing dependencies
# add PATH in start script to access new DLL (installed in a different directory)

<source lang=diff>
--- courier-authlib-0.61.0/makedat/Makefile.in 2008-05-24 16:21:09.000000000 +0200
+++ courier-authlib-0.61.0-cygwin/makedat/Makefile.in 2008-10-21 16:02:38.709166700 +0200
@@ -182,7 +182,7 @@
libexecdir = @libexecdir@
localedir = @localedir@
localstatedir = @localstatedir@
-makedatprog_target = @makedatprog_target@
+makedatprog_target = @makedatprog_target@$(EXEEXT)
makedatprogpath = @makedatprogpath@
mandir = @mandir@
mkdir_p = @mkdir_p@
@@ -198,7 +198,7 @@
target_alias = @target_alias@
top_builddir = @top_builddir@
top_srcdir = @top_srcdir@
-noinst_PROGRAMS = @makedatprog_target@
+noinst_PROGRAMS = @makedatprog_target@$(EXEEXT)
makedatprog_SOURCES = makedatprog.c
makedatprog_DEPENDENCIES = @dblibrary@
makedatprog_LDADD = @dblibrary@
--- courier-authlib-0.61.0/Makefile.in 2008-07-12 21:41:08.000000000 +0200
+++ courier-authlib-0.61.0-cygwin/Makefile.in 2008-10-23 22:59:03.843750000 +0200
@@ -213,9 +213,10 @@
LTCOMPILE = $(LIBTOOL) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) \
--mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) \
$(AM_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS)
-CCLD = $(CC)
+CCLD = $(CC) -no-undefined
+CCLDEXE = $(CC)
LINK = $(LIBTOOL) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) \
- --mode=link $(CCLD) $(AM_CFLAGS) $(CFLAGS) $(AM_LDFLAGS) \
+ --mode=link $(CCLDEXE) $(AM_CFLAGS) $(CFLAGS) $(AM_LDFLAGS) \
$(LDFLAGS) -o $@
SOURCES = $(libauthcustom_la_SOURCES) $(libauthldap_la_SOURCES) \
$(libauthmysql_la_SOURCES) $(libauthpam_la_SOURCES) \
@@ -452,9 +453,9 @@
README.authdebug.html

DISTCLEANFILES = dbobj.config README_authlib.html
-commonlibdep = libcourierauthcommon.la
+commonlibdep = libcourierauthcommon.la libcourierauth.la
commonldflags = -module -rpath $(pkglibdir) -export-symbols-regex 'courier_auth.*_init' -avoid-version
-commonlibadd = libcourierauthcommon.la
+commonlibadd = libcourierauthcommon.la libcourierauth.la
libcourierauthcommon_t = @CRYPTLIBS@
libcourierauthcommon_la_SOURCES = \
auth.h courierauth.h \
--- courier-authlib-0.61.0/authdaemond.in 2005-07-05 14:25:08.000000000 +0200
+++ courier-authlib-0.61.0-cygwin/authdaemond.in 2008-10-25 12:03:32.140625000 +0200
@@ -15,4 +15,10 @@
set -a
. @authdaemonrc@

+# Some shared libraries (DLL) are installed in @libdir@/bin
+# instead of @libdir@/@PACKAGE@
+# Setting LD_LIBRARY_PATH at runtime or LR_RUN_PATH at linktime doesn't
+# work in cygwin, only setting PATH works.
+export PATH=$PATH:@libdir@/bin
+
exec ${sbindir}/courierlogger -pid=@authdaemonvar@/pid $LOGGEROPTS -$1 @libexecdir@/courier-authlib/authdaemond
</source>

Install the patch

<source lang="bash">
$ patch -Np1 < courier-authlib-0.61.0-cygwin.patch
</source>

Configure the package without most authentication modules, keeping only '''userdb'''. Replace ''mailuser'' by an existing user name (in your /etc/passwd file). I used my own username.

<source lang="bash">
$ ./configure --disable-root-check --with-waitfunc=wait --without-authpam --without-authldap --without-authpwd \
--without-authshadow --without-authcustom --without-authpipe --without-authmysql --without-authpgsql --with-mailuser=mailuser \
--with-mailgroup=mkgroup-l-d
</source>

Take a long pause... and when ready build, check the result and install the libraries.

<source lang="bash">
$ make
$ make check
$ make install
</source>

'''NOTES''': list of related topics found during investigation for compilation
* authldap: configure: error: -lresolve is needed for res_query... http://www.cygwin.com/ml/cygwin/2005-04/msg00419.html

=Building and Installing Courier-IMAP=

Download and untar package [http://www.courier-mta.org/download.php#imap courier-imap] in your favorite sandbox.
Version used: [http://prdownloads.sourceforge.net/courier/courier-imap-4.4.1.tar.bz2 4.4.1.20080920]

<source lang="bash">
$ tar -xvjf courier-imap-4.4.1.tar.bz2
$ cd courier-imap-4.4.1
</source>

Apply the [{{#file:courier-imap-4.4.1-cygwin.patch}} patch] before configuration to correct following issues:

# incorrect usage of EXEEXT in ''makedat/Makefile''
# incorrect usage of EXTEXT in main ''Makefile''
# remove usage of /usr/lib/env in start/stop script (/usr/lib/env does not seem to work under Cygwin)

<source lang=diff>
--- courier-imap-4.4.1.20080920/makedat/Makefile.in 2008-08-24 19:52:51.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/makedat/Makefile.in 2008-10-21 20:43:27.500000000 +0200
@@ -182,7 +182,7 @@
libexecdir = @libexecdir@
localedir = @localedir@
localstatedir = @localstatedir@
-makedatprog_target = @makedatprog_target@
+makedatprog_target = @makedatprog_target@$(EXEEXT)
makedatprogpath = @makedatprogpath@
mandir = @mandir@
mkdir_p = @mkdir_p@
@@ -198,7 +198,7 @@
target_alias = @target_alias@
top_builddir = @top_builddir@
top_srcdir = @top_srcdir@
-noinst_PROGRAMS = @makedatprog_target@
+noinst_PROGRAMS = @makedatprog_target@$(EXEEXT)
makedatprog_SOURCES = makedatprog.c
makedatprog_DEPENDENCIES = @dblibrary@
makedatprog_LDADD = @dblibrary@
--- courier-imap-4.4.1.20080920/Makefile.in 2008-09-20 14:48:46.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/Makefile.in 2008-10-21 20:59:45.156250000 +0200
@@ -247,9 +247,9 @@
CLEANFILES = $(databin_SCRIPTS) $(man_MANS) $(sysconf_DATA) $(sbin_SCRIPTS)
databindir = $(datadir)
databin_SCRIPTS = mkimapdcert mkpop3dcert
-binPROGRAMS = imapd pop3d maildirmake maildiracl deliverquota maildirkw
-sbinPROGRAMS = imaplogin pop3login
-libexecPROGRAMS = makedatprog couriertcpd
+binPROGRAMS = imapd$(EXEEXT) pop3d$(EXEEXT) maildirmake$(EXEEXT) maildiracl$(EXEEXT) deliverquota$(EXEEXT) maildirkw$(EXEEXT)
+sbinPROGRAMS = imaplogin$(EXEEXT) pop3login$(EXEEXT)
+libexecPROGRAMS = makedatprog$(EXEEXT) couriertcpd$(EXEEXT)
bin_PROGRAMS = @binPROGRAMS_exec@
sbin_PROGRAMS = @sbinPROGRAMS_exec@
libexec_PROGRAMS = @libexecPROGRAMS_exec@
--- courier-imap-4.4.1.20080920/imapd.rc.in 2008-05-04 15:12:47.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/imapd.rc.in 2008-10-26 10:17:51.359375000 +0100
@@ -22,6 +22,15 @@
exit 1
fi

+# Location for some shared libraries (cygcourierauth*.dll) from authlib
+AUTHLIBDIR=/usr/local
+if test ! -f $AUTHLIBDIR/lib/bin/cygcourierauth.dll
+then
+ echo "$AUTHLIBDIR/lib/bin/cygcourierauth.dll not found."
+ exit 1
+fi
+export PATH=$PATH:$AUTHLIBDIR/lib/bin
+
TLS_CACHEFILE=""
. @sysconfdir@/imapd-ssl
. @sysconfdir@/imapd
@@ -35,7 +44,7 @@

umask $IMAP_UMASK
@ULIMIT@ $IMAP_ULIMITD
- @SETENV@ -i @SHELL@ -c " set -a ;
+ @SHELL@ -c " set -a ;
prefix=@prefix@ ;
exec_prefix=@exec_prefix@ ;
bindir=@bindir@ ;
--- courier-imap-4.4.1.20080920/pop3d.rc.in 2008-05-04 15:12:47.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/pop3d.rc.in 2008-10-26 10:18:02.187500000 +0100
@@ -22,12 +22,21 @@
exit 1
fi

+# Location for some shared libraries (cygcourierauth*.dll) from authlib
+AUTHLIBDIR=/usr/local
+if test ! -f $AUTHLIBDIR/lib/bin/cygcourierauth.dll
+then
+ echo "$AUTHLIBDIR/lib/bin/cygcourierauth.dll not found."
+ exit 1
+fi
+export PATH=$PATH:$AUTHLIBDIR/lib/bin
+
. @sysconfdir@/pop3d-ssl
. @sysconfdir@/pop3d

case $1 in
start)
- @SETENV@ -i @SHELL@ -c " set -a ;
+ @SHELL@ -c " set -a ;
prefix=@prefix@ ;
exec_prefix=@exec_prefix@ ;
bindir=@bindir@ ;
</source>

Install the patch

<source lang="bash">
$ patch -Np1 < courier-imap-4.4.1-cygwin.patch
</source>

Configure the package

<source lang="bash">
$ ./configure --disable-root-check --with-waitfunc=wait
</source>

Take a walk or some coffee... and when ready build the package and install it.

<source lang="bash">
$ make
$ make install
$ make install-configure
</source>

=Configuring Courier=

If not present yet, add <tt>/usr/local/bin</tt> and <tt>/usr/local/sbin</tt> to your path (ideally in <tt>/etc/bash.bashrc</tt>):

<source lang="bash">
PATH=/usr/local/bin:/usr/local/sbin:$PATH
</source>

==courier-authlib==

Create configuration file for '''authdaemond''' (<tt>/usr/local/etc/authlib/authdaemonrc</tt>)

$ cp /usr/local/etc/authlib/authdaemonrc.dist /usr/local/etc/authlib/authdaemonrc

And edit its content to change number of daemons (I use three):

<source lang=text>
# The number of daemon processes that are started.
daemons=3
</source>

==courier-imap==

Edit the file <tt>/usr/lib/courier-imap/etc/imapd</tt>, taking care to adapt '''IMAP_ULIMITD''' accordingly:
<source lang="bash">
ADDRESS=127.0.0.1 #Typ. use 0 or 127.0.0.1
...
IMAP_ULIMITD=2097152 #Set same value as returned by 'ulimit -v' or imapd.rc will complain
</source>

==Start-up Scripts==

Create link to authlib and imap daemon in <tt>/etc/rc.d/init.d</tt>
<source lang="bash">
ln -s /usr/local/sbin/authdaemond /etc/rc.d/init.d/courier-authdaemon
ln -s /usr/lib/courier-imap/libexec/imapd.rc /etc/rc.d/init.d/courier-imap
</source>

==IMAP accounts==

User accounts and settings are managed by ''userdb''. <tt>[http://linux.die.net/man/8/makeuserdb man makeuserdb]</tt> for details.

First create user mailbox (replace ''username'' with user account name) with <tt>maildirmake</tt>. Repeat for all users:
<source lang="bash">
mkdir -p /home/username #if don't exist yet...
cd /home/username
for i in "" .Drafts .Sent .Trash .Templates; do /usr/lib/courier-imap/bin/maildirmake ~username/MailDir/$i; done
# Create also default Maildir for new users
for i in "" .Drafts .Sent .Trash .Templates; do /usr/lib/courier-imap/bin/maildirmake /etc/skel/MailDir/$i; done
</source>

Next create the user authentication database <tt>/usr/local/etc/authlib/userdb</tt>:
<source lang="bash">
touch /usr/local/etc/authlib/userdb
chmod 700 /usr/local/etc/authlib/userdb
#Repeat for each username:
pw2userdb | grep "^username">>/usr/local/etc/authlib/userdb # Fill userdb from /etc/passwd user entry
userdb username set mail=~username/MailDir # Specify user MailDir location
userdbpw | userdb username set systempw # Set user account password
...
makeuserdb # Create binary database
</source>

You should now have the following files created:

* /usr/local/etc/authlib/userdb.dat
* /usr/local/etc/authlib/userdbshadow.dat

==Configuration Summary==

$ /usr/local/bin/courierauthconfig.exe --version
$ /usr/local/bin/courierauthconfig.exe --ldflags
$ /usr/local/bin/courierauthconfig.exe --cppflags
$ /usr/local/bin/courierauthconfig.exe --configfiles

=Running=

Install and configure syslogd. Run the [http://www.cygwin.com/setup.exe Cygwin setup] program. Locate, select and install inetutils (contains syslogd). Configure syslogd with the following command. It will create the /etc/syslogd.conf file and install a windows service using cygrunsrv

$ syslogd-config
$ net start syslogd

Start the authlib daemon with the following command:

$ /usr/local/sbin/authdaemond start

Stop the daemon with the following command:

$ /usr/local/sbin/authdaemond stop

You'll find log messages in /var/log/messages (default target file as specified in /etc/syslogd.conf)

Syslogd service and authdaemond should be started. See above.

Start the imap daemon with the following command:

$ /usr/lib/courier-imap/libexec/imapd.rc start

Stop the daemon with the following command:

$ /usr/lib/courier-imap/libexec/imapd.rc stop

You'll find log messages in /var/log/messages (default target file as specified in /etc/syslogd.conf)

=Testing=

$ /usr/local/sbin/authtest username CCCCCC
Authentication succeeded.

Authenticated: username (uid 1001, gid 10545)
Home Directory: /home/username
Maildir: /home/username/Maildir
Quota: (none)
Encrypted Password: XXXXXX
Cleartext Password: CCCCCC
Options: (none)

where username is your user name and XXXXXX is your encrypted password as provided in ''/usr/local/etc/authlib/userdb''. CCCCCC is your password in clear.

When authdaemond and imapd are running you can do a simple login test:

$ telnet localhost 143

and type the login command

a login <username> <password>

the server should reply

a OK LOGIN Ok.

to which you can reply with a logout

a logout

the session is terminated by the server

* BYE Courier-IMAP server shutting down
a OK LOGOUT completed
Connection closed by foreign host.

'''Hint:''' if the connection is immediately closed by foreign host it probably means that ''imapd'' does not have access to all required DLLs. Check your PATH.

==Troubleshooting==

Should you have a doubt on the execution of a command (ie: no output) you can check if the application is able to load required DLL. This error is not reported. Using ''strace'', windows will complain about DLL not found, if any. To solve this problem you may need to add some path in your PATH. For example, in order to test authlib with authtest:

$ strace /usr/local/sbin/authtest
# This will open a dialog box to complain about DLL.
$ export PATH=$PATH:/usr/local/lib/bin
$ /usr/local/sbin/authtest ausername

=Installing as a Windows Service=

==Startup Script==

If everything works fine you can use the following [{{#file:imapd}} startup script] to launch or stop imap and authlib daemons. It can be saved in /usr/local/sbin.

<source lang=bash>
#! /bin/sh
#
# Courier imap daemon startup script.

authlib=/usr/local/sbin/authdaemond
imapd=/usr/lib/courier-imap/libexec/imapd.rc

test ! -f $authlib && echo "File not found: $authlib" && exit 1;
test ! -f $imapd && echo "File not found: $imapd" && exit 1;

case $1 in
start)
$authlib start
$imapd start
;;
stop)
$imapd stop
$authlib stop
;;
*)
echo "Usage: $0 <start|stop>"
;;
esac
</source>

==Cygwin Service==

Create the [{{#file:imapd-srv.sh}} service script]

<source lang=bash>
#!/bin/sh
# File: imapd-service.sh
# Purpose: Courier imap daemon service script.
#
# NOTE: This script must be excutable by SYSTEM
username=username # <<<< Change me !!!
pidsleep=''

# Function to handle QUIT signal.
# Stop imapd daemon and kill background sleep.
function handleQuit
{
echo "STOPPING: /usr/local/sbin/imapd"
su $username -c "/usr/local/sbin/imapd stop"
test -n "$pidsleep" && kill -9 $pidsleep
exit
}

# Interruptible sleep
# Note: sleep is an external command (not builtin command), hence
# not interruptible by the QUIT signal.
function intsleep
{
sleep $1 &
pidsleep=$!
wait $pidsleep
}

echo "STARTING: /usr/local/sbin/imapd as $username at "`date`
su $username -c "/usr/local/sbin/imapd start"

trap "handleQuit" SIGQUIT

echo "WAITING: "$$
while true; do intsleep 1d; done
</source>

Change username and save it to your favorite location, says /usr/local/sbin. The service script must be executable by SYSTEM, ie: the user that run the windows services.

$ chmod +rx /usr/local/sbin/imapd-service.sh

Install the new service with the following command.

$ cygrunsrv --install imapd --desc "Courier IMAP daemon" --disp "CYGWIN imapd" \
--path /usr/local/sbin/imapd-service.sh --termsig QUIT --type auto --shutdown

Finally, just start the service

$ cygrunsrv --start imapd
or
$ net start imapd

You should now have couple of processes related to courier-imap

$ ps -a

'''NOTES''':
* If the service could not be started, it's probably because the service script is not executable by SYSTEM

$ net start imapd
The CYGWIN imapd service is starting.
The CYGWIN imapd service could not be started.

The service did not report an error.

More help is available by typing NET HELPMSG 3534.

* The combination of background sleep and builtin wait command is used to make the waiting statment interruptible. The original script found on the net, was simply using a 'sleep 1' which turned to consume ~10% of the CPU.

=Other Interesting Packages=

==Maildir==

'''maildirmake''' is part of the courier-imap package

$ /usr/lib/courier-imap/bin/maildirmake

==Mairix - index and search mail folders==
Run the [http://www.cygwin.com/setup.exe Cygwin setup] program. Locate, select and install the package mairix.

Create the configuration file in your home directory: ~/.mairixrc

base=/home/username
maildir=Maildir...
omit=Maildir/.Mairix**
mfolder=Maildir/.Mairix
database=/home/username/.mairix_database

The create the mairix index

$ mairix
and
$ mairix --fast-index # on daily base

See [[Mail_Tips]] for additional details.

==Offlineimap==
Requires python, so run the [http://www.cygwin.com/setup.exe Cygwin setup] program. Locate, select and install the package python.

Download offlineimap [http://software.complete.org/software/projects/list_files/offlineimap here]. I use [http://software.complete.org/software/versions/download/242?attachment_id=334 version 6.0.3]. Extract files in you favorite sandbox.

$ tar xvfz offlineimap_6.0.3.tar.gz
$ cd offlineimap

Apply the following [{{#file:offlineimap_6.0.3-cygwin.patch}} patch] to correct the following issues:

* [http://en.wikipedia.org/wiki/Maildir#Windows_software Suffix separator] character colon (:) is not allowed in Windows. Courier-imap for cygwin is using (!)
* While used by courier-imap for cygwin Maildir mail files should not use CRLF, rather only LF. The issue appears only in mail with attachment, that is the raw message is not interpreted and displayed as it in the mail body (under Thunderbird).

<source lang=diff>
--- offlineimap.conf 2008-08-13 20:56:04.000000000 +0200
+++ offlineimap-cygwin.conf 2008-11-06 10:44:31.066750700 +0100
@@ -215,6 +215,19 @@

restoreatime = no

+# Cygwin/Windows users may have issues with the Maildir mail filename
+# containing the suffix separator character (:). In this case, they
+# could change the suffix separator character to whatever required.
+# For example, Courier-IMAP for Cygwin is using (!)
+
+# suffixsep = !
+
+# Cygwin/Windows users may need to create Maildir mail filename in
+# binary mode. In this case, set the 'filemode' to 'binary'. Default is
+# 'text'. Courier-IMAP for Cygwin works better with binary mode.
+
+# filemode = binary
+
[Repository RemoteExample]

# And this is the remote repository. We only support IMAP or Gmail here.
--- offlineimap/folder/Maildir.py 2008-08-13 20:55:48.000000000 +0200
+++ offlineimap/folder/Maildir-cygwin.py 2008-11-06 10:44:05.344202700 +0100
@@ -23,7 +23,6 @@
import os.path, os, re, time, socket, md5

uidmatchre = re.compile(',U=(\d+)')
-flagmatchre = re.compile(':.*2,([A-Z]+)')

timeseq = 0
lasttime = long(0)
@@ -54,6 +53,12 @@
self.messagelist = None
self.repository = repository
self.accountname = accountname
+ self.filemode = {
+ 'binary': 'wb',
+ 'text': 'wt',
+ }[repository.getconf("filemode", 'text')]
+ self.suffixsep = repository.getconf("suffixsep",':')
+ self.flagmatchre = re.compile('%s.*2,([A-Z]+)' % self.suffixsep)
BaseFolder.__init__(self)

def getaccountname(self):
@@ -102,7 +107,7 @@
nouidcounter -= 1
else:
uid = long(uidmatch.group(1))
- flagmatch = flagmatchre.search(messagename)
+ flagmatch = self.flagmatchre.search(messagename)
flags = []
if flagmatch:
flags = [x for x in flagmatch.group(1)]
@@ -180,7 +185,7 @@
break
tmpmessagename = messagename.split(',')[0]
ui.debug('maildir', 'savemessage: using temporary name %s' % tmpmessagename)
- file = open(os.path.join(tmpdir, tmpmessagename), "wt")
+ file = open(os.path.join(tmpdir, tmpmessagename), self.filemode)
file.write(content)

# Make sure the data hits the disk
@@ -226,11 +231,11 @@
newpath = os.path.join(self.getfullname(), 'cur')
else:
newpath = os.path.join(self.getfullname(), 'new')
- infostr = ':'
- infomatch = re.search('(:.*)$', newname)
+ infostr = self.suffixsep
+ infomatch = re.search('(%s.*)$' % self.suffixsep, newname)
if infomatch: # If the info string is present..
infostr = infomatch.group(1)
- newname = newname.split(':')[0] # Strip off the info string.
+ newname = newname.split(self.suffixsep)[0] # Strip off the info string.
infostr = re.sub('2,[A-Z]*', '', infostr)
flags.sort()
infostr += '2,' + ''.join(flags)
</source>

$ patch -Np0 < offlineimap_6.0.3-cygwin.patch

Install offlineimap

$ python setup.py install

Configure offlineimap as described in [[Offlineimap]] and launch offlineimap. '''CAUTION:''' Add the NEW suffixsep and filemode options in the Maildir repository section of the configuration file.

$ offlineimap
or
$ offlineimap -1 -o -u Noninteractive.Basic
or
$ offlineimap -c /home/username/offlineimap.conf

Here is a simple [{{#file:offlineimap.conf}} offlineimap configuration file].

<source lang=text>
[general]
metadata = /home/username/.offlineimap
accounts = Test
maxsyncaccounts = 1
ui = Curses.Blinkenlights, TTY.TTYUI,
Noninteractive.Basic, Noninteractive.Quiet
ignore-readonly = no

[mbnames]
enabled = no

[ui.Curses.Blinkenlights]
statuschar = .

[Account Test]
localrepository = LocalExample
remoterepository = RemoteExample

[Repository LocalExample]
type = Maildir
localfolders = /home/username/Maildir
sep = .
restoreatime = no

suffixsep = ! # <<< NEW
filemode = binary # <<< NEW

[Repository RemoteExample]
type = IMAP
remotehost = imapserver
ssl = no
remoteport = 143
remoteuser = username
remotepass = XXXXXX
maxconnections = 1
holdconnectionopen = no
nametrans = lambda foldername: re.sub('^INBOX\.*', '.', foldername)
folderfilter = lambda foldername: not re.search('(SPAM$|Mairix)', foldername)
</source>

Courier-Cygwin

2008-11-07T09:39:36Z

Fuujuhi: /* Building and Installing Courier-IMAP */

=Prerequisites=

Courier-IMAP requires the installation of the [http://www.courier-mta.org/authlib/ Courier Authentication Library] and the following cygwin tools and packages

* <tt>patch</tt>, <tt>tar</tt>, <tt>make</tt>, <tt>gcc</tt>
* <tt>crypt</tt>
* <tt>libgdbm-devel</tt>
* <tt>libtool</tt>
* <tt>inetutils</tt>
* <tt>cygrunsrv</tt>

Run the [http://www.cygwin.com/setup.exe Cygwin setup] program. Locate, select and install the required packages.

=Building and Installing Courier Authentication Library=

Installation of ''Courier Authentication Library'' using the module '''userdb''' to manage mail accounts. All other modules are disabled.

Download and untar [http://www.courier-mta.org/download.php#authlib courier-authlib] in your favorite sandbox.
Version used: [http://prdownloads.sourceforge.net/courier/courier-authlib-0.61.0.tar.bz2 0.61.0]

<source lang="bash">
$ tar -xvjf courier-authlib-0.61.0.tar.bz2
$ cd courier-authlib-0.61.0
</source>

Apply the [{{#file:courier-authlib-0.61.0-cygwin.patch}} patch] before configuration to correct following issues:

# incorrect usage of EXEEXT in ''makedat/Makefile''
# add libtool flag (-no-undefined) required for DLL creation ([http://lists.cairographics.org/archives/cairo/2004-April/001125.html])
# add missing dependencies
# add PATH in start script to access new DLL (installed in a different directory)

<source lang=diff>
--- courier-authlib-0.61.0/makedat/Makefile.in 2008-05-24 16:21:09.000000000 +0200
+++ courier-authlib-0.61.0-cygwin/makedat/Makefile.in 2008-10-21 16:02:38.709166700 +0200
@@ -182,7 +182,7 @@
libexecdir = @libexecdir@
localedir = @localedir@
localstatedir = @localstatedir@
-makedatprog_target = @makedatprog_target@
+makedatprog_target = @makedatprog_target@$(EXEEXT)
makedatprogpath = @makedatprogpath@
mandir = @mandir@
mkdir_p = @mkdir_p@
@@ -198,7 +198,7 @@
target_alias = @target_alias@
top_builddir = @top_builddir@
top_srcdir = @top_srcdir@
-noinst_PROGRAMS = @makedatprog_target@
+noinst_PROGRAMS = @makedatprog_target@$(EXEEXT)
makedatprog_SOURCES = makedatprog.c
makedatprog_DEPENDENCIES = @dblibrary@
makedatprog_LDADD = @dblibrary@
--- courier-authlib-0.61.0/Makefile.in 2008-07-12 21:41:08.000000000 +0200
+++ courier-authlib-0.61.0-cygwin/Makefile.in 2008-10-23 22:59:03.843750000 +0200
@@ -213,9 +213,10 @@
LTCOMPILE = $(LIBTOOL) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) \
--mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) \
$(AM_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS)
-CCLD = $(CC)
+CCLD = $(CC) -no-undefined
+CCLDEXE = $(CC)
LINK = $(LIBTOOL) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) \
- --mode=link $(CCLD) $(AM_CFLAGS) $(CFLAGS) $(AM_LDFLAGS) \
+ --mode=link $(CCLDEXE) $(AM_CFLAGS) $(CFLAGS) $(AM_LDFLAGS) \
$(LDFLAGS) -o $@
SOURCES = $(libauthcustom_la_SOURCES) $(libauthldap_la_SOURCES) \
$(libauthmysql_la_SOURCES) $(libauthpam_la_SOURCES) \
@@ -452,9 +453,9 @@
README.authdebug.html

DISTCLEANFILES = dbobj.config README_authlib.html
-commonlibdep = libcourierauthcommon.la
+commonlibdep = libcourierauthcommon.la libcourierauth.la
commonldflags = -module -rpath $(pkglibdir) -export-symbols-regex 'courier_auth.*_init' -avoid-version
-commonlibadd = libcourierauthcommon.la
+commonlibadd = libcourierauthcommon.la libcourierauth.la
libcourierauthcommon_t = @CRYPTLIBS@
libcourierauthcommon_la_SOURCES = \
auth.h courierauth.h \
--- courier-authlib-0.61.0/authdaemond.in 2005-07-05 14:25:08.000000000 +0200
+++ courier-authlib-0.61.0-cygwin/authdaemond.in 2008-10-25 12:03:32.140625000 +0200
@@ -15,4 +15,10 @@
set -a
. @authdaemonrc@

+# Some shared libraries (DLL) are installed in @libdir@/bin
+# instead of @libdir@/@PACKAGE@
+# Setting LD_LIBRARY_PATH at runtime or LR_RUN_PATH at linktime doesn't
+# work in cygwin, only setting PATH works.
+export PATH=$PATH:@libdir@/bin
+
exec ${sbindir}/courierlogger -pid=@authdaemonvar@/pid $LOGGEROPTS -$1 @libexecdir@/courier-authlib/authdaemond
</source>

Install the patch

<source lang="bash">
$ patch -Np1 < courier-authlib-0.61.0-cygwin.patch
</source>

Configure the package without most authentication modules, keeping only '''userdb'''. Replace ''mailuser'' by an existing user name (in your /etc/passwd file). I used my own username.

<source lang="bash">
$ ./configure --disable-root-check --with-waitfunc=wait --without-authpam --without-authldap --without-authpwd \
--without-authshadow --without-authcustom --without-authpipe --without-authmysql --without-authpgsql --with-mailuser=mailuser \
--with-mailgroup=mkgroup-l-d
</source>

Take a long pause... and when ready build, check the result and install the libraries.

<source lang="bash">
$ make
$ make check
$ make install
</source>

'''NOTES''': list of related topics found during investigation for compilation
* authldap: configure: error: -lresolve is needed for res_query... http://www.cygwin.com/ml/cygwin/2005-04/msg00419.html

=Building and Installing Courier-IMAP=

Download and untar package [http://www.courier-mta.org/download.php#imap courier-imap] in your favorite sandbox.
Version used: [http://prdownloads.sourceforge.net/courier/courier-imap-4.4.1.tar.bz2 4.4.1.20080920]

<source lang="bash">
$ tar -xvjf courier-imap-4.4.1.tar.bz2
$ cd courier-imap-4.4.1
</source>

Apply the [{{#file:courier-imap-4.4.1-cygwin.patch}} patch] before configuration to correct following issues:

# incorrect usage of EXEEXT in ''makedat/Makefile''
# incorrect usage of EXTEXT in main ''Makefile''
# remove usage of /usr/lib/env in start/stop script (/usr/lib/env does not seem to work under Cygwin)

<source lang=diff>
--- courier-imap-4.4.1.20080920/makedat/Makefile.in 2008-08-24 19:52:51.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/makedat/Makefile.in 2008-10-21 20:43:27.500000000 +0200
@@ -182,7 +182,7 @@
libexecdir = @libexecdir@
localedir = @localedir@
localstatedir = @localstatedir@
-makedatprog_target = @makedatprog_target@
+makedatprog_target = @makedatprog_target@$(EXEEXT)
makedatprogpath = @makedatprogpath@
mandir = @mandir@
mkdir_p = @mkdir_p@
@@ -198,7 +198,7 @@
target_alias = @target_alias@
top_builddir = @top_builddir@
top_srcdir = @top_srcdir@
-noinst_PROGRAMS = @makedatprog_target@
+noinst_PROGRAMS = @makedatprog_target@$(EXEEXT)
makedatprog_SOURCES = makedatprog.c
makedatprog_DEPENDENCIES = @dblibrary@
makedatprog_LDADD = @dblibrary@
--- courier-imap-4.4.1.20080920/Makefile.in 2008-09-20 14:48:46.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/Makefile.in 2008-10-21 20:59:45.156250000 +0200
@@ -247,9 +247,9 @@
CLEANFILES = $(databin_SCRIPTS) $(man_MANS) $(sysconf_DATA) $(sbin_SCRIPTS)
databindir = $(datadir)
databin_SCRIPTS = mkimapdcert mkpop3dcert
-binPROGRAMS = imapd pop3d maildirmake maildiracl deliverquota maildirkw
-sbinPROGRAMS = imaplogin pop3login
-libexecPROGRAMS = makedatprog couriertcpd
+binPROGRAMS = imapd$(EXEEXT) pop3d$(EXEEXT) maildirmake$(EXEEXT) maildiracl$(EXEEXT) deliverquota$(EXEEXT) maildirkw$(EXEEXT)
+sbinPROGRAMS = imaplogin$(EXEEXT) pop3login$(EXEEXT)
+libexecPROGRAMS = makedatprog$(EXEEXT) couriertcpd$(EXEEXT)
bin_PROGRAMS = @binPROGRAMS_exec@
sbin_PROGRAMS = @sbinPROGRAMS_exec@
libexec_PROGRAMS = @libexecPROGRAMS_exec@
--- courier-imap-4.4.1.20080920/imapd.rc.in 2008-05-04 15:12:47.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/imapd.rc.in 2008-10-26 10:17:51.359375000 +0100
@@ -22,6 +22,15 @@
exit 1
fi

+# Location for some shared libraries (cygcourierauth*.dll) from authlib
+AUTHLIBDIR=/usr/local
+if test ! -f $AUTHLIBDIR/lib/bin/cygcourierauth.dll
+then
+ echo "$AUTHLIBDIR/lib/bin/cygcourierauth.dll not found."
+ exit 1
+fi
+export PATH=$PATH:$AUTHLIBDIR/lib/bin
+
TLS_CACHEFILE=""
. @sysconfdir@/imapd-ssl
. @sysconfdir@/imapd
@@ -35,7 +44,7 @@

umask $IMAP_UMASK
@ULIMIT@ $IMAP_ULIMITD
- @SETENV@ -i @SHELL@ -c " set -a ;
+ @SHELL@ -c " set -a ;
prefix=@prefix@ ;
exec_prefix=@exec_prefix@ ;
bindir=@bindir@ ;
--- courier-imap-4.4.1.20080920/pop3d.rc.in 2008-05-04 15:12:47.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/pop3d.rc.in 2008-10-26 10:18:02.187500000 +0100
@@ -22,12 +22,21 @@
exit 1
fi

+# Location for some shared libraries (cygcourierauth*.dll) from authlib
+AUTHLIBDIR=/usr/local
+if test ! -f $AUTHLIBDIR/lib/bin/cygcourierauth.dll
+then
+ echo "$AUTHLIBDIR/lib/bin/cygcourierauth.dll not found."
+ exit 1
+fi
+export PATH=$PATH:$AUTHLIBDIR/lib/bin
+
. @sysconfdir@/pop3d-ssl
. @sysconfdir@/pop3d

case $1 in
start)
- @SETENV@ -i @SHELL@ -c " set -a ;
+ @SHELL@ -c " set -a ;
prefix=@prefix@ ;
exec_prefix=@exec_prefix@ ;
bindir=@bindir@ ;
</source>

Install the patch

<source lang="bash">
$ patch -Np1 < courier-imap-4.4.1-cygwin.patch
</source>

Configure the package

<source lang="bash">
$ ./configure --disable-root-check --with-waitfunc=wait
</source>

Take a walk or some coffee... and when ready build the package and install it.

<source lang="bash">
$ make
$ make install
$ make install-configure
</source>

=Configuring Courier=

==courier-authlib==

Create authdaemonrc file: /usr/local/etc/authlib/authdaemonrc

$ cp /usr/local/etc/authlib/authdaemonrc.dist /usr/local/etc/authlib/authdaemonrc

Change number of daemons (I use three) and enable DEBUG_LOGIN

<source lang=text>
# The number of daemon processes that are started.
daemons=3

# DEBUG_LOGIN=2 - turn on debugging + log passwords too
DEBUG_LOGIN=2
</source>

See http://www.courier-mta.org/authlib/README.authdebug.html for details.

==courier-imap==

Setup courier-imap: $prefix/etc/imapd (where $prefix=/usr/lib/courier-imap)

Set ADDRESS to a valid value (0 or 127.0.0.1) and change IMAP_ULIMITD to a value accepted by the command ulimit. I simply used the existing value returned by

$ ulimit -v

==IMAP accounts==

User accounts and settings are managed by ''userdb''. See [http://linux.die.net/man/8/makeuserdb makeuserdb] man page for details, or used the freshly new install man page - man makeuserdb

Assuming ''username'' is your username, 1001 and 10545 are respectively the UID and GID of an existing user (part of your /etc/passwd and /etc/group). I use my own UID and GID. Create the mailuser home directory, if required, and create Maildir folder.

$ mkdir -p /home/username
$ cd /home/username
$ /usr/lib/courier-imap/bin/maildirmake Maildir

Create the userdb text file /usr/local/etc/authlib/userdb

$ touch /usr/local/etc/authlib/userdb
$ chmod 700 /usr/local/etc/authlib/userdb

Edit /usr/local/etc/authlib/userdb, one line per account

<source lang=text>
username[TAB]uid=1001|gid=10111|home=/home/username|mail=/home/username/Maildir|systempw=XXXXXX
</source>

where XXXXXX is the encrypted password created with

$ /usr/local/sbin/userdbpw.exe

Create the (binary) userdb

$ /usr/local/sbin/makeuserdb

You should now have the following files created:

* /usr/local/etc/authlib/userdb.dat
* /usr/local/etc/authlib/userdbshadow.dat

==Configuration Summary==

$ /usr/local/bin/courierauthconfig.exe --version
$ /usr/local/bin/courierauthconfig.exe --ldflags
$ /usr/local/bin/courierauthconfig.exe --cppflags
$ /usr/local/bin/courierauthconfig.exe --configfiles

=Running=

Install and configure syslogd. Run the [http://www.cygwin.com/setup.exe Cygwin setup] program. Locate, select and install inetutils (contains syslogd). Configure syslogd with the following command. It will create the /etc/syslogd.conf file and install a windows service using cygrunsrv

$ syslogd-config
$ net start syslogd

Start the authlib daemon with the following command:

$ /usr/local/sbin/authdaemond start

Stop the daemon with the following command:

$ /usr/local/sbin/authdaemond stop

You'll find log messages in /var/log/messages (default target file as specified in /etc/syslogd.conf)

Syslogd service and authdaemond should be started. See above.

Start the imap daemon with the following command:

$ /usr/lib/courier-imap/libexec/imapd.rc start

Stop the daemon with the following command:

$ /usr/lib/courier-imap/libexec/imapd.rc stop

You'll find log messages in /var/log/messages (default target file as specified in /etc/syslogd.conf)

=Testing=

$ /usr/local/sbin/authtest username CCCCCC
Authentication succeeded.

Authenticated: username (uid 1001, gid 10545)
Home Directory: /home/username
Maildir: /home/username/Maildir
Quota: (none)
Encrypted Password: XXXXXX
Cleartext Password: CCCCCC
Options: (none)

where username is your user name and XXXXXX is your encrypted password as provided in ''/usr/local/etc/authlib/userdb''. CCCCCC is your password in clear.

When authdaemond and imapd are running you can do a simple login test:

$ telnet localhost 143

and type the login command

a login <username> <password>

the server should reply

a OK LOGIN Ok.

to which you can reply with a logout

a logout

the session is terminated by the server

* BYE Courier-IMAP server shutting down
a OK LOGOUT completed
Connection closed by foreign host.

'''Hint:''' if the connection is immediately closed by foreign host it probably means that ''imapd'' does not have access to all required DLLs. Check your PATH.

==Troubleshooting==

Should you have a doubt on the execution of a command (ie: no output) you can check if the application is able to load required DLL. This error is not reported. Using ''strace'', windows will complain about DLL not found, if any. To solve this problem you may need to add some path in your PATH. For example, in order to test authlib with authtest:

$ strace /usr/local/sbin/authtest
# This will open a dialog box to complain about DLL.
$ export PATH=$PATH:/usr/local/lib/bin
$ /usr/local/sbin/authtest ausername

=Installing as a Windows Service=

==Startup Script==

If everything works fine you can use the following [{{#file:imapd}} startup script] to launch or stop imap and authlib daemons. It can be saved in /usr/local/sbin.

<source lang=bash>
#! /bin/sh
#
# Courier imap daemon startup script.

authlib=/usr/local/sbin/authdaemond
imapd=/usr/lib/courier-imap/libexec/imapd.rc

test ! -f $authlib && echo "File not found: $authlib" && exit 1;
test ! -f $imapd && echo "File not found: $imapd" && exit 1;

case $1 in
start)
$authlib start
$imapd start
;;
stop)
$imapd stop
$authlib stop
;;
*)
echo "Usage: $0 <start|stop>"
;;
esac
</source>

==Cygwin Service==

Create the [{{#file:imapd-srv.sh}} service script]

<source lang=bash>
#!/bin/sh
# File: imapd-service.sh
# Purpose: Courier imap daemon service script.
#
# NOTE: This script must be excutable by SYSTEM
username=username # <<<< Change me !!!
pidsleep=''

# Function to handle QUIT signal.
# Stop imapd daemon and kill background sleep.
function handleQuit
{
echo "STOPPING: /usr/local/sbin/imapd"
su $username -c "/usr/local/sbin/imapd stop"
test -n "$pidsleep" && kill -9 $pidsleep
exit
}

# Interruptible sleep
# Note: sleep is an external command (not builtin command), hence
# not interruptible by the QUIT signal.
function intsleep
{
sleep $1 &
pidsleep=$!
wait $pidsleep
}

echo "STARTING: /usr/local/sbin/imapd as $username at "`date`
su $username -c "/usr/local/sbin/imapd start"

trap "handleQuit" SIGQUIT

echo "WAITING: "$$
while true; do intsleep 1d; done
</source>

Change username and save it to your favorite location, says /usr/local/sbin. The service script must be executable by SYSTEM, ie: the user that run the windows services.

$ chmod +rx /usr/local/sbin/imapd-service.sh

Install the new service with the following command.

$ cygrunsrv --install imapd --desc "Courier IMAP daemon" --disp "CYGWIN imapd" \
--path /usr/local/sbin/imapd-service.sh --termsig QUIT --type auto --shutdown

Finally, just start the service

$ cygrunsrv --start imapd
or
$ net start imapd

You should now have couple of processes related to courier-imap

$ ps -a

'''NOTES''':
* If the service could not be started, it's probably because the service script is not executable by SYSTEM

$ net start imapd
The CYGWIN imapd service is starting.
The CYGWIN imapd service could not be started.

The service did not report an error.

More help is available by typing NET HELPMSG 3534.

* The combination of background sleep and builtin wait command is used to make the waiting statment interruptible. The original script found on the net, was simply using a 'sleep 1' which turned to consume ~10% of the CPU.

=Other Interesting Packages=

==Maildir==

'''maildirmake''' is part of the courier-imap package

$ /usr/lib/courier-imap/bin/maildirmake

==Mairix - index and search mail folders==
Run the [http://www.cygwin.com/setup.exe Cygwin setup] program. Locate, select and install the package mairix.

Create the configuration file in your home directory: ~/.mairixrc

base=/home/username
maildir=Maildir...
omit=Maildir/.Mairix**
mfolder=Maildir/.Mairix
database=/home/username/.mairix_database

The create the mairix index

$ mairix
and
$ mairix --fast-index # on daily base

See [[Mail_Tips]] for additional details.

==Offlineimap==
Requires python, so run the [http://www.cygwin.com/setup.exe Cygwin setup] program. Locate, select and install the package python.

Download offlineimap [http://software.complete.org/software/projects/list_files/offlineimap here]. I use [http://software.complete.org/software/versions/download/242?attachment_id=334 version 6.0.3]. Extract files in you favorite sandbox.

$ tar xvfz offlineimap_6.0.3.tar.gz
$ cd offlineimap

Apply the following [{{#file:offlineimap_6.0.3-cygwin.patch}} patch] to correct the following issues:

* [http://en.wikipedia.org/wiki/Maildir#Windows_software Suffix separator] character colon (:) is not allowed in Windows. Courier-imap for cygwin is using (!)
* While used by courier-imap for cygwin Maildir mail files should not use CRLF, rather only LF. The issue appears only in mail with attachment, that is the raw message is not interpreted and displayed as it in the mail body (under Thunderbird).

<source lang=diff>
--- offlineimap.conf 2008-08-13 20:56:04.000000000 +0200
+++ offlineimap-cygwin.conf 2008-11-06 10:44:31.066750700 +0100
@@ -215,6 +215,19 @@

restoreatime = no

+# Cygwin/Windows users may have issues with the Maildir mail filename
+# containing the suffix separator character (:). In this case, they
+# could change the suffix separator character to whatever required.
+# For example, Courier-IMAP for Cygwin is using (!)
+
+# suffixsep = !
+
+# Cygwin/Windows users may need to create Maildir mail filename in
+# binary mode. In this case, set the 'filemode' to 'binary'. Default is
+# 'text'. Courier-IMAP for Cygwin works better with binary mode.
+
+# filemode = binary
+
[Repository RemoteExample]

# And this is the remote repository. We only support IMAP or Gmail here.
--- offlineimap/folder/Maildir.py 2008-08-13 20:55:48.000000000 +0200
+++ offlineimap/folder/Maildir-cygwin.py 2008-11-06 10:44:05.344202700 +0100
@@ -23,7 +23,6 @@
import os.path, os, re, time, socket, md5

uidmatchre = re.compile(',U=(\d+)')
-flagmatchre = re.compile(':.*2,([A-Z]+)')

timeseq = 0
lasttime = long(0)
@@ -54,6 +53,12 @@
self.messagelist = None
self.repository = repository
self.accountname = accountname
+ self.filemode = {
+ 'binary': 'wb',
+ 'text': 'wt',
+ }[repository.getconf("filemode", 'text')]
+ self.suffixsep = repository.getconf("suffixsep",':')
+ self.flagmatchre = re.compile('%s.*2,([A-Z]+)' % self.suffixsep)
BaseFolder.__init__(self)

def getaccountname(self):
@@ -102,7 +107,7 @@
nouidcounter -= 1
else:
uid = long(uidmatch.group(1))
- flagmatch = flagmatchre.search(messagename)
+ flagmatch = self.flagmatchre.search(messagename)
flags = []
if flagmatch:
flags = [x for x in flagmatch.group(1)]
@@ -180,7 +185,7 @@
break
tmpmessagename = messagename.split(',')[0]
ui.debug('maildir', 'savemessage: using temporary name %s' % tmpmessagename)
- file = open(os.path.join(tmpdir, tmpmessagename), "wt")
+ file = open(os.path.join(tmpdir, tmpmessagename), self.filemode)
file.write(content)

# Make sure the data hits the disk
@@ -226,11 +231,11 @@
newpath = os.path.join(self.getfullname(), 'cur')
else:
newpath = os.path.join(self.getfullname(), 'new')
- infostr = ':'
- infomatch = re.search('(:.*)$', newname)
+ infostr = self.suffixsep
+ infomatch = re.search('(%s.*)$' % self.suffixsep, newname)
if infomatch: # If the info string is present..
infostr = infomatch.group(1)
- newname = newname.split(':')[0] # Strip off the info string.
+ newname = newname.split(self.suffixsep)[0] # Strip off the info string.
infostr = re.sub('2,[A-Z]*', '', infostr)
flags.sort()
infostr += '2,' + ''.join(flags)
</source>

$ patch -Np0 < offlineimap_6.0.3-cygwin.patch

Install offlineimap

$ python setup.py install

Configure offlineimap as described in [[Offlineimap]] and launch offlineimap. '''CAUTION:''' Add the NEW suffixsep and filemode options in the Maildir repository section of the configuration file.

$ offlineimap
or
$ offlineimap -1 -o -u Noninteractive.Basic
or
$ offlineimap -c /home/username/offlineimap.conf

Here is a simple [{{#file:offlineimap.conf}} offlineimap configuration file].

<source lang=text>
[general]
metadata = /home/username/.offlineimap
accounts = Test
maxsyncaccounts = 1
ui = Curses.Blinkenlights, TTY.TTYUI,
Noninteractive.Basic, Noninteractive.Quiet
ignore-readonly = no

[mbnames]
enabled = no

[ui.Curses.Blinkenlights]
statuschar = .

[Account Test]
localrepository = LocalExample
remoterepository = RemoteExample

[Repository LocalExample]
type = Maildir
localfolders = /home/username/Maildir
sep = .
restoreatime = no

suffixsep = ! # <<< NEW
filemode = binary # <<< NEW

[Repository RemoteExample]
type = IMAP
remotehost = imapserver
ssl = no
remoteport = 143
remoteuser = username
remotepass = XXXXXX
maxconnections = 1
holdconnectionopen = no
nametrans = lambda foldername: re.sub('^INBOX\.*', '.', foldername)
folderfilter = lambda foldername: not re.search('(SPAM$|Mairix)', foldername)
</source>

Courier-Cygwin

2008-11-07T09:35:20Z

Fuujuhi: /* Building and Installing Courier Authentication Library */

=Prerequisites=

Courier-IMAP requires the installation of the [http://www.courier-mta.org/authlib/ Courier Authentication Library] and the following cygwin tools and packages

* <tt>patch</tt>, <tt>tar</tt>, <tt>make</tt>, <tt>gcc</tt>
* <tt>crypt</tt>
* <tt>libgdbm-devel</tt>
* <tt>libtool</tt>
* <tt>inetutils</tt>
* <tt>cygrunsrv</tt>

Run the [http://www.cygwin.com/setup.exe Cygwin setup] program. Locate, select and install the required packages.

=Building and Installing Courier Authentication Library=

Installation of ''Courier Authentication Library'' using the module '''userdb''' to manage mail accounts. All other modules are disabled.

Download and untar [http://www.courier-mta.org/download.php#authlib courier-authlib] in your favorite sandbox.
Version used: [http://prdownloads.sourceforge.net/courier/courier-authlib-0.61.0.tar.bz2 0.61.0]

<source lang="bash">
$ tar -xvjf courier-authlib-0.61.0.tar.bz2
$ cd courier-authlib-0.61.0
</source>

Apply the [{{#file:courier-authlib-0.61.0-cygwin.patch}} patch] before configuration to correct following issues:

# incorrect usage of EXEEXT in ''makedat/Makefile''
# add libtool flag (-no-undefined) required for DLL creation ([http://lists.cairographics.org/archives/cairo/2004-April/001125.html])
# add missing dependencies
# add PATH in start script to access new DLL (installed in a different directory)

<source lang=diff>
--- courier-authlib-0.61.0/makedat/Makefile.in 2008-05-24 16:21:09.000000000 +0200
+++ courier-authlib-0.61.0-cygwin/makedat/Makefile.in 2008-10-21 16:02:38.709166700 +0200
@@ -182,7 +182,7 @@
libexecdir = @libexecdir@
localedir = @localedir@
localstatedir = @localstatedir@
-makedatprog_target = @makedatprog_target@
+makedatprog_target = @makedatprog_target@$(EXEEXT)
makedatprogpath = @makedatprogpath@
mandir = @mandir@
mkdir_p = @mkdir_p@
@@ -198,7 +198,7 @@
target_alias = @target_alias@
top_builddir = @top_builddir@
top_srcdir = @top_srcdir@
-noinst_PROGRAMS = @makedatprog_target@
+noinst_PROGRAMS = @makedatprog_target@$(EXEEXT)
makedatprog_SOURCES = makedatprog.c
makedatprog_DEPENDENCIES = @dblibrary@
makedatprog_LDADD = @dblibrary@
--- courier-authlib-0.61.0/Makefile.in 2008-07-12 21:41:08.000000000 +0200
+++ courier-authlib-0.61.0-cygwin/Makefile.in 2008-10-23 22:59:03.843750000 +0200
@@ -213,9 +213,10 @@
LTCOMPILE = $(LIBTOOL) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) \
--mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) \
$(AM_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS)
-CCLD = $(CC)
+CCLD = $(CC) -no-undefined
+CCLDEXE = $(CC)
LINK = $(LIBTOOL) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) \
- --mode=link $(CCLD) $(AM_CFLAGS) $(CFLAGS) $(AM_LDFLAGS) \
+ --mode=link $(CCLDEXE) $(AM_CFLAGS) $(CFLAGS) $(AM_LDFLAGS) \
$(LDFLAGS) -o $@
SOURCES = $(libauthcustom_la_SOURCES) $(libauthldap_la_SOURCES) \
$(libauthmysql_la_SOURCES) $(libauthpam_la_SOURCES) \
@@ -452,9 +453,9 @@
README.authdebug.html

DISTCLEANFILES = dbobj.config README_authlib.html
-commonlibdep = libcourierauthcommon.la
+commonlibdep = libcourierauthcommon.la libcourierauth.la
commonldflags = -module -rpath $(pkglibdir) -export-symbols-regex 'courier_auth.*_init' -avoid-version
-commonlibadd = libcourierauthcommon.la
+commonlibadd = libcourierauthcommon.la libcourierauth.la
libcourierauthcommon_t = @CRYPTLIBS@
libcourierauthcommon_la_SOURCES = \
auth.h courierauth.h \
--- courier-authlib-0.61.0/authdaemond.in 2005-07-05 14:25:08.000000000 +0200
+++ courier-authlib-0.61.0-cygwin/authdaemond.in 2008-10-25 12:03:32.140625000 +0200
@@ -15,4 +15,10 @@
set -a
. @authdaemonrc@

+# Some shared libraries (DLL) are installed in @libdir@/bin
+# instead of @libdir@/@PACKAGE@
+# Setting LD_LIBRARY_PATH at runtime or LR_RUN_PATH at linktime doesn't
+# work in cygwin, only setting PATH works.
+export PATH=$PATH:@libdir@/bin
+
exec ${sbindir}/courierlogger -pid=@authdaemonvar@/pid $LOGGEROPTS -$1 @libexecdir@/courier-authlib/authdaemond
</source>

Install the patch

<source lang="bash">
$ patch -Np1 < courier-authlib-0.61.0-cygwin.patch
</source>

Configure the package without most authentication modules, keeping only '''userdb'''. Replace ''mailuser'' by an existing user name (in your /etc/passwd file). I used my own username.

<source lang="bash">
$ ./configure --disable-root-check --with-waitfunc=wait --without-authpam --without-authldap --without-authpwd \
--without-authshadow --without-authcustom --without-authpipe --without-authmysql --without-authpgsql --with-mailuser=mailuser \
--with-mailgroup=mkgroup-l-d
</source>

Take a long pause... and when ready build, check the result and install the libraries.

<source lang="bash">
$ make
$ make check
$ make install
</source>

'''NOTES''': list of related topics found during investigation for compilation
* authldap: configure: error: -lresolve is needed for res_query... http://www.cygwin.com/ml/cygwin/2005-04/msg00419.html

=Building and Installing Courier-IMAP=

Download and untar package [http://www.courier-mta.org/download.php#imap courier-imap] in your favorite sandbox.
Version used: [http://prdownloads.sourceforge.net/courier/courier-imap-4.4.1.tar.bz2 4.4.1.20080920]

Apply the [{{#file:courier-imap-4.4.1-cygwin.patch}} patch] before configuration to correct following issues:

# incorrect usage of EXEEXT in ''makedat/Makefile''
# incorrect usage of EXTEXT in main ''Makefile''
# remove usage of /usr/lib/env in start/stop script (/usr/lib/env does not seem to work under Cygwin)

<source lang=diff>
--- courier-imap-4.4.1.20080920/makedat/Makefile.in 2008-08-24 19:52:51.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/makedat/Makefile.in 2008-10-21 20:43:27.500000000 +0200
@@ -182,7 +182,7 @@
libexecdir = @libexecdir@
localedir = @localedir@
localstatedir = @localstatedir@
-makedatprog_target = @makedatprog_target@
+makedatprog_target = @makedatprog_target@$(EXEEXT)
makedatprogpath = @makedatprogpath@
mandir = @mandir@
mkdir_p = @mkdir_p@
@@ -198,7 +198,7 @@
target_alias = @target_alias@
top_builddir = @top_builddir@
top_srcdir = @top_srcdir@
-noinst_PROGRAMS = @makedatprog_target@
+noinst_PROGRAMS = @makedatprog_target@$(EXEEXT)
makedatprog_SOURCES = makedatprog.c
makedatprog_DEPENDENCIES = @dblibrary@
makedatprog_LDADD = @dblibrary@
--- courier-imap-4.4.1.20080920/Makefile.in 2008-09-20 14:48:46.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/Makefile.in 2008-10-21 20:59:45.156250000 +0200
@@ -247,9 +247,9 @@
CLEANFILES = $(databin_SCRIPTS) $(man_MANS) $(sysconf_DATA) $(sbin_SCRIPTS)
databindir = $(datadir)
databin_SCRIPTS = mkimapdcert mkpop3dcert
-binPROGRAMS = imapd pop3d maildirmake maildiracl deliverquota maildirkw
-sbinPROGRAMS = imaplogin pop3login
-libexecPROGRAMS = makedatprog couriertcpd
+binPROGRAMS = imapd$(EXEEXT) pop3d$(EXEEXT) maildirmake$(EXEEXT) maildiracl$(EXEEXT) deliverquota$(EXEEXT) maildirkw$(EXEEXT)
+sbinPROGRAMS = imaplogin$(EXEEXT) pop3login$(EXEEXT)
+libexecPROGRAMS = makedatprog$(EXEEXT) couriertcpd$(EXEEXT)
bin_PROGRAMS = @binPROGRAMS_exec@
sbin_PROGRAMS = @sbinPROGRAMS_exec@
libexec_PROGRAMS = @libexecPROGRAMS_exec@
--- courier-imap-4.4.1.20080920/imapd.rc.in 2008-05-04 15:12:47.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/imapd.rc.in 2008-10-26 10:17:51.359375000 +0100
@@ -22,6 +22,15 @@
exit 1
fi

+# Location for some shared libraries (cygcourierauth*.dll) from authlib
+AUTHLIBDIR=/usr/local
+if test ! -f $AUTHLIBDIR/lib/bin/cygcourierauth.dll
+then
+ echo "$AUTHLIBDIR/lib/bin/cygcourierauth.dll not found."
+ exit 1
+fi
+export PATH=$PATH:$AUTHLIBDIR/lib/bin
+
TLS_CACHEFILE=""
. @sysconfdir@/imapd-ssl
. @sysconfdir@/imapd
@@ -35,7 +44,7 @@

umask $IMAP_UMASK
@ULIMIT@ $IMAP_ULIMITD
- @SETENV@ -i @SHELL@ -c " set -a ;
+ @SHELL@ -c " set -a ;
prefix=@prefix@ ;
exec_prefix=@exec_prefix@ ;
bindir=@bindir@ ;
--- courier-imap-4.4.1.20080920/pop3d.rc.in 2008-05-04 15:12:47.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/pop3d.rc.in 2008-10-26 10:18:02.187500000 +0100
@@ -22,12 +22,21 @@
exit 1
fi

+# Location for some shared libraries (cygcourierauth*.dll) from authlib
+AUTHLIBDIR=/usr/local
+if test ! -f $AUTHLIBDIR/lib/bin/cygcourierauth.dll
+then
+ echo "$AUTHLIBDIR/lib/bin/cygcourierauth.dll not found."
+ exit 1
+fi
+export PATH=$PATH:$AUTHLIBDIR/lib/bin
+
. @sysconfdir@/pop3d-ssl
. @sysconfdir@/pop3d

case $1 in
start)
- @SETENV@ -i @SHELL@ -c " set -a ;
+ @SHELL@ -c " set -a ;
prefix=@prefix@ ;
exec_prefix=@exec_prefix@ ;
bindir=@bindir@ ;
</source>

Install the patch

$ patch -Np1 < courier-imap-4.4.1-cygwin.patch

Configure the package

$ ./configure --disable-root-check --with-waitfunc=wait

Take a walk... and when ready execute the following command to build the package.

$ make

Time to go for some coffee... and then install the package with the usual

$ make install
$ make install-configure

=Configuring Courier=

==courier-authlib==

Create authdaemonrc file: /usr/local/etc/authlib/authdaemonrc

$ cp /usr/local/etc/authlib/authdaemonrc.dist /usr/local/etc/authlib/authdaemonrc

Change number of daemons (I use three) and enable DEBUG_LOGIN

<source lang=text>
# The number of daemon processes that are started.
daemons=3

# DEBUG_LOGIN=2 - turn on debugging + log passwords too
DEBUG_LOGIN=2
</source>

See http://www.courier-mta.org/authlib/README.authdebug.html for details.

==courier-imap==

Setup courier-imap: $prefix/etc/imapd (where $prefix=/usr/lib/courier-imap)

Set ADDRESS to a valid value (0 or 127.0.0.1) and change IMAP_ULIMITD to a value accepted by the command ulimit. I simply used the existing value returned by

$ ulimit -v

==IMAP accounts==

User accounts and settings are managed by ''userdb''. See [http://linux.die.net/man/8/makeuserdb makeuserdb] man page for details, or used the freshly new install man page - man makeuserdb

Assuming ''username'' is your username, 1001 and 10545 are respectively the UID and GID of an existing user (part of your /etc/passwd and /etc/group). I use my own UID and GID. Create the mailuser home directory, if required, and create Maildir folder.

$ mkdir -p /home/username
$ cd /home/username
$ /usr/lib/courier-imap/bin/maildirmake Maildir

Create the userdb text file /usr/local/etc/authlib/userdb

$ touch /usr/local/etc/authlib/userdb
$ chmod 700 /usr/local/etc/authlib/userdb

Edit /usr/local/etc/authlib/userdb, one line per account

<source lang=text>
username[TAB]uid=1001|gid=10111|home=/home/username|mail=/home/username/Maildir|systempw=XXXXXX
</source>

where XXXXXX is the encrypted password created with

$ /usr/local/sbin/userdbpw.exe

Create the (binary) userdb

$ /usr/local/sbin/makeuserdb

You should now have the following files created:

* /usr/local/etc/authlib/userdb.dat
* /usr/local/etc/authlib/userdbshadow.dat

==Configuration Summary==

$ /usr/local/bin/courierauthconfig.exe --version
$ /usr/local/bin/courierauthconfig.exe --ldflags
$ /usr/local/bin/courierauthconfig.exe --cppflags
$ /usr/local/bin/courierauthconfig.exe --configfiles

=Running=

Install and configure syslogd. Run the [http://www.cygwin.com/setup.exe Cygwin setup] program. Locate, select and install inetutils (contains syslogd). Configure syslogd with the following command. It will create the /etc/syslogd.conf file and install a windows service using cygrunsrv

$ syslogd-config
$ net start syslogd

Start the authlib daemon with the following command:

$ /usr/local/sbin/authdaemond start

Stop the daemon with the following command:

$ /usr/local/sbin/authdaemond stop

You'll find log messages in /var/log/messages (default target file as specified in /etc/syslogd.conf)

Syslogd service and authdaemond should be started. See above.

Start the imap daemon with the following command:

$ /usr/lib/courier-imap/libexec/imapd.rc start

Stop the daemon with the following command:

$ /usr/lib/courier-imap/libexec/imapd.rc stop

You'll find log messages in /var/log/messages (default target file as specified in /etc/syslogd.conf)

=Testing=

$ /usr/local/sbin/authtest username CCCCCC
Authentication succeeded.

Authenticated: username (uid 1001, gid 10545)
Home Directory: /home/username
Maildir: /home/username/Maildir
Quota: (none)
Encrypted Password: XXXXXX
Cleartext Password: CCCCCC
Options: (none)

where username is your user name and XXXXXX is your encrypted password as provided in ''/usr/local/etc/authlib/userdb''. CCCCCC is your password in clear.

When authdaemond and imapd are running you can do a simple login test:

$ telnet localhost 143

and type the login command

a login <username> <password>

the server should reply

a OK LOGIN Ok.

to which you can reply with a logout

a logout

the session is terminated by the server

* BYE Courier-IMAP server shutting down
a OK LOGOUT completed
Connection closed by foreign host.

'''Hint:''' if the connection is immediately closed by foreign host it probably means that ''imapd'' does not have access to all required DLLs. Check your PATH.

==Troubleshooting==

Should you have a doubt on the execution of a command (ie: no output) you can check if the application is able to load required DLL. This error is not reported. Using ''strace'', windows will complain about DLL not found, if any. To solve this problem you may need to add some path in your PATH. For example, in order to test authlib with authtest:

$ strace /usr/local/sbin/authtest
# This will open a dialog box to complain about DLL.
$ export PATH=$PATH:/usr/local/lib/bin
$ /usr/local/sbin/authtest ausername

=Installing as a Windows Service=

==Startup Script==

If everything works fine you can use the following [{{#file:imapd}} startup script] to launch or stop imap and authlib daemons. It can be saved in /usr/local/sbin.

<source lang=bash>
#! /bin/sh
#
# Courier imap daemon startup script.

authlib=/usr/local/sbin/authdaemond
imapd=/usr/lib/courier-imap/libexec/imapd.rc

test ! -f $authlib && echo "File not found: $authlib" && exit 1;
test ! -f $imapd && echo "File not found: $imapd" && exit 1;

case $1 in
start)
$authlib start
$imapd start
;;
stop)
$imapd stop
$authlib stop
;;
*)
echo "Usage: $0 <start|stop>"
;;
esac
</source>

==Cygwin Service==

Create the [{{#file:imapd-srv.sh}} service script]

<source lang=bash>
#!/bin/sh
# File: imapd-service.sh
# Purpose: Courier imap daemon service script.
#
# NOTE: This script must be excutable by SYSTEM
username=username # <<<< Change me !!!
pidsleep=''

# Function to handle QUIT signal.
# Stop imapd daemon and kill background sleep.
function handleQuit
{
echo "STOPPING: /usr/local/sbin/imapd"
su $username -c "/usr/local/sbin/imapd stop"
test -n "$pidsleep" && kill -9 $pidsleep
exit
}

# Interruptible sleep
# Note: sleep is an external command (not builtin command), hence
# not interruptible by the QUIT signal.
function intsleep
{
sleep $1 &
pidsleep=$!
wait $pidsleep
}

echo "STARTING: /usr/local/sbin/imapd as $username at "`date`
su $username -c "/usr/local/sbin/imapd start"

trap "handleQuit" SIGQUIT

echo "WAITING: "$$
while true; do intsleep 1d; done
</source>

Change username and save it to your favorite location, says /usr/local/sbin. The service script must be executable by SYSTEM, ie: the user that run the windows services.

$ chmod +rx /usr/local/sbin/imapd-service.sh

Install the new service with the following command.

$ cygrunsrv --install imapd --desc "Courier IMAP daemon" --disp "CYGWIN imapd" \
--path /usr/local/sbin/imapd-service.sh --termsig QUIT --type auto --shutdown

Finally, just start the service

$ cygrunsrv --start imapd
or
$ net start imapd

You should now have couple of processes related to courier-imap

$ ps -a

'''NOTES''':
* If the service could not be started, it's probably because the service script is not executable by SYSTEM

$ net start imapd
The CYGWIN imapd service is starting.
The CYGWIN imapd service could not be started.

The service did not report an error.

More help is available by typing NET HELPMSG 3534.

* The combination of background sleep and builtin wait command is used to make the waiting statment interruptible. The original script found on the net, was simply using a 'sleep 1' which turned to consume ~10% of the CPU.

=Other Interesting Packages=

==Maildir==

'''maildirmake''' is part of the courier-imap package

$ /usr/lib/courier-imap/bin/maildirmake

==Mairix - index and search mail folders==
Run the [http://www.cygwin.com/setup.exe Cygwin setup] program. Locate, select and install the package mairix.

Create the configuration file in your home directory: ~/.mairixrc

base=/home/username
maildir=Maildir...
omit=Maildir/.Mairix**
mfolder=Maildir/.Mairix
database=/home/username/.mairix_database

The create the mairix index

$ mairix
and
$ mairix --fast-index # on daily base

See [[Mail_Tips]] for additional details.

==Offlineimap==
Requires python, so run the [http://www.cygwin.com/setup.exe Cygwin setup] program. Locate, select and install the package python.

Download offlineimap [http://software.complete.org/software/projects/list_files/offlineimap here]. I use [http://software.complete.org/software/versions/download/242?attachment_id=334 version 6.0.3]. Extract files in you favorite sandbox.

$ tar xvfz offlineimap_6.0.3.tar.gz
$ cd offlineimap

Apply the following [{{#file:offlineimap_6.0.3-cygwin.patch}} patch] to correct the following issues:

* [http://en.wikipedia.org/wiki/Maildir#Windows_software Suffix separator] character colon (:) is not allowed in Windows. Courier-imap for cygwin is using (!)
* While used by courier-imap for cygwin Maildir mail files should not use CRLF, rather only LF. The issue appears only in mail with attachment, that is the raw message is not interpreted and displayed as it in the mail body (under Thunderbird).

<source lang=diff>
--- offlineimap.conf 2008-08-13 20:56:04.000000000 +0200
+++ offlineimap-cygwin.conf 2008-11-06 10:44:31.066750700 +0100
@@ -215,6 +215,19 @@

restoreatime = no

+# Cygwin/Windows users may have issues with the Maildir mail filename
+# containing the suffix separator character (:). In this case, they
+# could change the suffix separator character to whatever required.
+# For example, Courier-IMAP for Cygwin is using (!)
+
+# suffixsep = !
+
+# Cygwin/Windows users may need to create Maildir mail filename in
+# binary mode. In this case, set the 'filemode' to 'binary'. Default is
+# 'text'. Courier-IMAP for Cygwin works better with binary mode.
+
+# filemode = binary
+
[Repository RemoteExample]

# And this is the remote repository. We only support IMAP or Gmail here.
--- offlineimap/folder/Maildir.py 2008-08-13 20:55:48.000000000 +0200
+++ offlineimap/folder/Maildir-cygwin.py 2008-11-06 10:44:05.344202700 +0100
@@ -23,7 +23,6 @@
import os.path, os, re, time, socket, md5

uidmatchre = re.compile(',U=(\d+)')
-flagmatchre = re.compile(':.*2,([A-Z]+)')

timeseq = 0
lasttime = long(0)
@@ -54,6 +53,12 @@
self.messagelist = None
self.repository = repository
self.accountname = accountname
+ self.filemode = {
+ 'binary': 'wb',
+ 'text': 'wt',
+ }[repository.getconf("filemode", 'text')]
+ self.suffixsep = repository.getconf("suffixsep",':')
+ self.flagmatchre = re.compile('%s.*2,([A-Z]+)' % self.suffixsep)
BaseFolder.__init__(self)

def getaccountname(self):
@@ -102,7 +107,7 @@
nouidcounter -= 1
else:
uid = long(uidmatch.group(1))
- flagmatch = flagmatchre.search(messagename)
+ flagmatch = self.flagmatchre.search(messagename)
flags = []
if flagmatch:
flags = [x for x in flagmatch.group(1)]
@@ -180,7 +185,7 @@
break
tmpmessagename = messagename.split(',')[0]
ui.debug('maildir', 'savemessage: using temporary name %s' % tmpmessagename)
- file = open(os.path.join(tmpdir, tmpmessagename), "wt")
+ file = open(os.path.join(tmpdir, tmpmessagename), self.filemode)
file.write(content)

# Make sure the data hits the disk
@@ -226,11 +231,11 @@
newpath = os.path.join(self.getfullname(), 'cur')
else:
newpath = os.path.join(self.getfullname(), 'new')
- infostr = ':'
- infomatch = re.search('(:.*)$', newname)
+ infostr = self.suffixsep
+ infomatch = re.search('(%s.*)$' % self.suffixsep, newname)
if infomatch: # If the info string is present..
infostr = infomatch.group(1)
- newname = newname.split(':')[0] # Strip off the info string.
+ newname = newname.split(self.suffixsep)[0] # Strip off the info string.
infostr = re.sub('2,[A-Z]*', '', infostr)
flags.sort()
infostr += '2,' + ''.join(flags)
</source>

$ patch -Np0 < offlineimap_6.0.3-cygwin.patch

Install offlineimap

$ python setup.py install

Configure offlineimap as described in [[Offlineimap]] and launch offlineimap. '''CAUTION:''' Add the NEW suffixsep and filemode options in the Maildir repository section of the configuration file.

$ offlineimap
or
$ offlineimap -1 -o -u Noninteractive.Basic
or
$ offlineimap -c /home/username/offlineimap.conf

Here is a simple [{{#file:offlineimap.conf}} offlineimap configuration file].

<source lang=text>
[general]
metadata = /home/username/.offlineimap
accounts = Test
maxsyncaccounts = 1
ui = Curses.Blinkenlights, TTY.TTYUI,
Noninteractive.Basic, Noninteractive.Quiet
ignore-readonly = no

[mbnames]
enabled = no

[ui.Curses.Blinkenlights]
statuschar = .

[Account Test]
localrepository = LocalExample
remoterepository = RemoteExample

[Repository LocalExample]
type = Maildir
localfolders = /home/username/Maildir
sep = .
restoreatime = no

suffixsep = ! # <<< NEW
filemode = binary # <<< NEW

[Repository RemoteExample]
type = IMAP
remotehost = imapserver
ssl = no
remoteport = 143
remoteuser = username
remotepass = XXXXXX
maxconnections = 1
holdconnectionopen = no
nametrans = lambda foldername: re.sub('^INBOX\.*', '.', foldername)
folderfilter = lambda foldername: not re.search('(SPAM$|Mairix)', foldername)
</source>

Courier-Cygwin

2008-11-07T09:26:40Z

Fuujuhi: /* Prerequisites */ Cosmetics

=Prerequisites=

Courier-IMAP requires the installation of the [http://www.courier-mta.org/authlib/ Courier Authentication Library] and the following cygwin tools and packages

* <tt>patch</tt>, <tt>tar</tt>, <tt>make</tt>, <tt>gcc</tt>
* <tt>crypt</tt>
* <tt>libgdbm-devel</tt>
* <tt>libtool</tt>
* <tt>inetutils</tt>
* <tt>cygrunsrv</tt>

Run the [http://www.cygwin.com/setup.exe Cygwin setup] program. Locate, select and install the required packages.

=Building and Installing Courier Authentication Library=

Installation of Courier Authentication Library using the module userdb to manage mail accounts. All other modules are disabled.

Download and untar [http://www.courier-mta.org/download.php#authlib courier-authlib] in your favorite sandbox.
Version used: [http://prdownloads.sourceforge.net/courier/courier-authlib-0.61.0.tar.bz2 0.61.0]

Apply the [{{#file:courier-authlib-0.61.0-cygwin.patch}} patch] before configuration to correct following issues:

# incorrect usage of EXEEXT in ''makedat/Makefile''
# add libtool flag (-no-undefined) required for DLL creation
# add missing dependencies
# add PATH in start script to access new DLL (installed in a different directory)

Solution source: http://lists.cairographics.org/archives/cairo/2004-April/001125.html

<source lang=diff>
--- courier-authlib-0.61.0/makedat/Makefile.in 2008-05-24 16:21:09.000000000 +0200
+++ courier-authlib-0.61.0-cygwin/makedat/Makefile.in 2008-10-21 16:02:38.709166700 +0200
@@ -182,7 +182,7 @@
libexecdir = @libexecdir@
localedir = @localedir@
localstatedir = @localstatedir@
-makedatprog_target = @makedatprog_target@
+makedatprog_target = @makedatprog_target@$(EXEEXT)
makedatprogpath = @makedatprogpath@
mandir = @mandir@
mkdir_p = @mkdir_p@
@@ -198,7 +198,7 @@
target_alias = @target_alias@
top_builddir = @top_builddir@
top_srcdir = @top_srcdir@
-noinst_PROGRAMS = @makedatprog_target@
+noinst_PROGRAMS = @makedatprog_target@$(EXEEXT)
makedatprog_SOURCES = makedatprog.c
makedatprog_DEPENDENCIES = @dblibrary@
makedatprog_LDADD = @dblibrary@
--- courier-authlib-0.61.0/Makefile.in 2008-07-12 21:41:08.000000000 +0200
+++ courier-authlib-0.61.0-cygwin/Makefile.in 2008-10-23 22:59:03.843750000 +0200
@@ -213,9 +213,10 @@
LTCOMPILE = $(LIBTOOL) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) \
--mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) \
$(AM_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS)
-CCLD = $(CC)
+CCLD = $(CC) -no-undefined
+CCLDEXE = $(CC)
LINK = $(LIBTOOL) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) \
- --mode=link $(CCLD) $(AM_CFLAGS) $(CFLAGS) $(AM_LDFLAGS) \
+ --mode=link $(CCLDEXE) $(AM_CFLAGS) $(CFLAGS) $(AM_LDFLAGS) \
$(LDFLAGS) -o $@
SOURCES = $(libauthcustom_la_SOURCES) $(libauthldap_la_SOURCES) \
$(libauthmysql_la_SOURCES) $(libauthpam_la_SOURCES) \
@@ -452,9 +453,9 @@
README.authdebug.html

DISTCLEANFILES = dbobj.config README_authlib.html
-commonlibdep = libcourierauthcommon.la
+commonlibdep = libcourierauthcommon.la libcourierauth.la
commonldflags = -module -rpath $(pkglibdir) -export-symbols-regex 'courier_auth.*_init' -avoid-version
-commonlibadd = libcourierauthcommon.la
+commonlibadd = libcourierauthcommon.la libcourierauth.la
libcourierauthcommon_t = @CRYPTLIBS@
libcourierauthcommon_la_SOURCES = \
auth.h courierauth.h \
--- courier-authlib-0.61.0/authdaemond.in 2005-07-05 14:25:08.000000000 +0200
+++ courier-authlib-0.61.0-cygwin/authdaemond.in 2008-10-25 12:03:32.140625000 +0200
@@ -15,4 +15,10 @@
set -a
. @authdaemonrc@

+# Some shared libraries (DLL) are installed in @libdir@/bin
+# instead of @libdir@/@PACKAGE@
+# Setting LD_LIBRARY_PATH at runtime or LR_RUN_PATH at linktime doesn't
+# work in cygwin, only setting PATH works.
+export PATH=$PATH:@libdir@/bin
+
exec ${sbindir}/courierlogger -pid=@authdaemonvar@/pid $LOGGEROPTS -$1 @libexecdir@/courier-authlib/authdaemond
</source>

Install the patch

$ patch -Np1 < courier-authlib-0.61.0-cygwin.patch

Configure the package without most authentication modules, keeping only ''userdb''.

'''Note''': replace ''mailuser'' by an existing user name (in your /etc/passwd file). I used my own username.

$ ./configure --disable-root-check --with-waitfunc=wait --without-authpam --without-authldap --without-authpwd \
--without-authshadow --without-authcustom --without-authpipe --without-authmysql --without-authpgsql --with-mailuser=mailuser \
--with-mailgroup=mkgroup-l-d

Take a long pause... and when ready execute the following command.

$ make

Take again a long pause... then check the result and install the libraries

$ make check
$ make install

'''NOTES''': list of related topics found during investigation for compilation
* authldap: configure: error: -lresolve is needed for res_query... http://www.cygwin.com/ml/cygwin/2005-04/msg00419.html

=Building and Installing Courier-IMAP=

Download and untar package [http://www.courier-mta.org/download.php#imap courier-imap] in your favorite sandbox.
Version used: [http://prdownloads.sourceforge.net/courier/courier-imap-4.4.1.tar.bz2 4.4.1.20080920]

Apply the [{{#file:courier-imap-4.4.1-cygwin.patch}} patch] before configuration to correct following issues:

# incorrect usage of EXEEXT in ''makedat/Makefile''
# incorrect usage of EXTEXT in main ''Makefile''
# remove usage of /usr/lib/env in start/stop script (/usr/lib/env does not seem to work under Cygwin)

<source lang=diff>
--- courier-imap-4.4.1.20080920/makedat/Makefile.in 2008-08-24 19:52:51.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/makedat/Makefile.in 2008-10-21 20:43:27.500000000 +0200
@@ -182,7 +182,7 @@
libexecdir = @libexecdir@
localedir = @localedir@
localstatedir = @localstatedir@
-makedatprog_target = @makedatprog_target@
+makedatprog_target = @makedatprog_target@$(EXEEXT)
makedatprogpath = @makedatprogpath@
mandir = @mandir@
mkdir_p = @mkdir_p@
@@ -198,7 +198,7 @@
target_alias = @target_alias@
top_builddir = @top_builddir@
top_srcdir = @top_srcdir@
-noinst_PROGRAMS = @makedatprog_target@
+noinst_PROGRAMS = @makedatprog_target@$(EXEEXT)
makedatprog_SOURCES = makedatprog.c
makedatprog_DEPENDENCIES = @dblibrary@
makedatprog_LDADD = @dblibrary@
--- courier-imap-4.4.1.20080920/Makefile.in 2008-09-20 14:48:46.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/Makefile.in 2008-10-21 20:59:45.156250000 +0200
@@ -247,9 +247,9 @@
CLEANFILES = $(databin_SCRIPTS) $(man_MANS) $(sysconf_DATA) $(sbin_SCRIPTS)
databindir = $(datadir)
databin_SCRIPTS = mkimapdcert mkpop3dcert
-binPROGRAMS = imapd pop3d maildirmake maildiracl deliverquota maildirkw
-sbinPROGRAMS = imaplogin pop3login
-libexecPROGRAMS = makedatprog couriertcpd
+binPROGRAMS = imapd$(EXEEXT) pop3d$(EXEEXT) maildirmake$(EXEEXT) maildiracl$(EXEEXT) deliverquota$(EXEEXT) maildirkw$(EXEEXT)
+sbinPROGRAMS = imaplogin$(EXEEXT) pop3login$(EXEEXT)
+libexecPROGRAMS = makedatprog$(EXEEXT) couriertcpd$(EXEEXT)
bin_PROGRAMS = @binPROGRAMS_exec@
sbin_PROGRAMS = @sbinPROGRAMS_exec@
libexec_PROGRAMS = @libexecPROGRAMS_exec@
--- courier-imap-4.4.1.20080920/imapd.rc.in 2008-05-04 15:12:47.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/imapd.rc.in 2008-10-26 10:17:51.359375000 +0100
@@ -22,6 +22,15 @@
exit 1
fi

+# Location for some shared libraries (cygcourierauth*.dll) from authlib
+AUTHLIBDIR=/usr/local
+if test ! -f $AUTHLIBDIR/lib/bin/cygcourierauth.dll
+then
+ echo "$AUTHLIBDIR/lib/bin/cygcourierauth.dll not found."
+ exit 1
+fi
+export PATH=$PATH:$AUTHLIBDIR/lib/bin
+
TLS_CACHEFILE=""
. @sysconfdir@/imapd-ssl
. @sysconfdir@/imapd
@@ -35,7 +44,7 @@

umask $IMAP_UMASK
@ULIMIT@ $IMAP_ULIMITD
- @SETENV@ -i @SHELL@ -c " set -a ;
+ @SHELL@ -c " set -a ;
prefix=@prefix@ ;
exec_prefix=@exec_prefix@ ;
bindir=@bindir@ ;
--- courier-imap-4.4.1.20080920/pop3d.rc.in 2008-05-04 15:12:47.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/pop3d.rc.in 2008-10-26 10:18:02.187500000 +0100
@@ -22,12 +22,21 @@
exit 1
fi

+# Location for some shared libraries (cygcourierauth*.dll) from authlib
+AUTHLIBDIR=/usr/local
+if test ! -f $AUTHLIBDIR/lib/bin/cygcourierauth.dll
+then
+ echo "$AUTHLIBDIR/lib/bin/cygcourierauth.dll not found."
+ exit 1
+fi
+export PATH=$PATH:$AUTHLIBDIR/lib/bin
+
. @sysconfdir@/pop3d-ssl
. @sysconfdir@/pop3d

case $1 in
start)
- @SETENV@ -i @SHELL@ -c " set -a ;
+ @SHELL@ -c " set -a ;
prefix=@prefix@ ;
exec_prefix=@exec_prefix@ ;
bindir=@bindir@ ;
</source>

Install the patch

$ patch -Np1 < courier-imap-4.4.1-cygwin.patch

Configure the package

$ ./configure --disable-root-check --with-waitfunc=wait

Take a walk... and when ready execute the following command to build the package.

$ make

Time to go for some coffee... and then install the package with the usual

$ make install
$ make install-configure

=Configuring Courier=

==courier-authlib==

Create authdaemonrc file: /usr/local/etc/authlib/authdaemonrc

$ cp /usr/local/etc/authlib/authdaemonrc.dist /usr/local/etc/authlib/authdaemonrc

Change number of daemons (I use three) and enable DEBUG_LOGIN

<source lang=text>
# The number of daemon processes that are started.
daemons=3

# DEBUG_LOGIN=2 - turn on debugging + log passwords too
DEBUG_LOGIN=2
</source>

See http://www.courier-mta.org/authlib/README.authdebug.html for details.

==courier-imap==

Setup courier-imap: $prefix/etc/imapd (where $prefix=/usr/lib/courier-imap)

Set ADDRESS to a valid value (0 or 127.0.0.1) and change IMAP_ULIMITD to a value accepted by the command ulimit. I simply used the existing value returned by

$ ulimit -v

==IMAP accounts==

User accounts and settings are managed by ''userdb''. See [http://linux.die.net/man/8/makeuserdb makeuserdb] man page for details, or used the freshly new install man page - man makeuserdb

Assuming ''username'' is your username, 1001 and 10545 are respectively the UID and GID of an existing user (part of your /etc/passwd and /etc/group). I use my own UID and GID. Create the mailuser home directory, if required, and create Maildir folder.

$ mkdir -p /home/username
$ cd /home/username
$ /usr/lib/courier-imap/bin/maildirmake Maildir

Create the userdb text file /usr/local/etc/authlib/userdb

$ touch /usr/local/etc/authlib/userdb
$ chmod 700 /usr/local/etc/authlib/userdb

Edit /usr/local/etc/authlib/userdb, one line per account

<source lang=text>
username[TAB]uid=1001|gid=10111|home=/home/username|mail=/home/username/Maildir|systempw=XXXXXX
</source>

where XXXXXX is the encrypted password created with

$ /usr/local/sbin/userdbpw.exe

Create the (binary) userdb

$ /usr/local/sbin/makeuserdb

You should now have the following files created:

* /usr/local/etc/authlib/userdb.dat
* /usr/local/etc/authlib/userdbshadow.dat

==Configuration Summary==

$ /usr/local/bin/courierauthconfig.exe --version
$ /usr/local/bin/courierauthconfig.exe --ldflags
$ /usr/local/bin/courierauthconfig.exe --cppflags
$ /usr/local/bin/courierauthconfig.exe --configfiles

=Running=

Install and configure syslogd. Run the [http://www.cygwin.com/setup.exe Cygwin setup] program. Locate, select and install inetutils (contains syslogd). Configure syslogd with the following command. It will create the /etc/syslogd.conf file and install a windows service using cygrunsrv

$ syslogd-config
$ net start syslogd

Start the authlib daemon with the following command:

$ /usr/local/sbin/authdaemond start

Stop the daemon with the following command:

$ /usr/local/sbin/authdaemond stop

You'll find log messages in /var/log/messages (default target file as specified in /etc/syslogd.conf)

Syslogd service and authdaemond should be started. See above.

Start the imap daemon with the following command:

$ /usr/lib/courier-imap/libexec/imapd.rc start

Stop the daemon with the following command:

$ /usr/lib/courier-imap/libexec/imapd.rc stop

You'll find log messages in /var/log/messages (default target file as specified in /etc/syslogd.conf)

=Testing=

$ /usr/local/sbin/authtest username CCCCCC
Authentication succeeded.

Authenticated: username (uid 1001, gid 10545)
Home Directory: /home/username
Maildir: /home/username/Maildir
Quota: (none)
Encrypted Password: XXXXXX
Cleartext Password: CCCCCC
Options: (none)

where username is your user name and XXXXXX is your encrypted password as provided in ''/usr/local/etc/authlib/userdb''. CCCCCC is your password in clear.

When authdaemond and imapd are running you can do a simple login test:

$ telnet localhost 143

and type the login command

a login <username> <password>

the server should reply

a OK LOGIN Ok.

to which you can reply with a logout

a logout

the session is terminated by the server

* BYE Courier-IMAP server shutting down
a OK LOGOUT completed
Connection closed by foreign host.

'''Hint:''' if the connection is immediately closed by foreign host it probably means that ''imapd'' does not have access to all required DLLs. Check your PATH.

==Troubleshooting==

Should you have a doubt on the execution of a command (ie: no output) you can check if the application is able to load required DLL. This error is not reported. Using ''strace'', windows will complain about DLL not found, if any. To solve this problem you may need to add some path in your PATH. For example, in order to test authlib with authtest:

$ strace /usr/local/sbin/authtest
# This will open a dialog box to complain about DLL.
$ export PATH=$PATH:/usr/local/lib/bin
$ /usr/local/sbin/authtest ausername

=Installing as a Windows Service=

==Startup Script==

If everything works fine you can use the following [{{#file:imapd}} startup script] to launch or stop imap and authlib daemons. It can be saved in /usr/local/sbin.

<source lang=bash>
#! /bin/sh
#
# Courier imap daemon startup script.

authlib=/usr/local/sbin/authdaemond
imapd=/usr/lib/courier-imap/libexec/imapd.rc

test ! -f $authlib && echo "File not found: $authlib" && exit 1;
test ! -f $imapd && echo "File not found: $imapd" && exit 1;

case $1 in
start)
$authlib start
$imapd start
;;
stop)
$imapd stop
$authlib stop
;;
*)
echo "Usage: $0 <start|stop>"
;;
esac
</source>

==Cygwin Service==

Create the [{{#file:imapd-srv.sh}} service script]

<source lang=bash>
#!/bin/sh
# File: imapd-service.sh
# Purpose: Courier imap daemon service script.
#
# NOTE: This script must be excutable by SYSTEM
username=username # <<<< Change me !!!
pidsleep=''

# Function to handle QUIT signal.
# Stop imapd daemon and kill background sleep.
function handleQuit
{
echo "STOPPING: /usr/local/sbin/imapd"
su $username -c "/usr/local/sbin/imapd stop"
test -n "$pidsleep" && kill -9 $pidsleep
exit
}

# Interruptible sleep
# Note: sleep is an external command (not builtin command), hence
# not interruptible by the QUIT signal.
function intsleep
{
sleep $1 &
pidsleep=$!
wait $pidsleep
}

echo "STARTING: /usr/local/sbin/imapd as $username at "`date`
su $username -c "/usr/local/sbin/imapd start"

trap "handleQuit" SIGQUIT

echo "WAITING: "$$
while true; do intsleep 1d; done
</source>

Change username and save it to your favorite location, says /usr/local/sbin. The service script must be executable by SYSTEM, ie: the user that run the windows services.

$ chmod +rx /usr/local/sbin/imapd-service.sh

Install the new service with the following command.

$ cygrunsrv --install imapd --desc "Courier IMAP daemon" --disp "CYGWIN imapd" \
--path /usr/local/sbin/imapd-service.sh --termsig QUIT --type auto --shutdown

Finally, just start the service

$ cygrunsrv --start imapd
or
$ net start imapd

You should now have couple of processes related to courier-imap

$ ps -a

'''NOTES''':
* If the service could not be started, it's probably because the service script is not executable by SYSTEM

$ net start imapd
The CYGWIN imapd service is starting.
The CYGWIN imapd service could not be started.

The service did not report an error.

More help is available by typing NET HELPMSG 3534.

* The combination of background sleep and builtin wait command is used to make the waiting statment interruptible. The original script found on the net, was simply using a 'sleep 1' which turned to consume ~10% of the CPU.

=Other Interesting Packages=

==Maildir==

'''maildirmake''' is part of the courier-imap package

$ /usr/lib/courier-imap/bin/maildirmake

==Mairix - index and search mail folders==
Run the [http://www.cygwin.com/setup.exe Cygwin setup] program. Locate, select and install the package mairix.

Create the configuration file in your home directory: ~/.mairixrc

base=/home/username
maildir=Maildir...
omit=Maildir/.Mairix**
mfolder=Maildir/.Mairix
database=/home/username/.mairix_database

The create the mairix index

$ mairix
and
$ mairix --fast-index # on daily base

See [[Mail_Tips]] for additional details.

==Offlineimap==
Requires python, so run the [http://www.cygwin.com/setup.exe Cygwin setup] program. Locate, select and install the package python.

Download offlineimap [http://software.complete.org/software/projects/list_files/offlineimap here]. I use [http://software.complete.org/software/versions/download/242?attachment_id=334 version 6.0.3]. Extract files in you favorite sandbox.

$ tar xvfz offlineimap_6.0.3.tar.gz
$ cd offlineimap

Apply the following [{{#file:offlineimap_6.0.3-cygwin.patch}} patch] to correct the following issues:

* [http://en.wikipedia.org/wiki/Maildir#Windows_software Suffix separator] character colon (:) is not allowed in Windows. Courier-imap for cygwin is using (!)
* While used by courier-imap for cygwin Maildir mail files should not use CRLF, rather only LF. The issue appears only in mail with attachment, that is the raw message is not interpreted and displayed as it in the mail body (under Thunderbird).

<source lang=diff>
--- offlineimap.conf 2008-08-13 20:56:04.000000000 +0200
+++ offlineimap-cygwin.conf 2008-11-06 10:44:31.066750700 +0100
@@ -215,6 +215,19 @@

restoreatime = no

+# Cygwin/Windows users may have issues with the Maildir mail filename
+# containing the suffix separator character (:). In this case, they
+# could change the suffix separator character to whatever required.
+# For example, Courier-IMAP for Cygwin is using (!)
+
+# suffixsep = !
+
+# Cygwin/Windows users may need to create Maildir mail filename in
+# binary mode. In this case, set the 'filemode' to 'binary'. Default is
+# 'text'. Courier-IMAP for Cygwin works better with binary mode.
+
+# filemode = binary
+
[Repository RemoteExample]

# And this is the remote repository. We only support IMAP or Gmail here.
--- offlineimap/folder/Maildir.py 2008-08-13 20:55:48.000000000 +0200
+++ offlineimap/folder/Maildir-cygwin.py 2008-11-06 10:44:05.344202700 +0100
@@ -23,7 +23,6 @@
import os.path, os, re, time, socket, md5

uidmatchre = re.compile(',U=(\d+)')
-flagmatchre = re.compile(':.*2,([A-Z]+)')

timeseq = 0
lasttime = long(0)
@@ -54,6 +53,12 @@
self.messagelist = None
self.repository = repository
self.accountname = accountname
+ self.filemode = {
+ 'binary': 'wb',
+ 'text': 'wt',
+ }[repository.getconf("filemode", 'text')]
+ self.suffixsep = repository.getconf("suffixsep",':')
+ self.flagmatchre = re.compile('%s.*2,([A-Z]+)' % self.suffixsep)
BaseFolder.__init__(self)

def getaccountname(self):
@@ -102,7 +107,7 @@
nouidcounter -= 1
else:
uid = long(uidmatch.group(1))
- flagmatch = flagmatchre.search(messagename)
+ flagmatch = self.flagmatchre.search(messagename)
flags = []
if flagmatch:
flags = [x for x in flagmatch.group(1)]
@@ -180,7 +185,7 @@
break
tmpmessagename = messagename.split(',')[0]
ui.debug('maildir', 'savemessage: using temporary name %s' % tmpmessagename)
- file = open(os.path.join(tmpdir, tmpmessagename), "wt")
+ file = open(os.path.join(tmpdir, tmpmessagename), self.filemode)
file.write(content)

# Make sure the data hits the disk
@@ -226,11 +231,11 @@
newpath = os.path.join(self.getfullname(), 'cur')
else:
newpath = os.path.join(self.getfullname(), 'new')
- infostr = ':'
- infomatch = re.search('(:.*)$', newname)
+ infostr = self.suffixsep
+ infomatch = re.search('(%s.*)$' % self.suffixsep, newname)
if infomatch: # If the info string is present..
infostr = infomatch.group(1)
- newname = newname.split(':')[0] # Strip off the info string.
+ newname = newname.split(self.suffixsep)[0] # Strip off the info string.
infostr = re.sub('2,[A-Z]*', '', infostr)
flags.sort()
infostr += '2,' + ''.join(flags)
</source>

$ patch -Np0 < offlineimap_6.0.3-cygwin.patch

Install offlineimap

$ python setup.py install

Configure offlineimap as described in [[Offlineimap]] and launch offlineimap. '''CAUTION:''' Add the NEW suffixsep and filemode options in the Maildir repository section of the configuration file.

$ offlineimap
or
$ offlineimap -1 -o -u Noninteractive.Basic
or
$ offlineimap -c /home/username/offlineimap.conf

Here is a simple [{{#file:offlineimap.conf}} offlineimap configuration file].

<source lang=text>
[general]
metadata = /home/username/.offlineimap
accounts = Test
maxsyncaccounts = 1
ui = Curses.Blinkenlights, TTY.TTYUI,
Noninteractive.Basic, Noninteractive.Quiet
ignore-readonly = no

[mbnames]
enabled = no

[ui.Curses.Blinkenlights]
statuschar = .

[Account Test]
localrepository = LocalExample
remoterepository = RemoteExample

[Repository LocalExample]
type = Maildir
localfolders = /home/username/Maildir
sep = .
restoreatime = no

suffixsep = ! # <<< NEW
filemode = binary # <<< NEW

[Repository RemoteExample]
type = IMAP
remotehost = imapserver
ssl = no
remoteport = 143
remoteuser = username
remotepass = XXXXXX
maxconnections = 1
holdconnectionopen = no
nametrans = lambda foldername: re.sub('^INBOX\.*', '.', foldername)
folderfilter = lambda foldername: not re.search('(SPAM$|Mairix)', foldername)
</source>

Courier-Cygwin

2008-11-07T09:22:54Z

Fuujuhi: Only reorganizing/renaming titles (no content changes)

=Prerequisites=

Courier-IMAP requires the installation of the [http://www.courier-mta.org/authlib/ Courier Authentication Library] and the following cygwin tools and packages

* patch, tar, make, gcc
* crypt
* libgdbm-devel
* libtool
* inetutils
* cygrunsrv

Run the [http://www.cygwin.com/setup.exe Cygwin setup] program. Locate, select and install the required packages.

=Building and Installing Courier Authentication Library=

Installation of Courier Authentication Library using the module userdb to manage mail accounts. All other modules are disabled.

Download and untar [http://www.courier-mta.org/download.php#authlib courier-authlib] in your favorite sandbox.
Version used: [http://prdownloads.sourceforge.net/courier/courier-authlib-0.61.0.tar.bz2 0.61.0]

Apply the [{{#file:courier-authlib-0.61.0-cygwin.patch}} patch] before configuration to correct following issues:

# incorrect usage of EXEEXT in ''makedat/Makefile''
# add libtool flag (-no-undefined) required for DLL creation
# add missing dependencies
# add PATH in start script to access new DLL (installed in a different directory)

Solution source: http://lists.cairographics.org/archives/cairo/2004-April/001125.html

<source lang=diff>
--- courier-authlib-0.61.0/makedat/Makefile.in 2008-05-24 16:21:09.000000000 +0200
+++ courier-authlib-0.61.0-cygwin/makedat/Makefile.in 2008-10-21 16:02:38.709166700 +0200
@@ -182,7 +182,7 @@
libexecdir = @libexecdir@
localedir = @localedir@
localstatedir = @localstatedir@
-makedatprog_target = @makedatprog_target@
+makedatprog_target = @makedatprog_target@$(EXEEXT)
makedatprogpath = @makedatprogpath@
mandir = @mandir@
mkdir_p = @mkdir_p@
@@ -198,7 +198,7 @@
target_alias = @target_alias@
top_builddir = @top_builddir@
top_srcdir = @top_srcdir@
-noinst_PROGRAMS = @makedatprog_target@
+noinst_PROGRAMS = @makedatprog_target@$(EXEEXT)
makedatprog_SOURCES = makedatprog.c
makedatprog_DEPENDENCIES = @dblibrary@
makedatprog_LDADD = @dblibrary@
--- courier-authlib-0.61.0/Makefile.in 2008-07-12 21:41:08.000000000 +0200
+++ courier-authlib-0.61.0-cygwin/Makefile.in 2008-10-23 22:59:03.843750000 +0200
@@ -213,9 +213,10 @@
LTCOMPILE = $(LIBTOOL) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) \
--mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) \
$(AM_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS)
-CCLD = $(CC)
+CCLD = $(CC) -no-undefined
+CCLDEXE = $(CC)
LINK = $(LIBTOOL) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) \
- --mode=link $(CCLD) $(AM_CFLAGS) $(CFLAGS) $(AM_LDFLAGS) \
+ --mode=link $(CCLDEXE) $(AM_CFLAGS) $(CFLAGS) $(AM_LDFLAGS) \
$(LDFLAGS) -o $@
SOURCES = $(libauthcustom_la_SOURCES) $(libauthldap_la_SOURCES) \
$(libauthmysql_la_SOURCES) $(libauthpam_la_SOURCES) \
@@ -452,9 +453,9 @@
README.authdebug.html

DISTCLEANFILES = dbobj.config README_authlib.html
-commonlibdep = libcourierauthcommon.la
+commonlibdep = libcourierauthcommon.la libcourierauth.la
commonldflags = -module -rpath $(pkglibdir) -export-symbols-regex 'courier_auth.*_init' -avoid-version
-commonlibadd = libcourierauthcommon.la
+commonlibadd = libcourierauthcommon.la libcourierauth.la
libcourierauthcommon_t = @CRYPTLIBS@
libcourierauthcommon_la_SOURCES = \
auth.h courierauth.h \
--- courier-authlib-0.61.0/authdaemond.in 2005-07-05 14:25:08.000000000 +0200
+++ courier-authlib-0.61.0-cygwin/authdaemond.in 2008-10-25 12:03:32.140625000 +0200
@@ -15,4 +15,10 @@
set -a
. @authdaemonrc@

+# Some shared libraries (DLL) are installed in @libdir@/bin
+# instead of @libdir@/@PACKAGE@
+# Setting LD_LIBRARY_PATH at runtime or LR_RUN_PATH at linktime doesn't
+# work in cygwin, only setting PATH works.
+export PATH=$PATH:@libdir@/bin
+
exec ${sbindir}/courierlogger -pid=@authdaemonvar@/pid $LOGGEROPTS -$1 @libexecdir@/courier-authlib/authdaemond
</source>

Install the patch

$ patch -Np1 < courier-authlib-0.61.0-cygwin.patch

Configure the package without most authentication modules, keeping only ''userdb''.

'''Note''': replace ''mailuser'' by an existing user name (in your /etc/passwd file). I used my own username.

$ ./configure --disable-root-check --with-waitfunc=wait --without-authpam --without-authldap --without-authpwd \
--without-authshadow --without-authcustom --without-authpipe --without-authmysql --without-authpgsql --with-mailuser=mailuser \
--with-mailgroup=mkgroup-l-d

Take a long pause... and when ready execute the following command.

$ make

Take again a long pause... then check the result and install the libraries

$ make check
$ make install

'''NOTES''': list of related topics found during investigation for compilation
* authldap: configure: error: -lresolve is needed for res_query... http://www.cygwin.com/ml/cygwin/2005-04/msg00419.html

=Building and Installing Courier-IMAP=

Download and untar package [http://www.courier-mta.org/download.php#imap courier-imap] in your favorite sandbox.
Version used: [http://prdownloads.sourceforge.net/courier/courier-imap-4.4.1.tar.bz2 4.4.1.20080920]

Apply the [{{#file:courier-imap-4.4.1-cygwin.patch}} patch] before configuration to correct following issues:

# incorrect usage of EXEEXT in ''makedat/Makefile''
# incorrect usage of EXTEXT in main ''Makefile''
# remove usage of /usr/lib/env in start/stop script (/usr/lib/env does not seem to work under Cygwin)

<source lang=diff>
--- courier-imap-4.4.1.20080920/makedat/Makefile.in 2008-08-24 19:52:51.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/makedat/Makefile.in 2008-10-21 20:43:27.500000000 +0200
@@ -182,7 +182,7 @@
libexecdir = @libexecdir@
localedir = @localedir@
localstatedir = @localstatedir@
-makedatprog_target = @makedatprog_target@
+makedatprog_target = @makedatprog_target@$(EXEEXT)
makedatprogpath = @makedatprogpath@
mandir = @mandir@
mkdir_p = @mkdir_p@
@@ -198,7 +198,7 @@
target_alias = @target_alias@
top_builddir = @top_builddir@
top_srcdir = @top_srcdir@
-noinst_PROGRAMS = @makedatprog_target@
+noinst_PROGRAMS = @makedatprog_target@$(EXEEXT)
makedatprog_SOURCES = makedatprog.c
makedatprog_DEPENDENCIES = @dblibrary@
makedatprog_LDADD = @dblibrary@
--- courier-imap-4.4.1.20080920/Makefile.in 2008-09-20 14:48:46.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/Makefile.in 2008-10-21 20:59:45.156250000 +0200
@@ -247,9 +247,9 @@
CLEANFILES = $(databin_SCRIPTS) $(man_MANS) $(sysconf_DATA) $(sbin_SCRIPTS)
databindir = $(datadir)
databin_SCRIPTS = mkimapdcert mkpop3dcert
-binPROGRAMS = imapd pop3d maildirmake maildiracl deliverquota maildirkw
-sbinPROGRAMS = imaplogin pop3login
-libexecPROGRAMS = makedatprog couriertcpd
+binPROGRAMS = imapd$(EXEEXT) pop3d$(EXEEXT) maildirmake$(EXEEXT) maildiracl$(EXEEXT) deliverquota$(EXEEXT) maildirkw$(EXEEXT)
+sbinPROGRAMS = imaplogin$(EXEEXT) pop3login$(EXEEXT)
+libexecPROGRAMS = makedatprog$(EXEEXT) couriertcpd$(EXEEXT)
bin_PROGRAMS = @binPROGRAMS_exec@
sbin_PROGRAMS = @sbinPROGRAMS_exec@
libexec_PROGRAMS = @libexecPROGRAMS_exec@
--- courier-imap-4.4.1.20080920/imapd.rc.in 2008-05-04 15:12:47.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/imapd.rc.in 2008-10-26 10:17:51.359375000 +0100
@@ -22,6 +22,15 @@
exit 1
fi

+# Location for some shared libraries (cygcourierauth*.dll) from authlib
+AUTHLIBDIR=/usr/local
+if test ! -f $AUTHLIBDIR/lib/bin/cygcourierauth.dll
+then
+ echo "$AUTHLIBDIR/lib/bin/cygcourierauth.dll not found."
+ exit 1
+fi
+export PATH=$PATH:$AUTHLIBDIR/lib/bin
+
TLS_CACHEFILE=""
. @sysconfdir@/imapd-ssl
. @sysconfdir@/imapd
@@ -35,7 +44,7 @@

umask $IMAP_UMASK
@ULIMIT@ $IMAP_ULIMITD
- @SETENV@ -i @SHELL@ -c " set -a ;
+ @SHELL@ -c " set -a ;
prefix=@prefix@ ;
exec_prefix=@exec_prefix@ ;
bindir=@bindir@ ;
--- courier-imap-4.4.1.20080920/pop3d.rc.in 2008-05-04 15:12:47.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/pop3d.rc.in 2008-10-26 10:18:02.187500000 +0100
@@ -22,12 +22,21 @@
exit 1
fi

+# Location for some shared libraries (cygcourierauth*.dll) from authlib
+AUTHLIBDIR=/usr/local
+if test ! -f $AUTHLIBDIR/lib/bin/cygcourierauth.dll
+then
+ echo "$AUTHLIBDIR/lib/bin/cygcourierauth.dll not found."
+ exit 1
+fi
+export PATH=$PATH:$AUTHLIBDIR/lib/bin
+
. @sysconfdir@/pop3d-ssl
. @sysconfdir@/pop3d

case $1 in
start)
- @SETENV@ -i @SHELL@ -c " set -a ;
+ @SHELL@ -c " set -a ;
prefix=@prefix@ ;
exec_prefix=@exec_prefix@ ;
bindir=@bindir@ ;
</source>

Install the patch

$ patch -Np1 < courier-imap-4.4.1-cygwin.patch

Configure the package

$ ./configure --disable-root-check --with-waitfunc=wait

Take a walk... and when ready execute the following command to build the package.

$ make

Time to go for some coffee... and then install the package with the usual

$ make install
$ make install-configure

=Configuring Courier=

==courier-authlib==

Create authdaemonrc file: /usr/local/etc/authlib/authdaemonrc

$ cp /usr/local/etc/authlib/authdaemonrc.dist /usr/local/etc/authlib/authdaemonrc

Change number of daemons (I use three) and enable DEBUG_LOGIN

<source lang=text>
# The number of daemon processes that are started.
daemons=3

# DEBUG_LOGIN=2 - turn on debugging + log passwords too
DEBUG_LOGIN=2
</source>

See http://www.courier-mta.org/authlib/README.authdebug.html for details.

==courier-imap==

Setup courier-imap: $prefix/etc/imapd (where $prefix=/usr/lib/courier-imap)

Set ADDRESS to a valid value (0 or 127.0.0.1) and change IMAP_ULIMITD to a value accepted by the command ulimit. I simply used the existing value returned by

$ ulimit -v

==IMAP accounts==

User accounts and settings are managed by ''userdb''. See [http://linux.die.net/man/8/makeuserdb makeuserdb] man page for details, or used the freshly new install man page - man makeuserdb

Assuming ''username'' is your username, 1001 and 10545 are respectively the UID and GID of an existing user (part of your /etc/passwd and /etc/group). I use my own UID and GID. Create the mailuser home directory, if required, and create Maildir folder.

$ mkdir -p /home/username
$ cd /home/username
$ /usr/lib/courier-imap/bin/maildirmake Maildir

Create the userdb text file /usr/local/etc/authlib/userdb

$ touch /usr/local/etc/authlib/userdb
$ chmod 700 /usr/local/etc/authlib/userdb

Edit /usr/local/etc/authlib/userdb, one line per account

<source lang=text>
username[TAB]uid=1001|gid=10111|home=/home/username|mail=/home/username/Maildir|systempw=XXXXXX
</source>

where XXXXXX is the encrypted password created with

$ /usr/local/sbin/userdbpw.exe

Create the (binary) userdb

$ /usr/local/sbin/makeuserdb

You should now have the following files created:

* /usr/local/etc/authlib/userdb.dat
* /usr/local/etc/authlib/userdbshadow.dat

==Configuration Summary==

$ /usr/local/bin/courierauthconfig.exe --version
$ /usr/local/bin/courierauthconfig.exe --ldflags
$ /usr/local/bin/courierauthconfig.exe --cppflags
$ /usr/local/bin/courierauthconfig.exe --configfiles

=Running=

Install and configure syslogd. Run the [http://www.cygwin.com/setup.exe Cygwin setup] program. Locate, select and install inetutils (contains syslogd). Configure syslogd with the following command. It will create the /etc/syslogd.conf file and install a windows service using cygrunsrv

$ syslogd-config
$ net start syslogd

Start the authlib daemon with the following command:

$ /usr/local/sbin/authdaemond start

Stop the daemon with the following command:

$ /usr/local/sbin/authdaemond stop

You'll find log messages in /var/log/messages (default target file as specified in /etc/syslogd.conf)

Syslogd service and authdaemond should be started. See above.

Start the imap daemon with the following command:

$ /usr/lib/courier-imap/libexec/imapd.rc start

Stop the daemon with the following command:

$ /usr/lib/courier-imap/libexec/imapd.rc stop

You'll find log messages in /var/log/messages (default target file as specified in /etc/syslogd.conf)

=Testing=

$ /usr/local/sbin/authtest username CCCCCC
Authentication succeeded.

Authenticated: username (uid 1001, gid 10545)
Home Directory: /home/username
Maildir: /home/username/Maildir
Quota: (none)
Encrypted Password: XXXXXX
Cleartext Password: CCCCCC
Options: (none)

where username is your user name and XXXXXX is your encrypted password as provided in ''/usr/local/etc/authlib/userdb''. CCCCCC is your password in clear.

When authdaemond and imapd are running you can do a simple login test:

$ telnet localhost 143

and type the login command

a login <username> <password>

the server should reply

a OK LOGIN Ok.

to which you can reply with a logout

a logout

the session is terminated by the server

* BYE Courier-IMAP server shutting down
a OK LOGOUT completed
Connection closed by foreign host.

'''Hint:''' if the connection is immediately closed by foreign host it probably means that ''imapd'' does not have access to all required DLLs. Check your PATH.

==Troubleshooting==

Should you have a doubt on the execution of a command (ie: no output) you can check if the application is able to load required DLL. This error is not reported. Using ''strace'', windows will complain about DLL not found, if any. To solve this problem you may need to add some path in your PATH. For example, in order to test authlib with authtest:

$ strace /usr/local/sbin/authtest
# This will open a dialog box to complain about DLL.
$ export PATH=$PATH:/usr/local/lib/bin
$ /usr/local/sbin/authtest ausername

=Installing as a Windows Service=

==Startup Script==

If everything works fine you can use the following [{{#file:imapd}} startup script] to launch or stop imap and authlib daemons. It can be saved in /usr/local/sbin.

<source lang=bash>
#! /bin/sh
#
# Courier imap daemon startup script.

authlib=/usr/local/sbin/authdaemond
imapd=/usr/lib/courier-imap/libexec/imapd.rc

test ! -f $authlib && echo "File not found: $authlib" && exit 1;
test ! -f $imapd && echo "File not found: $imapd" && exit 1;

case $1 in
start)
$authlib start
$imapd start
;;
stop)
$imapd stop
$authlib stop
;;
*)
echo "Usage: $0 <start|stop>"
;;
esac
</source>

==Cygwin Service==

Create the [{{#file:imapd-srv.sh}} service script]

<source lang=bash>
#!/bin/sh
# File: imapd-service.sh
# Purpose: Courier imap daemon service script.
#
# NOTE: This script must be excutable by SYSTEM
username=username # <<<< Change me !!!
pidsleep=''

# Function to handle QUIT signal.
# Stop imapd daemon and kill background sleep.
function handleQuit
{
echo "STOPPING: /usr/local/sbin/imapd"
su $username -c "/usr/local/sbin/imapd stop"
test -n "$pidsleep" && kill -9 $pidsleep
exit
}

# Interruptible sleep
# Note: sleep is an external command (not builtin command), hence
# not interruptible by the QUIT signal.
function intsleep
{
sleep $1 &
pidsleep=$!
wait $pidsleep
}

echo "STARTING: /usr/local/sbin/imapd as $username at "`date`
su $username -c "/usr/local/sbin/imapd start"

trap "handleQuit" SIGQUIT

echo "WAITING: "$$
while true; do intsleep 1d; done
</source>

Change username and save it to your favorite location, says /usr/local/sbin. The service script must be executable by SYSTEM, ie: the user that run the windows services.

$ chmod +rx /usr/local/sbin/imapd-service.sh

Install the new service with the following command.

$ cygrunsrv --install imapd --desc "Courier IMAP daemon" --disp "CYGWIN imapd" \
--path /usr/local/sbin/imapd-service.sh --termsig QUIT --type auto --shutdown

Finally, just start the service

$ cygrunsrv --start imapd
or
$ net start imapd

You should now have couple of processes related to courier-imap

$ ps -a

'''NOTES''':
* If the service could not be started, it's probably because the service script is not executable by SYSTEM

$ net start imapd
The CYGWIN imapd service is starting.
The CYGWIN imapd service could not be started.

The service did not report an error.

More help is available by typing NET HELPMSG 3534.

* The combination of background sleep and builtin wait command is used to make the waiting statment interruptible. The original script found on the net, was simply using a 'sleep 1' which turned to consume ~10% of the CPU.

=Other Interesting Packages=

==Maildir==

'''maildirmake''' is part of the courier-imap package

$ /usr/lib/courier-imap/bin/maildirmake

==Mairix - index and search mail folders==
Run the [http://www.cygwin.com/setup.exe Cygwin setup] program. Locate, select and install the package mairix.

Create the configuration file in your home directory: ~/.mairixrc

base=/home/username
maildir=Maildir...
omit=Maildir/.Mairix**
mfolder=Maildir/.Mairix
database=/home/username/.mairix_database

The create the mairix index

$ mairix
and
$ mairix --fast-index # on daily base

See [[Mail_Tips]] for additional details.

==Offlineimap==
Requires python, so run the [http://www.cygwin.com/setup.exe Cygwin setup] program. Locate, select and install the package python.

Download offlineimap [http://software.complete.org/software/projects/list_files/offlineimap here]. I use [http://software.complete.org/software/versions/download/242?attachment_id=334 version 6.0.3]. Extract files in you favorite sandbox.

$ tar xvfz offlineimap_6.0.3.tar.gz
$ cd offlineimap

Apply the following [{{#file:offlineimap_6.0.3-cygwin.patch}} patch] to correct the following issues:

* [http://en.wikipedia.org/wiki/Maildir#Windows_software Suffix separator] character colon (:) is not allowed in Windows. Courier-imap for cygwin is using (!)
* While used by courier-imap for cygwin Maildir mail files should not use CRLF, rather only LF. The issue appears only in mail with attachment, that is the raw message is not interpreted and displayed as it in the mail body (under Thunderbird).

<source lang=diff>
--- offlineimap.conf 2008-08-13 20:56:04.000000000 +0200
+++ offlineimap-cygwin.conf 2008-11-06 10:44:31.066750700 +0100
@@ -215,6 +215,19 @@

restoreatime = no

+# Cygwin/Windows users may have issues with the Maildir mail filename
+# containing the suffix separator character (:). In this case, they
+# could change the suffix separator character to whatever required.
+# For example, Courier-IMAP for Cygwin is using (!)
+
+# suffixsep = !
+
+# Cygwin/Windows users may need to create Maildir mail filename in
+# binary mode. In this case, set the 'filemode' to 'binary'. Default is
+# 'text'. Courier-IMAP for Cygwin works better with binary mode.
+
+# filemode = binary
+
[Repository RemoteExample]

# And this is the remote repository. We only support IMAP or Gmail here.
--- offlineimap/folder/Maildir.py 2008-08-13 20:55:48.000000000 +0200
+++ offlineimap/folder/Maildir-cygwin.py 2008-11-06 10:44:05.344202700 +0100
@@ -23,7 +23,6 @@
import os.path, os, re, time, socket, md5

uidmatchre = re.compile(',U=(\d+)')
-flagmatchre = re.compile(':.*2,([A-Z]+)')

timeseq = 0
lasttime = long(0)
@@ -54,6 +53,12 @@
self.messagelist = None
self.repository = repository
self.accountname = accountname
+ self.filemode = {
+ 'binary': 'wb',
+ 'text': 'wt',
+ }[repository.getconf("filemode", 'text')]
+ self.suffixsep = repository.getconf("suffixsep",':')
+ self.flagmatchre = re.compile('%s.*2,([A-Z]+)' % self.suffixsep)
BaseFolder.__init__(self)

def getaccountname(self):
@@ -102,7 +107,7 @@
nouidcounter -= 1
else:
uid = long(uidmatch.group(1))
- flagmatch = flagmatchre.search(messagename)
+ flagmatch = self.flagmatchre.search(messagename)
flags = []
if flagmatch:
flags = [x for x in flagmatch.group(1)]
@@ -180,7 +185,7 @@
break
tmpmessagename = messagename.split(',')[0]
ui.debug('maildir', 'savemessage: using temporary name %s' % tmpmessagename)
- file = open(os.path.join(tmpdir, tmpmessagename), "wt")
+ file = open(os.path.join(tmpdir, tmpmessagename), self.filemode)
file.write(content)

# Make sure the data hits the disk
@@ -226,11 +231,11 @@
newpath = os.path.join(self.getfullname(), 'cur')
else:
newpath = os.path.join(self.getfullname(), 'new')
- infostr = ':'
- infomatch = re.search('(:.*)$', newname)
+ infostr = self.suffixsep
+ infomatch = re.search('(%s.*)$' % self.suffixsep, newname)
if infomatch: # If the info string is present..
infostr = infomatch.group(1)
- newname = newname.split(':')[0] # Strip off the info string.
+ newname = newname.split(self.suffixsep)[0] # Strip off the info string.
infostr = re.sub('2,[A-Z]*', '', infostr)
flags.sort()
infostr += '2,' + ''.join(flags)
</source>

$ patch -Np0 < offlineimap_6.0.3-cygwin.patch

Install offlineimap

$ python setup.py install

Configure offlineimap as described in [[Offlineimap]] and launch offlineimap. '''CAUTION:''' Add the NEW suffixsep and filemode options in the Maildir repository section of the configuration file.

$ offlineimap
or
$ offlineimap -1 -o -u Noninteractive.Basic
or
$ offlineimap -c /home/username/offlineimap.conf

Here is a simple [{{#file:offlineimap.conf}} offlineimap configuration file].

<source lang=text>
[general]
metadata = /home/username/.offlineimap
accounts = Test
maxsyncaccounts = 1
ui = Curses.Blinkenlights, TTY.TTYUI,
Noninteractive.Basic, Noninteractive.Quiet
ignore-readonly = no

[mbnames]
enabled = no

[ui.Curses.Blinkenlights]
statuschar = .

[Account Test]
localrepository = LocalExample
remoterepository = RemoteExample

[Repository LocalExample]
type = Maildir
localfolders = /home/username/Maildir
sep = .
restoreatime = no

suffixsep = ! # <<< NEW
filemode = binary # <<< NEW

[Repository RemoteExample]
type = IMAP
remotehost = imapserver
ssl = no
remoteport = 143
remoteuser = username
remotepass = XXXXXX
maxconnections = 1
holdconnectionopen = no
nametrans = lambda foldername: re.sub('^INBOX\.*', '.', foldername)
folderfilter = lambda foldername: not re.search('(SPAM$|Mairix)', foldername)
</source>

Courier-Cygwin

2008-11-06T13:28:50Z

Fuujuhi: /* Running */ Typo and added "net start syslogd"

Courier-Cygwin

2008-11-05T22:15:21Z

Fuujuhi: Fixed small typo (courrier -> courier)

=Pre-requisite=

Courier-IMAP requires the installation of the [http://www.courier-mta.org/authlib/ Courier Authentication Library] and the following cygwin tools and packages

* patch, tar, make, gcc
* crypt
* libgdbm-devel
* libtool
* inetutils
* cygrunsrv

Run the [http://www.cygwin.com/setup.exe Cygwin setup] program. Locate, select and install the required packages.

=Courier Authentication Library=

Installation of Courier Authentication Library using the module userdb to manage mail accounts. All other modules are disabled.

==Compilation and installation==

Download and untar [http://www.courier-mta.org/download.php#authlib courier-authlib] in your favorite sandbox.
Version used: [http://prdownloads.sourceforge.net/courier/courier-authlib-0.61.0.tar.bz2 0.61.0]

Apply the [{{#file:courier-authlib-0.61.0-cygwin.patch}} patch] before configuration to correct following issues:

# incorrect usage of EXEEXT in ''makedat/Makefile''
# add libtool flag (-no-undefined) required for DLL creation
# add missing dependencies
# add PATH in start script to access new DLL (installed in a different directory)

Solution source: http://lists.cairographics.org/archives/cairo/2004-April/001125.html

<source lang=diff>
--- courier-authlib-0.61.0/makedat/Makefile.in 2008-05-24 16:21:09.000000000 +0200
+++ courier-authlib-0.61.0-cygwin/makedat/Makefile.in 2008-10-21 16:02:38.709166700 +0200
@@ -182,7 +182,7 @@
libexecdir = @libexecdir@
localedir = @localedir@
localstatedir = @localstatedir@
-makedatprog_target = @makedatprog_target@
+makedatprog_target = @makedatprog_target@$(EXEEXT)
makedatprogpath = @makedatprogpath@
mandir = @mandir@
mkdir_p = @mkdir_p@
@@ -198,7 +198,7 @@
target_alias = @target_alias@
top_builddir = @top_builddir@
top_srcdir = @top_srcdir@
-noinst_PROGRAMS = @makedatprog_target@
+noinst_PROGRAMS = @makedatprog_target@$(EXEEXT)
makedatprog_SOURCES = makedatprog.c
makedatprog_DEPENDENCIES = @dblibrary@
makedatprog_LDADD = @dblibrary@
--- courier-authlib-0.61.0/Makefile.in 2008-07-12 21:41:08.000000000 +0200
+++ courier-authlib-0.61.0-cygwin/Makefile.in 2008-10-23 22:59:03.843750000 +0200
@@ -213,9 +213,10 @@
LTCOMPILE = $(LIBTOOL) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) \
--mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) \
$(AM_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS)
-CCLD = $(CC)
+CCLD = $(CC) -no-undefined
+CCLDEXE = $(CC)
LINK = $(LIBTOOL) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) \
- --mode=link $(CCLD) $(AM_CFLAGS) $(CFLAGS) $(AM_LDFLAGS) \
+ --mode=link $(CCLDEXE) $(AM_CFLAGS) $(CFLAGS) $(AM_LDFLAGS) \
$(LDFLAGS) -o $@
SOURCES = $(libauthcustom_la_SOURCES) $(libauthldap_la_SOURCES) \
$(libauthmysql_la_SOURCES) $(libauthpam_la_SOURCES) \
@@ -452,9 +453,9 @@
README.authdebug.html

DISTCLEANFILES = dbobj.config README_authlib.html
-commonlibdep = libcourierauthcommon.la
+commonlibdep = libcourierauthcommon.la libcourierauth.la
commonldflags = -module -rpath $(pkglibdir) -export-symbols-regex 'courier_auth.*_init' -avoid-version
-commonlibadd = libcourierauthcommon.la
+commonlibadd = libcourierauthcommon.la libcourierauth.la
libcourierauthcommon_t = @CRYPTLIBS@
libcourierauthcommon_la_SOURCES = \
auth.h courierauth.h \
--- courier-authlib-0.61.0/authdaemond.in 2005-07-05 14:25:08.000000000 +0200
+++ courier-authlib-0.61.0-cygwin/authdaemond.in 2008-10-25 12:03:32.140625000 +0200
@@ -15,4 +15,10 @@
set -a
. @authdaemonrc@

+# Some shared libraries (DLL) are installed in @libdir@/bin
+# instead of @libdir@/@PACKAGE@
+# Setting LD_LIBRARY_PATH at runtime or LR_RUN_PATH at linktime doesn't
+# work in cygwin, only setting PATH works.
+export PATH=$PATH:@libdir@/bin
+
exec ${sbindir}/courierlogger -pid=@authdaemonvar@/pid $LOGGEROPTS -$1 @libexecdir@/courier-authlib/authdaemond
</source>

Install the patch

$ patch -Np1 < courier-authlib-0.61.0-cygwin.patch

Configure the package without most authentication modules, keeping only ''userdb''.

'''Note''': replace ''mailuser'' by an existing user name (in your /etc/passwd file). I used my own username.

$ ./configure --disable-root-check --with-waitfunc=wait --without-authpam --without-authldap --without-authpwd \
--without-authshadow --without-authcustom --without-authpipe --without-authmysql --without-authpgsql --with-mailuser=mailuser \
--with-mailgroup=mkgroup-l-d

Take a long pause... and when ready execute the following command.

$ make

Take again a long pause... then check the result and install the libraries

$ make check
$ make install

'''NOTES''': list of related topics found during investigation for compilation
* authldap: configure: error: -lresolve is needed for res_query... http://www.cygwin.com/ml/cygwin/2005-04/msg00419.html

==Configuration==

Create authdaemonrc file: /usr/local/etc/authlib/authdaemonrc

$ cp /usr/local/etc/authlib/authdaemonrc.dist /usr/local/etc/authlib/authdaemonrc

Change number of daemons (I use three) and enable DEBUG_LOGIN

<source lang=text>
# The number of daemon processes that are started.
daemons=3

# DEBUG_LOGIN=2 - turn on debugging + log passwords too
DEBUG_LOGIN=2
</source>

See http://www.courier-mta.org/authlib/README.authdebug.html for details.

==IMAP accounts==

User accounts and settings are managed by ''userdb''. See [http://linux.die.net/man/8/makeuserdb makeuserdb] man page for details, or used the freshly new install man page - man makeuserdb

Assuming ''username'' is your username, 1001 and 10545 are respectively the UID and GID of an existing user (part of your /etc/passwd and /etc/group). I use my own UID and GID. Create the mailuser home directory, if required, and create Maildir folder.

$ mkdir -p /home/username
$ cd /home/username
$ /usr/lib/courier-imap/bin/maildirmake Maildir

Create the userdb text file /usr/local/etc/authlib/userdb

$ touch /usr/local/etc/authlib/userdb
$ chmod 700 /usr/local/etc/authlib/userdb

Edit /usr/local/etc/authlib/userdb, one line per account

<source lang=text>
username[TAB]uid=1001|gid=10111|home=/home/username|mail=/home/username/Maildir|systempw=XXXXXX
</source>

where XXXXXX is the encrypted password created with

$ /usr/local/sbin/userdbpw.exe

Create the (binary) userdb

$ /usr/local/sbin/makeuserdb

You should now have the following files created:

* /usr/local/etc/authlib/userdb.dat
* /usr/local/etc/authlib/userdbshadow.dat

==Running==

Install and configure syslogd. Run the [http://www.cygwin.com/setup.exe Cygwin setup] program. Locate, select and install inetutils (contains syslogd). Configure syslogd with the following command. It will create the /etc/syslogd.conf file and install a windows service using cygrunsrv

$ syslogd-config

Start the authlib daemon with the following command:

$ /usr/local/sbin/authdaemon start

Stop the daemon with the following command:

$ /usr/local/sbin/authdaemon stop

You'll find log messages in /var/log/messages (default target file as specified in /etc/syslogd.conf)

==Testing authlib==

$ /usr/local/sbin/authtest username CCCCCC
Authentication succeeded.

Authenticated: username (uid 1001, gid 10545)
Home Directory: /home/username
Maildir: /home/username/Maildir
Quota: (none)
Encrypted Password: XXXXXX
Cleartext Password: CCCCCC
Options: (none)

where username is your user name and XXXXXX is your encrypted password as provided in ''/usr/local/etc/authlib/userdb''. CCCCCC is your password in clear.

==Troubleshooting==

Should you have a doubt on the execution of a command (ie: no output) you can check if the application is able to load required DLL. This error is not reported. Using ''strace'', windows will complain about DLL not found, if any. To solve this problem you may need to add some path in your PATH. For example, in order to test authlib with authtest:

$ strace /usr/local/sbin/authtest
# This will open a dialog box to complain about DLL.
$ export PATH=$PATH:/usr/local/lib/bin
$ /usr/local/sbin/authtest ausername

==Courier Authentication Config==

$ /usr/local/bin/courierauthconfig.exe --version
$ /usr/local/bin/courierauthconfig.exe --ldflags
$ /usr/local/bin/courierauthconfig.exe --cppflags
$ /usr/local/bin/courierauthconfig.exe --configfiles

=Courier-IMAP=
==Compilation and installation==

Download and untar package [http://www.courier-mta.org/download.php#imap courier-imap] in your favorite sandbox.
Version used: [http://prdownloads.sourceforge.net/courier/courier-imap-4.4.1.tar.bz2 4.4.1.20080920]

Apply the [{{#file:courier-imap-4.4.1-cygwin.patch}} patch] before configuration to correct following issues:

# incorrect usage of EXEEXT in ''makedat/Makefile''
# incorrect usage of EXTEXT in main ''Makefile''
# remove usage of /usr/lib/env in start/stop script (/usr/lib/env does not seem to work under Cygwin)

<source lang=diff>
--- courier-imap-4.4.1.20080920/makedat/Makefile.in 2008-08-24 19:52:51.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/makedat/Makefile.in 2008-10-21 20:43:27.500000000 +0200
@@ -182,7 +182,7 @@
libexecdir = @libexecdir@
localedir = @localedir@
localstatedir = @localstatedir@
-makedatprog_target = @makedatprog_target@
+makedatprog_target = @makedatprog_target@$(EXEEXT)
makedatprogpath = @makedatprogpath@
mandir = @mandir@
mkdir_p = @mkdir_p@
@@ -198,7 +198,7 @@
target_alias = @target_alias@
top_builddir = @top_builddir@
top_srcdir = @top_srcdir@
-noinst_PROGRAMS = @makedatprog_target@
+noinst_PROGRAMS = @makedatprog_target@$(EXEEXT)
makedatprog_SOURCES = makedatprog.c
makedatprog_DEPENDENCIES = @dblibrary@
makedatprog_LDADD = @dblibrary@
--- courier-imap-4.4.1.20080920/Makefile.in 2008-09-20 14:48:46.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/Makefile.in 2008-10-21 20:59:45.156250000 +0200
@@ -247,9 +247,9 @@
CLEANFILES = $(databin_SCRIPTS) $(man_MANS) $(sysconf_DATA) $(sbin_SCRIPTS)
databindir = $(datadir)
databin_SCRIPTS = mkimapdcert mkpop3dcert
-binPROGRAMS = imapd pop3d maildirmake maildiracl deliverquota maildirkw
-sbinPROGRAMS = imaplogin pop3login
-libexecPROGRAMS = makedatprog couriertcpd
+binPROGRAMS = imapd$(EXEEXT) pop3d$(EXEEXT) maildirmake$(EXEEXT) maildiracl$(EXEEXT) deliverquota$(EXEEXT) maildirkw$(EXEEXT)
+sbinPROGRAMS = imaplogin$(EXEEXT) pop3login$(EXEEXT)
+libexecPROGRAMS = makedatprog$(EXEEXT) couriertcpd$(EXEEXT)
bin_PROGRAMS = @binPROGRAMS_exec@
sbin_PROGRAMS = @sbinPROGRAMS_exec@
libexec_PROGRAMS = @libexecPROGRAMS_exec@
--- courier-imap-4.4.1.20080920/imapd.rc.in 2008-05-04 15:12:47.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/imapd.rc.in 2008-10-26 10:17:51.359375000 +0100
@@ -22,6 +22,15 @@
exit 1
fi

+# Location for some shared libraries (cygcourierauth*.dll) from authlib
+AUTHLIBDIR=/usr/local
+if test ! -f $AUTHLIBDIR/lib/bin/cygcourierauth.dll
+then
+ echo "$AUTHLIBDIR/lib/bin/cygcourierauth.dll not found."
+ exit 1
+fi
+export PATH=$PATH:$AUTHLIBDIR/lib/bin
+
TLS_CACHEFILE=""
. @sysconfdir@/imapd-ssl
. @sysconfdir@/imapd
@@ -35,7 +44,7 @@

umask $IMAP_UMASK
@ULIMIT@ $IMAP_ULIMITD
- @SETENV@ -i @SHELL@ -c " set -a ;
+ @SHELL@ -c " set -a ;
prefix=@prefix@ ;
exec_prefix=@exec_prefix@ ;
bindir=@bindir@ ;
--- courier-imap-4.4.1.20080920/pop3d.rc.in 2008-05-04 15:12:47.000000000 +0200
+++ courier-imap-4.4.1.20080920-cygwin/pop3d.rc.in 2008-10-26 10:18:02.187500000 +0100
@@ -22,12 +22,21 @@
exit 1
fi

+# Location for some shared libraries (cygcourierauth*.dll) from authlib
+AUTHLIBDIR=/usr/local
+if test ! -f $AUTHLIBDIR/lib/bin/cygcourierauth.dll
+then
+ echo "$AUTHLIBDIR/lib/bin/cygcourierauth.dll not found."
+ exit 1
+fi
+export PATH=$PATH:$AUTHLIBDIR/lib/bin
+
. @sysconfdir@/pop3d-ssl
. @sysconfdir@/pop3d

case $1 in
start)
- @SETENV@ -i @SHELL@ -c " set -a ;
+ @SHELL@ -c " set -a ;
prefix=@prefix@ ;
exec_prefix=@exec_prefix@ ;
bindir=@bindir@ ;
</source>

Install the patch

$ patch -Np1 < courier-imap-4.4.1-cygwin.patch

Configure the package

$ ./configure --disable-root-check --with-waitfunc=wait

Take a walk... and when ready execute the following command to build the package.

$ make

Time to go for some coffee... and then install the package with the usual

$ make install
$ make install-configure

==Configuration==

Setup courier-imap: $prefix/etc/imapd (where $prefix=/usr/lib/courier-imap)

Set ADDRESS to a valid value (0 or 127.0.0.1) and change IMAP_ULIMITD to a value accepted by the command ulimit. I simply used the existing value returned by

$ ulimit -v

==Running==

Syslogd service and authdaemond should be started. See above.

Start the imap daemon with the following command:

$ /usr/lib/courier-imap/libexec/imapd.rc start

Stop the daemon with the following command:

$ /usr/lib/courier-imap/libexec/imapd.rc stop

You'll find log messages in /var/log/messages (default target file as specified in /etc/syslogd.conf)

==Testing==
When authdaemond and imapd are running you can do a simple login test:

$ telnet localhost 143

and type the login command

a login <username> <password>

the server should reply

a OK LOGIN Ok.

to which you can reply with a logout

a logout

the session is terminated by the server

* BYE Courier-IMAP server shutting down
a OK LOGOUT completed
Connection closed by foreign host.

'''Hint:''' if the connection is immediately closed by foreign host it probably means that ''imapd'' does not have access to all required DLLs. Check your PATH.

==All in one==
If everything works fine you can use the following [{{#file:imapd}} startup script] to launch or stop imap and authlib daemons. It can be saved in /usr/local/sbin.

<source lang=bash>
#! /bin/sh
#
# Courier imap daemon startup script.

authlib=/usr/local/sbin/authdaemond
imapd=/usr/lib/courier-imap/libexec/imapd.rc

test ! -f $authlib && echo "File not found: $authlib" && exit 1;
test ! -f $imapd && echo "File not found: $imapd" && exit 1;

case $1 in
start)
$authlib start
$imapd start
;;
stop)
$imapd stop
$authlib stop
;;
*)
echo "Usage: $0 <start|stop>"
;;
esac
</source>

==Cygwin service==
Create the [{{#file:imapd-srv.sh}} service script]

<source lang=bash>
#!/bin/sh
# File: imapd-service.sh
# Purpose: Courier imap daemon service script.
#
# NOTE: This script must be excutable by SYSTEM
username=username # <<<< Change me !!!
pidsleep=''

# Function to handle QUIT signal.
# Stop imapd daemon and kill background sleep.
function handleQuit
{
echo "STOPPING: /usr/local/sbin/imapd"
su $username -c "/usr/local/sbin/imapd stop"
test -n "$pidsleep" && kill -9 $pidsleep
exit
}

# Interruptible sleep
# Note: sleep is an external command (not builtin command), hence
# not interruptible by the QUIT signal.
function intsleep
{
sleep $1 &
pidsleep=$!
wait $pidsleep
}

echo "STARTING: /usr/local/sbin/imapd as $username at "`date`
su $username -c "/usr/local/sbin/imapd start"

trap "handleQuit" SIGQUIT

echo "WAITING: "$$
while true; do intsleep 1d; done
</source>

Change username and save it to your favorite location, says /usr/local/sbin. The service script must be executable by SYSTEM, ie: the user that run the windows services.

$ chmod +rx /usr/local/sbin/imapd-service.sh

Install the new service with the following command.

$ cygrunsrv --install imapd --desc "Courier IMAP daemon" --disp "CYGWIN imapd" \
--path /usr/local/sbin/imapd-service.sh --termsig QUIT --type auto --shutdown

Finally, just start the service

$ cygrunsrv --start imapd
or
$ net start imapd

You should now have couple of processes related to courier-imap

$ ps -a

'''NOTES''':
* If the service could not be started, it's probably because the service script is not executable by SYSTEM

$ net start imapd
The CYGWIN imapd service is starting.
The CYGWIN imapd service could not be started.

The service did not report an error.

More help is available by typing NET HELPMSG 3534.

* The combination of background sleep and builtin wait command is used to make the waiting statment interruptible. The original script found on the net, was simply using a 'sleep 1' which turned to consume ~10% of the CPU.

=Maildir=
'''maildirmake''' is part of the courier-imap package

$ /usr/lib/courier-imap/bin/maildirmake

=Mairix - index and search mail folders=
Run the [http://www.cygwin.com/setup.exe Cygwin setup] program. Locate, select and install the package mairix.

Create the configuration file in your home directory: ~/.mairixrc

base=/home/username
maildir=Maildir...
omit=Maildir/.Mairix**
mfolder=Maildir/.Mairix
database=/home/username/.mairix_database

The create the mairix index

$ mairix
and
$ mairix --fast-index # on daily base

See [[Mail_Tips]] for additional details.

=Offlineimap=
Requires python, so run the [http://www.cygwin.com/setup.exe Cygwin setup] program. Locate, select and install the package python.

Download offlineimap [http://software.complete.org/software/projects/list_files/offlineimap here]. I use [http://software.complete.org/software/versions/download/242?attachment_id=334 version 6.0.3]. Extract files in you favorite sandbox.

$ tar xvfz offlineimap_6.0.3.tar.gz
$ cd offlineimap

Apply the following [{{#file:offlineimap_6.0.3-cygwin.patch}} patch] to correct the following issues:

* [http://en.wikipedia.org/wiki/Maildir#Windows_software Suffix separator character] (:) is not allowed in Windows. Courier-imap for cygwin is using (!)
* While used by courier-imap for cygwin Maildir mail files should not use CRLF, rather only LF. The issue appears only in mail with attachment, that is the raw message is not interpreted and displayed as it in the mail body (under Thunderbird).

<source lang=diff>
--- offlineimap/folder/Maildir.py 2008-11-01 20:49:30.843750000 +0100
+++ offlineimap/folder/Maildir-cygwin.py 2008-11-03 22:54:51.437500000 +0100
@@ -23,7 +23,7 @@
import os.path, os, re, time, socket, md5

uidmatchre = re.compile(',U=(\d+)')
-flagmatchre = re.compile(':.*2,([A-Z]+)')
+flagmatchre = re.compile('!.*2,([A-Z]+)')

timeseq = 0
lasttime = long(0)
@@ -180,7 +180,7 @@
break
tmpmessagename = messagename.split(',')[0]
ui.debug('maildir', 'savemessage: using temporary name %s' % tmpmessagename)
- file = open(os.path.join(tmpdir, tmpmessagename), "wt")
+ file = open(os.path.join(tmpdir, tmpmessagename), "wb")
file.write(content)

# Make sure the data hits the disk
@@ -226,11 +226,11 @@
newpath = os.path.join(self.getfullname(), 'cur')
else:
newpath = os.path.join(self.getfullname(), 'new')
- infostr = ':'
- infomatch = re.search('(:.*)$', newname)
+ infostr = '!'
+ infomatch = re.search('(!.*)$', newname)
if infomatch: # If the info string is present..
infostr = infomatch.group(1)
- newname = newname.split(':')[0] # Strip off the info string.
+ newname = newname.split('!')[0] # Strip off the info string.
infostr = re.sub('2,[A-Z]*', '', infostr)
flags.sort()
infostr += '2,' + ''.join(flags)
</source>

$ cd offlineimap/folder
$ patch -Np2 < ../../offlineimap_6.0.3-cygwin.patch
$ cd ../..

Install offlineimap

$ python setup.py install

Configure offlineimap as described in [[Offlineimap]] and launch offlineimap

$ offlineimap
or
$ offlineimap -1 -o -u Noninteractive.Basic
or
$ offlineimap -c /home/username/offlineimap.conf

Here is a simple [{{#file:offlineimap.conf}} offlineimap configuration file].

<source lang=text>
[general]
metadata = /home/username/.offlineimap
accounts = Test
maxsyncaccounts = 1
ui = Curses.Blinkenlights, TTY.TTYUI,
Noninteractive.Basic, Noninteractive.Quiet
ignore-readonly = no

[mbnames]
enabled = no

[ui.Curses.Blinkenlights]
statuschar = .

[Account Test]
localrepository = LocalExample
remoterepository = RemoteExample

[Repository LocalExample]
type = Maildir
localfolders = /home/username/Maildir
sep = .
restoreatime = no

[Repository RemoteExample]
type = IMAP
remotehost = imapserver
ssl = no
remoteport = 143
remoteuser = username
remotepass = XXXXXX
maxconnections = 1
holdconnectionopen = no
nametrans = lambda foldername: re.sub('^INBOX\.*', '.', foldername)
folderfilter = lambda foldername: not re.search('(SPAM$|Mairix)', foldername)
</source>

MiscCrypto

2008-10-09T10:00:50Z

Fuujuhi: /* To unlock password-protected pdfs */

==Under Windows==
===[http://www.cryptool.com/ CryptTool]===
* Demonstration and Reference Program for Cryptography + making AES executables
* freeware
===[http://axcrypt.sourceforge.net/ AxCrypt]===
* AES-128 File Encryption, Compression and double-click Edit/View for secure document storage on Windows 95/98/ME/NT/2K/XP, locally or remote.
* GPL
===[http://www.chilkatsoft.com/ChilkatSfx.asp Zip 2 Secure EXE]===
* AES-128/192/256 bits File Encryption, Create self-extracting EXE files from zip files
* freeware
==[[LoopCrypt]]==
==[[Encfs]]==
==[[TrueCrypt]]==
==[[LUKS]]==

==Misc==
===To unlock password-protected pdfs===
* PDF documents may have up to 2 passwords:
** '''User password''': protects read access to the document, and when no ''Owner password'' is specified, also protects permission settings.
** '''Owner password''': protects full access to the document, and protects permission settings (printing, copying...)

* Use <tt>pdftops</tt> to convert a protected PDF to Postscript, and possibly combine it with <tt>ps2pdf</tt> to obtain an unrestricted pdf. Knowing the ''Owner password'' is the easiest since that allows removing all restrictions. If not, it also works but in that case the permission <tt>allow Printing</tt> must be set. If it is not the case, then ''Owner password'' is necessary. Also if ''User password'' is set (ie. pdf reader asks for a password at open), then it must be provided (providing ''Owner password'' is also ok of course).

<source lang="bash">
apt-get install gs-common xpdf-utils
pdftops <encrypted>.pdf <decrypted>.ps # Owner pwd set but unknown and 'allow printing' set
pdftops -upw <userpwd> <encrypted>.pdf <decrypted>.ps # Same as above but user password also set
pdftops -opw <ownerpwd> <encrypted>.pdf <decrypted>.ps # 'allow printing' not set -> owner pwd is necessary

pdftops -upw <password> <encrypted>.pdf -|ps2pdf - <decrypted>.pdf # To generate a new PDF immediately
</source>

* An other solution is to use ''PDF Toolkit'' <tt>pdftk</tt> (<tt>apt-get install pdftk</tt>). However unlike ''pdftops'' this method doesn't work when only ''user password'' is known but both passwords are set.

<source lang="bash">pdftk <encrypted>.pdf input_pw <password> output <decrypted>.pdf</source>

* <tt>pdftk</tt> can also be used to generate protected documents, and specifying permissions.

<source lang="bash">
pdftk <unprotected>.pdf output <protected>.pdf allow Printing owner_pw <ownerpwd> user_pw <userpwd> #Allow HQ printing
</source>

===pwsafe===
To use 'secure ram' (thereby locking the data in memory and preventing it being flushed to disk) you may like to run it setuid root:
$ sudo dpkg-statoverride --update --add --force root root 4755 /usr/bin/pwsafe

Outlook

2008-10-07T14:01:07Z

Fuujuhi: /* Macro Send & Save */

A page about Outlook on this wiki? Sounds weird isn't it?
 Well, blame my employer.

We migrated from Lotus Notes, with crappy hacked outdated client so Outlook is still better but...
 At least under Lotus Notes it was possible to add an IMAP account (BTW Lotus Notes crashed quite often when talking IMAP...)
 Here: Tools->Account Settings->''This feature has been disabled by your system administrator''

== References ==
* Microsoft Office 2007 Resource Kit (official documentation from Microsoft)
** [http://technet.microsoft.com/en-gb/library/cc178979.aspx Documentation page]
*** [http://go.microsoft.com/fwlink/?LinkId=85671 Security for the 2007 Office release]
*** [http://go.microsoft.com/fwlink/?LinkID=94264 Group Policy overview for the 2007 Office release]
** [http://go.microsoft.com/fwlink/?LinkId=78161 Administrative template (ADM,ADMX,...) files]
*** Use [http://www.7-zip.org/ 7-zip] to decompress this file without installing it. Don't miss the summary excel sheet inside.
* ''Configuring Microsoft Outlook 2003'', By Sue Mosher, Robert Sparnaaij, Charlie Pulfer, David Hooker.
** Excellent book for administering Outlook 2003. Content can be browsed/searched on [http://books.google.com/books?id=v8oZNEByTR0C Google Books].
* Excellent 3rd party page on Outlook policy: http://www.howto-outlook.com/howto/policies.htm
* Folder with interesting policy settings on Outlook [http://theether.net/download/Microsoft/Office/2007/].

== Unlocking ==

===Main registry keys===
Apparently important registry keys are under:
HKEY_CURRENT_USER\Software\Microsoft\Office\12.0\Outlook
HKEY_CURRENT_USER\Software\Microsoft\Windows NT\CurrentVersion\Windows Messaging Subsystem\Profiles\Outlook
HKEY_CURRENT_USER\Software\Policies\Microsoft\Office\12.0\Outlook
Some tips [http://www.hotline-pc.org/outlook.htm here] (french)

===IMAP settings===

We can remove the admin barrier on account management:
HKEY_CURRENT_USER\Software\Policies\Microsoft\Office\12.0\Outlook\Setup\ModifyAccounts
=0

Another way without modifying the Registry is to use the import feature:
 If e.g. Outlook Express client is also present on the Windows machine
* In Outlook Express:
** Wizard -> typed my IMAP&SMTP settings
* In Outlook:
** File -> Import and Export... -> Import Internet Mail Account Settings -> Microsoft Outlook Express or Microsoft Windows Mail -> Wizard allowing to edit the imported settings :-)
** Then write a new mail -> Account (below Send button) -> select your new account -> Send test mail -> Prompted to choose the sent folder, ok -> you've a multiple tabs config view of your new account, possibility to switch servers to TLS, changing name of the account, saying you need auth to talk to your SMTP etc etc
** If you need to tune IMAP settings later, go to registry and reset key (000000NN must be replaced with the number of your IMAP profile)
HKEY_CURRENT_USER\Software\Microsoft\Windows NT\CurrentVersion\Windows Messaging Subsystem\Profiles\Outlook\9375CFF0413111d3B88A00104B2A6676\000000NN\IMAP Sentitems
= 0

Note that I also tried importing from Eudora Light & Eudora but Outlook didn't see anything

=== Cached Exchange Mode ===
The following keys were set in my configuration, preventing to change the settings related to ''Cached Exchange Mode''. In particular the mode selection was disabled, ie. the options ''Download full items'', ''Download headers and then full items'', ''Download headers'' were all grayed out.
<source lang="reg">
[HKEY_CURRENT_USER\Software\Policies\Microsoft\Office\12.0\Outlook\Cached Mode]
"CachedExchangeMode"=dword:00000001
"Enable"=dword:00000001
"NoDrizzle"=dword:00000001
"NoFullItems"=dword:00000001
"NoHeaders"=dword:00000000
"NoManualOnlineSync"=dword:00000001
"SyncPFFav"=dword:00000001
</source>
To unlock all settings, and eg. allow download full headers, simply delete values ''CachedExchangeMode'', ''NoDrizzle'', ''NoFullItems'', ''NoHeaders'', and ''NoManualOnlineSync'' (ie. all but ''Enable'' and ''SyncPFFav'').

===Changing JunkMail levels===
HKEY_CURRENT_USER\Software\Policies\Microsoft\Office\12.0\Outlook\Options\Mail\JunkMailProtection
No Protection = 0xffffffff = 4294967295
Low = 6
High = 3
Trusted Lists Only = 0x80000000 = 2147483648
To enable the JunkMail otions in the Options->Preferences->E-Mail->Junk E-mail menu
HKEY_CURRENT_USER\Software\Policies\Microsoft\Office\12.0\Outlook\DisableAntiSpam
=0

===All in one registry file===
You can also put all the above registry modifications in one single file [{{#file:Outlook.reg}} Outlook.reg]
<source lang=reg>
Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\Software\Policies\Microsoft\Office\12.0\Outlook]
"DisableAntiSpam"=dword:00000000

[HKEY_CURRENT_USER\Software\Policies\Microsoft\Office\12.0\Outlook\Cached Mode]
"NoFullItems"=-
"NoDrizzle"=-
"NoHeaders"=-
"CachedExchangeMode"=-
"NoManualOnlineSync"=-

[HKEY_CURRENT_USER\Software\Policies\Microsoft\Office\12.0\Outlook\Options\Mail]
"JunkMailProtection"=-

[HKEY_CURRENT_USER\Software\Policies\Microsoft\Office\12.0\Outlook\Setup]
"ModifyAccounts"=dword:00000000
</source>

...and modify the registry at each startup with the following command:
regedit /s Outlook.reg

'''Note''': the hyphen after the registry value indicates a request to delete the value. See http://support.microsoft.com/kb/310516 for details.

==Group Policy and other configuration hints==

* [http://technet.microsoft.com/en-us/library/cc303396.aspx Configure Outlook security and protection features]
* [http://books.google.com/books?id=v8oZNEByTR0C&pg=PA232&lpg=PA232&dq=Outlook+policy+JunkMailProtection+registry&source=web&ots=NgPRrRu7ms&sig=0Fz7fiRm-hpc8GQTp9od3GtLLrM&hl=en&sa=X&oi=book_result&resnum=9&ct=result#PPA221,M1 Configuring Microsoft Outlook 2003]

To use the Group Policy for Microsoft Outlook 2007 you may need to install the Outlook Group Policy template [http://www.yobi.be/files/outlk12.adm Outlk12.adm]

# Copy the policy template Outlk12.adm into %WINDIR%\inf
# Open Group Policy Editor: Start-> Run-> gpedit.msc
# Add template:
## Select User Configuration
## Right-click on Administrative Templates
## Select Add/Remove properties-> Click Add button
## Browse to %WINDIR%\inf and select Outlk12.adm

==iCalendar==
Outlook does not handle correctly the iCal file (*.ics) with recurrent event.

Problem: when opening (eg.: double-click on the attachment) an iCal file with multiple events, ie. multiple VEVENT sections, Outlook tries to add a complete calendar. Your are prompted with a dialog box with the following question: '''Add this Internet Calendar to Outlook'''. It is probably not what you want. Most probably you just need to add the multiple events in your existing calendar.

The solution requires to save the iCal file on your hard drive and import it back using the import function.

# Save file as...
# File -> Import and Export...
# Select ''Import and iCalendar (*.ics) or vCalendar (.vcs)''
# Browse and select your saved iCal file, done.

Solution source: https://events.sfu.ca/manual/en/iCalExporter.htm.

==Moving PST location==
cf [http://azcarya.blogspot.com/2008/02/gmail-outlook-2007-and-imap.html this post]

# Close Outlook.
# Open Control Panel -> Choose Mails -> Click on Data files.
# Select the Account name and check for the location of the PST file. Leave the window open.
# Open the folder containing the PST. Move the PST to the desired location.
# Do NOT rename the file - if you do Outlook creates a new file in the default location again.
# Switch to the Data Files window (as opened in Step 2) and double click on the PST file location.
# This displays an error window, Ignore the error, and point to the new location. Close the windows.
# Restart Outlook.
# Outlook may create a new PST in the default location again, delete it, in such a case.

==Recovering PST password==
Password protection of PST is terribly wrong.

Get [http://www.nirsoft.net/utils/pst_password.html PstPassword, a free Outlook PST Password Recovery], launch it and launch your PST, you'll get instantly a password, actuallt many passwords, and they will all work! How is it possible? Read [http://www.nirsoft.net/articles/pst_password_bug.html this].

Note that McAfee detects the program as "Potentially Unwanted Program", a "[http://vil.nai.com/vil/content/v_150020.htm Generic PUP.z]":
 ''McAfee(R) Avert™ recognizes that this program may have legitimate uses in contexts where an authorized administrator has knowingly installed this application''
 But still no question, the program is simply erased by McAfee.
 One easy solution: open it under andLinux and run it with WINE :-)

Apparently this can work both on "compressible encryption" and "high encryption" types of PST.
==Macro Send & Save==
Lotus Notes can prompt the sender to specify what folder a message should be saved in. In Outlook, you can set the storage folder on the Options dialog. This VBA code reproduces something closer to the Notes behavior by popping up the Select Folder dialog when the user sends the message. (see [http://www.outlookcode.com/d/code/setsavefolder.htm here] and [http://www.outlookcode.com/codedetail.aspx?id=456 there])
 ''UPDATE:'' Avoid treating appointments as it fails.
<source lang=vb>
Private Sub Application_ItemSend(ByVal Item As Object, _
Cancel As Boolean)
Dim objNS As NameSpace
Dim objFolder As MAPIFolder
Set objNS = Application.GetNamespace("MAPI")
If TypeOf Item Is MailItem Then
Set objFolder = objNS.PickFolder
If TypeName(objFolder) <> "Nothing" Then
Set Item.SaveSentMessageFolder = objFolder
End If
End If
Set objFolder = Nothing
Set objNS = Nothing
End Sub
</source>
To install it:
* Alt-F11 in Outlook
* Open Project1 -> Microsoft Office Outlook Objects -> ThisOutlookSession (double-click)
* copy/paste code
* close & save
Then allow unsigned macros:
* Tools -> Macro -> Security -> Macro Security -> Warning for all macros
You'll have annoying popup at startup but there is no way around

==Interoperability between Exchange/Outlook & Linux==

===PST to Maildir===

* In Outlook:
** New -> Outlook Data File... -> Outlook 97-2002 Personal Folders file
** Copy/Move mails to it
** Right-Click on it -> Close it (probably safer)
* In Linux:
** Convert to mbox using readpst, then to Maildir using mb2md
<source lang=bash>
mkdir pst
readpst -r -o pst backup.pst
find pst -name mbox -size 0 -exec rm {} \;
find pst -mindepth 2 -type d -print0|xargs --null -I XXX mb2md -s "$(pwd)/XXX/mbox" -
find pst -mindepth 2 -name mbox -print0|xargs --null rm
</source>

I tried the PST Import plugin v1.2 for Thunderbird but it failed finding any mail in the PST
===Openchange===
I tried the one from Debian Experimental (version 1.0~svn679-1)
mapiprofile --address=A.B.C.D --username=name --password=passwd --domain=DOM --workstation=BLABLA --profile=test --create
-> segfault from libmapi0

I tried to compile the libmapi0 from the SVN, no segfault anymore but still failed
===Evolution===
Apparently ther is an attempt to support MAPI in a development branch but it has too much recent dependencies for me to compile it...
svn checkout http://svn.gnome.org/svn/evolution/branches/EXCHANGE_MAPI_BRANCH

svn diff http://svn.gnome.org/svn/evolution/trunk http://svn.gnome.org/svn/evolution/branches/EXCHANGE_MAPI_BRANCH

MiscCrypto

2008-10-01T16:01:42Z

Fuujuhi: /* To unlock password-protected pdfs */ user pwd/owner pwd

==Under Windows==
===[http://www.cryptool.com/ CryptTool]===
* Demonstration and Reference Program for Cryptography + making AES executables
* freeware
===[http://axcrypt.sourceforge.net/ AxCrypt]===
* AES-128 File Encryption, Compression and double-click Edit/View for secure document storage on Windows 95/98/ME/NT/2K/XP, locally or remote.
* GPL
===[http://www.chilkatsoft.com/ChilkatSfx.asp Zip 2 Secure EXE]===
* AES-128/192/256 bits File Encryption, Create self-extracting EXE files from zip files
* freeware
==[[LoopCrypt]]==
==[[Encfs]]==
==[[TrueCrypt]]==
==[[LUKS]]==

==Misc==
===To unlock password-protected pdfs===
* PDF documents may have up to 2 passwords:
** '''User password''': protects read access to the document, and when no ''Owner password'' is specified, also protects permission settings.
** '''Owner password''': protects full access to the document, and protects permission settings (printing, copying...)

* Use <tt>pdftops</tt> to remove password from a protected PDF. It works with either password, but when both passwords are set, it also works when only the ''user password'' is known (upon condition that permission <tt>allow Printing</tt> is set).

<source lang="bash">
apt-get install gs-common xpdf-utils
pdftops -upw <password> <encrypted>.pdf <decrypted>.ps # To generate a PostScript file (e.g. for further editing)
pdftops -upw <password> <encrypted>.pdf -|ps2pdf - <decrypted>.pdf # To generate a new PDF
pdftops -opw <password> <encrypted>.pdf -|ps2pdf - <decrypted>.pdf # When the Owner Password is known...
</source>

* An other solution is to use ''PDF Toolkit'' <tt>pdftk</tt> (<tt>apt-get install pdftk</tt>). However unlike ''pdftops'' this method doesn't work when only ''user password'' is known but both passwords are set.

<source lang="bash">pdftk <encrypted>.pdf input_pw <password> output <decrypted>.pdf</source>

* <tt>pdftk</tt> can also be used to generate protected documents, and specifying permissions.

<source lang="bash">
pdftk <unprotected>.pdf output <protected>.pdf allow Printing owner_pw <ownerpwd> user_pw <userpwd> #Allow HQ printing
</source>

===pwsafe===
To use 'secure ram' (thereby locking the data in memory and preventing it being flushed to disk) you may like to run it setuid root:
$ sudo dpkg-statoverride --update --add --force root root 4755 /usr/bin/pwsafe

Outlook

2008-09-25T09:46:12Z

Fuujuhi: /* Cached Mode */

Outlook

2008-09-25T09:34:03Z

Fuujuhi: /* Unlocking */ Cached mode - allow full header download

Outlook

2008-09-23T20:23:31Z

Fuujuhi: /* References */ Some interesting links