YobiWiki - User contributions [en]

SAGE & cryptology

2009-12-24T06:59:37Z

Newacct:

Back to [[Sage]]
==Discussions==
* [http://groups.google.com/group/sage-devel/browse_thread/thread/9557f299b1ec2fab?hl=en this thread in the ML]
* [http://www.informatik.uni-bremen.de/cgi-bin/cgiwrap/malb/blosxom.pl this blog] and especially [http://www.informatik.uni-bremen.de/cgi-bin/cgiwrap/malb/blosxom.pl/2008/03/21#sage-crypto-todo this post]
* [http://modular.math.washington.edu/sage/jsage/ JSAGE]
* [http://modular.math.washington.edu/sage/dev.html notes for developers]
==Docs==
* http://www.cryptlib.com/standards-compliance.htm
* [http://www.sagemath.org/doc/html/prog/index.html Sage Programming Guide]
* Sage on Google Groups
** [http://groups.google.com/group/sage-support sage-support]
** [http://groups.google.com/group/sage-devel sage-devel]
** [http://groups.google.com/group/sage-edu sage-edu]
* Python on Google Groups
** [http://groups.google.com/group/comp.lang.python/topics?hl=en comp.lang.python]

==General Remarks==
* When using Python code in Sage
** Sage uses an internal integer representation (''sage.rings.integer.Integer'') which differs with the ''int'' representation of Python and which can lead to errors in some circumstances. => cast ''sage'' integers to ''python'' integers before supplying them to a Python function in Sage
*** Use int(...) to cast a variable (ex. int(var) )
*** Use ...r for an immediate Python integer (ex. 384r)
*** more info [http://groups.google.com/group/sage-support/browse_thread/thread/49f5e14f36083782/f97f6997d5164da here]
*** one example: '1/2' in Python = 0, of type int (for now, it's foreseen to make it a real div, cf __future__ module, will be =0.5 of type float) but '1/2' in Sage = 1/2, of type sage.rings.rational.Rational , to solve it, either cast to int() or use operator // instead of / in Python

==TODO==
* check licensing of different packages and compare them to requirements for sage
** PyCrypto:
<blockquote> I've filed a bug in PyCrypto's bug tracker: https://bugs.launchpad.net/pycrypto/+bug/260130
The PyCrypto licensing status is a bit of a mess. It looks like a bunch of reference implementations were simply copied-and-pasted into the source tree, and each has its own licensing statement.
I recommend looking at each source file and making a judgment for yourself.
I'm slowly working on a new release of PyCrypto (I've just taken over from Andrew Kuchling). In the next release, I'll try to document things better, and fix the most obvious problems (I've already written a replacement for RIPEMD.c).
However, some of the software is unattributed. I assume that most of it was written by A.M. Kuchling, but I can't be totally sure. I'll try to contact Andrew and see if he can clear things up.
- Dwayne
</blockquote>
* analyse structure of integers and strings and see if the representation from sage is compatible with the one from python
** Integers: Integers in Sage are a different type than the Python integers. Problems can occur when executing standard python code in Sage. To avoid problems: add 'r' after a number to let it be interpreted as a Python integer. More info: Google Groups: [http://groups.google.com/group/sage-support/browse_thread/thread/49f5e14f36083782/f97f6997d5164da0] [http://groups.google.com/group/sage-support/browse_thread/thread/a815fa8c73411da3/6c967fbd741a31c7] Sage Tutorial on differences caused by the Sage preparser: [http://www.sagemath.org/doc/tut/node72.html]
** Strings: Strings in Sage are the same type as Python strings
* analyse the format of objects used by different libraries and see if they are compatible
** Almost all libraries used "binary strings" as input/output with some exceptions:
*** PyCrypto: RSA signatures are "long"
*** TLS Lite: RSA signatures are "array of bytes" (defined by the TLS Lite library)
* write a unified API for the different libraries
* write wrapper for internal C library
* check [http://code.google.com/p/keyczar/ keyczar], the new lib of Google, also [http://www.keyczar.org/pydocs/index.html available in Python]
===TODO for Phil===
* make some speed tests with [http://psyco.sourceforge.net/ psyco], claiming to run python code up to 5x faster but: Psyco does not support the 64-bit x86 architecture, unless you have a Python compiled in 32-bit compatibility mode. There are no plans to port Psyco to 64-bit architectures. This would be rather involved. Psyco is only being maintained, not further developed. The development efforts of the author are now focused on [http://codespeak.net/pypy/dist/pypy/doc/home.html PyPy], which includes Psyco-like techniques.
** so to be tried on the desktop and if efficient, for the IT box
** see also [http://www.sagemath.org/doc/html/prog/node38.html doc of Sage]
** here is an [http://bitecode.co.uk/2008/07/password-cracking-in-python-with-psyco/ example] and I tried on the long2string() fastest implementation, it works even on so small code:
<source lang=python>
def long2string(i):
s='0'+hex(i)[2:-1]
return s[len(s) % 2:].decode('hex')
timeit long2string(123456789012345678901234)
#100000 loops, best of 3: 2.7 µs per loop
import psyco
psyco.full()
timeit long2string(123456789012345678901234)
#1000000 loops, best of 3: 1.44 µs per loop
</source>
* Try also [http://www.msri.org/about/computing/docs/sage/prog/node7.html Pyrex]
Setup a subversion server and explain how to use it, for new Python code developments...

==Available==
===sage.crypto===
[http://www.sagemath.org/doc/html/ref/node171.html Sage Reference Manual] 
[http://www.sagemath.org/doc/html/const/index.html Constructions in Sage] 
* sage.crypto.all
* sage.crypto.cipher
* sage.crypto.classical
* sage.crypto.classical_cipher
** hillchipher
** substitutioncipher
** transpositioncipher
** vigenerecipher
* sage.crypto.cryptosystem
* sage.crypto.lfsr
** Module Level Functions
*** lfsr_autocorrelation(L, p, k)
*** lfsr_connection_polynomial(s)
*** lfsr_sequence(key, fill, n)
** Examples:
sage: F = GF(2)
sage: o = F(0)
sage: l = F(1)
sage: key = [l,o,o,l]; fill = [l,l,o,l]; n = 20
sage: s = '''lfsr_sequence'''(key,fill,n)
sage: '''lfsr_autocorrelation'''(s,15,7)
4/15
sage: lfsr_autocorrelation(s,int(15),7)
4/15

sage: F = GF(2)
sage: F
Finite Field of size 2
sage: o = F(0); l = F(1)
sage: key = [l,o,o,l]; fill = [l,l,o,l]; n = 20
sage: s = '''lfsr_sequence'''(key,fill,n); s
[1, 1, 0, 1, 0, 1, 1, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 1, 0]
sage: '''lfsr_connection_polynomial'''(s)
x^4 + x + 1
sage: berlekamp_massey(s)
x^4 + x^3 + 1
* sage.crypto.stream_cipher [http://www.sagemath.org/doc/html/ref/module-sage.crypto.stream-cipher.html]
** Class: LFSRCipher
** Class: ShrinkingGeneratorCipher
*** new(input): input = connection polynomial & initial state
*** decimating_cipher(self)
sage: FF = FiniteField(2)
sage: P.<x> = PolynomialRing(FF)
sage: LFSR = LFSRCryptosystem(FF)
sage: IS_1 = [ FF(a) for a in [0,1,0,1,0,0,0] ]
sage: e1 = LFSR((x^7 + x + 1,IS_1))
sage: IS_2 = [ FF(a) for a in [0,0,1,0,0,0,1,0,1] ]
sage: e2 = LFSR((x^9 + x^3 + 1,IS_2))
sage: E = ShrinkingGeneratorCryptosystem()
sage: e = E((e1,e2))
sage: e.decimating_cipher()
(x^9 + x^3 + 1, [0, 0, 1, 0, 0, 0, 1, 0, 1])
*** keystream_cipher(self)
...
sage: e.keystream_cipher()
(x^7 + x + 1, [0, 1, 0, 1, 0, 0, 0])
* sage.crypto.mq
** SBox Class: sage.crypto.mq.SBOX: see [http://wiki.sagemath.org/sage-2.10.4 wiki.sagemath.org]
*** S.difference_distribution_matrix()
*** S.maximal_difference_probability()
*** S.interpolation_polynomial()
*** S.polynomials(degree=2)
** Small Scale Variants of the AES (SR) Polynomial System Generator: cage.crypto.mq.sr [http://modular.math.washington.edu/sage/doc/html/ref/module-sage.crypto.mq.sr.html Reference Manual]
** Multivariate Polynomial Systems: sage.crypto.mq.mpolynomialsystem [http://modular.math.washington.edu/sage/doc/html/ref/module-sage.crypto.mq.mpolynomialsystem.html]

===PyCrypto, the Python Cryptography Toolkit===
Sage ships [http://www.amk.ca/python/code/crypto PyCrypto] ([http://www.dlitz.net/software/pycrypto/ new maintainer's page?]) which implements many standard cryptographic algorithms. It is not really meant for research/education/playing around but for production code but maybe something could be done to have easier access to it from within Sage. The docstring level documentation is horrible:
sage: import Crypto.Cipher.IDEA
sage: Crypto.Cipher.IDEA?
x.__init__(...) initializes x; see x.__class__.__doc__ for signature

Manual is available [http://www.amk.ca/python/writing/pycrypt/pycrypt.html here]. 
-> Contains also some info on how to extend the toolkit with new algorithms[http://www.amk.ca/python/writing/pycrypt/pycrypt.html#SECTION000900000000000000000] 
A blog about PyCrypto [http://pycrypto.blogspot.com here] 
 
PyCrypto mostly consist of C code with a Python wrapper. 
 
INPUT/OUTPUT:
* always "binary strings" where each character represents 8 bits 
* except for RSA signatures -> those are "long"s
 
* '''Hash functions'''([http://www.amk.ca/python/writing/pycrypt/pycrypt.html#SECTION000300000000000000000 Manual]): MD2, MD4, MD5, RIPEMD, SHA256, SHA, HMAC
** SHA1 example: [http://www.yobi.be:8300/home/pub/9 on Sage Notebook]
** HMAC example: [http://www.yobi.be:8300/home/pub/17/ on Sage Notebook]
** All hash functions support the API described by [http://www.python.org/dev/peps/pep-0247/ PEP 247]: after importing a given hashing module, call the '''new()''' function to create a new hashing object. You can now feed arbitrary strings into the object with the '''update()''' method, and can ask for the hash value at any time by calling the '''digest()''' or '''hexdigest()''' methods. The '''new()''' function can also be passed an optional string parameter that will be immediately hashed into the object's state.[http://www.amk.ca/python/writing/pycrypt/pycrypt.html#SECTION000300000000000000000] Using the argument '''digest_size''' you can get the digest size but its constant.
** MD5, SHA and HMAC are just the standard Python implementations
** MD2, MD4, SHA256 and RIPEMD-160 are C implementations wrapped by PyCrypto
* '''Block encryption algorithms'''([http://www.amk.ca/python/writing/pycrypt/pycrypt.html#SECTION000400000000000000000 Manual]): AES, ARC2, Blowfish, CAST, DES, Triple-DES, IDEA, RC5, RC2
** AES & DES example: [http://www.yobi.be:8300/home/pub/7 on Sage Notebook]
** All block cipher support the interface described in [http://www.python.org/dev/peps/pep-0272/ PEP 272]
** Chaining modes: ECB, CBC, CFB, PGP, OFB and CTR
** Possibilities: Define a new cipher object after importing the module and define the key, mode (cbc,cfb,ecb or pgp) and possible IV. The object gives you two methods: 'encrypt()' and 'decrypt()'. For AES: S-Box not modifiable, LookUp Tables are being used.
* '''Stream encryption algorithms''': ARC4, simple XOR
** RC4 example: [http://www.yobi.be:8300/home/pub/10/ on Sage Notebook]
** Support the interface described in [http://www.python.org/dev/peps/pep-0272/ PEP 272] = block cipher with block size = 1
* '''Public-key algorithms'''([http://www.amk.ca/python/writing/pycrypt/pycrypt.html#SECTION000600000000000000000 Manual]): RSA, DSA, ElGamal, qNEW
** RSA example: [http://www.yobi.be:8300/home/pub/16/ on Sage Notebook]
** Signature is ''Long'' instead of ''Binary String'' binascii doesn't provide long<->binary conversion[http://mail.python.org/pipermail/python-3000/2006-November/004453.html] Encrypted message is a "binary string"
** No PKCS#1 padding ''sign''() in RSA.py calls ''decrypt''() and that only does "return pow(ciphertext[0], self.d, self.n)" => no padding <-> TLS Lite has padding
** n, e and d are also provided as ''Long''
** Public key Modules '''construct'''(tuple): construct( (long(n),long(e),long(d)) ) '''generate'''(size, randfunc, progress_func=None) => public key object
**Public key Objects available methods: '''canencrypt'''(), '''cansign'''(), '''decrypt'''(tuple), '''encrypt'''(string, K), '''hasprivate'''(), '''publickey'''(), '''sign'''(string, K), '''size'''(), '''verify'''(string, signature)
* '''Protocols''': All-or-nothing transforms, chaffing/winnowing
* '''Miscellaneous''':
** '''Crypto.Util.number''' '''GCD'''(x,y),'''getPrime'''(N, randfunc),'''getRandomNumber'''(N, randfunc),'''inverse'''(u, v),'''isPrime'''(N)
** '''Crypto.Util.randpoo'''l[http://www.amk.ca/python/writing/pycrypt/pycrypt.html#SECTION000720000000000000000] The randpool module implements a strong random number generator in the RandomPool class. The internal state consists of a string of random data, which is returned as callers request it. The class keeps track of the number of bits of entropy left, and provides a function to add new random data; this data can be obtained in various ways, such as by using the variance in a user's keystroke timings.
*** Getting N random bytes:
sage: from Crypto.Util import randpool
sage: randfunc = randpool.RandomPool()
sage: randfunc.get_bytes(N)
-> returns 8-bit string consisting of N bytes
* '''Some demo programs''' (currently all quite old and outdated)

===OpenSSL===
[http://www.openssl.org/docs/crypto/crypto.html Manual] (incomplete, for example: no AES documentation) 
OpenSSL book on [http://books.google.com/books?id=FBYHEBTrZUwC Google Books] 
Functionality in OpenSLL:
* SYMMETRIC CIPHERS
:blowfish(3), cast(3), des(3), idea(3), rc2(3), rc4(3), rc5(3)
:Block Cipher Modes available: ECB, CBC, CFB, OFB
:Padding: only PKCS7 padding available?
:AES is same implementation as in pycrypto and only supports ECB and CBC (p. 175 in OpenSSL book)
* PUBLIC KEY CRYPTOGRAPHY AND KEY AGREEMENT
:dsa(3), dh(3), rsa(3)
* CERTIFICATES
:x509(3), x509v3(3)
* AUTHENTICATION CODES, HASH FUNCTIONS
:hmac(3), md2(3), md4(3), md5(3), mdc2(3), ripemd(3), sha(3)
:CBC-MAC and XCBC-MAC algorithms for OpenSSL are provided [http://books.google.com/books?id=FBYHEBTrZUwC&pg=PA205&lpg=PA205&dq=cbc-mac+openssl&source=web&ots=Am1o_i-tMJ&sig=Abx0hmynfuKMcrbgJH1qhoC1M3w&hl=en&sa=X&oi=book_result&resnum=1&ct=result#PPA205,M1 here].
* AUXILIARY FUNCTIONS
:err(3), threads(3), rand(3), OPENSSL_VERSION_NUMBER(3)
* INPUT/OUTPUT, DATA ENCODING
:asn1(3), bio(3), evp(3), pem(3), pkcs7(3), pkcs12(3)
* INTERNAL FUNCTIONS
:bn(3), buffer(3), lhash(3), objects(3), stack(3),txt_db(3)
 
Functionality of OpenSSL in Sage is provided via the PyOpenSSL wrapper. A more complete wrapper is M2Crypto but it is not available as a package for Sage. Still have to try to import it
====PyOpenSSL====
http://pyopenssl.sourceforge.net/pyOpenSSL.html/
*X509 objects
*509Name objects
*X509Req objects
*:X509Store objects
*PKey objects
*PKCS7 objects
*PKCS12 objects
*X509Extension objects
*NetscapeSPKI objects
 
Looks like less functionality than PyCrypto => PyCrypto seems like a better candidate to adjust, else we would have to extend the PyOpenSSL wrapper AND OpenSSL itself for any wanted extended functionality.
====M2Crypto====
[http://chandlerproject.org/Projects/MeTooCrypto Homepage] 
[http://www.pktek.net/docs/python/M2Crypto/M2Crypto-module.html API documentation] 
Oreilly: OpenSSL p. 258-266
 
INPUT/OUTPUT:
* always "binary strings" where each character represents 8 bits
 
* '''symmetric ciphers''' (in EVP module: AES, Blowfish, CAST5, DES, DESX, 3DES, IDEA, RC2, RC4, RC5)
** AES example: [http://www.yobi.be:8300/home/pub/8/ on Sage Notebook]
** RC4 example: [http://www.yobi.be:8300/home/pub/12/ on Sage Notebook]
*** other algo's that can be used: aes_128_x, aes_192_x, aes_256_x, bf_x, cast_x, des_x, desx_cbc, des_ede3(_x), des(_ede_x), idea_x, rc2_x, rc4(_40), rc5_32_16_12_x Where x is the chaining mode and (...) is optional
** EVP module (''from M2Crypto import EVP'')= message digests, symmetric ciphers and PK algo's
** PKCS7 padding is used
** A Cipher and Message Digest example: see [http://books.google.com/books?id=IIqwAy4qEl0C&pg=PA261&lpg=PA261&dq=EVP.MessageDigest&source=web&ots=m9061S6wTx&sig=WE-Sqje8LMCwpA370wACq1iROlo&hl=en&sa=X&oi=book_result&resnum=1&ct=result#PPA261,M1 here]
* '''message digests''' (MD5, SHA1, RipeMD-160) (in EVP module)
** SHA1 example: [http://www.yobi.be:8300/home/pub/9/ on Sage Notebook]
* '''HMAC'''
** example: [http://www.yobi.be:8300/home/pub/15/ on Sage Notebook]
** 2 possibilities: using the HMAC class or the ''hmac()'' function
** supports following hash functions: MD2/4/5,MDC2,SHA1,RipeMD-160 ''TLS Lite allows any hash functions conform to PEP 247 to be used''
** the API is not conform to the PEP 247 specifications ''The TLS Lite implementation is conform to PEP 247''
* '''RSA, DSA, DH''' (in EVP module)
** See OpenSSL book [http://books.google.com/books?id=FBYHEBTrZUwC&pg=PA263&vq=%22from+M2Crypto+import+DH,+DSA,+RSA,+RC4%22&source=gbs_search_r&cad=1_1&sig=ACfU3U2PSbjVhJdo2R3Osyz1zcRM8fx7Dw on google books]
* '''PKCS7 padding'''
** example: [http://www.yobi.be:8300/home/pub/18/ on Sage Notebook]
* SSL functionality to implement clients and servers.
* HTTPS extensions to Python's httplib, urllib, and xmlrpclib.
* Unforgeable HMAC'ing AuthCookies for web session management.
* FTP/TLS client and server.
* S/MIME.
* ZServerSSL: A HTTPS server for Zope.
* ZSmime: An S/MIME messenger for Zope.
 
More functionality than the PyOpenSSL wrapper, but not available as a Sage package. Importing in sage is easy.
=====Setup and import=====
sudo aptitude install openssl libssl-dev python-dev
''
$ python setup.py build
$ python setup.py install
''
sage: import sys
sage: sys.path.append('/usr/lib/python2.5/site-packages')
sage: import M2Crypto
or
$ sage -python setup.py install

===GnuTLS===
[http://www.gnu.org/software/gnutls/manual/html_node/index.html Manual] 
Standard package in sage. 
Mostly for certification and not for basic cryptography. 
*Support for TLS 1.1, TLS 1.0 and SSL 3.0 protocols
:Since SSL 2.0 is insecure it is not supported.
:TLS 1.2 is supported but disabled by default.
*Support for TLS extensions: server name indication, max record size, opaque PRF input, etc.
*Support for authentication using the SRP protocol.
*Support for authentication using both X.509 certificates and OpenPGP keys.
*Support for TLS Pre-Shared-Keys (PSK) extension.
*Support for Inner Application (TLS/IA) extension.
*Support for X.509 and OpenPGP certificate handling.
*Support for X.509 Proxy Certificates (RFC 3820).
*Supports all the strong encryption algorithms (including SHA-256/384/512), including Camellia (RFC 4132).
*Supports compression.
===Python-GnuTLS===
http://pypi.python.org/pypi/python-gnutls/1.1.5 
API reference built on local machine. 
Same story as with OpenSSL: C-library + python wrapper 
===TLS Lite===
[http://trevp.net/tlslite/ Homepage] 
[http://sourceforge.net/mailarchive/forum.php?forum_name=tlslite-users Mailing List Archive] 
TLS Lite hasn't been updated since February 21, 2005 
 
INPUT/OUTPUT:
* always "binary strings" where each character represents 8 bits
* exception for RSA: "array of bytes" instead of "binary strings"
 
Not available as Sage package, but it is '''pure python''' 
 
"TLS Lite is a free python library that implements SSL 3.0, TLS 1.0, and TLS 1.1. TLS Lite supports non-traditional authentication methods such as SRP, shared keys, and cryptoIDs in addition to X.509 certificates. TLS Lite is pure Python, however it can access OpenSSL, cryptlib, pycrypto, and GMPY for faster crypto operations. TLS Lite integrates with httplib, xmlrpclib, poplib, imaplib, smtplib, SocketServer, asyncore, and Twisted." 
Pure python implementations for:
*'''AES''':
**example: [http://www.yobi.be:8300/home/pub/6 on Sage Notebook]
**Interesting are AES.py, Python_AES.py and rijndael.py 
***rijndael.py originally [http://bitconjurer.org/ here] and was ported from [http://www.quadcap.com/products/qed/docs/source/_rijndael_8java-source.html this] java code
***Might be possible to modify SBox but '''only CBC-mode''' available (see Python_AES.py)
** No padding
*'''RC4'''
** example: [http://www.yobi.be:8300/home/pub/11/ on Sage Notebook]
*'''RSA'''
** example: [http://www.yobi.be:8300/home/pub/13/ on Sage Notebook]
** signature = PKCS1-SHA1
** input/output is array of bytes instead of binary string Use ''tlslite.utils.keyfactory.stringToBytes'' and ''tlslite.utils.keyfactory.bytesToString'' to convert between array of bytes and binary string
*'''TripleDES'''
** no pure python implementation. API available for cryptlib, openssl(m2crypto) and pycrypto
*'''HMAC'''
** example: [http://www.yobi.be:8300/home/pub/14/ on Sage Notebook]
** supports the API for Cryptographic Hash Functions ([http://www.python.org/dev/peps/pep-0247/ PEP 247])
** can use any hashing algorithm that is also conform to the PEP 247 specifications
** source has some comments: tlslite/utils/hmac.py
*tlslite.utils.cryptomath. ... interesting? base64ToBytes base64ToNumber base64ToString bytesToBase64 bytesToNumber gcd getBase64Nonce getRandomBytes getRandomNumber getRandomPrime getRandomSafePrime hashAndBase64 invMod isPrime lcm makeSieve mpiToNumber numberToBase64 numberToBytes numberToMPI numberToString numBytes powMod stringToBase64 stringToNumber

====Remarks====
=====Bug=====
When calling the key generation function "rsa = tlslite.utils.keyfactory.generateRSAKey(384,["python"])" it will throw an error. The TLS Lite code will also break in future versions of python (more info on the SourceForge link) 
Solutions:
* add an "r" to the number to cast it to a python Integer instead of a Sage integer. => rsa = tlslite.utils.keyfactory.generateRSAKey(384r,["python"])
* fix the code of TLS Lite: see the bugreport on [http://sourceforge.net/tracker/index.php?func=detail&aid=2027766&group_id=102605&atid=632288 SourceForge]
=====Importing in Sage=====
sage: import sys
sage: sys.path.append('/usr/lib/python2.5/site-packages')
sage: import tlslite
sage: from tlslite.api import *

===Python===
*conversion between "binary string" and "hexadecimal string"
** convert hexadecimal to appropriate string input via: ''sage: "A0B1C2".decode('hex')''
** convert string output to hexadecimal via: ''sage: "\xA0\xB1\xC2".encode('hex')''
* modules: hmac, md5, sha, hashlib[http://docs.python.org/lib/module-hashlib.html] (contains: md5(), sha1(), sha224(), sha256(), sha384(), and sha512())

===Misc===
*[http://echidna.maths.usyd.edu.au/~kohel/tch/Crypto/index.html book written on Cryptography by David Kohel], using SAGE
*[http://twhiteman.netfirms.com/des.html pyDes]: python implementation of DES and 3DES
*[http://www.cryptool.org/ Cryptool]: open source windows program for educational use of cryptography
*[http://www.libtom.org LibtomCrypt]: C library with lots of stuff with good documentation
*[https://sourceforge.net/projects/cryptopy/ CryptoPy]: has some code to analyze SBox AES Sbox Analysis - a simple analysis of the AES Sbox that determines the number and size of the permutation subgroups in the transformation. Could be extended to examine any Sbox ...
*[http://www.stuvel.eu/rsa RSA implementation in Python]
* [http://jclement.ca/software/pyrijndael/ PyRijndael]
** based on Phil Fresle's VB implementation [http://www.frez.co.uk/freecode.htm#rijndael] (doesn't provide more comments)
** two high level functions:
*** ''EncryptData''(key, data): Encrypts data using key and returns encrypted string. Uses 256 bit Rijndael cipher. Key is built from first 32 characters of password. A 32-bit message length is attached to beginning of ciphertext.
*** ''DecryptData''(key, data)
** example in the code:
PlainText="Hello World" *50
Key="Secret"
CipherText=EncryptData(Key,PlainText)
PlainText2=DecryptData(Key,CipherText)
print "PT :",PlainText
print "KY :",Key
print "PT2:",PlainText2
* Collection of Python Crypto stuff: [http://vermeulen.ca/python-cryptography.html here]
** PBKDF2 pure python implementation + example: [http://code.google.com/p/uliweb/source/browse/trunk/lib/beaker/crypto/pbkdf2.py?r=119 here]
** Public Key algo's in pure python: [http://pypi.python.org/pypi/asym/0.1.0 here]
*in Python:
** Python Enhancement Proposals
*** [http://www.python.org/dev/peps/pep-0247/ PEP 247]: API for Cryptographic Hash Functions
*** [http://www.python.org/dev/peps/pep-0272/ PEP 272]: API for Block Encryption Algorithms v1.0
*** [http://www.python.org/dev/peps/pep-0358/ PEP 358]: The "bytes" Object (not implemented yet)
*** [http://www.python.org/dev/peps/pep-3137/ PEP 3137]: Immutable Bytes and Mutable Buffer (not implemented yet)
**Random generators
*** module [http://docs.python.org/tut/node12.html#SECTION0012600000000000000000 random] (present in Sage)
*** module [http://docs.python.org/tut/node17.html#SECTION0017310000000000000000 _random] (Mersenne Twister) (present in Sage)
*** os.urandom(n) (present in Sage)
*** numpy.random core random tools of NumPy (present in Sage) (also via scipy.random)
** Resources for Python modules related to crypto:
*** [http://pypi.python.org/pypi?%3Aaction=search&term=crypt&submit=search Python Package Index]
*** [http://sourceforge.net/search/index.php?words=(%2Bpython+%2Bcrypt)&type_of_search=soft&pmode=0&words=(%2Bpython+%2Bcrypto)&Search=Search on SourceForge]
*** [http://py.vaults.ca/apyllo.py?find=crypt Vaults of Parnassus]
** [http://wiki.python.org/moin/CodingProjectIdeas/Libraries Python Coding Project Ideas]: M2Crypto & TLS Lite are mentioned there
* Blockcipher API for ECB, CBC, CFB & OFB: [http://www.nightsong.com/phr/crypto/blockcipher.tgz]

'''To be looked at:'''
* Python code for ripemd (PEP 247), rijndael, serpent, twofish, whirlpool (PEP 247), XTS
** code: [http://psionicist.online.fr/code/ here]
** blog explaining TrueCrypt using that code: [http://blog.bjrn.se/2008/01/truecrypt-explained.html here]
* [http://labix.org/python-mcrypt Python-mcrypt] python-mcrypt is a comprehensive Python interface to the mcrypt library, which is a library providing a uniform interface to several symmetric encryption algorithms. It is intended to have a simple interface to access encryption algorithms in ofb, cbc, cfb, ecb and stream modes. The algorithms it supports are DES, 3DES, RIJNDAEL, Twofish, IDEA, GOST, CAST-256, ARCFOUR, SERPENT, SAFER+, and more.
* [http://del.icio.us/tag/python%2Bcrypto on Del.icio.us]

==Wishes==
* [http://www.sagemath.org:9002/sage_trac/report/1?sort=ticket&asc=0 General trac]
* sage.crypto: block ciphers
* Someone needs to replace FiniteField_ext_pari with the two NTL implementations (they are much faster).
* elliptic and hyperelliptic curves over finite fields support is rather poor
* algebraic aspects received some attention for the cryptanalysis of symmetric cryptographic algorithms, i.e. the cryptanalyst expresses the cipher as a large set of multivariate polynomials and attempts to solve the system. The most common case over GF(2) is handled by PolyBoRi. This library is the backbone of BooleanPolynomialRing and friends. This class needs testing, documentation, extension and bugfixes. Basically someone should sit down and add all the methods of MPolynomial[Ring]_libsingular to BooleanPolynomial[Ring] which make sense, add a ton of doctests and test the hell out of the library to make sure no SIGSEGVs surprise the user.
* the module sage.crypto.mq is also relevant for the above.
* Univariate polynomials over GF(2) are still implemented via NTL's ZZ_pX class rather than GF2X. This should be changed. Also [http://trac.sagemath.org/sage_trac/ticket/2114 this ticket] has a link to gf2x a very small drop in replacement C library which claimed to be 5x faster than NTL. Though, a formal vote is needed to get it into Sage.
* At the end of the day everything boils down to linear algebra. So if you improve that, everybody wins. Sparse linear algebra mod p is still too slow (Ralf-Phillip Weinmann did some work here wrapping code from eclib), there isn o special implementation for sparse linear algebra over GF(2) (both blackbox and e.g. reduced echelon forms), dense LA over GF(2) needs Strassen multiplication/reduction, dense LA over GF(2^n) should probably get implemented.
==[[Sage ideal crypto toolbox|The ideal toolbox]]==

==[[Sage Cross Reference Table of Wishes and Availability | Cross Reference Table: wishes <-> availability]]==
==[[Sage Sandbox]]==
==[[PyCryptoPlus]]==

Sage Sandbox

2009-12-24T06:56:55Z

Newacct:

Back to [[SAGE & cryptology]]

==Helper functions==
Conversion from int or long to raw string and back is [http://bugs.python.org/issue1023290 discussed on the Python bug tracking list] but since the problem is not yet solved, let's see how to make our own helper functions in a clean and fast way: (measures with %timeit of ipython)
<source lang=python>
############################### long2string ########################################
def long2string(i):
l=[]
while i:
l[0:0]=chr(i&0xff)
i>>=8
return ''.join(l)
long2string(123456789012345678901234)
#46 µs per loop
long2string(1234567890123456789012345)
#51.8 µs per loop
###################################################################################
def long2string(i):
l=[]
while i:
l.append(chr(i&0xff))
i>>=8
return ''.join(l[::-1])
long2string(123456789012345678901234)
#15.1 µs per loop
long2string(1234567890123456789012345)
#16.4 µs per loop
###################################################################################
def long2string(i):
return ''.join([chr(i>>j & 0xFF) for j in range(int(math.log(i,2)//8)<<3,-1,-8)])
long2string(123456789012345678901234)
#15.5 µs per loop
long2string(1234567890123456789012345)
#16.7 µs per loop
###################################################################################
import math
def long2string(i):
return ('%0*x' % (int(math.ceil(math.log(i,2)/8)*2) , i)).decode('hex')
long2string(123456789012345678901234)
#4.77 µs per loop
long2string(1234567890123456789012345)
#4.83 µs per loop
###################################################################################
def long2string(i):
return ('%0*x' % (len(hex(i)) & -2 , i)).decode('hex').lstrip('\x00')
long2string(123456789012345678901234)
#3.64 µs per loop
long2string(1234567890123456789012345)
#3.78 µs per loop
###################################################################################
('%0*x' % (32 , 123456789012345678901234)).decode('hex')
#2.06 µs per loop
('%0*x' % (32 , 1234567890123456789012345)).decode('hex')
#2.01 µs per loop
#Ok if you want fixed length buffer, e.g. 32 bytes
###################################################################################
def long2string(i):
s=hex(i)[2:].rstrip('L')
if len(s) % 2:
s='0'+s
return s.decode('hex')
long2string(123456789012345678901234)
#2.88 µs per loop
long2string(1234567890123456789012345)
#3.22 µs per loop
###################################################################################
def long2string(i):
s='0'+hex(long(i))[2:-1]
return s[len(s) % 2:].decode('hex')
long2string(123456789012345678901234)
#3.04 µs per loop
long2string(1234567890123456789012345)
#3.03 µs per loop
###################################################################################
def long2string(i):
s=hex(long(i))[2:-1]
if len(s) % 2:
s='0'+s
return s.decode('hex')
long2string(123456789012345678901234)
#2.75 µs per loop
long2string(1234567890123456789012345)
#3.06 µs per loop
###################################################################################
#If we are sure our argument is a long we can remove the cast in the 2 previous codes:
def long2string(i):
s='0'+hex(i)[2:-1]
return s[len(s) % 2:].decode('hex')
long2string(123456789012345678901234)
#2.66 µs per loop
long2string(1234567890123456789012345)
#2.62 µs per loop
###################################################################################
def long2string(i):
s=hex(i)[2:-1]
if len(s) % 2:
s='0'+s
return s.decode('hex')
long2string(123456789012345678901234)
#2.42 µs per loop
long2string(1234567890123456789012345)
#2.76 µs per loop
###################################################################################
# There is still room ;-)
'1a249b1f10a06c96aff2'.decode('hex')
#990 ns per loop
###################################################################################
############################### string2long ########################################
def string2long(s):
i=0
for c in s:
i=(i<<8)+ord(c)
return i
string2long('\x01\x05n\x0f6\xa6D=\xe2\xdfy')
#8.91 µs per loop
###################################################################################
int('\x01\x05n\x0f6\xa6D=\xe2\xdfy'.encode('hex'),16)
#3.15 µs per loop
###################################################################################
int('\x01\x05n\x0f6\xa6D=\xe2\xdfy'.decode('hex'),16)
#1.64 µs per loop
###################################################################################
long('\x01\x05n\x0f6\xa6D=\xe2\xdfy'.decode('hex'),16)
#1.38 µs per loop
</source>

==Learning Python==
Some code to make some padding functions available. The code is used as a way to learn python, so it contains things that don't make sense or could be done better.

{{#fileanchor: helper.py}}
<source lang=python>
from __future__ import division #http://www.python.org/dev/peps/pep-0238/
import math

def roundUp (n, p):
"""Round an integer up to the nearest multiple

A given integer n will be round up to the nearest multiple of p

Example:
>>> roundUp(13,8)
16
"""
return int(math.ceil(n/p)*p)
#return (n+p)/p*p
</source>
Download code: [{{#filelink: helper.py}} helper.py]

{{#fileanchor: padding.py}}
<source lang=python>
import random

import sys
from optparse import OptionParser

try:
import helper
except ImportError:
print 'ERROR: helper.py should be placed in the same location as padding.py'
raise ImportError

class BlockOperator:
"""Class for BlockOperator

Only holds a variable blocksize. Class only exist for testing purpose.
"""

def __init__(self, bs=8):
self.blockSize = bs

class Padding(BlockOperator):
"""Class for padding functions

Inherits from the BlockOperator class

padding info here: http://en.wikipedia.org/wiki/Padding_(cryptography)

Example:
from padding import *
padder = Padding()
padded = padder.pad('test','bitpadding)
padder.unpad(padded,'bitpadding)

After changing the source:
del padder
del Padding
from padding import *
"""

def pad (self, toPad, algo):
#private function has to be accessed by it's public name when using getattr...
return getattr(self,"_%(classname)s__%(algo)s" % {'classname':self.__class__.__name__,'algo':algo})(toPad)

def unpad (self, padded, algo):
#there is no switch statement in python
#discussion with alternatives: http://simonwillison.net/2004/May/7/switch/
if algo == 'bitPadding':
return self.__bitPadding_unpad(padded)
elif algo == 'zerosPadding':
return self.__zerosPadding_unpad(padded)
elif algo == 'PKCS7':
return self.__PKCS7_unpad(padded)
elif algo == 'ANSI_X923':
return self.__ANSI_X923_unpad(padded)
elif algo == 'ISO_10126':
return self.__ISO_10126_unpad(padded)
raise NotImplementedError()

def __bitPadding (self, toPad ):
padded = toPad + '\x80' + '\x00'*(self.blockSize - len(toPad)%self.blockSize -1)
return padded

def __bitPadding_unpad (self, padded ):
if padded.rstrip('\x00')[-1] == '\x80':
return padded.rstrip('\x00')[:-1]
else:
return padded

def __zerosPadding (self, toPad ):
totalLength = helper.roundUp(len(toPad),self.blockSize)
return toPad.ljust(totalLength,'\x00')

def __zerosPadding_unpad (self, padded ):
return padded.rstrip('\x00')

def __PKCS7 (self, toPad ):
"""Pad a binary string

Input:
toPad: binary string to be padded
self.blockSize: the padded result will be a multiple of the self.blockSize(default=8)
Output:
return a binary string
"""

pattern = self.blockSize - len(toPad)%self.blockSize
patternstring = chr(pattern)
amount = self.blockSize - len(toPad)%self.blockSize
pad = patternstring * amount
return toPad + pad

def __PKCS7_unpad (self, padded ):
pattern = padded[-1]
length = ord(pattern)
#check if the bytes to be removed are all the same pattern
if padded.endswith(pattern*length):
return padded[:-length]
else:
return padded
print 'error: padding pattern not recognized'

def __ANSI_X923 (self, toPad ):
bytesToPad = self.blockSize - len(toPad)%self.blockSize
pattern = '\x00'*(bytesToPad -1) + chr(bytesToPad)
return toPad + pattern

def __ANSI_X923_unpad (self, padded ):
length = ord(padded[-1])
#check if the bytes to be removed are all zero
if padded.count('\x00',-length,-1) == length - 1:
return padded[:-length]
else:
print 'error: padding pattern not recognized %s' % padded.count('\x00',-length,-1)
return unpadded

def __ISO_10126 (self, toPad):
bytesToPad = self.blockSize - len(toPad)%self.blockSize
pattern1 = ''.join(chr(random.randrange(256)) for x in range(bytesToPad-1))
pattern2 = chr(bytesToPad)
return toPad + pattern1 + pattern2

def __ISO_10126_unpad (self, padded):
return padded[:-ord(padded[-1])]

def main():
usage = "usage: %prog [options] [texttopad]"
parser = OptionParser(usage=usage)
parser.add_option("-b", "--blocksize", dest='blocksize', default=8, help="specify the bloksize", type='int')
(options, args) = parser.parse_args()

if len(args) > 1:
parser.error("Program takes maximum 1 argument")

try:
testbench(args[0].decode('string_escape'),options.blocksize)
except IndexError:
testbench(blocksize=options.blocksize)

def testbench(toPad='test',blocksize=8):
testbench = ('bitPadding','zerosPadding','PKCS7','ANSI_X923','ISO_10126')
padder = Padding(blocksize)
print "String to be padded: %r, with length %i\n" % (toPad,len(toPad))
for test in testbench:
padded = padder.pad(toPad,'%s'%test)
print "padded: %r" % padded
unpadded = padder.unpad(padded,'%s'%test)
print "unpadded: %r" % unpadded
if unpadded == toPad:
print "%s test OK!\n" % test
else:
print "%s test not OK!\n" % test

if __name__ == "__main__":
main()
</source>
Download code: [{{#filelink: padding.py}} padding.py]

Mediawiki RawFile

2009-10-19T08:02:38Z

Newacct: /* Hook to intercept the raw output */

==Very short introduction==
Just have a look to the 2 [[Mediawiki_RawFile#Short_example|examples]] to see how to use the extension
 and to the [[Mediawiki_RawFile#Installation|Installation]] section to see how to install the extension in your [[MediaWiki]] server
==Introduction==
Originally the idea was to be able to download directly a portion of code as a file.
 I've numerous code examples in my wiki and I wanted an easy way to download them, easier than a copy/paste!
 But from there it was rather easy to get something very close to [http://en.wikipedia.org/wiki/Literate_programming literate programming] just by allowing multiple blocks referring to the same file, which will be concatenated together at download time.

* It must work with pre, nowiki, js, css, code, source, so let's make it general: take the tag that comes after the parser function we'll create and select data up to the closing tag.
* There are two distinct functionalities provided by the extension:
** the parser that will convert a magic word into a link to the download URL
** an extended ?action=raw that will strip the raw output to keep the desired code

==Syntax==
There are 3 kinds of elements to add to the wiki language:
* anchors that will flag which code blocks belong to a specific file
** '''<code><nowiki>{{#fileAnchor: myscript.sh}}</nowiki></code>'''
** Not visible in the regular wiki display
* links that will allow to download the file
** '''<code><nowiki>{{#fileLink: myscript.sh}}</nowiki></code>'''
** or '''<code><nowiki>{{#fileLink: myscript.sh|Another Page}}</nowiki></code>'''
** The first syntax is used when the file is on the same page. It is transformed into new regular wikicode that will be eventually transformed to real URLs: <code><nowiki>{{fullurl:{{PAGENAME}}|action=raw&file=myscript.sh}}</nowiki></code> <code><nowiki>http://wiki.yobi.be/index.php?title=Mediawiki_RawFile&action=raw&file=myscript.sh</nowiki></code>
** The second syntax indicates that the file is on another local wiki page. It is eventually transformed into the real URLs: <code><nowiki>{{fullurl:{{PAGENAME}}|action=raw&file=myscript.sh}}</nowiki></code> <code><nowiki>http://wiki.yobi.be/index.php?title=Another_Page&action=raw&file=myscript.sh</nowiki></code>
* a shortcut notation mixing both an anchor and download link, handy for regular use, when a single code block is used and when the download link can be at the same position as the anchor
** '''<code><nowiki>{{#file: myscript.sh}}</nowiki></code>'''

===Short example===
The extension works with any block such as pre, nowiki, js, css, code, source,...
 This example is using the syntax highlighting <nowiki><source></nowiki> tag provided by [http://www.mediawiki.org/wiki/Extension:SyntaxHighlight_GeSHi SyntaxHighlight extension] (using [http://qbnz.com/highlighter/ GeSHi Highlighter])
 If you didn't install that extension on your MediaWiki, you can try the example by using <nowiki><pre></nowiki> instead of <nowiki><source></nowiki>.

<pre>Let's save the following code [{{#file: myscript.sh}} as myscript.sh]
<source lang=bash>
#!/bin/bash

echo 'Hello world!'
exit 0
</source>
</pre>
will give:
----
Let's save the following code [{{#file: myscript.sh}} as myscript.sh]
<source lang=bash>
#!/bin/bash

echo 'Hello world!'
exit 0
</source>

===Complete example===
And a full example with anchors & link:

<pre>
Let's start with the Bash usual header:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
#!/bin/bash
</source>
Then we'll display a welcome message:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
echo 'Welcome on earth!'
</source>
And we finally exit cleanly:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
exit 0
</source>
[{{#filelink: myotherscript.sh}} myotherscript.sh is now available for download below the code]
</pre>
will give:
----
Let's start with the Bash usual header:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
#!/bin/bash
</source>
Then we'll display a welcome message:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
echo 'Welcome on earth!'
</source>
And we finally exit cleanly:
{{#fileanchor: myotherscript.sh}}
<source lang=bash>
exit 0
</source>
[{{#filelink: myotherscript.sh}} myotherscript.sh is now available for download below the code]

==The code (the ultimate example)==
Which you can of course download just by following [{{#filelink: RawFile.php}} this link :-)]

So let's explain a bit the code in a Literate Programming way...
===Hooks===
First some hooks for our functions...

We will create:
* a [http://www.mediawiki.org/wiki/Manual:Parser_functions Parser Function] (see also [http://meta.wikimedia.org/wiki/Help:Parser_function here]), with help of
** [http://www.mediawiki.org/wiki/Manual:%24wgExtensionFunctions $wgExtensionFunctions] to define the setup function
** [http://www.mediawiki.org/wiki/Manual:Magic_words Magic Words]
** [http://www.mediawiki.org/wiki/Manual:Hooks/LanguageGetMagic LanguageGetMagic] hook to initialize the magic words
* a [http://www.mediawiki.org/wiki/Manual:Hooks/RawPageViewBeforeOutput RawPageViewBeforeOutput] hook to intercept the raw output

{{#fileanchor: RawFile.php}}
<source lang=php>
<?php

if (defined('MEDIAWIKI')) {

$wgExtensionFunctions[] = 'efRawFile_Setup';
$wgHooks['LanguageGetMagic'][] = 'efRawFile_Magic';
$wgHooks['RawPageViewBeforeOutput'][] = 'fnRawFile_Strip';

</source>

===Setup function===
For the wiki parsing to create download links, file and fileLink are equally treated, while fileAnchor will be simply left out.
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Setup() {
global $wgParser;
$wgParser->setFunctionHook( 'file', 'efRawFile_Render' );
$wgParser->setFunctionHook( 'filelink', 'efRawFile_Render' );
$wgParser->setFunctionHook( 'fileanchor', 'efRawFile_Empty' );
}
</source>

===Hook to initialize the magic words===
We add the magic words here: the first array element indicates if it is case sensitive, in this case it is not case sensitive. We could add extra elements to create synonyms for our parser function.
 Unless we return true, other parser functions extensions will not get loaded.
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Magic( &$magicWords, $langCode ) {
$magicWords['file'] = array( 0, 'file' );
$magicWords['filelink'] = array( 0, 'filelink' );
$magicWords['fileanchor'] = array( 0, 'fileanchor' );
return true;
}
</source>

===Parser functions of the magic words===
The transformation rule to replace link shortcuts to actual links for download, handling an optional local wiki page title if present.
 The input parameters are wikitext with templates expanded, the output should be wikitext too
 '''TODO''': what error to send out if there is no filename given?
 '''EDIT''': It seems that [http://svn.wikimedia.org/viewvc/mediawiki?view=rev&revision=27667 commit 27667] (1.11 -> 1.12) changed the default parser, which breaks the recursive parsing. Thanks to Tim Starling for helping me to get around the problem!
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Render( &$parser, $filename = '', $titleText = '') {
if( $titleText == '' )
$title = $parser->mTitle;
else
$title = Title::newFromText( $titleText );
return $title->getFullURL( 'action=raw&file='.urlencode( $filename ) );
}
</source>
And the other one, just removing the anchors from the rendered wiki page.
 Curiously enough if the function doesn't exist at all the effect is exactly the same, MW doesn't throw any error.
 But let's keep things clean...
{{#fileanchor: RawFile.php}}
<source lang=php>
function efRawFile_Empty( &$parser, $filename = '') {
return '';
}
</source>

===Hook to intercept the raw output===
This part of the code doesn't look that nice because we've to parse the raw wiki page ourselves to retrieve the code sections we want.

First let's see if <code>?action=raw</code> was used in the context of this extension: in that case we receive the filename as GET parameter, otherwise we simply return from our extension with return value=true which means we authorize the raw display (originally the hook was created to add an authentication point)
{{#fileanchor: RawFile.php}}
<source lang=php>
function fnRawFile_Strip(&$rawPage, &$text) {
if (!isset($_GET['file']))
return true;
$filename=$_GET['file'];
</source>
By default the downloadable file will still be handled by the ob_gzhandler session made by Mediawiki. To avoid output buffering and gzipping, one can uncomment the following line:
{{#fileanchor: RawFile.php}}
<source lang=php>
// Uncomment the following line to avoid output buffering and gzipping:
// wfResetOutputBuffers();
</source>
Raw action already set the headers with some client cache pragmas and is supposed to be displayed in the browser but in our case we want to make this "page" a downloadable file so we overwrite the headers which were defined and we add a few more, to ensure there is no caching on the client (it's very hard for the client to force a refresh on a file download, contrary to a web page) and to provide the adequate filename.

{{#fileanchor: RawFile.php}}
<source lang=php>
header("Content-disposition: attachment;filename={$filename}");
header("Content-type: application/octetstream");
header("Content-Transfer-Encoding: binary");
header("Expires: 0");
header("Pragma: no-cache");
header("Cache-Control: no-store");
</source>
Then we'll strip the output, first we've to locate the anchors but there are anchors that could be protected in literal blocks like nowiki.
 So we'll mask the literal blocks before searching for the anchors (we mask with the same string length because we'll retrieve an offset that we will use on the initial string and offsets must match)
 '''TODO''': should we care also of source, js, css, pre,... blocks?
{{#fileanchor: RawFile.php}}
<source lang=php>
$maskedtext=preg_replace('/<nowiki>.*?<\/nowiki>/e',
'ereg_replace(".","X",$0)',
$text);
</source>
Now we can search for the anchors (or the short version, in which case we only keep the first hit, no multiple blocks support)
 And we free the memory used for the masked version
 '''TODO''': instead of cowardly returning if we don't find our anchors, we should cancel the headers and return a proper error page
{{#fileanchor: RawFile.php}}
<source lang=php>
if (preg_match_all('/{{#fileanchor: *'.$filename.' *}}/i', $maskedtext, $matches, PREG_OFFSET_CAPTURE))
$offsets=$matches[0];
else if (preg_match_all('/{{#file: *'.$filename.' *}}/i', $maskedtext, $matches, PREG_OFFSET_CAPTURE))
$offsets=array($matches[0][0]);
else
// We didn't find our anchor, let's output all the raw...
return true;
unset($maskedtext);
</source>
$text is both input & output so we copy it and start with an empty output.
{{#fileanchor: RawFile.php}}
<source lang=php>
$textorig=$text;
$text='';
</source>
For each anchor found we've to isolate the content of the next block.
{{#fileanchor: RawFile.php}}
<source lang=php>
foreach ($offsets as $offset) {
</source>
Let's remove the text up to the tag following the anchor
 '''TODO''': the next tag could be a , which we should skip
{{#fileanchor: RawFile.php}}
<source lang=php>
$out = substr($textorig, $offset[1]);
$out = substr($out, strpos($out, '<'));
</source>
What type of tag do we have?
 Note that we're looking to the word directly following '<' up to '>' or a space, e.g. if there are arguments to the tag.
 '''TODO''': once again, better handling of errors than just returning.
{{#fileanchor: RawFile.php}}
<source lang=php>
if (!preg_match('/^<([^> ]+)/', $out, $matches))
return true;
$key = $matches[1];
</source>
OK, let's extract the text up to the closing tag
 We skip the first carriage return after the opening tag, if any
 We look for the closing tag and we take what's in between.
 '''TODO''': once again, better handling of errors than just returning.
{{#fileanchor: RawFile.php}}
<source lang=php>
$begin = strpos($out, '>')+1;
if (ord(substr($out,$begin,1))==10)
$begin++;
if (preg_match_all('/<\/'.$key.'>/', $out, $matches, PREG_OFFSET_CAPTURE))
$text .= substr($out, $begin, $matches[0][0][1]-$begin);
else
// error, we could not find end of bloc
$text .= substr($out, $begin);
}
</source>
 No need to deal with a Content-Length header because Mediawiki will do it for us, moreover more properly than we could if the output is sent gzipped, which is the default.
 So that's it, $text contains our file!
{{#fileanchor: RawFile.php}}
<source lang=php>
return true;
}
</source>

===Credits===
There is an official way to register the extension in a Mediawiki installation, so that it will be visible on the [[Special:Version]] page.
 Let's say the extension is in the category of parser hooks even if there is also a hook on Raw action.
{{#fileanchor: RawFile.php}}
<source lang=php>
$wgExtensionCredits['parserhook'][] = array('name' => 'RawFile',
'version' => '0.3',
'author' => 'Philippe Teuwen',
'url' => 'http://www.mediawiki.org/wiki/Extension:RawFile',
// 'url' => 'http://wiki.yobi.be/wiki/Mediawiki_RawFile',
'description' => 'Downloads a RAW copy of <nowiki><tag>data</tag></nowiki> in a file '.
'Useful e.g. to download a script or a patch '.
'It also allows what is called [http://en.wikipedia.org/wiki/Literate_programming Literate Programming]');
}

?>
</source>
And finally registration of the extension at the Mediawiki website according to the [http://www.mediawiki.org/wiki/Manual:Extensions Extensions Manual].

So this extension has now [http://www.mediawiki.org/wiki/Extension:RawFile its own page on the official Mediawiki site].

==Installation==
Download [{{#filelink: RawFile.php}} RawFile.php] and save it under the MediaWiki directory as extensions/RawFile/RawFile.php

Add at the end of LocalSettings.php:
<source lang=php>
require_once("$IP/extensions/RawFile/RawFile.php");
</source>
==Status==
If you use the extension properly the code is fully functional but it's rather raw on error handling.
==ChangeLog==
'''0.3'''
* Added optional parameter to <code>#fileLink</code> to indicate that the file is on another local wiki page
'''0.2'''
* Fix problem with Content-Length mismatch when transport is gzipped (default for Mediawiki if client supports it)
'''0.1'''
* Initial version
==Known bugs==
Jani Uusitalo reported the following issue:
 For some reason, if you use Epiphany's 'Save as' instead of a direct left-click, the downloaded file is a single byte. In Firefox the links work just fine, so this is probably an Epiphany bug. Uncommenting the <code>// wfResetOutputBuffers();</code> line didn't help. If anyone knows a workaround to help Epiphany users without breaking the support of the other browsers, please speak up :-)

==Questions and feedback==
If you've any trouble, questions or suggestions, you can [[User:PhilippeTeuwen|contact me]].