View source for BMP PCM polyglot

=BMP & PCM=
This is a Poc (Proof of Concept) to create a file that can be seen as image (BMP) and played as sound (RAW PCM).

So it's a kind of polyglot file.
<br>It's a bit comparable to steganography but here the sound doesn't need to be extracted first, the file can be just played as such, provided that you tell to the player what are the sound specs (sampling rate, channels, bit-depth).

The trick is the following:
<br>We start from a 16-bit BMP and 16-bit WAV and merge samples to create a 32-bit BMP/PCM.
<br>16-bit BMP can be created e.g. with The Gimp, choosing 16-bit R5G6B5 encoding.
<br>16-bit WAV can be mono or stereo, of arbitrary sampling rate, non compressed, little endian.

Initial RGBA masks are for R5G6B5: 0000F800, 000007E0, 0000001F, 00000000
<br>When grouping pixels and sound samples together we'll extend the sound from 16-bit little endian to 32-bit little endian so we add low 16-bits, under audible level, and we'll put the pixels there.
<br>So we adapt the BMP masks to tell where the color components are now.

To conclude: image will be below audible level and sound will be ignored by BMP masks.

Example:
<br>[[File:Pf.bmp|800px|link=http://wiki.yobi.be/images/9/9c/Pf.bmp]]
<br>Click on the image to download the combined BMP.

Sound can be played e.g. with
<source lang=bash>
wget -O - http://wiki.yobi.be/images/9/9c/Pf.bmp|aplay -r 44100 -c2 -f S32_LE
</source>
Or with Audacity: import as raw and specify sampling rate=44100, stereo, 32-bit little endian

[{{#filelink: BMPPCM.py}} BMPPCM.py is available for download here]
==Beginning==
{{#fileanchor: BMPPCM.py}}
<source lang=python>
#!/usr/bin/env python

from struct import unpack, pack
import wave

PCM_LE = True  # PCM 32-bit should be Little Endian or Big Endian?
bmp_in ='a.bmp'
wav_in ='a.wav'
bmp_out='b.bmp'
# BMP created with Gimp as 16-bit R5G6B5
f=open(bmp_in).read()
# WAV created with mpg123 -w a.wav a.mp3 (stereo)
w=wave.open(wav_in, 'rb')
</source>
==Parsing BMP==
{{#fileanchor: BMPPCM.py}}
<source lang=python>
class bmp(): pass
class bmpheader(): pass
class bmpdib(): pass
b=bmp()
b.header=bmpheader()
b.dib=bmpdib()

fheader            =f[0:14]
b.header.magic     =fheader[0:2]
assert b.header.magic == "BM"
b.header.filesize, =unpack('<I', fheader[2:6])
b.header.unused1,  =unpack('<H', fheader[6:8])
b.header.unused2,  =unpack('<H', fheader[8:10])
b.header.offdata,  =unpack('<I', fheader[10:14])
fdib               =f[14:b.header.offdata]
fimg               =f[b.header.offdata:]
b.dib.dibsize,     =unpack('<I', fdib[0:4])
assert b.dib.dibsize == len(fdib)
assert b.dib.dibsize >= 56 # at least BITMAPV3HEADER
b.dib.width,       =unpack('<i', fdib[4:8])
b.dib.height,      =unpack('<i', fdib[8:12])
b.dib.planes,      =unpack('<H', fdib[12:14])
b.dib.bpp,         =unpack('<H', fdib[14:16])
assert b.dib.bpp == 16
b.dib.comp,        =unpack('<I', fdib[16:20])
assert b.dib.comp == 3 # BI_BITFIELDS
b.dib.imgsize,     =unpack('<I', fdib[20:24])
assert b.dib.imgsize == b.header.filesize - b.header.offdata
b.dib.hppm,        =unpack('<I', fdib[24:28])
b.dib.vppm,        =unpack('<I', fdib[28:32])
b.dib.colors,      =unpack('<I', fdib[32:36])
b.dib.icolors,     =unpack('<I', fdib[36:40])
b.dib.Rmask,       =unpack('<I', fdib[40:44])
b.dib.Gmask,       =unpack('<I', fdib[44:48])
b.dib.Bmask,       =unpack('<I', fdib[48:52])
b.dib.Amask,       =unpack('<I', fdib[52:56])
b.dib.remaining    =fdib[56:]
b.img              =list(unpack('<'+'H'*(b.dib.imgsize*8/b.dib.bpp), fimg))
</source>
==Making BMP 32-bit==
And shifting filter masks if needed
{{#fileanchor: BMPPCM.py}}
<source lang=python>
b.dib.bpp=32
if PCM_LE:
    b.dib.Rmask <<=16
    b.dib.Gmask <<=16
    b.dib.Bmask <<=16
    b.dib.Amask <<=16
</source>
==Reading enough sound samples==
{{#fileanchor: BMPPCM.py}}
<source lang=python>
assert w.getnchannels() <= 2     # 1 or 2 channels
assert w.getsampwidth() == 2     # 16-bit
assert w.getcomptype()  == 'NONE'
assert w.getnframes() * w.getnchannels() >= len(b.img)
s=list(unpack('<'+'h'*len(b.img), w.readframes(len(b.img) / w.getnchannels())))
</source>
==Recreating samples==
from pixels & sound samples
{{#fileanchor: BMPPCM.py}}
<source lang=python>
if PCM_LE:
    for i in xrange(len(b.img)):
        b.img[i], = unpack('<I', pack('<hH', s[i], b.img[i]))
else:
    for i in xrange(len(b.img)):
        b.img[i], = unpack('<I', pack('<H', b.img[i]) + pack('>h', s[i]))
</source>
==Fixing BMP headers==
with new size
{{#fileanchor: BMPPCM.py}}
<source lang=python>
b.dib.imgsize = len(b.img) * b.dib.bpp / 8
b.header.filesize = b.dib.imgsize + b.header.offdata
</source>
==Packing back BMP==
{{#fileanchor: BMPPCM.py}}
<source lang=python>
b2=b.header.magic+pack('<IHHIIiiHHIIIIIIIIII', b.header.filesize, b.header.unused1, b.header.unused2, b.header.offdata,
b.dib.dibsize, b.dib.width, b.dib.height, b.dib.planes, b.dib.bpp, b.dib.comp, b.dib.imgsize,
b.dib.hppm, b.dib.vppm, b.dib.colors, b.dib.icolors, b.dib.Rmask, b.dib.Gmask, b.dib.Bmask, b.dib.Amask)
b2+=b.dib.remaining
p={8:'B', 16:'H', 32:'I'}[b.dib.bpp]
b2+=pack('<'+p*(b.dib.imgsize*8/b.dib.bpp), *b.img)
open(bmp_out, 'wb').write(b2)
print '%s written!' % bmp_out
print 'You can play it as signed 32-bit PCM, e.g.:'
print 'cat %s | aplay -r %i -c %i -f S32_%s' % (bmp_out, w.getframerate(), w.getnchannels(), ("BE", "LE")[PCM_LE])
</source>

[{{#filelink: BMPPCM.py}} BMPPCM.py is available for download here]

=Image in sound=
@angealbertini [https://twitter.com/angealbertini/status/504016678305288192 proposes] to hide an image in an audio spectrum so both techniques can be combined.

[http://www.nicolasfournel.com/audiopaint.htm AudioPaint] allows to make easily such sounds. But we've to be a bit careful to have a sound file of the right size if we want to put it in the BMP file.
<br>E.g. choose channel routing: L=brightness / R=none, linear scale, 22050Hz (sampling rate / 2) and choose a duration such that the samples of the left channel will be equal or slightly larger than the number of pixels: 
 duration = image width * height / sampling rate
<br>Then once WAV is saved, we isolate the left channel
 sox audiopaint.wav -c 1 a.wav remix 1
So we have a proper sound file to be combined with the BMP.

Example with 
<br>[[File:Corkami.bmp|link=http://wiki.yobi.be/images/b/b3/Corkami.bmp]]
<br>Click on the image to download the combined BMP.
<br>To see the spectrogram:
<source lang=bash>
sox -t raw -r 44100 -c 1 -e signed -b 32 Corkami.bmp -n spectrogram -m -x 555 -y 512 -z 24 -Z -36
</source>
[[File:CorkamiSpectrogram.png]]
<br>Size 555x512 is the initial image size 400x369 scaled up such that y is a power of 2 (for FFT)
<br>Note that sampling rates are not really important as the absolute time and frequency scaling doesn't mean anything here.

=Image in sound: add color=
Here is another PoC based on the same picture as in our first test, but now the spectrogram of the picture will be the picture itself, in colors!
<br>It's a bit tricky because AudioPaint and Python wave module only support stereo and I want 3 channels, to encode RGB of course, and sox displays three spectrograms instead of one combined.
<br>To display the spectrogram, I'll use [http://www.baudline.com/index.html baudline].
<br>Because baudline works in waterfall mode (time is on vertical axis, going downwards) we'll rotate first the image anti-clockwisely.
<br>Then in AudioPaint I choose a duration such that total of the three channels will be slightly larger than the amount of pixels:
 1600*1034/44100/3 <~ 12.51 seconds
<br>I make two WAV with AudioPaint, a first one with L=red R=green, saved as rg.wav, and a second one with L=blue R=none, saved as b.wav
<br>Then I recombine the three channels with sox
 sox -M rg.wav b.wav -c3 a.wav remix 3 2 1
<br>I can now use BMPPCM.py, slightly modified to read manually the sound samples as Python wave doesn't support three channels.
<br>Result is here: http://wiki.yobi.be/images/d/d6/Pfrgb.bmp
<br>Seen in baudline (as raw 44100Hz 3 channels 32-bit linear little-endian, channel mapping RGB and color aperture from -32 to -64dB, drift integrator beam width 126 slices)
<br>[[File:Pfrgbspectrogram.png]]

=Troubleshooting=
==Gimp==
It seems Gimp >= 2.8.8 has a bug when opening our crafted BMP (dark greenish image) while Gimp <= 2.8.6 is ok.
<br>See my [https://bugzilla.gnome.org/show_bug.cgi?id=735557 bugreport & patch], now [https://git.gnome.org/browse/gimp/commit/?h=gimp-2-8&id=c7c4475eaf578557e6413713ea911012900905d5 integrated] for next release.

==Saturated sound?==
If the sound is saturated, you probably imported the PCM as 32-bit float or big endian or 16-bit instead of 32-bit signed little endian.
<br>(unless you're trying to play the files with image in spectrogram, there is nothing to listen to there, only in the very first image)
<br>Thanks @cbrocas ;-)

=Misc=
If you want to know more about the bodypainting picture used, see http://www.feelnumb.com/2012/07/19/the-classic-1997-pink-floyd-back-catalogue-poster/