Bright Yellow Cow :: Blog

Demo: a shared counter running on GAE

Yes, you can now do comet in PHP.

There still has to be an external service to wake up the client at the appropriate time, but in this case it presents an interface known as an 'Event Variable'. This service is completely general, and never needs to know anything about your application. It is also trivial to implement, and can be hosted very cheaply.

An Event Variable is simple. It is either fired (set) or not, and you can wait for it to be set. If it is already set when you ask, the call returns immediately. If it is not set then the call blocks until someone calls 'set'. There is no way to 'unset' an event variable. You create them, use them and throw them away when done.

The server http://doorbell.verieda.com provides a REST api for this service. There are three operations provided:

• To create an event variable, pick a large random number

• To wait for an event variable to be set, GET /<bignumber>

• To set an event variable, POST /<bignumber>

To try it out on the command line:

curl http://doorbell.verieda.com/<any number>
# in another window
curl -d '' http://doorbell.verieda.com/<any number>

In order for this to avoid XSS restrictions in the browser, the GET request cannot be made using XMLHttpRequest. The solution is to dynamically create a <script> DOM node, which doesn't have the same restrictions. In order to get control back into the application, the result of the GET request is the piece of javascript 'doorbell(<bignumber>);'. The application will implement the 'doorbell' method to catch the completion.

This is all really simple, here is an example:

Read the source of the demo for more information. It provides a handy abstraction using Mochikit's deferreds called 'doDeferredOnDoorbell' which you use like this:

function getBla() {
    d = loadJSONDoc(“/mysite/getnewmessages...”);
    d.addCallBack(function (result) {
        if [result is 'yes there is new stuff'] {
          // handle it as usual
        } else { 
            // we got a 'wait for an event' signal
            d = new Deferred();
            d.addCallBack(function (result) {
                getBla(); // Try the poll again
            });
            doDeferredOnDoorbell(d, thedoorbellid);
        }
    });
}

So there you go, now any application can support Comet requests. Unlike other solutions, it is shared-nothing, and doesn't require deploying a Stomp MQ server.

The event variable abstraction is also easy to use. Clients poll the server. When the server returns 'nothing', change it to say 'nothing, and don't ask again until this event has been triggered'. When the server gets an update, write it to the database as before, then set the event variables in order to wake the clients up.

Instead of polling in a loop and waiting for a timeout, the clients poll and wait for an event to fire before trying again.

“My intention of the project was directed to make the plug as slim as possible and follow the British Standard regulation at the same time.”

Behold:

Interestedly, I learnt that the gate on a 13A socket is not operated by the earth pin as I had always assumed, but it instead requires something to be pushed in both live and neutral simultaneously. There is no substitute to trying something.

# On your existing machine
# dpkg --get-selections > packages

This will dump out all the packages you have installed on an existing machine. I went through and deleted anything that I didn't want to install my new machine. You can then take this file to the next machine and tell apt that you want the packages installed:

# scp packages remote.machine.example.com:
# cat pacakges | dpkg --set-selections
# apt-get dselect-upgrade

My guess was that something was madly spawning processes that were never around for long enough for top to pick them up. To check this assumption, and find out what the processes are:

# ps axr

'a' lists the processes for all users, 'x' includes daemon processes and 'r' only shows running processes.

Encoding the Data

mencoder -o screencast-40.flv -of lavf -ovc lavc -oac mp3lame \
  -lameopts cbr:aq=0:br=32:mode=3 \
  -lavcopts vcodec=flv:vbitrate=8:trell:keyint=20 -fps 10 \
  -ofps 2.5 -vf scale=640:480 mf://output/*.png -audiofile\
   output/*.wav -lavfopts format=flv -af channels=1 #-frames 25

flvtool2 -UP screencast-40.flv

mencoder -o screencast-240.flv -of lavf -ovc lavc -oac mp3lame \
  -lameopts cbr:aq=0:br=64:mode=3 \
  -lavcopts vcodec=flv:vbitrate=176:trell:keyint=20 -fps 10 \
  -vf scale=640:480 mf://output/*.png -audiofile output/*.wav \
  -lavfopts format=flv -af channels=1 # -frames 100

flvtool2 -UP screencast-240.flv

mencoder -o screencast-500.flv -of lavf -ovc lavc -oac mp3lame \
  -lameopts cbr:aq=0:br=64:mode=3 \
  -lavcopts vcodec=flv:vbitrate=436:trell:keyint=20 -fps 10 \
  -vf scale=640:480 mf://output/*.png -audiofile output/*.wav \
  -lavfopts format=flv -af channels=1 # -frames 100

flvtool2 -UP screencast-500.flv

Hosting the streaming video

I just discovered SimpleCDN. Not only are they a cheap CDN, but they also provide flv and h264 pseudo-streaming, which is what you want to stream video. Right now they give you $15 of free hosting. Sweet. You want their 'StormFront' offering.

Flash Video Player

For this we will use Flowplayer and two plugins. The first is to support pseudostreaming and the second to do bandwidth detection.

FLV/h264 switching

if (flashembed.isSupported([9, 115]) ) {
// do h264
} else {
//do flv
}

Bandwidth detection

Use this plugin. Annoyingly I couldn't get the bandwidth test file to stream from the CDN, so it had to come from my webserver. This sucks, but shouldn't matter too much since the point of congestion will be at the user's end not mine.

First set the page size to match the output aspect ratio. For example 21.0 cm by 28.0 cm has the same aspect ratio as 1024x768, then use GraphicsMagic to turn those into a set of PNGs.

$ convert -density 600 -resize 1024x768 slides.pdf slides%d.png

The -density 600 means that the image is sampled at a high resolution first, which is needed because there isn't and anti-aliasing otherwise.

We can still get access to control the printout mode and duplex settings. These are passed in with the '-o' option to lpr. We can query the settings available for a printer using

# lpoptions -l

PageSize/Media Size: Letter *A4 Photo Photo5x7 PhotoTearOff…
PageRegion/Media Size: Letter A4 Photo Photo5x7 PhotoTearOff…
PrintoutMode/Printout Mode: Draft Draft.Gray *Normal 
                           Normal.Gray High High.Gray Photo
InputSlot/Media Source: *Default PhotoTray Upper Lower…
Duplex/Double-Sided Printing: DuplexNoTumble DuplexTumble *None
DryTime/Additional Dry Time: *Zero Five Ten Fifteen Twenty TwentyFive Thirty
Quality/Resolution, Quality, Ink Type, Media Type: *FromPrintoutMode…

So to print in draft mode we want to call lpr with “-o PrintoutMode=Draft”

Here's my Haskell recipe:

import Graphics.Rendering.Cairo
import Graphics.UI.Gtk.Cairo

import System.Process

inch = (*) 72

myDraw :: Render ()
myDraw = do
	setSourceRGB 1 1 0
	setLineWidth 5
	
	moveTo 120 60
	lineTo 60 110
	lineTo 180 110
	closePath
	
	stroke
	
	setSourceRGB 0 0 0
	setLineWidth 1
	moveTo (inch 5) (inch 5)
	lineTo (inch 6) (inch 6)
	closePath
	stroke
	
	moveTo 20 20
	rotate (pi/4)
	l <- createLayout "Haskell and Cairo is cool."
	showLayout l
	

main = do
	let tempFileName = "/tmp/foo.ps"
	withPSSurface tempFileName 595 842 $ \surface -> renderWith surface myDraw
	h <- runProcess "lpr" ["-o","PrintoutMode=Draft",tempFileName] Nothing Nothing Nothing Nothing Nothing
	e <- waitForProcess h
	print e

You call it 'ugly hack' I call it 'process isolation safety'☻

“: No such file or directory
”

This is terribly annoying because:

• Some broken version control tools convert between Unix and dos line endings when they feel like it

• All editors for the last 10 years have handled line endings transparently

• Line endings are not displayed by default in most editors

• Keeping windows line endings out of version control systems is a PITA

Once you know what is causing the problem the solution is trivial. Either just switch the line ending type in your text editor or use dos2unix:

# sudo apt-get install tofrodos
# dos2unix <myscript>
# ./myscript

Done!

Well nearly:

# [hg|cvs|svn] blame myscript
# Send an email rant, cc'd to the whole development organisation ☻

My primary interest is in removing some of the administration that goes with the traditional stack of

• Virtual Machine (I use Bytemark. My friends use them. Bytemark are awesome.)

• Debian

• Apache (or lighttpd/nginx)

• <Your application goes here>

• Django/Turbogears/Ror

• Postgresql/MySQL

Now, everything that isn't the actual value-adding, money making part of the application is a pure waste of resources.

Scalability? This application doesn't need scalability past what I can get for £5 grand from dell.com, but there is real potential value to be had removing the need for me to write apache config files. It isn't that I can't write config files, just that customers don't pay for them.

So with this background I decided to give Google's cloud offering a trial. I went to a great presentation by a guy from MSR about the Azure platform. All I can say is that Microsoft are targeting exactly this market and their demonstration rocked. Remember how Visual Basic became the most successful language ever? Azure is like that. Unfortunately I don't run windows on my desk, so I'm locked out of that party.

Back to GAE. Every web framework these days is optimised for the first 10 minutes of the experience, and this is no different. Within a few minutes I had an application hosted at appstot.com , so decide to make it a proper application and put it behind my own domain name.

Ok, so I have to prove I own the domain, which is weird because if I don't own the domain how would I point it at Google?

Do I have to host a small file, like with the search webmaster stuff? No I have to sign up with Google Apps, which costs $50. Eventually I find that there is a non-business free version with some restrictions. Will that do? I don't know.

But Google Apps wants to take all my corporate email and stick it on their servers. I don't want that. I want to run a website.

Very quickly you see Google's lock-everyone-in attitude take hold again. Will App. Engine be the same price tomorrow? Will I still be able to keep my corporate email out of their prying eyes tomorrow if I go down this route? Since moving away from the platform would be madly expensive, that is a risk too far.

I think the only way is to start documenting the exact behaviour of the interfaces between, say the ext3 filesystem, mysql, or an IDE block device in a formal, machine readable way.

This way MySQL can say 'if the file system obeys these properties, then I'll provide this interface' and ext3 will say 'if the hard disk obeys these properties, then I'll provide this interface'. Once this is in place is should be easy to check that the properties provided by each layer are enough to satisfy the layer above.

It won't eliminate bugs, of course. What it will mean is that we can assemble systems without worrying 'is this safe to run over NFS', because there will be an actual model of the behaviour of the NFS server and some requirements imposed by the program to check against.

It isn't obvious which language to express the formal model in, but temporal logic would be an obvious choice. Could the valid behaviours be expressed in Ruby or Python? It'd be great if they could.

import Data.Ord
thenComparing :: (Ord b) => (a -> a -> Ordering) -> (a -> b) -> a -> a -> Ordering 
thenComparing prev f a b = let x = prev a b in if x /= EQ then x else comparing f a b

It is used like this:

order = comparing foo `thenComparing` bar
res = sortBy order list

What you need to do is to fire up ghci and tell it to drop into a debugger when an exception that will propagate to the top level is about to be thrown, using the '-fbreak-on-error' flag, then :trace the execution of your function. For example, here is our application:

# cat app.hs
complicatedInnerFunction x = read x :: Int
toplevelApp = complicatedInnerFunction

Imagine now if you will about 50 thousand lines of boilerplate between the 'read' code that solves the problem and the interface. If you have coded in Java before, this is much easier. Lets try it out:

# ghci /tmp/app.hs 
GHCi, version 6.8.2: http://www.haskell.org/ghc/  :? for help
Loading package base ... linking ... done.
[1 of 1] Compiling Main             ( /tmp/app.hs, interpreted )
Ok, modules loaded: Main.
*Main> toplevelApp "ff"
*** Exception: Prelude.read: no parse

Not really any help. Now to tell it break when an exception will propagate to the top level:

*Main> :set -fbreak-on-error

*Main> :trace toplevelApp "ff"
Stopped at <exception thrown>
_exception :: e = GHC.IOBase.ErrorCall ['P','r','e',....]

Now we can find out how it got here:

[<exception thrown>] *Main> :history
-1  : complicatedInnerFunction (/tmp/app.hs:2:29-41)
-2  : complicatedInnerFunction (/tmp/app.hs:2:0-41)
<end of history>

That's given us a line number that actually caused the problem, which in many cases is enough to track the problem down, or we can actually use the debugger:

[<exception thrown>] *Main> :back

Logged breakpoint at /tmp/app.hs:2:29-41
_result :: Int
x :: String

This has bound the variables into our scope, so we can inspect them

[-1: /tmp/app.hs:2:29-41] *Main> print x
"ff"
[-1: /tmp/app.hs:2:29-41] *Main> 
length x
2

Full chapter and verse is on the GHC Docs

It is true that a bank would not be able to port their (low latency) stock trading systems to Amazon Elastic cloud, but there is no reason that an entrepreneur would not start offering cloud computing in an environment with a low latency connection to major stock exchanges such as in Canary Wharf, London. It would surely cost a lot more than CPU time in a consumer-grade data centre, and there would have to be higher levels of security in place, but none of that seems to fundamentally change the market for utility computing.

The other example that was cited is when there is a large amount of data being created. For the scientific community it might be hard to move the experiment to the data centre. In these cases we might expect to find that smaller data centres are built close to the thing generating the data. For a university this might mean that a new physics lab would be designed with space for a few containers of servers. A science park that wanted to attract Biotech start-ups would want to offer a local cloud with very high bandwidth connections to the offices. A Hollywood film might rent a container load of servers and bring it to the filming location to handle the data from digital film cameras.

The utility model for computation works in all these cases, not just the 'cheapest page hit' that has characterised the current offerings.

Today I wanted to go though my code and have a look at each of them, and it turns out that Jedit's support for jumping to the source of a compiler error message is just what is needed. First you need the 'Console' plugin installed, then just open a console window and type: (The -n produces line numbers)

find . -iname '*.hs' | xargs grep -n TODO

Bring up the Error List view (Plugins → ErrorList → Error List), and you can quickly click through each item marked TODO and instantly jump to it. Magic.

$ debootstrap --arch i386 etch . http://ftp.uk.debian.org/debian
cp /etc/passwd /buildmachinechroot/etc
cp /etc/shadow /buildmachinechroot/etc

# chroot /buildmachinechroot
# Set the locale
$ apt-get install locales
$ dpkg-reconfigure locales

# Install the packages required to build
# apt-get install ghc6 ...

# Bug whoever wrote the README that the list of required packages is not correct

JSON is a serialization format that (for some reason) encodes Unicode strings, numbers, lists and records as a Unicode string. Which is good, as long as the only data types your application deals with are one of those above.

What about:

char* (c)
ByteString or (Haskell)

Which doesn't have a mapping to JSON. Strangely, many of the programs written over the last—oh—20 years deal in 'lists of bytes'. Consider JPEG image data or a million other examples.

There are workarounds: you could convert a list of bytes to a list of integers and encode that, or use base64 or use uuencode. You could even pick 256 Unicode code points and map the bytes to those. U+0000 to U+00FF would seem to be one option, equally U+0001 to U+0100.

I'm not saying that it isn't possible, just that a 'standard' that doesn't define how to encode the most common, universal data type ever is a wee bit broken.

In general, any standard that includes pseudo-formalisms on its homepage is broken.

begin;
insert into bar (email) values ('foo');
select last_value from bar_id_seq;
commit;

This is valid behaviour in the standard isolation model of Postgres (and most likely MySql too, but I haven't tested it). Here's how to reproduce it. Open two psql consoles, and run these commands:

create table bar (id serial primary key, email text);

Begin;

Begin;

insert into bar (email) values ('foo');

 insert into bar (email) values ('ff');

select last_value from bar_id_seq;
 last_value 
------------
          2
(1 row)

...assume that 'foo' has id 2...

Commit;

select * from bar;
 id | email 
----+-------
  1 | foo
(1 row)

Commit;

select * from bar;
 id | email 
----+-------
  1 | foo
  2 | ff
(2 rows)

Some database libraries (JDBC for example) allow you to query the values generated, which is one solution to the problem. Another is to not make the id type 'serial' at all, but define the sequence explicitly:

create sequence bar_id_seq;
create table bar (
	id	bigint primary key,
	... 
);

You then issue two queries: the first to mint a new id, and then use that to insert into the table:

select nextval(bar_id_seq');
insert into bar (id,...) values (?,...)

A second solution is to the the Postgresql extension 'returning':

insert into foo (id,email) values (default,'bla') returning id;
 id 
----
  1
(1 row)

Blender will treat a sequence of images as a video: In the VSE timeline do add → images, select the directory with the images in, press 'a' to select them all and click 'Select Images' to import them.

Capturing

# Start x11vnc with 'Copyrect' disabled, since pyvnc2swf doesn't know about it
x11vnc -nowcr

# Start recording
pyvnc2swf-0.9.5/pyvnc2swf/vnc2swf.py -r 10 -o recording.swf myhost:0
# ...Write a wiki in 19 minutes...

# output 1 frame every 3 seconds, to see which bits of the desktop to display
pyvnc2swf-0.9.5/pyvnc2swf/edit.py -R 30 -o recording-overlay2.bmp -t bmp  recording.swf 

Now look at recording-overlay*.bmp in gimp and decide that the 640x480 at 1224+184 is our openoffice session, which we want to show.

mkdir openoffice
pyvnc2swf-0.9.5/pyvnc2swf/edit.py -o openoffice/office.bmp -C 640x480+1224+184 -t bmp  recording.swf 

After repeating this for other windows, we now we have a bunch of directories that contain view of of the various important windows on our main (large) desktop. Go to blender and edit these into a movie.

Capturing From UltraVNC

It appears that pyvncswf doesn't do a good job of negotiating the VNC connection settings with UltraVNC. This bit me when recording a Windows session (running on Amazon EC2, but that's another story). The answer here is to use a slightly different incantation:

# Start recording (UltraVNC)
pyvnc2swf-0.9.5/pyvnc2swf/vnc2swf.py -r 10 -e 4 -o recording.vnc myhost:0

Encoding

Blender can encode the output directly using ffmpeg, but I prefer to use mencoder, since it has more flexibility. First export the movie from blender as a sequence of PNGs, then encode using mencoder:

# encode the video (lavc gives better quality than x264, somehow)
mencoder -o vid.avi -of avi -ovc lavc -oac faac -lavcopts \
 vcodec=libx264:vbitrate=80:keyint=50:vpass=1 \
 -faacopts mpeg=4:object=2:br=80 -fps 10 mf://blender-output/*.png \
 -audiofile blender-output/0001_0237.wav 

# split the audio and video
mencoder -ovc copy -oac copy -of rawaudio -o vid.aac vid.avi
mencoder -ovc copy -oac copy -of rawvideo -o vid.h264 vid.avi

# mux the two together in a mp4 stream
MP4Box vid.mp4 -new -add vid.h264 -hint -inter 500 -add vid.aac

# encode flv
mencoder -o vid.flv -of lavf -ovc lavc -oac mp3lame -lavcopts \
 vcodec=flv:vbitrate=100:trell:keyint=50 -fps 10 mf://blender-output/*.png \
 -audiofile blender-output/0001_0237.wav -lavfopts format=flv
flvtool2 -UP vid.flv

These can be played in flv-scrubber or JW Player. Note that 'keyint=50' is required to make seeking work.

Overlays

Overlays can be created by creating a transparent png in Gimp/Inkscape and adding the image to the Blender VSE. In order to make it work right, three settings have to be made on the Sequencer button bar. With the image strip selected:

1. Enable 'Premul' in the 'filters' section, to make Alpha blending work

2. Change 'Replace' to 'Alpha Over' in the 'Edit' section

3. Enable 'Use Translate' to stop the image scaling, and adjust the positioning of the overlay with the 'X-Ofs' and 'Y-Ofs' controls below

Reducing File sizes

The disk space taken up by raw bmp files is quite large (but so are modern disks). Pyvnc doesn't like producing output in other formats (at least on my machine). It is quick to convert them to PNGs though:

mogrify -format png *.bmp

or

find . -iname '*.bmp' | xargs mogrify -format png

The 'Real' solution would be to teach blender how to read vnc dump files. Making pyvnc/edit produce PNGs directly is probably easier, but that will have to wait for another day.

Firstly, there is the concept of a 'table name', but since there is no way to have two tables in one cluster, this provides nothing beyond a forwards compatibility hook for the FB guys.

The next concept is a 'Column Family': these have to be chosen when the cluster is started, and since there is no way to migrate data from one cluster to another, you better get them right first time. Each column family has a name and is either a 'super column' or a 'normal column'.

The database provides a (very restrictive) set of operations. I'm using [ ] to mean 'list of' and ( , ) for tuple construction. Anyone exposed to the ML series of languages will be familiar with the notation:

For a Normal Column

When 'family' is defined to be a normal column in the server configuration:

insert (family,key1,key2,value,timestamp) → ()

get (family,key1,key2) → (value,timestamp)

get_range (family,key1) → [(value,timestamp)]

For a Super Column

When 'family' is defined to be a super column in the server configuration:

insert (family,key1,key2,key3,value,timestamp) → ()

get (family,key1,key2) → [(key3,value,timestamp)]

get_range (family,key1) → [(key2,[(key3,value,timestamp)]]

Note that there isn't (at least in my understanding) an operation on super columns like

lookup: (family, key1, key2, key3) → (value, timestamp)

Note that you can only get back a list of (key3,value,timestamps) from a super column. This makes super columns nearly the same as just storing a list of (key3,values) in a normal column, except that new items can be added freely, and the timestamps are per value.

Remove

I don't understand remove operations yet:

# insert an item
# ./Cassandra-remote insert users fred edges:a:g fff 48

# check it got added
# ./Cassandra-remote get_slice_super users fred edges -1 -1
[ {'name': 'a', 'columns': [{'columnName': 'g', 'value': 'fff', 'timestamp': 48}]}]


# Delete it
# ./Cassandra-remote remove users fred edges:a
None

# Check it has gone
# ./Cassandra-remote get_slice_super users fred edges -1 -1
[]

# Add it back
# ./Cassandra-remote insert users fred edges:a:g fff 50

# ... and it doesn't get added?
# ./Cassandra-remote get_slice_super users fred edges -1 -1
[]

Update: Jonathan Ellis notes that some of this is getting old now, specifically that remove is fixed. When the first release is ready, I'll refresh this analysis. Personally I prefer Cassandra over Hadoop/HBase because Hadoop has a central single point of failure.

Consider this problem: I want to ask the user for four numbers, A,B,C and D such that A+B = C±2. D can be any number.

This has got to be as simple as GUI problems get, surely?

Firstly, I'm going to tell you that the GUI has to look like this:

You might suggest that I rework the GUI to ask for independent things, so ask for values for A, B and 'C-A-B', and report the value of C back to the user. In this case that would be possible, but that isn't a nice user interface, and I could give more complex examples where it isn't possible to do that at all. They might walk in a say 'I'd like to buy £1's worth of apples, please,' or maybe 'I'd like 1lb of apples, please'.

Ok, so lets consider all the possible things the user can put in those boxes:

• They may enter a correct set of values

• They may have entered numbers into all the boxes, but A+B isn't nearly equal to C

• They may have entered '45foo' into box C, so we can't even start our validation.

• They may have entered 'foo' into box D, but A=1; B=2; C=27, so our A+B=C validation rule fails.

Now consider some things we want to support:

• Loading and saving

• Undo

• Display a pretty graphical representation of the setup, which is only possible when the form validates.

In the general case getting all the validation to pass may be tricky for a user. They surely want the support of undo while this is taking place. It would be nice if they could save where they were and show a friend it they got stuck.

Assuming we have a single underlying datamodel, the critical issue here is about what states we let the underlying data model get into. Most sane people wouldn't store the values as strings (unless they believe in XML...), which means that you can't click 'Save' when a box contains '45foo'. You might want undo to work for this series of operations though:

Set D to 'foo123'
Set C to '34'
Set A to '12'
Undo

If the undo history is in terms of the application Model, then undo will reverse all three operations in this case and only 1 operation in this case:

Set D to '123'
Set C to '34'
Set A to '12'
Undo

Hardly the expected result.

“Ah”, you say “The solution is to stop the user typing in invalid characters in the first place!” This, alas doesn't get us very far (imagine if I said that the numbers had to have 4 digits: how would delete work?). It doesn't even work in this case, since “-” isn't a number but “-1” is. Or consider entering a decimal number:

“1000.0” Oops meant to enter 10.000
“10.00.0” Insert the new decimal point
“10.000” Remove the old one

Forcing the last two operations to be the other way round is not what the user would expect. How are they meant to find out why doesn't the dot key doesn't work?

Fundamentally, the idea of the GUI view being a true function of the underlying state (Model) is flawed, since it assumes that there is always a simple route to get from one valid state to another. In many cases this might involve changing two things at once (in the A+B=C case) or just put funny constraints on the way things are done, such as we saw in the decimal number editing case above.

My feeling is the answer is to consider the validation problem as constructing a series of successively more constrained representations of the state. Think about editing a program in your favourite language:

You start with a source file that is a list of Unicode code points. This gets converted to a sequence of tokens by the lexer, which then gets parsed, type checked, and linked. Every stage tries to build a higher level representation of the input, which can optionally fail.

But this model is totally different to MVC. In MVC the master representation is at the highest level of abstraction, and the views are derived from that. In a compiler the master representation is the lowest possible level of abstraction (a list of characters), and the high level representations needed for execution and compilation are created as necessary.

Turns out that this is easy to fix. Just create a line in your .hgrc:

$ cat ~/.hgrc

[defaults]
log  = --limit=5

...for exactly the same reason that they used Flash to distribute the video telling us why it is better than Flash. Everyone has a Flash player.

Eclipse has three (or more) settings that control which JVM is used. You can run eclipse, compile your code and run your code all under different environments if you want to. First lets change the JVM used to run eclipse:

$ cat java_home 
# This file determines the search order the Eclipse Platform uses to find a
# compatible JAVA_HOME. This setting may be overridden on a per-user basis by
# altering the JAVA_HOME setting in ~/.eclipse/eclipserc.

#/usr/lib/jvm/java-6-openjdk

#/usr/lib/jvm/java-gcj

#/usr/lib/kaffe/pthreads

/usr/lib/jvm/java-6-sun
/usr/lib/jvm/java-1.5.0-sun
/usr/lib/j2se/1.5
/usr/lib/j2se/1.4
/usr/lib/j2sdk1.5-ibm
/usr/lib/j2sdk1.4-ibm
/usr/lib/j2sdk1.6-sun
/usr/lib/j2sdk1.5-sun
/usr/lib/j2sdk1.4-sun

Note the three 'Non-JVMs' have been commented out. You can check which is used by your eclipse by doing Help → about → Configuration Details. You are looking for a line like “-vm /usr/lib/jvm/java-6-sun/bin/java” any mention of gjc is wrong.

Now we can tell eclipse about the Sun JRE. Do Window → Preferences → Java → Installed JREs. Add yours, and make it the default.

Also right-click on your Java project, and do Java Build Path → Libraries. Remove the GCJ JRE and do Add Library → JRE System Library. Make sure the workspace JRE is the Sun one, and click ok.

It is also possible for the JRE used to run your app to be wrong. Go to the 'Run' dialog box, and check the JRE in there is correct too.

...And you're good to go!

The people who develop it are geniuses, but the resulting code is some of the nicest I've ever seen. As someone who is interested in hardware, I wanted to find an example of the register set interface between some common PC peripheral and the host processor. Let's have a look at how the OLPC finds out the state of the battery.

We are going to need three tools: cscope, jedit and the kernel source. First off unpack the kernel and tell cscope to build an index of it:

# cscope -bRs .

That will save us a few seconds later on. Cscope has a weird user interface, but don't let that put you off.

Ok, lets find the driver that understands the OLPC's battery monitor. It is a driver, for the power management code, for the OLPC battery, so 'drivers/power/olpc_battery.c' doesn't seem such a bad place to start.

That starts with a whole bunch of #defines talking about voltage, current and temperature, so we appear to be on the right track. The whole thing is less than 500 lines long, so a quick skim of it reveals that olpc_bat_get_property seem to be important. It doesn't seem to do any IO directly, but there are lots of calls to 'olpc_ec_cmd' e.g.:

olpc_ec_cmd(EC_BAT_STATUS, NULL, 0, &ec_byte, 1);

Ok, lets chase that down. Fire up cscope:

# cscope -d   # -d means 'use the index we build earlier

We enter 'olpc_ec_cmd' in the 'Find definition' box and hit enter. The definition is in arch/x86/kernel/olpc.c:

/*

 * This allows the kernel to run Embedded Controller commands.  The EC is

 * documented at <http://wiki.laptop.org/go/Embedded_controller>, and the

 * available EC commands are here:

 * <http://wiki.laptop.org/go/Ec_specification>.  Unfortunately, while

 * OpenFirmware's source is available, the EC's is not.

 */

int olpc_ec_cmd(unsigned char cmd, unsigned char *inbuf, size_t inlen,

                unsigned char *outbuf,  size_t outlen)

Ah, it seems like it takes a command, a input buffer and an output buffer. Skimming the code reveals that it sends the command byte to a port (a serial port FIFO?), streams the buffer then reads the response back into outbuf.

We can inspect the details in olpc_battery.c to find out how any specific command works, but the interface is not memory-mapped at all as we had first assumed: it uses a synchronous message passing interface instead.

So there you go. Next time you are interested in 'how it works' try Linux. It contains a functional description of nearly every piece of PC hardware ever made.

We're going to use the Graphics.GD bindings to the GD library written by Bjorn Bringert to do all the work:

> import Graphics.GD
> import Numeric (showHex, readHex)
> import System.Environment
> 
> 
> main = do
> 	[prefix, strSize,fg,bg] <- getArgs
> 	let size = read strSize
> 	img <- newImage (size,size)
> 	doAt img fg bg size (prefix++"_tl.png") (size-1,size-1)
> 	doAt img fg bg size  (prefix++"_tr.png") (0,size-1)
> 	doAt img fg bg size  (prefix++"_bl.png") (size-1,0)
> 	doAt img fg bg size  (prefix++"_br.png") (0,0)
> 	
> doAt img fg bg size name pos = do
> 	fillImage (torgb bg) img
> 	drawFilledEllipse pos (size*2-1,size*2-1) (torgb fg) img
> 	savePngFile name img
> 
> torgb s = 
> 	let
> 		[r,g,b] = fromHexStr s
> 	in
> 		rgb r g b
> 
> 
> fromHexStr :: String -> [Int]
> fromHexStr (a:b:tl) = (fromHex [a,b]):(fromHexStr tl)
> fromHexStr [] = []
> fromHex :: String -> Int
> fromHex s = 
> 	case readHex s of
> 		[(val,"")] -> fromIntegral (val)
> 		other -> error $! show other
> 

Usage is simple: "./makeCorners l 20 3D7AB8 FFFFFF" will make 4 images with quadrants of a circle in them, coloured #3D7AB8 on a white background. And that's it!

:wrap=soft:maxLineLen=90:

Simply put the above magic incantation anywhere in the first or last 10 lines of your document, and jEdit will pick it up automatically. Hit save and it will update. Don't forget to include the trailing colon.