Homebrew Coding

How to recruit on LOM

2012-01-21T22:38:00.000+01:00

I've had an email about how to recruit on my Android Lords of Midnight remake - "the War of the Solstice". That means other people are probably stuck too and haven't asked. Here are the basics.

From the start of the game, go Northwest, Northwest, Northeast and you'll stand on the same spot as the Lord of Shadows. Here I've done it as Morkin, but you can use any of the starting group.

Now touch the name part of the text (where it says "Morkin. He stands in the Forest of Shadows, looking Northeast."). This brings up the "Think" screen.

On this screen, touch the right hand part of the text. This brings up the "Seek" screen, where you can do the recruiting.

On here, touch the option to recruit the Lord of Shadows. That's it!

Silly Predictions For Android Versions In 2012

2012-01-13T22:28:00.000+01:00

Image via WikipediaLast week Google released the latest numbers with the percentage of each Android version accessing the Android Market. What really jumped out at me is the rise and rise of Gingerbread (2.3.x), and the fall of Froyo (2.2). In the past I've been a bit skeptical of Google's commitment to older versions of Android. Looking at the way things advance, it makes no sense for them to dedicate resources to try and make life easier for developers on older devices. Every year the release 2 versions ago more or less disappears, or that's the way things have been tending to go for the last couple of years at least.

Here's a graph of how the relative percentage has changed over the last 18 months. I've occasionally jotted down the percentage numbers in a spreadsheet, and the rest I got from the Wayback Machine. I've lumped minor releases together, so 2.3.1 and 2.3.2 are in the 2.3 bucket.

Version 2.1 (Eclair) is slowly disappearing, despite being at 55% back in July 2010. 2.2 (Froyo) is at 30% now, though in mid 2011 it peaked at 65% of the Google-approved devices online. Right now 2.3 (Gingerbread) is on the rise. I think the poor showing of 3.0 (Honeycomb) confirms that the driving force behind these numbers is mobile phones, rather than tablets. For those not paying attention, Honeycomb is only available on tablets, you see.

Given all that, here's my predict-o-graph of how things will go in 2012. The yellow area marks my completely arbitrary prediction values of how the percentages will change. I've pulled the numbers out my /dev/backside.

If everything follows the same pattern as previous years, by early summer 2012 Gingerbread will have peaked at about 60% of the market. By the end of 2012, Froyo will be on fewer than 10% of devices, Gingerbread will be on the slide, and the newest Android release - Ice Cream Sandwich, version 4.0 - will be taking over. I don't think Honeycomb is going to grow much this year, and the relative percentage of devices with 3.x may even shrink. No new tablets should be released with Honeycomb this year anyway - if they are released, they won't sell very well or will be upgraded to ICS pretty quickly - and most new or new-ish phones will have or get ICS too.

As for me, and the games I've put out on Android... I'll continue to target 1.6+, since I don't really do anything that warrants a newer release and 2.2. is all I've got (curse you HTC and your locked bootloaders!) I think that by the end of this year I may have to look at upgrading, by then a 2nd hand Samsung may have come down to the kind of cheapskate price I'm willing to pay :-)

Non-ARM Android Apps

2011-11-27T21:32:00.001+01:00

I've had 2 requests now for my Lords of Midnight remake on x86-based Android devices. A while ago I updated the scons-android build system I use to support multiple architectures, but I didn't actually upload an x86 version to the Market. I thought nobody actually owned one of these mythical devices. Turns out that times are changing.

Back in July, Google added multiple APK support to the Market. Sadly, the multiple APK idea is a bit limited in what you can do. The APKs have to be different in one of 3 key features only. These features are Open GL texture compression type, screen size and API level. Differences in other features - such as whether the device has a touch screen or not, has a hardware keyboard or not, the type of processor it has, and so on, are all ignored. If the only differences between the APKs are in these "minor" features then they are considered identical by the Market software and will not be allowed to coexist.

The explanation I've read as to why this is the case is that the Market client application on devices doesn't provide enough information to the Market server for it to make use of all the feature filters. This makes some sense, as there have only really been multiple ARM ABIs available since June 2010 with release 4b of the NDK. But the Market does already know how to hide non-compatible APKs from you, so I get the feeling this is more a case of "we can't be bothered to do this" rather than "we can't do this".

As it stands, in order to do an x86 version of an app that uses native code while still maintaining compatibility with the majority ARM devices, you need to provide a so-called "fat binary". This is an APK that contains a copy of the ARM version (versions) of your library and the x86 version too. I think it's a bit of a shame that users with one type of processor have to pay the download-tax due to the inclusion of its "evil twin" opposing processor library. Fortunately the Android installation tools are smart enough not to copy useless libraries onto the device for use at run-time, so it is only the increased download size that is a problem.

With this in mind, I've uploaded a new version of War of the Solstice and Chaos that support x86 too. Since I don't own anything that I can actually test these on it is a stab in the dark - it may work, but it will likely just crash, let me know with your 1 star reviews! ;-)

Screenshot of the strange Android Market upload warnings

One thing that worried me slightly is that when you upload an ARM/x86 dual APK to the Market it says "this apk requests 2 native platforms that will be used for filtering". That makes it sounds like you need a strange ARM and x86 device to make the game work. I hope that this is just a typo, and that the final APK works on just about everything.

3D Space Game Bug Fixes

2011-11-01T18:00:00.001+01:00

Back in July I wrote that I was working on getting rid of the bugs in 3D Space Game. It has certainly taken me a while to get most of them squashed, but I've just uploaded a new version to the Android Market that fixes many crashes (hopefully!). You can also view the source code on the googlecode site.

One of the nasty side effects of the old version was it's battery hogging behaviour. This was caused by my dreadfully inefficient Game Boy Advance-like emulation layer that was baked into the game. Over the last few months I've also improved my android-ndk-profiler so it works a lot better on older devices, and I've used it here to try and get better performance. Even with that I still did most of my tuning and debugging on the SDL and Linux port - Android is still missing something like valgrind for native code and GDB doesn't work prior to 2.3.

For the umpteenth time I learned the following lesson: if an (Android) application crashes, chances are it is because I am writing past the end of a buffer into something important! Let's see if the lesson sticks this time :-)

Android Upload Fail

2011-08-30T23:05:00.000+02:00

Nobody spotted the *ahem* "deliberate" mistake I made with the last release of War of the Solstice - I updated the textual list of changes on the Market, but completely failed to update the APK file to 1.3! In fact, I'd uploaded it to the developer's console, but didn't press the right combination of buttons to get it to actually go live.

This is the danger of testing via side-loading - I never check that the actual version on the market is what I expect. It's all too much like hard work. Fixed it now though.

Another War of the Solstice update

2011-07-25T20:48:00.001+02:00

Despite my prediction, it actually took 2 comments to get the infamous "this game crashes my HTC Desire". The good news is that I've uploaded a new version that may fix the problem.

You can read about the fixes on the Market, the biggy was a couple of buffer overflows that caused random crashes. Once again, valgrind and some carefully crafted unit tests to the rescue. The trick to embedded coding is to get the code running on a real PC :-)

There's a new feature that's inspired by a pretty novel feature in the original Lords of Midnight - a screen shot record! On the original LOM, you could print out the current screen. I bet nobody actually did that though, as you needed a Speccy printer which were rarer than hens' teeth. Anyway, on this Android version pressing volume up or down at any time captures the current screen and saves it to the SD card. Then you can view the screens in your "Gallery" application.

I've also been practicing and finally beat the game with a carefully planned "Ultimate Victory", defeating Doomdark by taking Ushgarak and destroying the Ice Crown in the same game! To celebrate I added in the text for this, which was never contemplated in the original game. It's probably a bit of a let down after an epic 90-day war of attrition, so don't get too excited.

Next up, summer hols permitting, I'm working on "3D Space Game" to fix the crashes there too. I've got it crashing on my phone now, which is half the battle. I get the feeling I'm making some nasty assumptions about video ram because the crashes are inexplicable and "trace-less".

Updates to The War of the Solstice

2011-07-06T22:09:00.000+02:00

I've uploaded a bug fix version of The War of the Solstice (my remake of "The Lords of Midnight") to the Android Market. This adds a way to exit the game, both returning to the title screen and from there actually quitting the game completely so it no longer runs. There are also a bunch of bug fixes.

Fix minor Lord-under-control bug when Luxor is killed and the Moonring lost. Previously if Luxor was killed at night while you controlled another lord, you would still be able to control that lord the next day despite the loss of the moonring. This bug may actually have been in the Speccy original too, seems that way from looking at the z80 code.
Added fading transition between some screens - this was always in the GBA version, but I never got round to implementing it on Android
There was no way to just "think" before on the Android version, so I've changed touch-text-on-look-view to "think". Think is really just when you want to get a more detailed view of what is at a location without using up the location's seekable object/event. So if you're at a town, you don't rest if you just hit the text, whereas before you would use up the town's ability to let you rest if you first used "seek".
Fix sometimes corrupt gfx on the left side of the screen when panning left/right. This was a hangover from the GBA port, I miscalculated the offset for the screen scrolling by one block.
Better screen repaint for less CPU use. Now it does less work if you stay in the same place, only using CPU when it has to. It still isn't the lowest it can go, but it is much better now.
Fixed textual representation of all numeric values. This was a real mess up. Let's just say I wasn't careful enough with my use of C preprocessor macros and leave it at that. This meant that before instead of 10 saying "ten" it said "no". As in "Luxur has no warriors". All other numbers were a factor of 100 out. The actual values were correct though, it was just the representation that was messed up.
Fixed white rectangle shield bug after pause/resume. This is an Android platform thing and had been the bane of my Androidy coding existence. I'm not sure I've really fixed it 100%, but it does seem to work on my phone (so you can all come round and use that... yeah right)
Added Exit Game option, Back on title screen quits. Now you don't have to use an app-killer (which are pointless anyway, but whatever). Also if you get completely stuck, or are going for a Morkin win and he gets himself killed it's much easier to restart.

There's also hardware keyboard support in this version. It mimics the Spectrum setup 100%, so now all those letters and numbers on the menus make sense. I've only tested this on an emulator though, so expect it to break horribly on the real deal. For the record, as well as the aforementioned menu shortcuts here are the keyboard controls:

Q - move forwards
R - think
T - choose
E - look
U - night
S - save
D - load
1-8  - change to face direction 1:N,2:NE,3:E,4:SE,5:S,6:SW,7:W,8:NW

That is all, hope you like it. I've been playing this one quite a bit and despite knowing all there is to know about the game, I'm still crap at it! :-)

NDK Profiler Improvements

2011-06-27T22:22:00.007+02:00

I've uploaded a new version of my Android NDK profiler library. The major change in this version is much better timing accuracy through the use of ptrace. This requires Android 2.3 aka Gingerbread to work properly, so if like me you're sporting Froyo it's time to upgrade (if you can...)

If you do use Froyo or earlier then you will get timing information, but it will not be precise. In particular it will give too much time to functions called from other functions, time that is not attributed to the calling function. The actual call counts are still accurate, and it can still give you an idea of bottlenecks.

I've also added a new minimal example that is included in the source release. It's a silly example with 3 buttons and is pretty fragile, but should give you an idea of what you need to do to add profiling. There's also a new snippet of Make code that you can include to avoid copy-pasting all those TARGET_thumb_release_CFLAGS lines.

The Lords of Midnight remake for Android

2011-06-26T18:56:00.000+02:00

I've released another Android game, this time The Lords of Midnight gets the remake treatment. Once more it's a port from a Nintendo platform, this time the GBA as I never got round to updating it for the DS. It's so close to the Spectrum original that you can almost smell the rubber keyboard. Check out that colour clash.

I've called the remake "The War of the Solstice", as it looks like Mike Singleton together with Chris Wild may be making an official game for iPod and possibly Android at some point. Rumours have been going on for several months now with nothing actually released, so it may come to nothing. They are concentrating on the iPlatforms first too, so hopefully my remake will not have any impact on their sales. I can't blame them for doing iPhone first; if you were to charge people for games on Android it'd be a support nightmare. There are too many devices with too many random reasons to crash or otherwise go wrong.

That's why this is yet another freemake. Hurrah! No adverts either (I'd edit /etc/hosts to block the advert networks myself if I actually used my Android phone for anything). I bet the first comment on the Market for this is going to be "this crashes on my HTC Desire".

Anyway, this remake fixes at least 1 major bug in the AI that was in my original remake (but not in the original game, oh man!), as well as adding touch controls throughout. I've learned my lesson from Elite: people want touch controls. The only exceptions here are back (for obvious back-like things) and the menu button, which is need to save the game. Everything else is fat finger friendly. Go get it.

3D Space Game Source Code Release

2011-06-13T00:05:00.000+02:00

I've received a few emails recently asking about the source code for the ahem Elite port that's in the Android Market. The TL;DR summary: the source is now available and it includes updates to the DS and GBA versions with all core bug fixes that have gone into the Android version too.

The game began life on the Game Boy Advance and the source was always available in tar-ball form from my now-defunct Geocities remakes site (later moved to a new home). From there I ported the same code to the DS, though I didn't really make the most of the platform. There was no touch input and no use of the 2nd screen at all. The source was in a svn repository now though, on Google Code.

When I ported to the DS I did it the wrong way really: I ripped out the GBA parts and added in the DS bits instead. This meant that the Android port had to at least rip out the DS bits. I knew I didn't want to abandon the DS and GBA just yet, so as well as taking out the platform-specific code in the core part of the game, I added in hooks that each platform would have to implement in its own way. At first I only did the Android, Linux/SDL and DS parts of these hooks, but in the end I've re-done the GBA part too.

There's not a lot of documentation on building the source yet, I'll update that later this week if I get time. If you're interested in compiling all this, you'll need the Android development kits as well as devkitARM. The devkitARM dependency is for grit, so if you get that some other way and don't want to build the DS/GBA versions then you can do that too. Run "scons --help-variables" from the top level directory to see what things you can alter about the build. Or you can just browse the source code and marvel at the hideousness of it all.

As I've pretty much moved on from DS and GBA coding these versions are probably not as well tested as they ought to be. Let me know of any problems you encounter and I'll see what I can do. Patches are welcome :-) Also: saves are, sadly, not backwards compatible with previous releases. This is due to the addition of a new flag that stores the type of game - "Classic" or "Elite-A" mode. Elite-A is also know as "difficultly turned up to 11 mode".

Bug Fixes in Android NDK Profiler

2011-05-22T22:55:00.000+02:00

One of the biggest outstanding bugs in my Android NDK profiler library was that it would produce the gmon.out file but when you ran it through gprof you would see "no accumulate time" and the timing information was nowhere to be found. Now you can thank Justin Wick for debugging the root cause of the problem and providing a fix.

The library size I was using was too small to trigger the problem. It turned out that on larger libraries the calculation that inserted the addresses into the hystogram buckets had an overflow bug that caused the information to be lost. This was a left over from using the profiling code from Visual Boy Advance, since on the GBA the address space tends to be much smaller especially as it doesn't use random address layouts.

I tested this out using the Android port of FreeCiv, which prior to the change didn't show timing information, and with the change it now shows the times for each call. Once again, thanks to Justin for debugging the problem and figuring this out.

Using armeabi-v7a

The other problem you may have seen mentioned in the comments was with armeabi-v7a. In theory you can combine armeabi and armeabi-v7a together and things just work. Sadly I've had reports that doing this causes code compiled as armeabi-v7a to crash.

I finally got round to looking at what gcc spits out when you use the APP_ABI variable and set it to armeabi-v7a. There was a slight surprise in store for me that would explain the crashes. In a nutshell, when I said in my original post that I had to save all the registers in the profiling glue code, I had made an assumption that the link register (lr) did not have to be saved. With Thumb2 it looks like I do have to save that one as well, since gcc forgets that calling __gnu_mcount_nc (the profiling glue) is a function call that wrecks lr.

Look at this bit of C code:

int foo(int x)
{
        return x * 3;
}

When I compile it with -Os -mthumb --save-temps it gives the following:

foo:
        lsl     r3, r0, #1
        add     r0, r3, r0
        bx      lr

Now if I add in -pg to get the profiling calls I get this:

foo:
        push    {r4, lr}
        push    {lr}
        bl      __gnu_mcount_nc
        lsl     r3, r0, #1
        add     r0, r3, r0
        pop     {r4}
        pop     {r1}
        bx      r1

You can see it pushes lr twice - once for the use in the actual function, and a second time for __gnu_mcount_nc to do what it likes with it. Now if I compile for thumb2 using -march=armv7-a -mthumb I get this:

foo:
        @ link register save eliminated.
        push    {lr}
        bl      __gnu_mcount_nc
        movs    r3, #3
        mul     r0, r3, r0
        bx      lr

That comment is quite interesting. If you squint a bit it almost looks like a subtle compiler bug, since when you add profiling it should consider the call to __gnu_mcount_nc just like another function call and save lr. But as I haven't been able to test this on a real device yet, I'm not 100% sure that it's the only problem with using armeabi-v7a in combination with the profiling library. Plus once I saw what was going on I've been able to work around it anyway. If you get the chance to try this out, and are using armeabi-v7a let me know how the latest release works out for you.

Black screen of death on HTC Desire

2011-05-20T18:30:00.000+02:00

It seems that my version of Elite on Android is not behaving very well on the HTC Desire. The comments speak for themselves:

Black screen on start up, freezes phone, have to remove battery on HTC desire. I loved this game back in the day too.
by starbuck (May 20, 2011)

Don't download this app, this app has crashed my f*cking htc desire
by Marco (May 19, 2011)

I am giving it a 4 , but it has crashed a few times and clearer instructions would help. Desire hd
by Thomas (May 7, 2011)

Intro music played but black screen and phone locked. Had to remove battery. Desire
by Robert (May 6, 2011)

Black screen lock-up on starting. Had to remove battery to recover. Not good. Desire.
by Grampian (April 26, 2011)

If there are any HTC Desire owners who want to help fix this problem then please get in touch. I'm obviously doing something wrong, but the platform shouldn't crash to such an extent that you have to take the battery out! Especially when the game does run on some HTC Desires as well as on many other types of phone.

I don't know which is more frustrating: the crashes themselves or the attitude of the people leaving these drive-by 1 star reviews. Writing such a comment may make them feel better (hey! my free stuff doesn't work!) but in the long term it doesn't help anybody.

So if you have a Desire and are experiencing crashes, please, please try and send me a "logcat" of the crash so I can work out what's going on and fix it. There's an application in the Market called aLogcat that lets you capture the log and email it out. Check that no personal information is in there (there shouldn't be any) and send me the info if you can. Otherwise there is nothing I can do, short of blacklisting Desires.

On a happier note: A new version of Chaos is out that fixes all the bugs that were reported, except one. The Nintendo DS and GBA versions are based on this same code. Fixes on all platforms are:

Make sound fx volume more consistent
Fix dismount wizard bug
Speed up magic bolt
Add a message when Meditate's side effect kicks in (it freezes the wizard)

The one bug that I can't reproduce is (surprise) Android only, and seems like another "Android hates me" oddity:

black screen after pause/resume

I've made a change that may help, but who can tell. Interestingly, a lot of commercial games seem to just kill the activity when you move away from it instead of dealing with this issue.

Android NDK Profiler Example

2011-04-30T20:29:00.000+02:00

I've added an example to the android-ndk-profiler project based on the native-activity from Google's Android SDK. As Google Code doesn't let you mix'n'match licences, and importing a bunch of random code into my SVN seemed a bit crap, I've not included the original source in the repository.

What I have done is supply a patch and a script to copy the code from your NDK installation, patch it up to use profiling, update the build scripts and then build the example. You'll need to have the "patch" program installed to get this working, or apply the changes manually.

Since this is Android 2.3 only, I've only been able to test it on the emulator. Also the emulator doesn't like armeabi-v7a, so that remains completely untested - I'd stick to using APP_ABI := armeabi for profiling for now.

Hopefully this'll clear up any misunderstandings like those mentioned in the comments on the last post I made on the subject. Full details are on this wiki page about getting the example working.

Why are my textures not showing up on Android?

2011-04-09T18:11:00.000+02:00

What a disaster the initial week of "3D Space Game" has been! I recieved lots of 1 star reviews with vague hints about what the problem was - "doesn't work", "goofed up". When I realised there was a problem and added a note on the market asking people to email their problems before "down voting" I got a couple of very helpful bug reports.

It turns out that on the Galaxy S and other larger-screened devices all of the textures I created from PNG images were not showing up. This meant that the control buttons and menu icons just show as white rectangles, completely unusable.

The usual reason for the white rectangle effect is that the size of the texture you're trying to use is not a power of 2. Textures have to be 32x32, 64x32, 128x512, etc - you can't have a 257x123 sized texture (unless it's on the Android emulator, which doesn't care about texture size). I even mentioned this in my "lessons learned" post in January. The code I use creates a texture straight from the PNG like this:

Bitmap bmp = BitmapFactory.decodeResource(getResources(), R.drawable.my_image);
GLUtils.texImage2D(GL10.GL_TEXTURE_2D, 0, bmp, 0);

The size of the texture is taken from the PNG, which I had carefully made sure was a power of 2. On my HTC Magic the textures show up fine, so I thought everything was working correctly.

Well, it turns out that the BitmapFactory uses a default set of options that includes a "resize" flag. The flag is by default set to true, or "resize the image to screw up Open GL textures", as it should be known. The fix for this was to create my own set of Options and set the flag to false like this:

BitmapFactory.Options opts = new BitmapFactory.Options();
opts.inScaled = false;
Bitmap bmp = BitmapFactory.decodeResource(getResources(),
                R.drawable.my_image, opts); /* pass in opts */
GLUtils.texImage2D(GL10.GL_TEXTURE_2D, 0, bmp, 0);

Problem solved. Let's hope for a reversal of the 1-star trend, eh readers?

3D Space Game for Android

2011-04-02T11:39:00.001+02:00

This is pretty much "Elite for Android". It's a port of a port of a remake of a fan-tweaked version of Elite. I first wrote this as a remake of Angus Duggans's Elite A for the GBA (Elite A being a tweaked version of the original Elite on the BBC Micro), ported that to the DS and now here it is on Android. It's not been without problems due to a severe lack of buttons on a phone. This version adds in "classic mode" to play the game without the Elite A modifications.

I started the Android port back in October 2010 and it was more or less complete by December 2010. I've spent the months since then squashing bugs and refining the controls. Hopefully it will work on a lot of devices - I've been testing on 2009 vintage hardware - but there's no way to know for sure. Try it and let me know. In particular Samsung really threw a spanner in the works by not supplying a d-pad or trackball on the Galaxy S.

There are still some known problems in this first release:

Yellow line under title is not always shown
Sprite text still shown on panels
Long press should show route to planet (bit flakey, ditto Galactic Hyperspace)
Text occasionally does not update

Most of these are due to the GBA/DS heritage, but none are show stoppers. I'll be updating with fixes as people complain :-) Also, the Nintendo DS and GBA "backports" of this code are not ready, so this is Android-only for now. There are still some outstanding problems that I need to fix on the DS, and the GBA code is further off, but I'm starting to get project fatigue here and need a change.

This video shows the gameplay, it's a little choppy - if anyone knows a good way to video off an Android device I'd love to know. I used droid@screen for this, which seems to be the best option out there, but it doesn't record video. I had to record the X11 window separately using xvidcap and this made things slightly worse. Anyway, on with the show:

Profiling Android NDK Code

2011-03-27T01:27:00.001+01:00

Profiling code on embedded systems is tricky. The Google tools for profiling Android applications only cover Java code, not native C code. So I've created a new project on Google Code for profiling Android NDK shared libraries. I had the idea when reading this post on the GBAdev.org forum - there coder "sverx" asked how to profile code, and I remembered Visual Boy Advance did it pretty well on the GBA, but there was no way to do so on the DS - nor on Android. Well, the Android situation just improved :-)

I read the VBA source, lifted the relevant chunks of gprof-inspired code, hacked in some timing information, and added a touch of assembly to tie together what gcc -pg does and what the profile-creating code does. If you look at a chunk of the output from gcc with -pg, it looks like this:

0001184c <do_raisedead_cast>:
   1184c:   b5f8        push    {r3, r4, r5, r6, r7, lr}
   1184e:   b500        push    {lr}
   11850:   f004 f830   bl  158b4 <__gnu_mcount_nc>
   ...
   1189c:   3a01        subs    r2, #1
   1189e:   701a        strb    r2, [r3, #0]
   118a0:   bdf8        pop {r3, r4, r5, r6, r7, pc}

So the compiler pushes the link register (lr) and the __gnu_mcount_nc function has to pop it off the stack. This isn't a normal calling convention - usually each function uses the stack symmetrically, pushing and popping the same number of times. Also I learned that the mcount function must preserve all the registers. You can't smush a single one or Bad Things happen (I've seen a lot of weird crashes lately, I can tell you). This little conundrum took me quite some time to figure out. It can probably be done better, but the implementation I came up with was this:

        push {r0-r3}
        push {lr}
        ldr r0, [sp, #20]  @ r0 = lr pushed by calling routine
        mov r1, lr    @ address of calling routine
        bl profCount
        pop {r0}
        mov lr, r0
        pop {r0-r3}
        @ pop previous address
        add sp, #4
        bx lr

Since registers r4+ must be preserved by the called function, I can ignore those when the code calls out to profCount. Keeping r0-r3 clean was the problem, as usually these are "scratch" registers and you're free to mess them up. The function profCount takes the C function-just-called and its caller addresses as arguments for use in generating the profile. Anyway, the results are pretty good for the code I've tested it on - you get the usual call counts and timing information that gprof provides. Here's a bit of the information from Chaos...

Flat profile:

Each sample counts as 0.01 seconds.
  %   cumulative   self              self     total           
 time   seconds   seconds    calls  ms/call  ms/call  name    
 65.31      0.32     0.32      288     1.11     1.11  android_render_screen
 18.37      0.41     0.09      450     0.20     0.20  clear_square
  4.08      0.43     0.02        3     6.67     6.67  clear_bg
  2.04      0.44     0.01       25     0.40     0.40  print_text16
  2.04      0.45     0.01        3     3.33     3.33  draw_decor_border_internal
  2.04      0.46     0.01        2     5.00    33.00  splash_touch
  2.04      0.47     0.01        1    10.00    52.00  options_back
  2.04      0.48     0.01        1    10.00    54.80  show_options
  2.04      0.49     0.01                             chaos_start
  0.00      0.49     0.00   442368     0.00     0.00  draw_8x8_tile
  0.00      0.49     0.00    28800     0.00     0.00  platform_set_tile_entry
  0.00      0.49     0.00     3646     0.00     0.00  platform_set_text_map_entry
  0.00      0.49     0.00     2671     0.00     0.00  platform_key_pressed
  0.00      0.49     0.00     1186     0.00     0.00  platform_set_palette_entry

index % time    self  children    called     name
                0.32    0.00     288/288         chaos_gl_render [2]
[1]     65.3    0.32    0.00     288         android_render_screen [1]
                0.00    0.00  442368/442368      draw_8x8_tile [22]
                0.00    0.00     286/286         InterruptProcess [30]

Functions with no "calls", "self" or "total" are because they got called via JNI. The "draw_8x8_tile" call count there is accurate - it is called for each 8x8 tile on the screen in each frame. There are 32 x 24 tiles, and 2 layers of tiles. 32 * 24 * 2 * 288 = 442368, as expected. And the android_render_screen count is correct, since at 30 fps that is around 9 seconds of use and I only ran the game for a few seconds to check.

More details on the android-ndk-profiler page, including a guide on how to use the library.

How to use Bitbucket's free private Mercurial repositories with Git

2011-02-13T17:55:00.000+01:00

Update: Since I wrote this post, bitbucket has added native Git support. Are you spotting a pattern here? Anyway, I'll leave the rest of the post here for the historians.

When Atlassian bought bitbucket in September 2010 one of the first things they did was to allow single users an unlimited amount of private repositories for free. If you want to create an backup of your own code without making the source available this seems like a difficult offer to beat. I've been using it for some months to do exactly this.

The only problem is that bitbucket uses Mercurial, what if you prefer git as your day to day version control? No worries as the Atlassians might say, you can bridge from Git to Mercurial using hg-git and a bit of fiddling. Here's what I did to set this all up, in case I forget later.

I use Ubuntu 10.04 and although hg-git is in the repositories it is an older version. Instead I installed python-dulwich via apt-get install python-dulwich, cloned the repository at https://bitbucket.org/durin42/hg-git and then added a suitable entry to my ~/.hgrc file:

hggit = /path/to/hg-git/hggit

The first gotcha is that hg-git can't clone a local file:// repository. When you try it, you get an error like this:


/tmp$ hg clone file:///home/rich/projects/test
destination directory: test
importing Hg objects into Git
fatal: 'file:///home/rich/projects/test' does not appear to be a git repository
abort: the remote end hung up unexpectedly

The workaround is to serve the repositories using the git-daemon and then clone them using the default git network protocol. I have the code I want to back-up in a directory ~/projects, so I start the git daemon to export all of them:

git daemon --verbose --export-all --base-path=~/projects/ --export-all

That is not particularly secure, but it isn't available outside my home network and means I can clone each repository using hg with the hg-git plugin enabled:

hg clone git://localhost/name name-hg

Once I have a local Mercurial clone, it is easy to set up the push to bitbucket. That is just vanilla Mercurial configuration. In name-hg/.hg/hgrc I added the bitbucket path so the full configuration looks like this:

[paths]
default = git://localhost/name
bb = http://user:password@bitbucket.org/user/project

Now name and project depend on the local name and the remote bitbucket project name, but hopefully you get the idea. I push this out to the server with hg push bb. Next time I want to copy all the changes I've made in the git repository since the last backup, I fire up the git-daemon, sync up the mercurial repository then push out the changes. I have a script that does all this, which boils down to the following:


#!/bin/sh

# backup git repos to bitbucket private repos...
cd ~/projects
git daemon --verbose  --export-all --base-path=~/projects/ --export-all &

for name in project1 project2 ; do
    # echo "did this once... not needed again"
    # hg clone git://localhost/${name} ${name}-hg
    cd ~/projects/${name}-hg || exit 1
    hg pull || exit 1
    hg update || exit 1
    hg push bb || exit 1
done

This way I can use git for day to day hacking, and have a secure backup on a server somewhere for if the worst happens.

Chaos 1.7 and Android Market on the web

2011-02-05T21:52:00.000+01:00

I added Chaos 1.7 to the Android Market the other day. It's now passed 3000 downloads and I've had several emails and 5 star reviews from Chaos fans, so I'm pretty pleased that it's gone down that well. The only downer is that it really is for fans only, I just don't have any idea how to teach anyone brought up on modern games how to fight their way through an 8-bit world. Patience is needed, I guess.

This release was the first time I'd linked to the brand new web-based Android Market. Until now I've always pointed to Appbrain. I think Appbrain has been struggling to keep up with the new additions that Google have made - like bigger icons, more screenshots and promo graphics. Appbrain has also had some down time and synchronisation issues, with comments being out of date or not appearing. I'm not sure what will happen to the 'brain. They have done a great service for Android users and it would be sad to see them disappear, but their business model was the classic snatching nickels from the path of an oncoming steamroller. I do hope they stick around, if only to keep Google on their toes.

Anyway, back to Chaos. The changes in this release are:

Remember preferences
Several performance improvements for better battery life
More consistent frame rate on newer Android devices
Attack sound no longer plays when sound is disabled in the options

Saving the preferences was suggested in the comments on the Market. I hadn't added it before because I didn't know how to save stuff on Android. Turns out it is really easy and doesn't even need extra permissions. Actually, I'd already learned how to save for the next game I'm doing so a nice bit of "copy paste" later and it was done. Ah, good old copy and paste. The best form of code re-use.

The performance improvements included not redrawing every single square every single frame, which for some reason I was doing before. If you spot any odd graphical artefacts because of this, do let me know.

The frame rate fixes were due to my recent discovery that newer Android devices are locked to a maximum of 30 frames-per-second. Hopefully Chaos now runs at the same speed across the board. If anyone wants to buy me a 2nd generation Android phone to check this on, please get in touch ;-)

The attack sound fix was a silly non-interesting bug. I fixed it ages ago but also got a prod by email to actually release a version with the fix in place.

I've updated the Nintendo DS and GBA versions too. These don't save the preferences, and I haven't tested them on hardware so let me know if they do not work at all. Apart from that, the DS and GBA are compiled with optimizations now (somehow this got lost in the last GBA/DS release) and should go faster. The GBA version was particularly sluggish last time round.

Lessons Learned with Android

2011-01-16T19:38:00.002+01:00

Around the summer of 2010 I decided to switch from hacking around on the Nintendo DS to hacking around on Android.

Android piqued my interest for the reaons I outlined at the time, but also because it has been changing at a breakneck pace especially if you compare it to the DS. I bought a HTC Magic off of ebay to test out my code. At the time the Magic had been out for a year or so, yet it was already considered out of date! To everyone's surprise Google updated the version of Android running on the phone to the very recent Froyo 2.2.1 at the end of December 2010. Meanwhile, Nintendo only updated the firmware on the DS to block flash cartridges that allow homebrew. From a fun-filled hacking point of view Android is a more entertaining prospect.

The distribution method - upload to the Market, and users are automatically notified of updates - is really great. As a user it's always nice to see applications that have been updated and to read the "recent changes" section to see what's new. As a developer, there's no need to advise people that you've released a bug fix. You just upload a new apk file, update the recent changes section, and you're done.

Here are notes on some of the things I've learned while coding for Android.

The emulator is not great for games

When I wrote about my switch to Android I said "coding for Android feels like cheating. The emulator is highly accurate". I must have been viewing the new (to me) platform through rose tinted glasses there. The reality is that the emulator became useless as soon as I started using OpenGL to render Chaos. On hardware I was getting close to 60 frames per second (FPS), but on the emulator I was lucky to get above 10 FPS. This made testing on anything but hardware impossibly slow.

Fortunately I had my experience with emulators from my DS days and I knew that the best way to test embedded code is to get that code running natively on desktop Linux. The majority of the time I compile and test the Linux build, only occasionally switching to Android to check things still work. All the real gameplay testing I do is on hardware though, as the feel of using a touch screen is nothing like the feel of clicking things with a mouse.

There is an inaccuracy in the OpenGL emulation that could catch you out if you never tested on hardware. On the emulator you don't need to call glTexParameterf and set GL_TEXTURE_MIN_FILTER and GL_TEXTURE_MAG_FILTER to GL_LINEAR or similar, which sets up the texture "mip maps". If you fail to do this on a real device, your textures will only show up as white polygons.

I think Android should use an approach more akin to the iPhone or Windows Phone, where you compile to a native simulation of the application. Instead of emulating a full Android phone and running your app, you have the Android APIs call into native code directly. It's probably too late for Google to change this now and I have my own workarounds. Hey, at least if your game runs fast on the emulator, it will run very well on the real deal :-)

Each Android device is a precious snowflake

A game can run well on the phone you have tested with, but this doesn't guarantee anything. Soon after you put the game on the Market, you can be sure someone will post a 1-star review stating that the game doesn't run on their particular Android device. The Market dashboard for developers has a tab for "Error reports" that never shows anything, but it would be nice if it somehow gave a logcat dump of crashes and force closes.

If you want people to get in touch with you instead of posting a 1-star review, you really need to explicitly state that on your game's description. Otherwise you will never get to the bottom of errors. I think the instant gratification of giving a free game 1 star trumps the hassle of getting in touch with the author. It's perfectly understandable really and something other people have noticed with free app users vs paying customers - people using free things actually have higher expectations!

The SoundPool class, which is used to play back sampled sounds and seems ideal for games, will randomly crash on some hardware platforms and some versions of Android if you use MP3. MP3 is one of the supported media formats, but there are some weird issues on some phones. For example Chaos crashed on an HTC Hero, which has almost identical hardware to a Magic, running similar versions of Android (2.2.1 "official" vs 2.2.1 cyanogen mod). Random internet forum posts show that OGG files work better, but that WAV files can crash if looped. When I switched from using MP3 to OGG the crash reports I was seeing dried up.

When managing the life-cycle of a native application, strange things can happen with your OpenGL context. According to the documentation all you need to do is call the "onPause()" and "onResume()" methods of your GLSurfaceView when the user switches away and then returns to the game. However, on Android 1.6 at least you can get in a state were the "onSurfaceChanged()" method is not called again, so you don't have a hook to setting up the correct GL state. This seems to have gone away with Android 2.2, or at least the game's already set-up context is not ruined in the same way. On 1.6 I'd seen Chaos have a wrecked graphical display after the phone went to sleep, but since the 2.2 update this hasn't happened to me.

You should use 30 FPS as a baseline

A few weeks ago I wrote about getting a good frame rate on Android using the NDK. On my HTC Magic I was getting nearly 60 FPS with that approach. Well, it turns out that new devices are limited to 30 fps. So if I assume a 60 fps rate (as seen on the DS), I would make everything run half as fast. The solution is to limit the frame rate to 30 fps even on "faster" (older!) hardware.

An advantage to running at a lower frame rate is that it saves the battery, but it still seems a bit backwards to me.

More ratings after an update

Almost every time I've posted a new version of Chaos it has received a few new ratings on the Market. I imagine that this is because people download the game then go off and play it, but forget to go back to the Market and rate it. When a new version appears, they return to the Market and now they know it's amazing (!) so rate it 5 stars and install the update.

Don't abuse this phenomenon though. If you upload lots of minor versions with no user-visible changes, people will quickly tire and start rating your game worse. I saw this effect in a few comments for the game "My Paper Plane 2", which had dozens of minor updates with no new features. One reviewer said "Minus 2 stars for frequency of updates and no explanation."

Eclipse is not great

I don't know if the issues I've had are related to the ADT plugin, communication with the emulator or Eclipse itself, but the integration of the whole Android SDK + Eclipse IDE is flaky. Often the text editing GUI hangs for 30+ seconds when you type a "." and auto-complete kicks in, with the CPU grinding away at 100%.

Other times Eclipse fails to connect to the emulator to debug or upload new apk files, and you have to restart the emulator. The emulator takes minutes to restart, it is slower to boot than even the hardware phone. Then there's the built in logcat view that will miss messages if they pass by too quickly. The command line adb logcat equivalent will show you what you missed.
I haven't tried the GUI builder nor XML editor, so cannot comment. At least in the last ADT version opening the XML layout files no longer causes a NullPointerException.

As a result of all of this, I continue to do all my development using Vim. For auto-complete I use CTags and Vim's omni completion. This is enough for C code. When editing Java I just write out all my imports the old fashioned way, like we did back before IDEs hid this crufty aspect from us all. I can always say it is helping me learn the API...

I have a Vim command :AdbReinstall that loads the APK file onto the emulator or device. I can build using :make. It's great that the Android SDK is flexible enough to allow me to do this, there are even instructions on how to set parts of it up in the official documentation. I doubt it is as easy to do for iPhone or Windows development.

Filtering based on the hardware used

One interesting characteristic of the Android Market that has matured a lot in the last few months is the way applications are filtered based on the hardware features they use.

As an example, Chaos uses the touch screen and is compiled for arm-eabi. The touch controls are actually optional - if you want you can just about get by with the trackball or d-pad and the buttons - but originally I didn't state that in the manifest. This meant that Chaos would not be seen on x86 Android devices (are there any?) nor on devices without a touch screen (examples?). This information was all auto-detected by the Market server - I didn't have to mention anything anywhere.

Once I decided that Chaos was usable without a touch screen, I added a "uses-feature" section to the manifest stating that the touch screen was optional.

There may be hardware fragmentation, but this part at least is handled in a quite developer-friendly way.

The next game I'm doing will use the compass and accelerometer for tilt controls. As far as I can tell, I won't have to state anything in the manifest to avoid disappointing folk with phones without these features, the Market will add the filters automatically (we'll see!).

Touch screen is bad for controls

The lack of physical buttons on Android means that several genres of games are not really possible, or you need to add a virtual d-pad and buttons on screen that react to touch. Unfortunately there seems to be a horrible bug in the way Android processes events when the user leaves their finger on the touch screen. When you hold your finger on the screen it leads to the touch input completely swamping the CPU to the detriment of the game's frame rate. The onTouchEvent method is called every 10 milliseconds with what amount to non-events, such as subtle changes in pressure detected by the screen's sensor. Even if all you do in the event handler is store the touch position and type, to later process in the main thread, the application will lose out on framerate as the touch event thread is given priority.

Now I may be doing something wrong here, but several similar-looking questions on stackoverflow.com seem to agree with my experience. There's a bug report on the Android project that has been fixed in Gingerbread. Sadly the lower-end devices that really need this fix will never see Android 2.3, at least not officially. Meanwhile, all the "sleep" workarounds proposed don't seem to make things any better.

There we are then. Android has some issues when compared to a closed hardware platform like the DS. You can't be 100% sure that what runs on your device will run on everyone else's. This is scary, but then there's a good chance your game will work OK. And it will probably work fine with different Android versions, screen resolutions, types of processor and hardware features. I think for free games, or if you have a free demo for people to try before they buy, then this is an acceptable compromise.

After my albeit limited experience with this, I understand now why a limited hardware platform is going to be more popular for games. If you compare the amount and variety of games released on the Apple hardware (iP*d/iPhone) to Android, the difference is huge. Giving stuff away for free is obviously not too cost effective (having embedded adverts can mitigate this, but adverts are only viable on huge hits) and creating a free demo takes time away from creating the core game. So given the choice, developers would rather have people pay for the game, right? If you expect people to pay for your game, then you'd like to think it works on their device. When you write for a platform with a limited range of hardware devices, you only have to test on those devices. With the DS, the DSi and DSiXL are identical platforms - Nintendo is very careful about backwards compatibility. You don't need need to test GBA games on a DS to check they run correctly, for example. These things just work.

Now if we consider Android, there's no way any regular developer could test all possible devices. The best you could hope for is to test on maybe some representative devices. If you support old versions of Android, a first generation HTC phone. Then maybe something like a Nexus One, then a Sony phone as they seem to have quite different Open GL hardware, maybe a Motorola and a Samsung due to their popularity. Factor the tablet-sized Android devices and we're talking quite an investment.

This doesn't include the various Android versions that may run on these devices - you probably have to target a lower API level and hope not to slip up and include newer Java API calls. Those would result in run time errors on older versions than your target API. When you write a game primarily in C it's harder to slip up like this, but still possible. For example, getting the name of the SD card directory to write to changed a lot in Froyo. Given all of this, the openness of Android is definitely off-putting if you're only in it for the money. As a tinkerer into this stuff, it certainly keeps me on my toes. Maybe the new Gingerbread SDK with its "look ma! no Java!" C-only Activities and touch screen bug fixes will lead to more Android 2.3-only (paid) games. Especially if the Sony Play-not-station phone runs this version and has any kind of popularity.

Who am I kidding? It's more likely that the iP*s will continue their rise in popularity with gamers, while Android becomes the new N-Gage. Hey I wonder if the 3DS will be hackable? :-)

Changes in Chaos 1.4

2010-12-19T23:06:00.001+01:00

I've uploaded another version of Chaos to the Market. This adds the oft-requested "Quit" feature. Now you can leave a game at any time by selecting the option from the game menu. Once on the splash screen, if you press "back" the game will exit and you'll be returned to the home screen, or wherever you were when you started the game.

The second change reduces the size of the frame width around the board. In the GBA version I had increased this to 16 pixels (2 GBA hardware tiles) so that scrolling was less pointless. When the whole game fits on a single screen with space to spare, having such a large border doesn't make sense. This decrease in the total game board size means the graphics are slightly bigger than before once they've been scaled up. Here is a screen shot showing the before and after versions of the game screen so you can compare:

You can just about see the extra pixels used on the left and right of the screen. Now the game reaches to the ! on the "USB attached" icon and to the 2 of the 22:00 on the clock. We're only talking a few pixels here, but it's better than nothing.

I fixed a minor bug that caused the game to show the meditate spell incorrectly on the "examine spell" screen. No biggie there.

The last change I've made is to reduce the size of the APK file. What can I say? I was using uncompressed sound files, and now they are MP3-compressed. The APK size went from 512K to 160K, and the installed size went from around 800K to 380K. This was such a simple change that it reminded me of the Daily WTF article about the software company that added a pointless loop in the code that they reduced by an order of magnitude every now and again for an "amazing speedup".

Finally, I've released the GBA and Nintendo DS versions based on this same code base. I had a C++ semi-rewrite that ran on the DS, the old C code for the GBA, and then this version of the GBA C code tidied up for Android. The Android version also incorporated the bug fixes and improvements from the DS version, including the new spells. Now that I've finally figured out how to write cross platform code without too many hacks, all of these are based off of the same common C core.

I've created a new site with the Nintendo DS and Game Boy Advance downloads as well as some instructions on how to play on Android.

The hardest part about testing the GBA version was dusting off my old Windows 98 laptop, finding the "Flash Advance" adapter (the drivers came on floppy disk!) and copying the GBA file onto the aging flash cartridge. My current laptop doesn't have a printer port, so there was no other way to get the GBA files onto the flash cartridge. Here's a photo of my workshop set up on the dining room table.

The photo was taken using a DSi of course :-) Suffice to say my dear wife was not impressed. I was accused of being like Jean Michel Jarre, surrounded by gadgets.

Chaos 1.3

2010-12-05T01:17:00.001+01:00

I've uploaded a new version of Chaos to the Android Market. The only bug fix is the missing attack sound, but there are a couple of Android-specific cleanups in there. First off, I've now forced the game to run in landscape mode. Swapping between landscape and portrait was more of a tech demo than an actual useful feature. In addition, maybe people didn't realise you can rotate the screen to get a better view. I never used portrait mode anyway, it was too small. The second change is that you can now install the game to the SD card if you are running Android 2.2. I only have Android 1.6, so let me know if this works or fails spectacularly for you.

Spell select screen

For this release I upgraded my tools to the latest Android SDK, which is designed for Android 2.2. Prior to this upgrade I had assumed that I needed to use the old toolkits to compile for old versions of Android, I didn't realise that I could compile for 1.6 using the latest tools too. The explanation on the developers' documentation page is pretty confusing. As usual, looking for the API strings on Google Code Search turns up real world uses of the features and the correct implementation.

Speaking of confusing versioning, how about this tongue twister: the Android SDK is currently at API 8, revision 2. This is for Android 2.2. The SDK tools are at revision 7. These SDK tools are not tied to a version of Android, but API 7 is compatible with Android 2.1 in other places. In order to run on Android 1.6, you have to use "minSdkVersion=4", which corresponds to API 4. The latest version of the Android NDK is revision 4b. But the 4 here doesn't mean it only works with Android 1.6. In fact the NDK includes the libraries and header files for API versions 3, 4, 5 and 8, which correspond to Androids 1.5, 1.6, 2.0 and 2.2. Note that it doesn't include different headers, etc. for API levels 6 or 7.

And then you wonder why it took me so long to figure out how to use the latest SDK to compile for older versions of Android! :-P

Responding to Android Market Comments

2010-12-01T23:08:00.003+01:00

The Android Market lets anyone comment on any of the applications available. Sadly there is no way to respond to a comment on the Market itself, so here is my feedback to your comments on Chaos...

by Neko_Noskire (November 28, 2010) ★★★★☆
This is what trading card games should have been like. A way to get out of the game would be nice.

Seems this comment was later deleted, but it still shows up on AppBrain. Adding a way to quit is actually quite tricky given the way the Android Activity lifecycle works with applications using native code. At the moment you can just press the Home button to go back to the home screen. This pauses the game and, assuming the OS does not kill the Chaos process, you can restart the game by pressing its icon again. This can lead to some surprising resets, though the alternative (always hard reset when you hit the Home button) is worse, I think. I don't particularly like games that do the forced hard reset anyway, it seems to go against the spirit of multi tasking. Or maybe Neko wanted a way to completely abandon a game that you'd already started? Hmm, that might be it and that would be doable. One for the list of things to do.

by Peter (November 21, 2010) ★★★☆☆
Pretty good, but needs landscape mode & speeding up; missing "attack" sound

I wonder if Peter uses an EVO 4G? Rumour has it that the frame rate is capped at 30 frames per second, which would cause Chaos to run slower than it should do. I use the frame redraw to sync the timers in the game, you see. As for the attack sound missing, I'll have to check that. Disclaimer: I often play with the sound switched off. Heh heh. The landscape mode comment is strange - just rotate your phone, the screen adjusts. Maybe I should lock the game to landscape only? The portrait mode version is not very useful as you end up with a tiny area to play on and big black borders at the top and bottom.

by Jon (November 20, 2010) ★★★★★
I now no longer need any other game. Awesome retro fun!

Thanks Jon! :-)

by Martin (November 1, 2010) ★☆☆☆☆
Naja, schwaches Game. Keine gescheite Steuerung :(

Translation: Well, weak game. No intelligent control. Er, okay. I think an '80s videogame magazine tautology will do here: If you don't like games like Chaos, then this game is not for you.

by Kieron (October 31, 2010) ★★★★★
YES! 1 of my fav Spectrum games that I still play on emus, Marvin works well but the keyboard mapping isn't great for phone. This port solves it! Thanks

This is why I wrote the GBA version originally too - I could play Chaos on FooN, but the controls were unwieldy. Almost a remake of a remake then.

by darren (October 31, 2010) ★★★★★
Amazing cool sexy fun ! Hint ! Kill the warlock on level seven by using your blood diamond in his water

Right... Not sure what darren has been imbibing (blood-diamond tainted water perhaps?) but at least he rated the game 5 stars. This may have been an odd sort of spam actually, it appeared almost immediately after uploading the game to the Market. I can't see what the spam is trying to achieve though. Very strange.

So there we go. Keep the feedback coming - Chaos is in maintenance mode while I hack on my next Android game, but I'll still add features if they sound good and aren't too tricky.

Changes in Chaos 1.1

2010-11-07T20:06:00.003+01:00

I've updated version 1.1 of Chaos in the Android Market. This version contains 2 bug fixes:

Computer opponents could never move again after casting Meditate
2 sources of random crashes fixed

The first bug was a silly one, easy to fix. The other one, well, I didn't want to believe the random crashes. I was hoping it was caused by cosmic rays or something. After it happened to me again when I was finally about to win I decided to get to the bottom of things.

I added a "stress mode" to the PC test version I have, and left 8 computer controlled wizards to battle it out all day watched by valgrind. The results were that there are 2 places were I ended up reading well outside the memory I had allocated. I haven't seen a repeat of the error since then, so hopefully the problem is fixed now.

Chaos on Android!

2010-10-31T18:06:00.002+01:00

I've just uploaded my remake of Chaos to the Android Market. It's free to download and is advert free. Hopefully this link will work market://search?q=pname:chaos.app or you can download via appbrain.com here.

This is pretty much a straight port of the GBA version of Chaos I wrote a few years back. I've added in some features from the Nintendo DS version I also did, such as the "Meditate" spell (new spells at the cost of no movement for the wizard that round) and finishing off the sleep and blind spells that almost existed in the Spectrum classic. The effects of Sleep are obvious, but Blind reduces movement, attack and defence to minimum. There's a random chance of either wearing off at the end of each cast round.

It has touch screen controls or can be controlled using the trackball or control pad if you have them. I've been testing this on my HTC Magic with Android 1.6. I hope it runs okay on newer phones with bigger screens, but let me know if you have issues.

Android Native Coding in C

2010-10-13T02:00:00.001+02:00

When I read questions like "How to do code sharing between Android and iOS" on stackoverflow.com, and the answers are variations on the theme: "the NDK is not for creating cross platform apps", it makes me sad. The NDK is an excellent way to write cross platform games. Here is a little insight into the approach I've taken with my so-far unreleased Chaos port.

To code a game in C on Android you have to first write a Java Activity with a View. This can be either a regular View or one that is OpenGL-ified, this explanation uses the GLSurfaceView. Then you use the Java Native Interface (JNI) to call from Java into your C code. You compile your C code using the Android Native Development Kit. The remaining problem is then: how can I draw pixels?

You have 2 options (3 if you are willing to target Android 2.2+): drawing pixels to a Canvas, drawing to an OpenGL ES texture, or drawing directly to the pixel buffer of a Bitmap. This last option is similar to option 1, but is faster and available on Android 2.2 "Froyo" only.

Assuming you want to draw a screen that is smaller than the Android native screen size and scale this up, the OpenGL version is the fastest and most compatible of the 3 choices. Using OpenGL from C code is actually cleaner than from Java, as you do not need to worry about adding gl. to all the GL function calls (gl.glActiveTexture for example, where gl is an instance of javax.microedition.khronos.opengles.GL10). You also don't have to worry about the arrays you pass to GL functions being on the Java heap rather than being the required native arrays. This means you don't have to deal with all the ByteBuffer calls that clog up the Java OpenGL examples.

You will need at least 3 native functions to draw in OpenGL: the "main loop" that runs your game's code, the "screen resized" and the "screen rendered".

The main loop can be the classic while (1) { update_state(); wait_vsync();}. The screen resized function is called when the Android device is rotated or otherwise needs a new screen setting up. The screen rendered function is called once per frame.

The main loop and the render functions both accept no arguments. The screen resized or set up code accepts a width and height argument. The Java native declarations for these calls will look like this:


private static native void native_start();
private static native void native_gl_resize(int w, int h);
private static native void native_gl_render();

static {
       System.loadLibrary("mybuffer");
}

Now you have to write these in C and somehow register them with the Dalvik VM (or "Dalek VM" as I often misread it. Exterminate!) Dalvik uses the same approach to binding native methods as the Java VM does; it opens a native library with dlopen() and looks for the symbol JNI_OnLoad and functions with "mangled" names that match the native declarations. The library loaded here will be "libmybuffer.so". You can either implement your functions with the mangled names or using a call to "RegisterNatives" in the JNI_OnLoad function. There are many JNI tutorials on the net, I won't rewrite how to do that here. Whichever way you choose, you still need to have the native declarations in your Java source code. My examples are using RegisterNatives, as it gives cleaner C function names.

In the constructor of your Java GLSurfaceView, you should call the main loop (in C) from a separate thread - this way the loop does not block the main thread of your Android application and the Android OS won't kill it for being non-responsive. It is important that the main C code does not do OpenGL manipulation as that can also crash the application. All GL manipulation is done in the renderer call. The main-loop thread can change whatever C state it likes; the render call later reads this state to create the final rendered screen.


public GlBufferView(Context context, AttributeSet attrs) {
       super(context, attrs);
       (new Thread() {
               @Override
               public void run() {
                       native_start();
               }
       }).start();
       setRenderer(new MyRenderer());
}

The implementation of your GLSurfaceView.Renderer class simply delegates to the native functions and should look like this:

class MyRenderer implements GLSurfaceView.Renderer {
       @Override
       public void onSurfaceCreated(GL10 gl, EGLConfig c) { /* do nothing */ }

       @Override
       public void onSurfaceChanged(GL10 gl, int w, int h) {
               native_gl_resize(w, h);
       }

       @Override
       public void onDrawFrame(GL10 gl) {
               native_gl_render();
       }
}

The onSurfaceCreated method is not used, the onSurfaceChanged method is what the OpenGL implementation really uses to indicate a screen should be set up properly. The method onDrawFrame is called once per frame, at a rate of between 30-60 FPS (if you're lucky).

Now you can forget about Java until you need to do input, but that's another story, and write the rest of your game in C. The native_gl_resize method should grab a texture and set up the simplest rendering scenario it can. Experimentation has shown that this is not too shabby:


#define TEXTURE_WIDTH  512
#define TEXTURE_HEIGHT 256
#define MY_SCREEN_WIDTH  272
#define MY_SCREEN_HEIGHT 208
static int s_w;
static int s_h;
static GLuint s_texture;

void JNICALL native_gl_resize(JNIEnv *env, jclass clazz, jint w, jint h)
{
       glEnable(GL_TEXTURE_2D);
       glGenTextures(1, &s_texture);
       glBindTexture(GL_TEXTURE_2D, s_texture);
       glTexParameterf(GL_TEXTURE_2D,
                       GL_TEXTURE_MIN_FILTER, GL_LINEAR);
       glTexParameterf(GL_TEXTURE_2D,
                       GL_TEXTURE_MAG_FILTER, GL_LINEAR);
       glShadeModel(GL_FLAT);
       glColor4x(0x10000, 0x10000, 0x10000, 0x10000);
       int rect[4] = {0, MY_SCREEN_HEIGHT, MY_SCREEN_WIDTH, -MY_SCREEN_HEIGHT};
       glTexParameteriv(GL_TEXTURE_2D, GL_TEXTURE_CROP_RECT_OES, rect);
       glTexImage2D(GL_TEXTURE_2D,             /* target */
                       0,                      /* level */
                       GL_RGB,                 /* internal format */
                       TEXTURE_WIDTH,          /* width */
                       TEXTURE_HEIGHT,         /* height */
                       0,                      /* border */
                       GL_RGB,                 /* format */
                       GL_UNSIGNED_SHORT_5_6_5,/* type */
                       NULL);                  /* pixels */
       /* store the actual width of the screen */
       s_w = w;
       s_h = h;
}

You can also call glDisable to turn off fog, depth, and other 3D functions, but it doesn't seem to make too much difference. The glEnable(GL_TEXTURE_2D); call enables textures. You need this as you'll be drawing your pixels into a texture. glGenTextures and glBindTexture get a handle to a texture and set it as the currently used one. The 2 glTexParameterf calls are needed to make the texture actually show up on hardware. Cargo cult coding here. Without these the texture is just a white square. Similarly, the glShadeModel and glColor4x are needed to have any chance of your texture showing up either on hardware or on the emulator. Presumably if the screen has no color it is not drawn at all.

The rect[4] array and associated glTexParameteriv call will crop the texture to the rectangle size given. The MY_SCREEN_XX values depend on your "emulated" screen size, but should be smaller than the texture. The TEXTURE_XXX sizes should be power-of-2 (256, 512, 1024) to work on hardware. Anything else may work on the emulator, but will fail miserably on the real thing. The rectangle is inverted here to get the final texture to show the right way round. The call to glTexImage2D allocates the texture memory in video ram, passing NULL means nothing is copied there yet. The native Android pixel type is RGB565, which means 5 bits of red, 6 of green and 5 of blue. How close to the Nintendo DS or GBA pixel format, just 1 bit different! Using this colour type speeds ups the frame rate from less than 30 FPS to a more respectable 50-60 FPS.

Now the render code. This uses the glDrawTexiOES function call, which is an OpenGLES extension to render a texture straight to the screen. It is the fastest way to do things as there is no real 3D going on, it is just drawing your texture straight to screen.


void JNICALL native_gl_render(JNIEnv *env UNUSED, jclass clazz UNUSED)
{
       memset(s_pixels, 0, S_PIXELS_SIZE);
       render_pixels(s_pixels);
       glClear(GL_COLOR_BUFFER_BIT);
       glTexSubImage2D(GL_TEXTURE_2D,          /* target */
                       0,                      /* level */
                       0,                      /* xoffset */
                       0,                      /* yoffset */
                       MY_SCREEN_WIDTH,        /* width */
                       MY_SCREEN_HEIGHT,       /* height */
                       GL_RGB,                 /* format */
                       GL_UNSIGNED_SHORT_5_6_5, /* type */
                       s_pixels);              /* pixels */
       glDrawTexiOES(0, 0, 0, s_w, s_h);
       /* tell the other thread to carry on */
       pthread_cond_signal(&s_vsync_cond);
}

The memset clears out old pixel values. If you were careful and kept track of dirty areas, only refreshing those, this could be omitted. I'm keeping things simple here though and clearing the screen each time. The render_pixels routine does whatever it takes to draw your game's pixels into the s_pixels array in the RGB565 format. The glClear call is not strictly necessary, but it may help to speed up the pipeline as the hardware knows not to worry about keeping any old values. Experimentation shows that leaving it in doesn't harm the framerate at least. The glTexSubImage2D call will copy the s_pixels data into video memory, only updating the area indicated rather than the whole thing. If you do update the whole texture it is actually faster to call glTexImage2D. Finally, glDrawTexiOES will draw the texture to the screen, scaled to the screen size.

The final pthread_cond_signal is to tell our vsync call to wake up. I haven't mentioned this yet, but in order to have a GBA or DS-like coding experience, it is vital to wait on the screen refresh. The implementation of this is simple, as Android lets you play with all the usual pthread calls from the world of Linux. You create a mutex and condition at the start of the main, and have the implementation of waitvblank lock the mutex and wait for a signal on the pthread condition.

#define UNUSED  __attribute__((unused))

static void wait_vsync()
{
       pthread_mutex_lock(&s_vsync_mutex);
       pthread_cond_wait(&s_vsync_cond, &s_vsync_mutex);
       pthread_mutex_unlock(&s_vsync_mutex);
}

void JNICALL native_start(JNIEnv *env UNUSED, jclass clazz UNUSED)
{
       /* init conditions */
       pthread_cond_init(&s_vsync_cond, NULL);
       pthread_mutex_init(&s_vsync_mutex, NULL);

       while (1) {
               /* game code goes here */
               wait_vsync();
       }
}

That ensures that main loop can wait on screen redraws, which avoids tearing.

Obviously if you want to write code from scratch that will only ever run on Android, there is no point jumping through these hoops. Just use Java and forget about the NDK. However, if you want to port existing code to Android, or you don't want to write new code that is tied to a single platform, this approach is IMO the best way to go about it. It makes Android an almost decent platform for writing old-skool games on :-)

I've made a compilable example of this code available on github here.

A new era?

2010-10-01T23:45:00.001+02:00

TL;DL I've bought an Android phone and will probably stop writing DS stuff.

As you can probably guess by the lack of updates, I haven't been particularly active writing code for the DS during the last year. The main reason for this is a complete lack of free time lately. Or perhaps it's just bad time management. The second reason is my dodgy SD card randomly failing to connect via USB. It gets frustrating having to plug in and unplug the card multiple time just to get it to register. Either way, I think I've reached the end of DS development for me. Never say never, but I don't have the enthusiasm I once did for the platform.

Want some more reasons? How about Nintendo's action against flash cards in the UK? I don't doubt that Nintendo is going to try and extend the ban in the rest of Europe during the next year, until the release of the 3DS. Because of this, the number of people able to run programs will remain constant or even decrease.

Speaking of unauthorized copying... I've always known that many flash card owners are into downloading and playing games that they ought to have paid for, but this means that if you write for the DS you're competing with commercial games directly. If you can get Zelda for free, why bother trying out some free homebrew? In other words, the homebrew scene is too tightly coupled to the scene that likes downloading and sharing commercial games, with the negatives that this brings. You can see this when homebrew games are released for download in the RAR file format, shared on rapidshare-type sites, without source code, or in a way that violates the GPL.

Many users don't care about the nuances of running homebrew games, improving the state of things, or doing it the "right way". As an example, I have an RSS feed for alerts about Bunjalloo. I get tired of reading about torrents and rapidshare downloads for really old versions of it, especially when you could download the latest release Free from the googlecode site anyway!

I bought a DSi when it came out in Europe. I think it is a pretty good console. The freebies from the Nintendo store are top notch, and the DSiWare titles are not too shabby. Nevertheless Nintendo has not really moved with the times. If you compare the price of DSiWare titles to Android or iPhone paid applications, the differences are laughable. Eurogamer does a semi-regular review roundup of downloadable games where you can see the disparity. A DSiWare title typically costs 7-9€, whereas the iPhone equivalent is 2-4€. Sometimes the difference is 30€ to 4€ (Ace Attorney). Anyone can write an iPhone or Android title, but no system is in place that lets the general public write for Nintendo's platform. On the DSi we have 2 free titles. There are thousands of free games and applications on the major phone platforms. Granted a lot of them are crap, but hey! it's free crap.

Maybe I could live with this state of affairs if the DSi hardware were homebrew programmable, but it isn't. Dave Murphy continues his efforts to hack on the DSi, but it is fairly obvious that Nintendo want their hardware locked down. The latest exploit does look promising. It rewrites part of the DSi's wifi module flash chip to "jailbreak" into DSi mode from an exploitable DSi-only game. Sadly the error that permits this exploit has already been fixed on newer DSi models (including the DSiXL). With Nintendo dead set against us, why bother anymore? It's getting tedious. Outside of the excellent devkitARM, the development tools are not as good as in the GBA era. There's still no developer-friendly emulator on par with Visual Boy Advance.

Perhaps if there were more DSi-only cartridge titles around, the companies creating flash cards would have found a way to run games in DSi mode. As it is, there's no need. This highlights the sad symbiosis of the homebrew hackers and the folks into unauthorized copying. We need flashcards to run our homebrew, they need us to make their flashcards "legitimate" in the eyes of the law (and even that wasn't enough in the UK).

Where now then? I don't think the DSi situation is going to change. I've moved with the times (2009 times...) and have bought a 2nd hand Android phone to do embedded stuff. I'll miss writing for Nintendo hardware though. As far as 2D games go, the DS is superior in every way, Nintendo designed it for gaming after all.

On the DS you have layered backgrounds with multiple modes, hardware sprites, and vblank power saving between frames. Android has a generic screen buffer type display that is not ideal for writing old skool 2D games. You have to do pixel pushing yourself, and it sends the CPU skyrocketing, while the battery flatlines. The emulator spins at 100% CPU even on the samples from Google, which is not good.

Apart from that, coding for Android feels like cheating. The emulator is highly accurate, you have a visual debugger, can connect to the running platform and kill runaway processes, there's a CPU usage monitor, unit testing is part of the SDK, the API is huge and you can use all of the standard Java libraries... what's not to like? Well, Java is a bit tedious, but the NDK leaves the door open for writing good old C or even ARM assembly.

I was starting to get a bit tired of the DS scene. Seeing my own code up and running on the Android emulator has reignited my enthusiasm again. Let's see how long it lasts (I give it a week).

Wi-Fi on the Nintendo DS

2010-01-04T04:00:00.000+01:00

I've had a couple of people ask me about which Wi-Fi settings work with Bunjalloo. That means there are probably more folk out there who haven't asked but are not sure either. So here's the lowdown.

The original Nintendo DS or DS lite has 3 different ways to connect to a wireless network:

Connect to an unencrypted Wi-Fi access point
Connect to a Wi-Fi access point encrypted with WEP
Connect to a Nintendo USB Wi-Fi Connection

The Nintendo DSi adds an Advanced Settings with the following feature:

Connect to a Wi-Fi access point encrypted with WPA

Bunjalloo will only work with unencrypted or WEP encrypted Wi-Fi settings. It does not work with the DSi Advanced Settings options (WPA), nor does it work with the Nintendo USB Wi-Fi Connection.

These limitations are partly due to the dswifi library, which sgstair created by reverse engineering the original DS Wi-Fi hardware. It's remarkable that it works at all, let alone that it works so well while being easy to use. But the price of being unofficial is that it does not support all the hardware features that official titles do, notable it's missing support for the USB Wi-Fi device. The lack of DSi/WPA support is simply because applications booted from the current generation of DS flash cards cannot access any DSi features at all, Wi-Fi or otherwise.

As for the Nintendo USB Wi-Fi Connection... well, that's a bit of a white elephant really. The advantage is that you can use your network with WPA and still have your DS connect via the USB dongle that's plugged in to your regular PC. For me though having to turn on my PC to use the DS with Wi-Fi defeats the object of a low overhead, quick-to-boot-and-connect gizmo. The final straw is that the USB Wi-Fi program is only available for Windows.

On a regular DS system you need a Wi-Fi-enabled game in order to change the Wi-Fi settings. Mario Kart or Animal Crossing will do, for example. The DSi fixes this shortcoming by including a way to configure the Wi-Fi straight from its system menu. From the DSi's main menu you tap System Settings, then the Internet option. It looks something like this (if your screens are covered in lint, as mine appear to be here):

From the internet screen, tap Connection Settings then tap a connection to configure it:

This is more or less the same deal as on a regular DS, but built in to the DSi's firmware. You don't need to start a game to alter the settings.

Open access points show the signal strength and SSID name. Encrypted connections also show a padlock icon. Gold padlocks are for WEP encrypted access points, which you can use with regular DS games (if you have the WEP keys). If the padlock is grey though, well that means the access point uses WPA encryption and you will only be able to connect using the DSi's "advanced settings".

Here's what it says if you click on a grey padlock from the regular DSi Internet settings screen:

"Unable to connect to this access point using the current settings. To connect to this access point, use Advanced Setup"

For all you old-skool DS owners who might be curious, here is what the DSi says about the WPA connection. These are found in the "Advanced Settings" option of the DSi's Internet settings:

"Allows use of high-security proxy settings. Cannot be used with Nintendo DS-compatible functions."

In other words, you can't expect to use Mario Kart's online mode directly with your WPA protected network. This type of connection will only work with the DSi's built-in browser, the Nintendo DSiWare store, and other DSi-only titles.

So there you have it. The DSi's WPA Wi-Fi settings are not backwards compatible with regular DS titles, so they cannot be used by homebrew applications either. If you want to use the DS to its full potential, you will have to run an access point with the security set to "dubious" (WEP). It'll probably be enough to stop your neighbours accidentally nicking your broadband, but won't keep determined Wi-Fi thieves out. The official help for all this can be found on the Nintendo Customer Service site.

Unofficial 2009 in Review

2009-12-30T18:29:00.004+01:00

Intro

Shortly we will be entering the Space Year 2010, so here is my unofficial review of this last year from the point of view of a DS fan and part-time homebrew coder. Please don't feel offended if I've left out your game or application. There's been a lot to get through.

This roundup was partly done from my recollections of homebrew in the last 12 months, and partly from trawling through my Google Reader feeds. I haven't been able to try out all of the releases this year so let me know in the comments if you feel particularly hard done by.

While creating this post, PDRoms has been very useful. This site does a really great job of keeping the whole scene ticking over. Kojote has consistently posted interesting stuff, especially when you consider that he also covers a load of other portable platforms. Keep up the good work dude!

Other sites of note are Drunken Coders and NintendoMax (That's their DS RSS feed). Writing your own DS programs seems to be all the rage in France and NintendoMax's site has had the scoop on the latest news most of the year. We Hack DSi has also been an interesting read, though their posts are rather less frequent and tend to mix in some of the less salubrious aspects of the scene.

I've tried to link to the latest release for each game or application where I can. This way you don't end up downloading an initial, buggier release if there is a newer one available. This does mean that the text of the link ("Version 0.9" or whatever) doesn't always match up with the latest release on the target page, so keep that in mind if you are a history weeny.

January

The year began with news reports on the latest revision of the DS hardware, the DSi, which was already out in Japan. With 2 cameras and better wifi, it promised to be an interesting (if rather conservative) upgrade and everyone was looking forward to the releases in Europe and the US.

The homebrew scene started the year off with the arrival of the deadline for the Winter 2008 Compo. Nyarla's Snowride shoot 'em up came in first place, with the judges being particularly impressed by its graphics, music and Christmassy theme. It is a great game too, definitely download it if you haven't had the change to play it yet. Damn those penguins! It has the added advantage of being seasonal if you play it now. Bonus.

Around the middle of January Okiwi was officially abandoned, marking the end of the DS browser wars. Writing a web browser for a console with just 4MB of memory and a TCP library that is not as battle hardened as its big-brother PC counterparts is an exercise in frustration, so I know exactly where Pedro was coming from. I've abandoned Bunjalloo several times already, but keep coming back for more punishment.

Towards the end of the month the moonbooks project resurfaced after having the website hacked the previous year. In a strange homage to Geocities (perhaps) it still shows "under construction".

There were lots of homebrew releases in January. Some that stood out for me were:

Powder 110 - Jeff continued to improve his fantastic rogue-like throughout 2009, albeit at a slower rate than in previous years.

Moonshell 2.0 - several beta releases marked the return of the ubiquitous menu/launcher/mp3/video player

AemioDA - vector graphics emulator with an unpronouncable name, but it lets you play classics like Tempest, Asteroids and Lunar Lander. Lunar Lander sadly seems to ignore all my attempts to control the craft. Bummer.

Brix - ace physics game where you have to make towers topple over while carefully balanced sticks of dynamite land in the right place. Plays much better than my description, luckily.

maketens - mathsy fun with the numbers 1 to 5. An original puzzler that is fully playable, but lacks that last little bit of polish to make it a real classic. Still good though.

I did want to say DS1942 at this point too, which is an emulator for the arcade classic 1942. I have been completely unable to get this to run though, with the emulator complaining about a missing "srb-03.m3" despite it being right there. Ho hum. You might have more luck. It did mark the start of a 2009 trend; stand alone emulators for individual arcade games, or sets of games. More later.

February

No hardware news for February, but the NDS homebrew scene was in full swing. Several of the games released in January got more updates.

I have fond memories of playing through Tetrominout, which is an amusing cross between Tetris and Breakout. Certainly it was a surprising twist on the usual "hello world" titles that new game coders often cut their teeth on. Also out this month was a new version of the devkitARM toolchain and libraries.

Other homebrew of note in February were:

And's PDF reader, which bent the laws of computer science to squeeze the rather bloated PDF format onto the DS.

Glubies Planet. We don't often see 3D homebrew games on the DS, so this one stood out from the crowd. The puzzley gameplay isn't bad either.

March

March was a slow month, with only a few updates to already released programs including And's PDF reader.

QuirkDS wasn't really new but it did get a big update right at the end of March. It's a remake of the old Game Boy title Kwirk and plays a bit like Sokoban (a bit, not really much). I'm completely hopeless at it, so good job it has a level skip. If anyone knows how you're supposed to move the yellow blocks that appear from about level 3 onwards, then please let me know in the comments.

News began to trickle through of a great looking remake of the Commodore 64 shooter Warhawk.

April

The big news as we entered the second quarter of the year was the launch of the Nintendo DSi in Europe and the US. Almost right away we received confirmation of our worst fears: the current generation of flash devices didn't work on the new console!

It wasn't really that much of a shock of course, as Nintendo had threatened this already. In Japan there had already been confirmation of the bad news too. Still, it hit home when the console was finally out over here.

The Acekard had a working flash card that they had not-so-secretly been developing since the Japanese launch. Soon after this, R4 and EZ Flash released new flash cards that bypassed the DSi's stricter security. Homebrew was back on the cards.

Sadly all of these devices only gave (and still give) access only to the DS hardware, not the new DSi features. Although there would later be exploits to get small programs running in DSi mode, at the end of the year we are still in the dark when it comes to details about the DSi hardware from an unofficial developer's point of view. This means that there is still no solution for connecting to the net with the DSi's faster network hardware, nor can we use secure WPA Wifi connections, nor can we slurp out all those photos and all that data that is stored on the built in RAM.

Homebrew out this month included an update to @gentakojima's DSTwitter app. Pretty cool, even if the jury is still out on the usefulness of Twitter. You should follow me on twitter here, if that's your bag. I don't say much though, mostly just complain about Dr Who. Don't try and visit gentakojima's old Acdrtux site though, you just get Rick Roll'd.

DSPack added to the growing list of arcade emulators released this year, this time emulating Pac-Man and derivatives from the MAME collection. Finding ROMS for these emulators proved to be a bit trickier - not everyone has the, er, technical know-how to back up the collection of arcade games that they keep in their spare room. Yes. Ahem.

The emulation is excellent. That just leaves the limitations of the DS screen to contend with. It would've been nice if there had been an option to keep the sceen fixed in one spot, losing the top banner that takes up about a third of the top screen. As it is, not all of the screen fits on at once, and the play area scrolls to show Pac on the bottom screen at all times. I suppose a fixed screen would have meant the yellow one traversing the area in the middle of the 2 screens, which could have been quite confusing. Oh well. Good games, great emulator, worth checking out.

May

In May we got another update to devkitARM and its support libraries. There were no major additions this time round, but it was a solid maintenance release.

Powder had its Nelson release in May. And I suppose it was quite unlucky for us Powder fans; we wouldn't get another update until well into November.

June

After more than 6 months of silence, Wee Basic made a come back. This novel application lets you code in BASIC directly on your DS. Perhaps it is about as far from the cutting edge of computer science as you can get, but I found it a nostalgic amusement. You know what would make something like this even cooler? Auto-completion and some online help. You should see my .vim directory though, so perhaps my expectations are a bit far fetched.

Atmos was another cool-looking DS puzzle game that came out in June. The initial releases were multiplayer only, but the idea seemed original. The game could be described as a cross between othello and one of those colour matching games like Columns. Sadly the only record of its existence now is that page on PDRoms. Let this be a lesson for would-be coders: whatever you do, don't just post your game exclusively to some random forum! Especially if it requires log-in to download the title. You have probably spent a great deal of time writing your game, the least you can do is to take a few more minutes to create a more permanent home for your work on Google Sites or some other free web hosting service (yeah, like Geocities. Ha.)

Speaking of downloads, GameUp looks like a really great way to keep up to date with the latest DS homebrew. I've only discovered it recently, and it hasn't been updated since July, but the idea is a winner. It's a front end to a web site where you can download and rate homebrew titles, so no need to faff around with flash cards and swapping them between the PC and the DS. A bit like an unofficial "AppStore" application, I suppose. The only real problem is that it depends on the GameUp website being active and people updating the database with new homebrew. Sadly it seems that new programs have not been posted for quite some time.

July

The first rumours of DSi homebrew started to appear this month. It would be a while before more details were released, and all we got to see at this point was a video. Skeptics everywhere battled wits on the YouTube comments thread.

If we want to talking about games we could actually play, then the amazingly good Warhawk was finally released in July. Based on the C64 classic and coded entirely in ARM assembly, it was one of the best pieces of homebrew to come out all year. In fact for my money it was only topped by the group's second release later in the year...

The release of Line Wars DS added to the underrepresented 3D space game genre on the DS. Its author, Patrick Aalto, ported the game from the original x86 assembly code to C for the DS. He cheated by having the original source code, I bet ;-) The fact that he wrote that original code is not to be sneezed at, of course. Line Wars runs at a really smooth frame rate, making use of the DS's hardware 3D and lighting effects. Unlike Elite, in LW you take part in single "missions" that get progressively more difficult, with the focus here being on the action, the shootin' and the blastin'. Well worth the download either way.

August

One of the highlights of the month of August, at least for us DSi owners, was the release of a new system menu. This upgrade, version 1.4, added Facebook support to the built in camera program. By connecting to a local Wifi network and then entering your Facebook account details to the DSi, you could upload any of the really terrible quality photos you'd taken. The DSi browser was also updated to a new minor version. Though details on the exact changes are non-existent, I suspect one of the changes was to make logging in on web-sites a lot more difficult; the updated browser seems to forget cookies almost instantly.

However, this was not the most dramatic feature of System Menu 1.4. Oh no. What really caught the eye was the new "blocks all DSi flash cards" feature that nobody asked for. And so began the next step in the cat-and-mouse game that card manufacturers and Nintendo were playing. Thanks to good design and forethought most of these next-gen flash cards had a way to update the firmware. They managed to patch their way out of the 1.4 straight jacket within a few days, but it did look pretty bleak for a while back there.

Other news in August came from WinterMute, who released his hack for DSi Classic Word Games. This hack allows a small amount of assembly code to run on the DSi in actual DSi mode. It exploits an error in the save game handling of a commercial title, which means that its viability for running DSi code in a more mainstream way is a tad limited - you have to own Classic Word Games and the hardware to upload save games to the cartridge to make use of the exploit. Cool nonetheless.

Regular DS homebrew releases continued in August. There were quite a few demos and unfinished early releases of games, but also updates to some old favourites, including DronDS. DronDS is a Tron light-cycles game that takes place in full 3D (on a 2D plane of course, you can't start riding up the walls). In the latest releases you can even play online against other Dronners. I've never coincided with anyone, but then I'm antisocial. If you have friends on IRC or even on Twitter (hey look, a use for Twitter!) then maybe you could gather a posse and have a game.

September

From the makers of Warhawk, a sneak preview video of another remake "Manic Miner - The Lost Levels" had us pining for its release back in September.

Nano Lua caused a flurry of activity as well this month. Lua is a great little language (even if it does have its warts. 1-indexed arrays, I am looking at you) and Nano Lua enabled people to write games for their DSes without getting their hand dirty with C or C++. I've not had chance to check Nano Lua out in detail - it seems to be a bit lacking on documentation, natch - but people have released more things with it than with the previous DS-Lua efforts, so it must be doing something right.

October

The floodgates opened again in October. Blockman Gets, a pacman/puzzle game, combined 2 genres (classic arcade and puzzle) to produce an infuriatingly difficult mind bender. No ghosts on the first level, and yet it's more difficult than the original!

For novelty value, Mario Bros Lemmings DS was pretty hard to beat. These are Mario themed levels for Lemmings DS, complete with Koopers, Goombas, Bowser, and Mario, all rendered in their full 8-bit NES glory. I suspect some sort of automatic process was used, as the levels don't really stand out as brilliant Lemmings puzzles (too many ∞ abilities, some levels cause the game to hang, unbalanced requirements), but the amusement factor is high and all the SMB1 levels seem to be in there.

Now I'm not a big fan of Pang, but Pang DS does a good job of emulating the arcade game. Once more, make sure you only use carefully created backups of those Pang bootleg arcade boards you have next to the washing machine. Don't just Google "Pang MAME ROMs" or anything, okay?

GBA veteran coder FluBBa ported his SEGA Master System/Game Gear emulator over to the DS. It's called S8DS and is very good, as you might expect. The DS has a large enough screen that the squashed graphics from the GBA version are a thing of the past. S8DS uses the usual DLDI/libFAT features, so there's no need to inject the MS/GG games into the NDS ROM - you can just read them from your flash card. There are loads of great MS games out there, which of course you er, back up from the original cartridges you own (this is getting silly). I spent many a happy hour wasting batteries playing Fantastic Dizzy on the Game Gear.

I didn't see many people mention Red when it came out in October, but it is really good. Although it is a quasi-official conversion of a Flash game, it feels at home on the DS. Using the stylus, you control the defences of a base at the bottom of the screen. This base appears to be some sort of last stand against an onslaught of meteors on a red planet somewhere. Story be damned! The base is armed with a cannon, and by shooting balls that look like a deadly paper/spit combination at the oncoming rocks, you can deflect them away from the base. By charging up the cannon, you can shoot bigger wads at the meteors, giving you a better chance of deflecting them. There are some power-ups in there as well to mix things up a bit. It plays like a faster, more physicsy version of Missile Command. You have no excuses, go and play this one.

Last, but by no means least, is the epic Manic Miner in the Lost Levels. Created by the same chaps that created Warhawk earlier in the year, together with input from Amiga Power's Stuart Campbell, MMLL gathers all the best bits from the myriad of Manic Miner ports and packages it up with all the pazzazz of a first-party Nintendo game. You get first class graphics and sound, snazzy menus, unlockable bonuses, hidden levels, high scores, time attacks, historical notes, insider jokes, 80s film references, sly digs at the C64... it's all there. It is impossible to imagine how they could have done more justice to good ol' miner Willy. And I'm not even especially fond of the original Manic Miner! (ducks the pick-axe thrown from the back row), but the way it is presented here, with its skippable levels and modern day graphics make it much more accessible. All this without compromising the classic pixel-perfect, difficulty-turned-up-to-11 gameplay. In a word: "stunning".

November

Woopsi had many updates throughout the year, and it's good to see Ant updating his GUI library. I still haven't completely ruled out a port to the Woopsi library for Bunjalloo - maybe in 2010? For me all it's really lacking is some way to support UTF-8, which is a tricky problem when you don't want to support STL collections.

Another teaser was released from the team behind the Manic Miner remake in November. This time they are were going to have a crack at The Detective. I'll admit now that I had not heard of the game before, and that despite being an ex-Commodore 64 owner. Still, with their pedigree even if the original was pants I'm looking forward to this one.

From the creator of retro-platform/puzzler Platdude, Munky Blocks also came out in November. I would describe the gameplay as somewhere between Sokoban and Columns, with a bit of Yoshi thrown in. Though the gameplay is simple to grasp and has only a few concepts, the levels start to get fiendish quite quickly. Viewed in side-on 2D, you control a Monkey/Cat thing that can swallow blocks and climb up platforms. The idea is to place like-coloured blocks next to one another in groups of 3 or more so that they disappear. When you have swallowed a block, you can then walk around with it in your gut, but you can only climb up a single floor level when you're this full. After regurgitating the block you can move freely again. Later levels mix things up with switches that open doors, keys you can collect to unlock other doors, and so on. A very good game, and strangely claustrophobic I found.

In other, non-playable news, the first DSi-mode game was announced, although it is only in video form. Perhaps 2010 will see this taken to the next level? Let's hope so.

A new emulator was announced by Patrick Aalto (of Line Wars fame) back at the start of November. This time the emulation target was the x86! Specifically, the 80286 with MCGA graphics and SoundBlaster audio. Quite an achievement. Full details of DSx86 are on Patrick's site. The current tech demo includes the DOS version of Line Wars, though his blog also shows screenshots of other programs and games running, including SYSINFO and Paratrooper. Hopefully 2010 will see more improvements and news.

December

I'm writing this in December. Past tense confusion overload.

Another great puzzle game appeared at the start of December, called Cogito. This has nothing to do with version control, but is a tricky sliding block puzzle game. It is somewhere between a Rubik's cube and those plastic sliding block puzzles. You have to align rows of coloured blocks on the bottom screen in the way that they are shown on the top screen. The trick is that you can only move entire rows at once. It is simple, but much more complicated to do than you first expect. Another great game.

Red Temple probably came out at the end of November, but it got renamed and had a new release right at the start of December too, so here it is. Besides, it's my blog so I'll put it where I like :-) Now I really quite liked Red Temple. The game starts off with a few effects on the start screen, then you are taken away to the world overview screen to select your start level. Once chosen, the game proper starts off. It is a straight forward collect-'em-up where you play a snake/worm thing that goes around eating a load of fruit. As snakes and worms often do, I imagine.

The art work is a colourful enough tile-based 2D affair (described rather unfairly as "crap" by the author, they aren't that bad! At least they are original, which counts for a lot IMO), and the sounds are made up of some quality coder-generated-sounds and grunts. At least that's what it sounded like - the "level win" and "you're dead" sounds being especially amusing. It has a nice Amiga PD game feel to it, the sounds reminding me of the credits files that occasionally described how the authors went about making each sound ("falling in water - dropping an orange in a mug of coffee", that sort of thing). Not bad, not bad at all. There are a few videos on youtube if you don't fancy downloading the game to try it. But you should.

The great-looking Ripholes in Rubbish was a surprise release. It came out of nowhere pretty much completed. The game is a simple platform game, with an added twist that on some screens you can interact with the background. The "rip" in the name comes from this idea - you can rip holes in the background and go through to the other side. The later levels use this technique as part of the puzzles to get past otherwise impossible sections. Other puzzles involve capturing and moving clouds. The graphics use a distinctive hand-drawn look and the sound is varied and original. Sadly the game has a few bugs that cause it to either hang or do strange things, such as all objects on levels disappearing suddenly. Still, there have been further updates improving the stability, and as it is the game is a playable work of art.

The last big release of the year was headsoft's The Detective Game. I'd not even heard of the original game, so this was something of an unknown. The graphics are great (Cook's bouncing chest aside) and the game is classic 8-bit "WTF do I have to do?!". It is an admirable remake. Sadly without the heritage of Manic Miner it doesn't have that elusive wow-factor, as they say on TV. I must say that Ben (AKA headcaze) has been great in fixing a few glitches that made the game unplayable on the EZ Flash cartridges, so extra kudos there.

Outro

Barring any last minute surprises, my game of the year award would go to Manic Miner in the Lost Levels. Not an easy choice, there have been a lot of fine chunks of code running on my DS this year.

I hope the 2010 has as good a crop of games as this year has had. I know a lot of DS coders bemoan the death of the system, saying that everyone has moved on to smart phones, but I'm sure the ol' DS still has a few years left in her yet. Happy holidays, and happy homebrewing!

Bunjalloo release v0.8

2009-12-23T22:16:00.000+01:00

I've just uploaded a new release of Bunjalloo. This contains all the performance improvements I've been working on recently as well as a few other changes. Full details are in the changelog.

One of the changes is a fix to the updater. Sadly I broke it in the last release. This means you will have to update manually by downloading the zip file and unpacking it to your flash card. I've added some automated tests during this release to make sure I don't mess it up again.

You can download the latest version here.

Setting environment variables in Java

2009-11-19T19:00:00.000+01:00

Have you ever noticed that the Java System class doesn't let you set environment variables? You can retrieve them using getenv() but there is no equivalent setenv() function.

First off, what is the environment? The manual entry for environ(7) describes the environment as:

an array of strings [that] have the form name=value. Common examples are USER (the name of the logged-in user), HOME (A user's login directory) and PATH (the sequence of directory prefixes that many programs use to search for a file known by an incomplete pathname).

It turns out that when you start up the JVM, it copies this environment into its own Map of Strings. The actual container it uses is an unmodifiable map, probably to be extra safe.

So in a running Java application we have 2 environments: the JVM copy that you can read via System.getenv() and the underlying environment that lives in the C library.

If we want to change the JVM's copy, we can do so using reflection. Or at least we should be able to, just as long as our code is not running in a sandbox. In that case you'd be right out of luck. Anyway, the code to fetch a modifiable copy of the environment could look like this:

import java.lang.reflect.Field;
import java.util.Map;
import java.util.HashMap;
public class Environment {
  @SuppressWarnings("unchecked")
  public static Map<String, String> getenv() {
    try {
      Map<String, String> unomdifiable = System.getenv();
      Class<?> cu = unomdifiable.getClass();
      Field m = cu.getDeclaredField("m");
      m.setAccessible(true);
      return (Map<String, String>)m.get(unomdifiable);
    }
    catch (Exception e) { }
    return new HashMap<String, String>();
  }
}

Calling System.getenv() returns us an UnmodifiableMap of Strings containing the copy of the C environment. In order to get to its squishy modifiable heart, we access the m member variable which is modifiable. This is because UnmodifiableMap uses the proxy pattern - it implements the Map interface, but delegates all calls for retrieving values off to a member variable Map that contains the values and does all the real work. The set methods throw UnsupportedOperation exceptions. By accessing that m we can change the JVM's copy of the environment from under its very nose. Heh heh.

Not so fast. On Windows this is slightly different. The environment there contains a different Map that allows you to search for variables in a case-insensitive way. System.getenv() returns one Map (the unmodifiable one, same as Linux) whereas System.getenv("value") looks up the value in the case-insensitive Map. In order to create a robust update-env implementation we should update both of these Maps. This needs some more reflection to get the Map in question out of the ProcessEnvironment, probably like this:

@SuppressWarnings("unchecked")
public static Map<String, String> getwinenv() {
  try {
    Class<?> sc = Class.forName("java.lang.ProcessEnvironment");
    Field caseinsensitive =
      sc.getDeclaredField("theCaseInsensitiveEnvironment");
    caseinsensitive.setAccessible(true);
    return (Map<String, String>)caseinsensitive.get(null);
  }
  catch (Exception e) {
  }
  return new HashMap<String, String>();
}

An example of how this may be useful without going any further would be if you have the DISPLAY environment variable set but you cannot use it (for example you connect via ssh -X, start a background process, then disconnect closing the X session connection too). In this situation even if you tell java.awt to run headless it may see that DISPLAY is set, try to use it and throw an exception. By clearing DISPLAY we can use headless methods to create off-screen graphics, to send them to a printer or to save a fake screen shot to file maybe for automated testing.

However this is not enough to affect the environment for any child processes. They will still only see the original, unmodified environment that the JVM craftily made a copy of at start-up. The comment in java.lang.ProcessEnvironment says it all:

// We cache the C environment.  This means that subsequent calls
// to putenv/setenv from C will not be visible from Java code.

Grrrr! By far the easiest approach here is to just use the ProcessBuilder. This lets you change the environment before launching the child process. Win. End of post.

No! I really want to change the underlying C environment for no good reason!

In order to change that we have to resort to either JNI or JNA. Lets start with JNA, it is the easier of the two and needs less tools. Just download a jar file and with the Java Compiler you're good to hack.

In case you have not heard of it, JNA is to Java what ctypes is to Python - a byte-code library that lets you open native, compiled, shared libraries and call the C routines within directly from Java code. JNI on the other hand requires you to create C++ functions, compile these, then call them from Java.

Armed with the JNA classes we can wrap the standard setenv() and unsetenv() functions from the C library.

import com.sun.jna.Library;
import com.sun.jna.Native;
public class Environment {
  public interface LibC extends Library {
    public int setenv(String name, String value, int overwrite);
    public int unsetenv(String name);
  }
  static LibC libc = (LibC) Native.loadLibrary("c", LibC.class);
}

That works fine on Linux, but on Windows we have to take a different approach. There neither setenv nor unsetenv exist, instead we have to call _putenv. This function accepts a "name=value" string, and if we pass "name=" we can delete a variable from the environment. Unfortunately this multi-platform approach messes up the code a fair amount. Here is one way to do it:

// Inside the Environment class...
public interface WinLibC extends Library {
  public int _putenv(String name);
}
public interface LinuxLibC extends Library {
  public int setenv(String name, String value, int overwrite);
  public int unsetenv(String name);
}
static public class POSIX {
  static Object libc;
  static {
    if (System.getProperty("os.name").equals("Linux")) {
      libc = Native.loadLibrary("c", LinuxLibC.class);
    } else {
      libc = Native.loadLibrary("msvcrt", WinLibC.class);
    }
  }

  public int setenv(String name, String value, int overwrite) {
    if (libc instanceof LinuxLibC) {
      return ((LinuxLibC)libc).setenv(name, value, overwrite);
    }
    else {
      return ((WinLibC)libc)._putenv(name + "=" + value);
    }
  }

  public int unsetenv(String name) {
    if (libc instanceof LinuxLibC) {
      return ((LinuxLibC)libc).unsetenv(name);
    }
    else {
      return ((WinLibC)libc)._putenv(name + "=");
    }
  }
}

static POSIX libc = new POSIX();

Here we use JNA to load either libc on Linux or the msvcrt DLL (which contains _putenv) on Windows. There are ugly casts in there, and other OSes are left as an exercise to the reader, but this means that I can call POSIX.setenv() or unsetenv() and have it work.

To complete the picture, the JNI equivalent of this would be:

public class Environment {
  public static class LibC {
    public native int setenv(String name, String value, int overwrite);
    public native int unsetenv(String name);
    LibC() {
      System.loadLibrary("Environment_LibC");
    }
  }
  static LibC libc = new LibC();
}

The call to System.loadLibrary() loads a dynamic/shared library. On Linux it looks for "libEnvironment_LibC.so" and on Windows "Environment_LibC.dll". The implementation of those native calls could be like this C++ code:

#include "Environment_LibC.h"
#include <stdlib.h>
#ifdef WINDOWS
#include <string>
#endif

struct JavaString
{
  JavaString(JNIEnv *env, jstring val):
    m_env(env),
    m_val(val),
    m_ptr(env->GetStringUTFChars(val, 0)) {}

  ~JavaString() {
    m_env->ReleaseStringUTFChars(m_val, m_ptr);
  }

  operator const char*() const {
    return m_ptr;
  }

  JNIEnv *m_env;
  jstring &m_val;
  const char *m_ptr;
};

JNIEXPORT jint JNICALL Java_Environment_00024LibC_setenv
(JNIEnv *env, jobject obj, jstring name, jstring value, jint overwrite)
{
  JavaString namep(env, name);
  JavaString valuep(env, value);
#ifdef WINDOWS
  std::string s(namep);
  s += "=";
  s += valuep;
  int res = _putenv(s.c_str());
#else
  int res = setenv(namep, valuep, overwrite);
#endif
  return res;
}

JNIEXPORT jint JNICALL Java_Environment_00024LibC_unsetenv
(JNIEnv *env, jobject obj, jstring name)
{
  JavaString namep(env, name);
#ifdef WINDOWS
  std::string s(namep);
  s += "=";
  int res = _putenv(s.c_str());
#else
  int res = unsetenv(namep);
#endif
  return res;
}

You generate the header files using javah - that gives you the strange function names needed - and compile the code as C++ to produce a shared library:

javac Environment.java
javah Environment
g++ -shared Environment_LibC.cc -o libEnvironment_LibC.so -I$(JAVA_HOME)/include/linux -I$(JAVA_HOME)/include/

This library needs to be in one of the usual places to work - somewhere where it can be found by dlopen(). So either in /usr/lib, a directory where ldconfig looks, or in one of the paths in LD_LIBRARY_PATH. This depends on the OS you are using. For Windows, Solaris or Mac you need a whole different set of flags and incantations. You can see why I lean towards JNA, even though it has its own problems. For the record this is how to compile the JNI library on Windows using MinGW:

g++ -DWINDOWS -Wl,--kill-at -shared Environment_LibC.cc -o Environment_LibC.dll -I$(JAVA_HOME)/include/win32 -I$(JAVA_HOME)/include

That --kill-at switch is a real gotcha. Without it the function symbol that the MinGW compiler produces is not the one that the JVM was expecting. On Windows the library itself must be in the current directory, or in one of the directories listed in the PATH variable.

As you can see we repeat the whole setenv-and-unsetenv-do-not-exist dance and use _putenv() for both. Here I fudge it with a bit of ifdeffing. Meh.

Now that we have a way to call the C library's setenv() and unsetenv() (or equivalent), let's wrap it all up. Here are the final setenv() and unsetenv() functions that update the C environment and the Java one too:

// inside the Environment class...
public static int unsetenv(String name) {
  Map<String, String> map = getenv();
  map.remove(name);
  Map<String, String> env2 = getwinenv();
  env2.remove(name);
  return libc.unsetenv(name);
}

public static int setenv(String name, String value, boolean overwrite) {
  if (name.lastIndexOf("=") != -1) {
    throw new IllegalArgumentException(
        "Environment variable cannot contain '='");
  }
  Map<String, String> map = getenv();
  boolean contains = map.containsKey(name);
  if (!contains || overwrite) {
    map.put(name, value);
    Map<String, String> env2 = getwinenv();
    env2.put(name, value);
  }
  return libc.setenv(name, value, overwrite?1:0);
}

Curiously enough ProcessEnvironment is wired so as to validate the values that you add to the "unmodifiable" Map, but the case insensitive equivalent on Windows is not validated. If you try and add an invalid environment variable, such as one with a name that contains =, only the unmodifiable map will throw an IllegalArgumentException. This makes it fairly robust as the nasty name doesn't trickle down to the underlying C environment, but for Windows we have to do an extra check manually.

I've uploaded a tarball with all the files mentioned on here together with the dependencies to my GBA Remakes site. So now you've no excuse to not go off setting environment variables like mad.

Of course you shouldn't really do any of this. This post was 60% "if you really need to", 40% "might be useful". ProcessBuilder is the way to go for changing the environment in child processes.

The only thing that you might want to do is change the environment in a running Java process, but even then it is probably easier to create a wrapper script or batch file that launches your program and fiddle the environment prior to launching the JVM. Using reflection to access all those inner member variables is pretty flaky - if they change their name in some future version, your code will stop working. Happy hacking :-)

Optimisations

2009-11-11T23:22:00.003+01:00

If you have been reading the changes that I've been pushing out to Bunjalloo lately, you will have noticed that there are quite a few that are aimed at optimising parts of the code. There was a lot of low-hanging fruit in there and the rendering of some 200+ comment threads on reddit was starting to annoy me.

There's an old rule when it comes to optimising your code; don't. 2nd verse, same as the first. The 3rd rule of optimisation is to profile your code first. So that's what I did.

The good old way to see where the bottle necks are is to use gprof. You compile with profiling on by passing a couple of flags to gcc, run the program and it spits out a profile upon exit. You then run the gprof command line tool and that interprets the profile to give you a table of hot spots. From commit 5beaffd581ed it looked something like this:

Each sample counts as 0.01 seconds.
%   cumulative   self              self     total
time   seconds   seconds    calls  ms/call  ms/call  name
10.00      0.02     0.02   103228     0.00     0.00  Font::valueToIndex(unsigned int) const
10.00      0.04     0.02    68648     0.00     0.00  std::_List_iterator<HtmlElement*>::operator--()
7.50      0.06     0.01   701479     0.00     0.00  std::less<unsigned int>::operator()(unsigned int const&, unsigned int const&) const

Right there you can see that Font::valueToIndex was taking a lot of time, but I already knew that that method could be improved. The other C++ STL calls were trickier to track down. I use all sorts of STL containers, so the problem could have been anywhere...

A couple of years ago Google released their performance analysis tools. One of the advantages google-perftools' CPU profiler has over gprof is that the output shows function calls line-by-line in a nifty graphical representation. Running the profiler on the same code gave this output:

Here is the original full-size image

From there, I could trace the call graph back from the "slow" STL call to my code see that the use of a std::set was causing problems. It was overkill anyway and a simple change improved things there.

The different output in the perftools analysis is probably due to the way it works compared to gprof. With gprof, you get a call to the profiling library added to each of your program's functions, which means you need to recompile all of your code. With the Google tools you don't need to recompile your code, you just have to link in the profiler library to your executable. It then hooks in to the start of the program and uses timers and signals to instrument the code, adding traces at the current point of execution. The result is that code that takes longer has more "hits", but the result is not exactly the same as gprof's output.

This is why being able to compile NDS code on the PC is a really good idea. These things are not impossible on the DS of course - counting the hblanks your slow function takes by changing the background color is a classic technique - but in general running unit tests, debugging, and doing performance analysis are much tougher to do on an embedded device.

Enough waffle. All this means that the next version of Bunjalloo may be a bit nippier, hopefully.

NDS thumbnail preview in Nautilus

2009-11-06T22:23:00.005+01:00

When you insert a cartridge into a Nintendo DS, the start menu shows a little image or logo of the game. Homebrew games for the DS can also add their own logos and these show up when you view them in Moonshell or whatever menu your cartridge happens to use.

Ages ago I wrote a little script to show these images as the preview icon for NDS files in the Gnome file browser Nautilus. The script is about a hundred lines of Python and it's available here.

The setup file in that download tarball associates the icon-extracting command to the MIME-type of the Nintendo DS file. If you use an older distro then you may have to associate the script with the generic application/octet-stream MIME type. The script filters by file name anyway. This association is not done using the usual Nautilus scripting method, where you add a script to ~/.gnome2/nautilus-scripts/, but rather using gconf variables.

If you need to uninstall the icon preview for some reason, you can run:

gconftool-2 -u /desktop/gnome/thumbnailers/application@x-nintendo-ds-rom

Oh, and while we're here I recommend you go and play the absolutely brilliant nostalgia-fest that is Manic Miner in the Lost Levels.

Game Boy Remakes moves home

2009-08-16T16:40:00.008+02:00

Geocities has been around in one form or another since about 1995, but Yahoo has recently announced that the web hosting service will close this October. With this in mind, I've moved all my Game Boy Advance remakes and BBC emulation to a new home on Google Sites.

Do you remember when people said "don't forget to update your bookmarks!" when they changed their website address? Now nobody does that because everyone uses a search engine to find what they need. That means that when Geocities closes, all the search-juice for the old site will be lost. So here I'm sowing a little search seed, which will hopefully grow and let people find these remakes when Yahoo finally pulls the plug. BTW, closing down Geocities seems a really odd thing to do. The limits on bandwidth were tiny (4MB per hour!), the sites are full to the brim with adverts. It must have been costing Yahoo nothing to host all of them.

Anyway the migration was completely manual, but hopefully I haven't missed anything and the new web site should have all of the old content on it.

As I moved all of the content over, I was reading through a lot of the stuff I had up on the old site. I really miss remaking these old games, I'd love to get back into that again one more time. Reading through old assembly code figuring out the algorithms used and rewriting them in C is pretty good fun. And at the end you get a game to play too. I have a few ideas for a remake, but I don't want to promise anything. I know from experience that announcing something that is a work in progress, or even less than that, can suck the life out of the effort.

Migration to Mercurial

2009-07-20T20:00:00.001+02:00

Image by ms sdb via Flickr

I've taken the plunge and decided to integrate all of the Bunjalloo code onto the Google Code site. This meant I migrated from the original Github-hosted Git repository to make use of the new Mercurial support that GC added not so long ago.

The main reason I migrated was to get all the issue tracking, wiki and code changes on to one site. I really had 2 choices: migrate the main site to Github and use the issues and wiki on there, or migrate the code to Google Code and use Mercurial. I quite like Github and I think it is amazing that there are so many brilliant free hosting services, but I really prefer the Google Code interface from a user's point of view. It is generally less cluttered.

So what does this change get us? First off, we lose github's famous "social coding" features. I had one fork in 2 years, with 0 additional commits. No great loss there then. You can still email patches to the discussion list of course, and of course with either DVCS the author information is retained. I gain the fact that I can now reference changes a bit easier in bug reports and close tickets straight from commit messages. Assuming I even write any more code or fix any bugs ;-)

I had a couple of pet peeves with Mercurial, mostly that I really missed gitk's features. Luckily I discovered what I assume everyone else must use: Tortoise HG. This provides "hgtk log" on Linux that works a lot better than the "hg view" history viewer, which is a port of a really ancient version of gitk. The most important feature that hgtk has is a refresh button so I don't have to keep killing and restarting the application each time I make some changes.

Some other areas that I saw as weaknesses were to do with Mercurial's history fiddling, or lack thereof. However I decided that I'm probably better off not messing with history too much anyway. Lately I've tended to avoid doing that in Git and I get more done, even if the revision log is not as clean as it could be. I finally understood the way to do this using Mercurial Queues anyway, even if it is a bit more fiddly.

My final missing feature was the "commit -v" feature, which shows you the patch of the commit that you are making without having to open up a separate console. This hasn't been fixed, but I've worked around it by writing a Vim script to do something similar. Pressing "K" shows the diff of the current tree in a new buffer. This actually works out pretty well as I can see a patch and write a comment at the same time, rather than having to jump back to the top as I did with the "commit -v" thing.

To do the actual Git to Mercurial conversion I used the "hg convert" extension that is included with core Mercurial. That worked flawlessly and made switching really easy. The conversion guide on the Google Code support site has detailed steps on what to do when converting from Subversion, but I'll describe a few gotchas that I found with the Git to Hg transition.

The recently released Mercurial 1.3 included a tiny patch that I wrote to generate a slightly nicer log for git conversions, so I used that version. You see Git tracks both the author of a patch and the person who made the actual commit, but Mercurial only tracks the "user". The user is equivalent to Git's "committer" by default, while author information is assumed to be the same dude. When you ran hg convert, it added a line like "committer: A.Committer " to the Mercurial log message for every commit, even if the author and committer were one and the same in the original git repository. It looked a bit silly when 99.9% of the commits were my own. So my patch made sure that the "committer" line only got added if the author of the original change was not the same as the commiter of the change.

Another interesting niggle: Git has 2 different types of tag; lightweight and annotated. Annotated tags can also be gpg signed, but that wasn't the case in my repository. The difference between lightweight and annotated tags is pretty subtle. As far as I understand it, a lightweight tag is simply a reference to a commit ID, while an annotated tag also has its own blob in the Git database.

By default "git tag mytag HEAD" will create a tag of the lightweight variety. This is apparently the Wrong Thing To Do and one of the few places that Git's default behaviour is not the best option. You really should pass the "-a" option to create an annotated tag. Suffice to say that I used the default lightweight type of tags for quite a while until I discovered my mistake. The "hg convert" extension doesn't convert lightweight tags at all, it only converts the chunky annotated kind. This is possibly by design (maybe by misunderstanding?) as it would be easy to fix the convert extension to convert either type of tag.

The easiest workaround for me was to just convert my git lightweight tags to annotated tags in the source git repository using the "--force" option to overwrite the old ones. The convert process picked these up and converted them over correctly. Interestingly enough Justin Williams had posted about a similar problem and his timing was perfect to ask it over on StackOverflow.com.

Now that I've used DVCS a bit more and the novelty of branching has worn off, I decided that I wanted the minimal number of heads in my new Mercurial repository. I also wanted to maintain as much of the released history as possible. Luckily the history was mostly linear. I did create a couple of branches for maintenance releases of Bunjalloo early on, but after about version 0.4 I just made releases from the trunk.

Originally the repository was in Subversion and I pulled in the tag branches with git-svn too. This lead to a few branch stubs with a single commit ("creating tag blah") with a corresponding git tag that I must have created at some point later on. I used the Mercurial Queues extension to trim these out of the history where applicable so that the final repo has just 2 heads - the main trunk and an old, closed maintenance branch from the 0.3 days.

Oh, when you install Mercurial from source on Ubuntu (possibly on any Debian derivative?) it rather inexplicably creates an /etc/mercurial/hgrc file that enables all of the extensions. This lead me to (re)discover a bug with the inotify extension when used in conjunction with Mercurial Queues. My solution was to simply disable the inotify extension (in fact just removing the /etc/mercurial directory and enabling what you need in $HOME/.hgrc is a better idea overall).

Anyway, feel free to check out the code and send me your patches to fix all of those open issues! :-)

Bunjalloo 0.7.5

2009-07-08T22:30:00.005+02:00

Image by beneneuman via Flickr

I've just put up yet another new version of Bunjalloo. This one fixes a load of bugs that caused lots of top pages to be broken. In particular you can log in to GMail again. Yay!

Changes in 0.7.5:

Improvements to caching - logging in to GMail works again
Clicking preference icon goes straight to preferences
Fix encoding problems that caused crashes
Fixed lots of non-ascii character keyboard bugs
Fix configuration changes that use escapable % characters

You may have to manually fix the download path in your configuration settings. This is because the download path could have become messed up and show all path separators as %2F instead of /.

In the next release I want to fix cookies so that you don't have to enter your name and password into all the sites that you log in to. I changed the password on my google account from something like "password" to a good strong one and it's a pain typing it in all the time on the DS ;-)

Creative Zen Mozaic on Ubuntu 9.04 Jaunty

2009-07-06T23:24:00.011+02:00

Image by ♪ Sleeping Sun ♪ via Flickr

I've just been given a Creative Zen Mozaic portable audio player for my birthday (it's a bit early, but I'm not complaining!). I've had a Zen Micro for a few years now and it's a pretty decent gizmo. Small enough to fit in my pocket, has stood up to several falls, has a UI that gets out of the way and it works with Gnomad2.

Update for Ubuntu 9.10: The Mozaic now runs out of the box, no changes needed. I'll leave this here for posterity.

After plugging the new Mozaic in to my laptop, which runs the latest release of Ubuntu (9.04 Jaunty Jackalope at the time of writing) I discovered that horror! it wasn't recognised by Gnomad2. It seems like it's a common problem. The solution is sort of linked to on that bug report, but the help is in French and the file you have to change has moved since that was written. You couldn't make up a better "open source documentation sucks" anecdote even if you tried.

So.. using your favourite editor open up the file /lib/udev/rules.d/45-libmtp8.rules and add the magic lines:

# Creative ZEN Mozaic
ATTR{idVendor}=="041e", ATTR{idProduct}=="4161", SYMLINK+="libmtp-%k", MODE="660", GROUP="audio"

You'll have to do that with super powers, so use something like this to open the file with the correct permissions using gedit:

gksu gedit /lib/udev/rules.d/45-libmtp8.rules

Add the lines right before the first "#Creative ZEN" line that is already in there, just to be on the safe side. This can be derived from first priciples actually, because if you run lsusb it'll say something like:

Bus 001 Device 004: ID 041e:4161 Creative Technology, Ltd

Which ties in pretty nicely to the line you have to add to the 45-libmtp8.rules file.

Anyway, after adding that unplug the Mozaic and plug it back in again. That forces udev to reread the settings you have just changed. Start Gnomad2 and hopefully it will recognise your player and list all the tracks on it. Yipee!

Like buses

2009-07-02T22:01:00.003+02:00

What a disaster! You wait ages for a release of Bunjalloo, then 2 arrive in 2 days. Let me tell you how this happened.

I use version control for all of the main code. All my own code, that is. I also have a big bunch of downloadable 3rd party libraries that Bunjalloo needs. The build system only deals with my own code, it doesn't handle the 3rd party libraries that I install to $DEVKITPRO/libnds and assume are pretty much fixed. So stuff like libpng, jpeg, zlib, the matrix ssl library, all of these have to be downloaded, compiled and installed separately. I have a few patches that I've applied on top of these libraries too. All of these steps are handled by a script that downloads the source files, patches them if needed, compiles and installs. These patches are changes that would either never be accepted in the core library - removing printfs for example, or changing the Makefiles for the DS - or in the case of matrix, the upstream authors don't seem to have any real "community" or way to send them patches.

These libraries are all relatively stable, and when I upgrade devkitPro I can just run my script to install the latest versions. I've also put a tarball of the compiled code on the Google Code site to make your life easier if you want to compile Bunjalloo from code.

There are some libraries here that I've not yet mentioned and that I mostly take "as is". These are the core devkitPro libraries for the NDS - libnds, libfat, and friends. But! I have also been patching dswifi since about Bunjalloo v0.7 to fix an issue with non blocking sockets that just can't be worked around.

Thanks to an oversight I didn't apply the patch when I installed the latest dswifi 0.3.9, which meant that the sockets are not dealt with properly. I assume a socket has connected before it really has, and start shoving data in before it's ready. This has the result that after a few page loads sockets just stop connecting, presumably because I've filled up some buffers with junk. Pages no longer load and the whole thing grinds to a halt.

The solution? Short term, I've patched dswifi again and uploaded v0.7.4. Longer term, I've added a bug report to devkitpro on sourceforge. Hopefully my patch can get integrated and I don't have to apply the changes manually each time I upgrade the DS toolchain. I should have done this a while ago really.

Bunjalloo 0.7.3 released

2009-07-01T22:02:00.002+02:00

I've released a new version of Bunjalloo on the Google Code site. This is a fairly minor release that mostly updates the code for devkitARM r26 and the various library updates. Full details are in the ChangeLog over on the wiki.

So it's a minor release, but Bunjalloo itself has gone through quite a lot since 0.7.2 came out in December. Development of new features went on hiatus while I tried to incorporate the Woopsi GUI library. After that failed to really work - mostly down to design decisions on both parts; Woopsi uses the DS hardware for really fast scrolling whereas I want flicker free double buffering - I started to go down this crazy architecture astronomy path that went nowhere either.

Now I want to get back to basics. I want to start releasing versions a bit more regularly. I'm going to try getting a version out every month from now on, even if it is crap! Let's see how long I can last. You may notice the new banner logo that can be seen when using moonshell and other menu programs. That is thanks to Sam and I'd like to get some more community contributions. The only way for that to happen is if Bunjalloo is alive and kicking, and I think regular releases is the best way for that.

EZ Flash Vi and libnds 1.3.6

2009-06-28T19:51:00.002+02:00

The version of libnds and the default ARM7 core that was released as part of devkitpro last week (24 June 2009) has a problem with the NDS touch screen controls on the EZ Flash Vi cartridge.

The symptom is seen best if you run the nds-example "touch_test.nds" from the devkitpro distribution. The keys shows 00002000 in the top right, as if you were holding down the stylus, but in fact no touches are registered. Until an official fix is released by the ezflash coders, the workaround is to use devkitpro's own homebrew menu application. It's GPL code so there's no real conspiracy going on here! In fact it'd be nice to see this menu become the defacto standard in the homebrew world, but it is very rudimentary and only shows a textual listing of the files on your card.

So on to the instructions to get the latest homebrew working. I'll assume that right now you've downloaded and installed all the latest devkitPro stuff and you're sat there wondering why your touch screen controls don't work.

First download the latest EZ Flash DLDI driver from http://www.ezflash.cn/software.htm. You'll have to download the "Kernel 1.90" zip file, extract it and find the Ez5s.dldi file. Now you have to download the Homebrew Menu code. This is only available in the devkitPro Subversion repository.


svn co http://devkitpro.svn.sourceforge.net/viewvc/devkitpro/trunk/projects/nds/HomebrewMenu/
cd HomebrewMenu
make

Once that is compiled, you have to patch it with the previously extracted Ez5s.dldi file (if you use a high capacity SD card, use the Ez5sdhc.dldi instead). Do that using the dlditool in $DEVKITARM/bin:


$DEVKITARM/bin/dlditool /path/to/ez5190ob11/moonshell/Ez5s.dldi HomebrewMenu.nds

Now copy that file onto your card, but copy it as the ez5sys.bin file. This means that when the card boots up, it will boot our HomebrewMenu instead of the regular EZ Flash/Moonshell executable.

Hopefully the next release of libnds will fix the issue so that homebrew written with it is compatible with all cartridges out of the box. Even better would be if the Homebrew Menu started to see regular updates and became a really cool and useful file browser.

Lint and CMake

2009-06-15T23:12:00.003+02:00

I've just updated my cmake-lint project with version 1.2. This fixed a bug that I introduced when I added in the ability to specify defaults in a .cmakelintrc file. This was one of those silly errors that using a compiled language would have caught, but in the end better unit tests would have caught it too. I'm not really a rabid TDD-weenie but automated tests that check for basic breakage are really handy. Python makes it dead easy to mock out unwanted and unrelated code too. Unit testing C++ is hellish in comparison.

As I was saying, cmake-lint is a really basic lint tool to root out common bad practices in CMake files and suggest some better practices. It also comes with some handy Vim plugins for working with CMake. One is a better version of the omni completion code I previously posted. The other is a :CMakeHelp command that helps you learn what all the CMake commands mean. There's no documentation for these plugins, but it should be obvious how they work. In Vim, type :CMakeHelp followed by the name of a command, or partial name and tab to complete. When you edit a CMakeLists.txt file, type CTRL-X CTRL-O to start up the omni completion.

I wrote these Vim scripts because CMake is a pretty good build system but there aren't that many IDE tools available. Actually, CMake is about my favourite build system ATM, it's quite a close call with Waf though. CMake's cross-compiling is probably what just pips Waf for any new stuff I do.

CMake lets you set up a regular build that compiles for the PC, and just by passing in the name of a file that describes a different compiler via the command line you can compile the same code for the DS. With Waf you have to faff about writing more code, cloning environments, fiddling with variants... That gets a bit messy and it's all manual, especially if you want to compile with debug flags too.

Waf does ease the pain of integrating multiple modules inside the same source tree though, and it doesn't compile templates down to Make (which seems slightly crazy). Although gyp also does similar craziness for Google Chrome, but converts python templates to Makefiles, SCons input files (!) or proprietary IDE project files. Makes you wonder why they didn't just go with CMake and add to it. I would have liked to have seen a CMake generator that spat out SConstructs, for example.

Mercurial

2009-04-26T16:21:00.010+02:00

Google Code is going to add Mercurial support soon, in fact it's already available for some pilot projects. I'm more familiar with Git though, so I needed a good translation guide and I found the Git HG Rosetta Stone. That isn't bad, but there are still some gaps, so I thought I'd share my experience fiddling around with Mercurial. To try and pad out some of those gaps. This is a big long dull technical entry. Sorry.

The first thing I tried was to convert a small Git repository to Mercurial. The way to do this is to use the convert extension.

Now a bit of a rant about extensions here. This goes for Mercurial, but I think it applies to Bazaar too. Anyway, the issue I have is that in order to activate extensions in Mercurial, you have to add a line to a file in your home directory called .hgrc. This seems a bit anti intuitive to me. If these extensions are available in the core, and they are installed and everything, then why do I have to activate them by changing a preferences file? It's probably to keep the core set of commands simple or something, but it adds needless complexity to the whole process.

The documentation on the conversion page mentions converting from CVS, Subversion and Darcs, but there is no mention of converting from Git. I wonder if this is because nobody would really want to convert from Git to something else? ;-) Anyway, the conversion was easy for a trivial repository, just run "hg convert /path/to/repo".

This creates a new directory repo-hg that contains your shiny new Mercurial-converted repository. The directory is initially empty, except for the .hg directory. That's equivalent to the .git directory in a git repository. The odd thing here is that running "hg status" on this empty directory reports no changes - compare this to a "git status" in the same situation, where it would say to you "hey, you've deleted all of your files!", and it seemed a little strange. You have to run "hg checkout" to get all your files back in the directory, which is pretty much the same as the "git reset --hard" idiom that you'd use in the same situation.

Converting my Bunjalloo repository wasn't as easy. I've used a few branches, and some of the history came from subversion originally and there are strange tag/branches there. Mercurial doesn't seem to like this at all and just creates a load of branch-less "things". There's probably a good way to do this, but it would need some investigation.

After compiling and running "hg status" I had a load of unknown files, marked with things like "? build/foo.o". I tried the simplest solution: "mv .gitignore .hgignore". This almost worked, but it game me the following strange error:


could not start inotify server: /tmp/repo-hg/.hgignore: invalid pattern (relre): *.[oa]
abort: /tmp/repo-hg/.hgignore: invalid pattern (relre): *.[oa]

By default Mercurial uses Python's regular expression syntax, not globs. That probably makes some sense, it means it was less effort to code I bet. The error message about inotify is a strange red herring though. The fix was to add a "syntax: glob" line to the file.

After this I made a change in my code to see how Mercurial handles the normal workflow. Git's diff command shows differences in colour. Well, that's not quite true - by default it doesn't, you have to set a preference by running something like "git config color auto". Mercurial does colour too, you have to edit the .hgrc file adding "hgext.color=".

All these properties and preferences can either go in your global $HOME/.hgrc file or in the repository-local .hg/hgrc file. There's no core "hg config" command, but there is an extension to do it. It is a second class citizen that isn't shipped with the core Mercurial and requires an extra not-core-Python library (I suspect this is why it isn't in the default Mercurial). In git you can run the core command "git config --global" to add a config option to your global file, or by default it changes the repository local configuration in the .git directory. It makes things a bit more newbie friendly than editing strange dot files. I can't imagine an ex-Subversion user being comfortable editing a file below .hg, for example. In subversion you just don't go futzing about in the .svn directories.

Once all that was done, I decided to commit the change I had made. Here hg doesn't use git's staging area. That's fine. It is something that works well, but a lot of people don't get it. Mercurial uses the classic approach to this - pass what you want to commit on the command line - which works too. What I do miss here from git is the ability to run "commit -v". With the -v option, you get shown the patch that you are committing as well as the file names. This saves running "$VCS diff" in a separate shell, which is what I always ended up doing with Subversion. A definite regression from git here.

After committing a change or two, I sometimes run gitk to see what is going on overall. Mercurial ships "hgk" which is a really ancient version of gitk tuned for Mercurial commands instead of Git ones. Unfortunately it isn't turned on by default and, you guessed it, you have to edit the .hgrc file. Not only that, but the examples given in the Mercurial wiki are not entirely correct, at least when you install from source. They all state silly paths for hgk.py - the real location is $PREFIX/lib/python$VERSION/site-packages/hgext/hgk.py, where PREFIX is usually /usr/local and VERSION is the Python version, probably 2.5. But even if you set that in your .hgrc, "hg view" doesn't work. You need to copy the file "hgk" from the source contrib directory into your path. This is quite awkward. If something ships with the core, it should be installed and Just Work (TM).

Sadly hgk is not very as good as the current gitk. It's missing a lot of the search options, the "you are here" yellow ball on the current
checkout, status bars for slower updates, and probably some other features that I've forgotten for the moment. Running help says "About gitk" and shows that it is version 1.2 (C) 2005 - the current gitk must be about version 1.80 now and is in active development. The hgk equivalent is about 4 years behind and appears to be more or less abandoned.

The other graphical interface that git has is the "git gui" command. This is really needed for managing the index well, adding chunks of patches and so on. The index is handled automatically in Mercurial, so that's a big chunk of usage that you wouldn't need a gui for. There is an extension - hg record - that emulates the "git add -i" behaviour for adding partial patches. A hg gui would be handy for using that recording extension, if nothing else, but there's nothing included by default. The only likely candidate is hgct, which has not been updated for a couple of years.

Another command I use a lot is "git rebase -i". This lets you squash up commits, change the order of unrelated commits, drop patches, break
up monolithic changesets into smaller ones, and so on. It is really good and easy to use. You run "git rebase -i someid" and get back a
list of commits to do stuff with, something like this:


pick 84d3267 Add the key release waits back in
squash 63656f4 This seems to work on desmume at least
pick d34b4f3 WIP, damn sprites do not show up...
edit c9b6718 Mostly works

# Rebase 7c1db7e..c9b6718 onto 7c1db7e
#
# Commands:
#  p, pick = use commit
#  e, edit = use commit, but stop for amending
#  s, squash = use commit, but meld into previous commit
#
# If you remove a line here THAT COMMIT WILL BE LOST.
# However, if you remove everything, the rebase will be aborted.
#

Mercurial doesn't have rebase, but on the other hand it does have the incomprehensible queues system. I've read the documentation, but I just don't get it at all. You need to remember a whole bunch of new Q commands, and you need to initialise the repository to tell it that you want to use queues. Then you can add, push and pop patches. I dunno. It just feels too heavyweight and requires too much planning up front.

The rebase command was added to the git core late in the game, but it integrates seamlessly with everything else (except submodules, but
git submodules have other problems anyway). You don't need to plan to use git-rebase, you just do it. The visualisation is done in your editor - you don't have to push and pop patches to get them in the right order. You just move them about as lines of text. If it all goes wrong, you can run "rebase --abort" and you don't get screwed over by having all these strange extra patch commits, which is what MQs tends to do.

Now to push things to a new bitbucket account, since Google Code doesn't let everyone use Mercurial yet. Creating the empty repository on Bit Bucket was dead simple. Pushing was equally simple - hg push http://bitbucket.org/quirky/repo. You enter a username and password, which is far easier than the git equivalent, where you have to generate a ssh key, perhaps having to fiddle with ~/.ssh/config to tell it that SSH should use the user "git" when it connects to the github server, and so on. I can see how this would be simpler for someone used to Subversion over HTTP. The Mercurial documentation on the push command seems easier to follow than the git equivalent, too.

Running that line will just push changes to the remote repository - it doesn't set up the remote branch, the equivalent of a "git remote add origin git@github.com:user/repo.git". To do that you have to edit .hg/hgrc and add a "[paths]" section pointing to the remote URL. The convention here is to set the URL to a value "default". i.e.


[paths]
default = http://example.com/repo

To see which changes have not been pushed to the remote repo with git, you can use gitk or git log --decorate. Both of these show where the remote head is and where the local one is. By inspection you can tell which commits are missing from the remote branch. The equivalent in Mercurial is to use "hg outgoing", which tells you which changes have yet to be pushed. Ah, another difference for the gitk/hgk list here: hgk doesn't show the remote branches at all. Anyway, Mercurial's outgoing command is handy. You could mostly emulate it in git using "git log origin/master..HEAD", but you need to know the name of the remote branch, which is surprisingly non-trivial in some cases.

Another feature of Git that I tend to use is for creating patches to send to other people. I mostly use this together with git-svn. That way I can clone a Subversion repository, make my local commits and then send patches of those commits off in bug reports or to mailing lists or whatever. This is "git-format-patch" and "git-send-email". Mercurial has a single command for that, "hg email". It's pretty much the same, but also sends email by default (like git-send-email). Sometimes you need to attach patches to HTML forms rather than emailing them, and so generating files is my preferred approach. By default git generates one file per commit - in fact I think that is the only way it works- Mercurial on the other hand only ever generates one file, with multiple patches inside. A nice option in Mercurial is "-o", which automatically generates the patches that are not upstream on the remote branch. Anyway, the command lines are subtly different, and the output is similar-but-not-the-same. Git has its single-patch approach, where you get out what you put in, plus it is less irritating in the number of questions it asks you (none). Mercurial creates a patch-bomb mbox and asks you for a To: email address, even if you don't want to email the file to anyone. It is probably a personal preference thing, but I like the way Git does it better here. They are both usable and scale better than "svn diff > ~/whatever.patch".

Another gotcha that caught me out was that "hg add" without arguments adds all untracked files for commit. This is like "git add .", but normally commands without arguments don't do naughty things. I didn't like the default behaviour here, it was too surprising. Especially if your hgignore file is not set up right. To fix it, you have to run "hg revert -a", which forgets about the added files.

Summary

Here's a big ol' list with the subtle, and not-so-subtle differences that bugged me. I really wanted to use a table, but blogger doesn't do tables without adding a bunch of blank lines...

Git gitignore
Hg .hgignore, syntax: glob to get the same behaviour as git

Git .git/config, ~/.gitconfig, use git-config to modify the values
Hg .hg/hgrc, ~/.hgrc, hg config is a non-core extension

Git git commit -v
Hg hg diff | less; hg commit

Git gitk
Hg hg view - there is some set up involving .hgrc and placing hgk in the path.

Git git rebase
Hg Mercurial Queues, kind of.

Git git push URL ; git remote add origin URL
Hg hg push URL; $EDITOR .hg/hgrc ; [paths] default = URL

Git gitk, git log origin/master..HEAD
Hg hg outgoing

Git git format-patch RANGE
Hg hg email -m filename -o

Git git gui
Hg Nothing equivalent. There's "hg record" for "git add -i", but it is less user friendly.

Git git add . ; Note the dot
Hg hg add ; No dot needed. Take care with that! You'll have to run "hg revert" to fix the mess

The main arguments for using Mercurial over Git are better Windows support and ease of use. I can't comment on how well either work on Windows, but for Linux Mercurial worked more or less as expected. The second argument about hg being much easier than git... I don't know. Maybe. I'm tainted now that I more or less grok DVCS. But given the difficulty in setting up most of the core plugins and the lack of good core GUIs (gitk/git-gui), I disagree. The basic work flow is more or less the same, but Git has this batteries included feel while Mercurial is more limited. Mostly by default, by enabling optional extensions there isn't that much real difference. Besides, I think that for someone used to Subversion who has never used DVCS, either tool would be overwhelming at first.

What I do know is that the GUI tools that come with Git are a huge help for getting up to speed with what is going on "under the hood". Seeing all the commits and their hierarchy in gitk is far easier than reading the log or viewing the gitweb page. Especially if you start doing rebases and history changing stuff. Maybe Mercurial's one-branch approach and lack of history-munging commands makes all these extras unnecessary. Who knows. Time will probably tell.

Either way it's time to get stuck in to learning Mercurial. I'll definitely start using Mercurial when Google Code offers it to everyone, because the current github/GC integration works but it's a bit ad-hoc. It's missing automatic links from issues to commits, the source tab looks a bit wrong, the Updates feed doesn't update, little things like that. I like the Github UI for code - you get the last change right there on the front page - but for non coders, I think it could be a bit daunting. Google Code's interface is cleaner and easier to navigate. Also subversion is missing features like real tags (no the /tags directory is not the same) and a couple of projects I've got on GC still use the default Subversion just because it's easier to set up and everything. Mercurial will be a nice bonus feature.

Elite DS version 0.6

2009-04-07T23:22:00.004+02:00

I've just cooked up a new release of Elite DS. This adds the music back in that I had to take out when my own GBA XM library proved too tricky/time consuming to port to the DS. I've updated the code so that it compiles with devkitARM r25 and the sound now uses devkitpro's default ARM7 core.

From a technical POV the changes aren't very interesting, but I'll waffle on about them anyway!

Prior to this release, I was using the DS's PSG sound system. This is the bleep-bleep-bleep square/triangle wave sound system that Nintendo has used in its handhelds since the dawn of time. I think it is pretty much unchanged since the original Gameboy ;-) The good thing about this "format" is that it is small - you can say "use this pitch", "play it!", "hold it!" in just a few bytes and the sound lasts ages. You just change the pitch after a given length of time to add new exciting effects. The bad part is that it sounds terrible! Also, the sound format is completely arbitrary/homegrown as you have to just poke registers with values to make the DS go BEEP.

Additional amusement creeps in as you have to poke the sound registers on the ARM7, but the sound events trigger on the ARM9. Like when you get shot or fire a laser. I used the inter-processor FIFO to pass the "bleep now dammit!" messages from the '9 to the '7. Most of the sounds were taken from running Elite under emulation in pocket beeb, capturing the sound values that were written to the BBC sound registers, converting them into the equivalent values for the DS registers and then using these values in Elite DS. They sound more or less like the original BBC sounds this way. This is not necessarily a good thing :-)

The new sound effects were easier to create. I used this simple (to use) sound generator. I spent a pleasant afternoon creating random sounds, some of which may be slightly above the "terrible" level. I forget where I stole the Elite music files from, I think they are nabbed from the NES version. Of course on the DS these all use the fantastic maxmod library. I wish this had been available years ago, it is very easy to use and works exceptionally well.

Ah, there was one random bug that had me scratching my head for a while - I used to use a bit of assembler to clear the screen. This sets all the ARM registers to zero and creates an unrolled loop to blit these zeroes to the video RAM. For some reason it caused crashes on hardware when I started using maxmod. The only thing I can think of is that somehow registers were getting clobbered that maxmod uses in an interrrupt, or vice versa. My fix was to replace that clear screen routine with a DMA fill. Speed is not an issue on the DS compared to the GBA :-) It's a bit cargo-culty but I was desperate and with that register-clobbering explanation I sort of convinced myself. Anyway, I've been playing 0.6 all afternoon and it hasn't shown any more crashes (of the game; there have been colisions between myself and a few Cobra Mk IIIs). Enjoy!

DSi thoughts

2009-04-05T15:56:00.007+02:00

I'm fairly weak and couldn't resist buying a new DSi on Friday when it was released here in Spain (and indeed, the rest of Europe). Here are my impressions of the new console. Keep in mind that I have never owned a DS lite, if that makes a difference.

First of all the bad news: my EZ Flash V doesn't work with the DSi. I knew this would be the case, but had to check anyway. The card is recognised on the menu, but when selected an error message immediately appears. Oh well, I still have the phat DS for homebrew until someone hacks the DSi.

The console itself has a most peculiar finish to it. It's as if the plastic used is "fuzzy". I don't really know how to describe the feeling that you get when you touch the outer case, but it sets my teeth on edge somewhat. Rather like touching polystyrene or chalk.

Once you boot up, the main menu is a new version of the old firmware. I'd completely forgotten about the new-world paranoia too, all those messages about health and safety that loopy's firmware hacks got rid of so many years ago are of course back on the 'i. And there doesn't seem to be a way to get the console to boot directly to the inserted game. You always have to go through the menu.

This menu also has options for the camera, the sound thing, connect to the DSi shop, as well as the old DS download play and pictochat that have been there since day 1 (and nobody actually uses). In addition, any downloaded games from the store appear on this menu in their own icon.

A new feature that was long overdue is in the way the wifi-settings are changed. Rather than have every game that needs to connect to the internet reimplement the settings screen, the firmware itself has a way to update them. This is a good move, although all the existing DS games will still use a different method to the new built in one, even if the settings themselves are stored the same way in the internal memory.

The first thing I did was to download the new Web Browser application for free from the DSi shop. The browser is based on Opera 9.50 and is pretty nifty. More on that in a minute though. First off, it seems that the firmware itself is updateable. The first time you connect to the "DSi Ware" shop, it updates the system. This means that any future firmware-hacks to get homebrew running will be an inevitable arms war with Nintendo. A bit sad really. If the shop were done correctly (i.e. like the Apple app store or the Android equivalent) then the DSi could have been an amazing platform for homebrew. But Nintendo are real control freaks, they make Apple look like some bunch of free for all hippies. For example, before you connect to the DSi Ware store, you have to agree to some sort of Excentric Unenforceable Licence Agreement (or "EULA" as they are known). One of the clauses states that if you sell your DSi you must erase any software you have downloaded from the shop first, as the right to use the software is not transferable. It's quite clear that Nintendo do not want to sell you games, but to sell you a licence to play a game on a given console.

I must say that when you download a game or program from the DSi Ware store, the "progress bar" is great!

I haven't really tried out the built in sound program/toy thing, so no comments on that.

The cameras that are built in are low res - I've seen ambiguous reports on the interwebs that one of them is 0.3 megapixels and the other is 3.0 megapixels. I can confirm that they are both lame-oh 0.3 megapixels. Never mind, they work quite well for the amusing games and stuff that are available at the moment.

You can store your photos onto either internal storage, of which there is about 32MB, or onto SD cards that fit into the side of the console. The internal storage is shared with applications that you download from the store. ~~Once again, Nintendo have screwed up by making it impossible to save these games to the SD card! They made this mistake on the Wii and have repeated it here with the DSi.~~ Update: I have discovered the option hidden away in the settings menu to copy the applications over to the SD card. By default they are stored internally, but it seems that lessons have been learnt since the Wii.

Speaking of cock ups.... There is no way, via either the camera tool or via the Web Browser, to upload your photos to any sort of web site or anything. For example, in gmail the "attach file" control is greyed out - the browser doesn't implement the input type="file" form element.

The camera software lets you trade images locally, i.e. with other DSi users in the same room, but that is all. Hilariously there is another paranoid legalese message when you try to do this: "you acknowledge that you are entitled to distribute these photos and frames and agree that they may be modified, distributed and/or posted by the recipient and/or third parties." Given that you can only exchange the photos with someone stood next to you, this seems a bit overkill to me. A console that has a camera and is wifi enabled, but has no way to send those photos via the intertubes seems like a huge missed oportunity to me. I look forward to some homebrew program that will allow us to do this. Oh. No. Homebrew doesn't run on the DSi. Gah!

You get 1000 "free" points for the store when you first connect to it. Free as in "you just payed 170 euros for the console, the least we can do is give you some cheapo games for it". Apart from a couple of rehashes (Pyoro, Paper Plane) and some puzzle games (I'm more than fed up with brain draining), there is a Wario game that makes use of the camera, "Warioware: Snapped!" it is called. I wish I had read this review (rating: 4/10) before downloading it though. It is amusing for a few minutes, but has no replay value and is not even very practical to play. I get the feeling that this is going to be the pinnacle of camera based games on the DSi, however. It plays similar to the eye-toy games on the Playstation, with the added irritation of the screen being tiny. On the plus side, the replay mode at the end of each round is amusingly done.

The first DS game I plugged in to the 'i was Etrian Odyssey II. It ran fine, so at least the much touted region lock out is not applied retro-actively. I'm on to the 4th stratus, which (if you forgive the minor spoiler) has some quite colourful cherry blossoms and trees. On the DSi they really are extra colourful. The new screen is a lot clearer than the phat one. Even on brightness 1, the lowest, it is brighter than the phat DS screen. The size is slightly larger too, but not so much that it makes any difference. Even side by side it is difficult to notice the difference. I suppose it couldn't be made too big at the same resolution or games would start looking pixelated.

So all in all what do I think of the DSi? Well, as an upgraded DS it is OK if you don't have a DS lite, which I don't. The lack of uploadable photos from the camera is a missed opportunity, what with everyone having Facebook accounts and whatnot nowadays. The browser is cool, even if it does run out of memory when reading pages on blogger.com. In column mode it feels a lot like Bunjalloo ;-) of course these Opera guys have a lot more resources than I do to get things right. They also have Nintendo breathing down their necks, so get things (like no file uploads) wrong too. I wonder if their will be updates to the browser? Given that it is stored in RAM, rather than ROM, it is possible that new versions will appear. Given how tightly Nintendo control everything, together with their restrictive software quality control process - which would make even minor upgrades an expensive effort for Opera -, I won't hold my breath.

Avoid hard coded variant directories by using a SCons Repository

2009-04-02T22:35:00.006+02:00

One of the things about SCons that I didn't like was that by default it built your object files in the same directory as your source code. This means you end up with this:

src/foo.c
src/foo.o

Your ${VCS} status output would be full of this sort of thing:

Untracked files:
? src/foo.o

Tab completion would stop at vi src/foo., grep would add spurious "Binary file foo.o matches". Yuck, what a mess.

Then you discover VariantDir in the SCons documentation and this looks great. You add variant_dir="build", quickly followed by duplicate=False, and all is well. However, this hard coding of "build" is a bit lame. Think about autotools - you could build out-of-source to any directory you liked...

mkdir /tmp/build
cd /tmp/build
/path/to/source/configure && make

Wouldn't it be great if SCons could do this?

It can!

This is a little known feature that I recently stumbled upon completely by chance. Assuming the code is in ~/projects/code, the magic incantation is as follows:

scons -Y ~/projects/code -f ~/projects/code/SConstruct

This uses a SCons secret weapon - Repositories. As a bonus, this particular feature is not even mentioned in the documentation. It means you can compile into any directory using the original source code. The explanation is that the options do the following:

-Y REPOSITORY Search REPOSITORY for source and target files.
-f FILE Read FILE as the top-level SConstruct file.

By saying use this SConstruct to build the code and if you don't find a file, look here, you emulate the autotools/CMake out-of-source behaviour almost perfectly. If the command line looks long, just add --interactive. That way hitting "b" and enter will rebuild.

Hang on, because there are 2 caveats.

First, you have to be careful that when you specify files that are not handled natively by SCons that you use the right names, i.e. in your own Builders. Also internal include directories should use # so that they mean "from the top of the source tree" rather than from the top of the build tree. That means env.Append(CPPPATH=['#mylib']) instead of just "mylib".

Secondly, and more annoyingly, the build directory can't be below the source directory. SCons gets all confused and starts putting another build directory in your build directory (so you can build while you build). But then crashes out saying it can't find the source code. Boo!

Apart from that, you're good to go.

Vim omni completion for CMake

2009-03-14T17:07:00.005+01:00

As CMake uses its own language, learning what commands are available can be a bit of a pain. I've written this really basic Vim omni completion plugin that takes some of the effort out of getting to know the API. Install the file to the autoload directory in your .vim directory, i.e in ~/.vim/autoload/, and add the following line to your .vimrc file

  autocmd FileType cmake set omnifunc=cmakecomplete#Complete

Now when you edit a CMakeLists.txt file, it should do omni completion when you summon help with the Ctrl-X Ctrl-O enchantment. Maybe someone will find it useful.

I've only tested this with the latest CMake and Vim versions, it is quite likely that I assume things that do not work in earlier editions of these programs. Patches welcome ;-)

Edit: Updated to add property and module completion

Notes on Eclipse CDT

2009-02-14T12:57:00.013+01:00

It has been a while since I last tried out Eclipse CDT. I've been using Eclipse at work recently for Java and RCP stuff, so I thought I'd give CDT another go. I prefer Vim, but I don't mind trying new things. So I thought I'd try it out for my homebrew stuff.

Eclipse for Java is actually pretty good. When you write Eclipse plug-ins you usually have to implement a load of seemingly random interfaces. There are project wizards in Eclipse that generate all the necessary stubs and what-not for you. Then once your project is underway, the editor has all sorts of auto-completion and javadoc "hints" to help you figure out the API. The facetious among you may say that all these crutches are necessary because the Eclipse API is not discoverable enough. I might not argue with you, but I've seen some terrible C++ libraries in my time too.

Now I am not a big Java coder. I mostly write C++, with a bit of Python. In Vim I can emulate a lot of what Eclipse does for Java, but for C++. Exuberant CTAGS is a marvelous help for doing this. Coupled with Vim's omni-complete, you can have a lot of context sensitive auto-completion for C++ code. The completion in Vim also gracefully degrades to just regular context-insensitive completion. So even if it doesn't complete your class based on its available member variables and methods, Vim completes based on the contents of the current buffer, or loaded buffers, or included files. Most of the time this achieves what you want.

By the way, to set up omni-completion in Vim for C++ you need to do a couple of things. First you need a modern Vim. Version 7.1 or so should do. Then you need to either pass these flags to ctags, or create a $HOME/.ctags file:

--c++-kinds=+p
--fields=+iaS
--extra=+q

Now run "ctags -R ." in your source code tree. That creates a tags file with the index that Vim uses to auto complete. Armed with that knowledge, you can create tags files for the STL to have auto-completed vectors, lists, strings and so on. Using g] in Vim it jumps to methods, variables and defines.

OK, back to the story. I'm trying to use Eclipse CDT for C++ to see how it compares to my Vim setup. This article from IBM describes how to compile UNIX software from Eclipse using the C++ IDE. Instead of using make directly, it recommends using ant to call configure and then make. That's quite novel and it "works" insomuch that your code compiles, but it isn't very good for proper coding. The main issue is that the errors from ant are not parsed into clickable quick fix errors. In other words, if you screw up and your code doesn't compile the errors are not highlighted in the editor as they should be.

Aha! But I don't use make. I use waf for bunjalloo. The waf interface is almost identical to make. In Vim I use "set makeprg=waf" so that running ":make" compiles my code. I realised that I could do the same thing in Eclipse, and here's how.

Before starting anything I did what I normally do to install new programs on Ubuntu; I opened up Synaptic. Sadly Ubunutu 8.04 has an ancient version of Eclipse (3.2.2! That's from the middle ages, held together with wattle and daub). I made like I was using Windows and went off to the Eclipse web site to download the latest release, which is 3.4.1 at the time of writing. Excellently this is just a big fat tarball, so you can unpack it in /opt or wherever you like and it runs. No dependencies and with batteries included.

Now I had to get my code into Eclipse. There's a bit of irritation involved with this. Eclipse insists on copying your files into its own workspace, even if you already have a perfectly good space for working available. My plan here was to create a new workspace, use the import tools to copy the code there, then manually copy the secret Eclipse project files back to my regular directory. I could then import the original directory as an already existing project. Convoluted, but I just don't trust Eclipse not to bork up my files while it whispers softly "it's for your own good..."

Step 1. Start Eclipse and tell it the workspace to use

eclipse -data /tmp/workspace

This was a throwaway workspace that I wouldn't use once the project files had been created.

Step 2. Create a new C++ project in Eclipse. This has to be an empty Makefile project.
This created an empty project that would not be managed by Eclipse's internal builder. Eclipse just calls "make all" and crosses its fingers.

Step 3. Add the source code. I used the Import > General > File System option.

This copied all the code into the /tmp/workspace/project folder.

Step 4. Set up waf as an alternative to make

To do this, I only needed to change "make" to "waf", and change the default incremental build target to be "build" instead of "all". This is done from the Project > Properties > C/C++ Build. In Builder Settings, uncheck "use default command" and change "make" to "waf". From the Behaviours tab, change the "all" target to "build".

At this point I was almost done. However first I had to add the configuration step in. Waf, like autotools, first configures the environment to see where your compilers, libraries and other tools are installed. This is done by running "waf configure" from the command line. I added a new Make target "configure" to do this. That's right click on the project > Make Targets > Create... In the dialog, I added "configure" in both the target fields.

I ran the "configure" target first, then ran "build" as normal. With this done, the whole thing was compiled.

Now that everything was set up, I could use the project files created by Eclipse. These are called .project and .cproject. The .project file is pretty light weight and describes the project basics. But the .cproject file is a mess. 276 lines of incomprehensible XML, with all sorts of redundant and machine-specific configuration information encrusted in it. Yuck. It also changes "randomly" when you alter minor details in the project. This makes it really unsuitable for storing in version control, but without it the whole CDT project is useless. A hairy dilemma, and one that doesn't occur in the Eclipse Java IDE, where the .project file is pretty stable.

I just completely ignored this problem and I copied the project metadata files to the original location of the project in $HOME/src/ndsdev/ anyway. This is to make sure I can re-import the project using just these files. I restarted Eclipse, using a different workspace now (the default $HOME/workspace), and used the "import existing project" option. This all went well. I now had all my code in Eclipse in its original location, compiling, running, and so on.

I then wanted to fiddle around, see what Eclipse could do with C++. The first problem I hit was a lot of "unresolved inclusion" errors. This occurs when the #include files cannot be found. As I wasn't using Eclipse's managed build, my project compiles OK but Eclipse is ignorant to the actual internal paths my Makefiles, er, wscripts use. To fix this, I had to add the project include paths using the Properties > C/C++ General > Paths and Symbols setup page.

Next up, I thought I'd try out the auto completion. What a disappointment! Compared to the Java equivalent, which has an almost 6th sense ability to guess what you are trying to write, the CDT version is hobbled. My Vim+CTAGS solution works much better. As an example, say we have this code:

class Test {
public:
Test();
int getValue() const {
 return m_
}
private:
int m_value;
};

the cursor is at the end of "m_" and I press Ctrl+Space to auto-complete, but I get told that there are no proposals. What about that m_value just 4 lines down, eh? Gah. You can "fix" this by placing the definition of m_value above where we were trying to use it, but that's just silly.

Another niggle I have is with the cut and paste mechanism. In Vim, yy, dd and pp all copy, cut and paste entire lines - from the very left of the screen to the end of the line. In Eclipse however, the keyboard shortcut for cutting a line is not enabled by default, copying a line with ctrl-alt-down results in a copy directly below where the line was, which you can then ctrl-x cut. But pasting is awkward - the cursor jumps about unexpectedly to an indented place, and it is very easy to add unwanted blanks at the end of text. Perhaps this just requires getting used to, but it feels weird and icky.

The other nice feature in Java is the documentation hints for filling out constructors and functions. There's an option to support Doxygen (Properties>C++ General), but it doesn't seem to make any difference. I've Doxygenized most of my headers, and Woopsi also uses the convention, but when I hover over a method that has documentation it just shows the implementation and leading comment. Fair enough, you get some idea of what is needed, but it'd be nicer if it showed the actual documentation.

Finally I tried some of that cool refactoring that the hip kids are all talking about. I found method that was looking a bit full, selected a portion of code and hit Refactor > Extract function. This was in a class method and the code used member variables. Ideally refactor would create a new method and add the prototype to the header file (no chance!). The dialog was disconcerting:

The dialog says Function Name, the text entry says Method name. Which is it? Function or method? A function is a named section of code. A method is a function that is part of a class. I had a bad feeling about this. Rather surprisingly, it did what I expected! It extracted the code, placed it in a new method and added the prototype to the header. Wow! Unfortunately the header file got a bit mangled, with all the comments stripped out. I tracked down a minimal bit of code that causes this and opened a bug report. It does mean that you probably have to be extra careful about using this feature blindly, but it's better than nothing. UPDATE: The bug has been fixed in record time! I opened it on the 12th, and it was marked fixed on the 17th.

Eclipse has a lot of other cool features - I haven't mentioned mylyn or the debugger integration here - but the C++ IDE itself is still a bit rough around the edges. This is probably to be expected if we consider that it's a C++ editor written in Java. That isn't meant as Java bashing, let me explain. The people who write the Java IDE version of Eclipse undoubtedly use this Java IDE to edit the Java code. The people writing the C++ IDE also need to write Java code. They don't necessarily need to write C++ though. Likewise, the users of the Java IDE edition of Eclipse are presumably capable Java coders, so it is more likely that they would be able to investigate the code and provide patches for bugs they find. Users of the C++ IDE are coders who probably know Java, but there are no guarantees, and it may not be their preferred language. They are less likely to dig into the guts of the editor to add functionality or fix bugs. Maybe. It's the dog food thing again.

Another factor is the size of the target audience. In the case of the C++ IDE, this is a lot smaller. According to the eclipse download page, there have been around 4.5 million downloads of the various Java flavours of the IDE, but less than 700,000 downloads of the C++ version. That doesn't take into account updates of the Java version to the C++ equivalent, but the opposite is also true (updates of the C++ IDE to add Java components) so it's probably a pretty accurate reflection. The Java IDE is more than 6 times as popular. More eyes means more bug fixes and polish.

So will I use Eclipse for C++ from now on? Probably not. I'll check back when the next release comes out, as the improvement over the last CDT release is major - the indexer hung last time I tried to load the bunjalloo code base into it and refactoring now has more than rename variable. Hopefully the next release will fix more niggles and add more features. But I've used Vim for years and it is an extremely optimal way to edit text once you get used to the interface. :wq

WIP Woopsi screenshot

2009-02-07T17:05:00.002+01:00

I've started the port to Woopsi now and it's going alright-ish. There are still a lot of bugs to iron out and old features to re-implement, but I think being able to view 2 web pages at the same time is going to make a huge difference. This screenshot shows a test page and the Google code hosting site. The page title is shown on the screen title bar. Each web page is a borderless window gadget. Clicking a link opens it in the same window. Shift-clicking (clicking with the stylus while holding a shoulder button) opens the web page on the opposite screen. So you can have 2 web pages open at once. This is an artificial limit, but given the memory limits I think is a good compromise.

Now for some technical crap! I'm using git's submodule support here to track Woopsi's development on sourceforge (via git svn...). Sounds complicated, but it's not really. And in the end it means I get bug fixes for free, but can also add my own hacks on top of the main Woopsi code. I've added my own build scripts for example, but most of the rest of my bug fixes have been committed by ant to his svn repository. Nice!

The main issues I've seen so far are to do with my "meta-gadget" that keeps track of all the text and image gadgets. There are some scrolling issues and position problems there. The other problem is with unicode of course, but I knew that was going to be a sticking point when I started.

I still need to figure out how I'm going to add in the back-forward-stop buttons. I think these will have to be in a separate window, but I get the feeling that that would be awkward. I'll probably end up adding them to each window under the main display. It'll cut off some screen space, but not too much. In the current version the buttons are shown on top of the text and are slightly transparent. This is a bit ugly though and you can't really read the text underneath anyway. Woopsi's gadget heirachy means that it might be easier to add these as separate controls and forget about it.

GUI plans

2009-01-22T21:19:00.004+01:00

This week Pedro Estébanez wrote what he claims will be the last entry in his Okiwi blog. It's a shame that there'll be no more updates to Okiwi, but I think we all new that this was the case after the quiet period following his initial release. A web browser is a lot of work.

I've looked at Okiwi's code and, apart from some monster thousand-line functions, it looked OK. The quad-tree rendering algorithms are certainly miles better than Bunjalloo's naive approach to showing words and images on the screen. But what counts is releasing stuff that's useful too. Okiwi didn't have all the minimal extras to get some websites to work (I'm thinking cookies and forms here) and these things need to be thought about from the start. I'll admit that I'm not the best at getting new releases out, especially now with a 1-year-old running about the house, but I've managed a few releases over the last year or so.

So what does this mean? Well with DSOrganize also abandoned, it looks like I've succeeding in boring the competition into early retirement! Go me! Heh. Now I just have to keep going. Since I wrote about abandoning Bunjallo a few weeks back, I've received quite a few emails and comments saying "don't give up dude!". And I can't really give up, I'm a code addict. Not everything I hack on gets released of course, there are many false starts and dead ends before I get something that's worth releasing.

My current way of wasting time is trying to integrate, or to port (or whatever), Bunjalloo to the Woopsi GUI library. There are a couple of problems with this, but it would have its advantages too. First off, the bad news. Woopsi is ASCII only. That's a really big stumbling block, but I think I'll be able to fix it given my experience writing UTF-8 decoders already. The other problem is that I'll have to rewrite the current rendering "engine" to this new Woopsi API. I have no idea if this'll go well or if it'll annoy me to such a degree that I'll throw the whole attempt in the bin.

The advantages of using Woopsi are that I'll get this widget toolkit that was designed specifically for the DS for free. The current widgets are OK, but there are some design issues and writing multi-window style configuration screens more difficult than it ought to be. Writing the configuration screens has been the most irritating and dull part of the whole Bunjalloo experience so far. A couple of core classes were getting ugly and it was clear to me that I'd got something wrong in the design. A change of tack will get things rolling again hopefully, while making it easier to add new features. If I rewrite the lot in Woopsi, then changing to an options screen wouldn't hide the current web page, for example. And I could even add multi-tab browsing (well, multi-window but it boils down to the same thing). This would be my killer feature of a Woopsi port.

The current progress on the port is not very impressive, sadly. I can show the start page, without images or links. Hmph. But the command line version can connect and show web pages (albeit just as crudely). This isn't starting from scratch you see - I have all the back end stuff still in place, and the design has been improved to allow multi-tab/window to be possible, memory permitting. The biggest UI shock is that each web page would be rendered on only one screen, rather than across both. I don't know if Woopsi allows windows to span both screens, but the fact is that I never really find myself looking at the top screen when browsing pages anyway. It just feels like dead space. Perhaps I'll be able to add the controls to a window on the top screen, since swapping screens is very easy with Woopsi - you just click a little icon in the top right and the screens switch places. This way the current controls for stop, forward, back, etc., could be moved out of the way when browsing. We'll see how it goes.

So what do you think? A crazy idea? Or a flash of brilliance? (Clue: it's a crazy idea). I'd recommend downloading the latest Woopsi demo, play around with it a bit, then try to imagine what a mini web browser with that kind of "feel" would be like. I think it'd be pretty cool.

Upgrading to devkitARM r24 (on Linux)

2008-12-15T22:18:00.004+01:00

The latest version of the devkitARM toolchain was released last week, and it was a biggie. A whole new API for sprites and backgrounds was added to libnds. A new default ARM7 binary was added that automatically handles wifi, sound and sleep mode. The other big addition was a new fangled sound library that adds mod playback and fancy sound effects.

This is my post on the upgrade process for Bunjalloo on Linux (Ubuntu 8.04 in particular). Hey, if you read all this and get through it, then you could probably help out hacking on Bunjalloo too :-)

A summary for the impatient: not without hassles, but the gain is worth the short term pain.

I assume you have a directory devkitPro somewhere. This will contain devkitARM and libnds. You should set an environment variable DEVKITPRO to point to this directory. Something like this will do:

export DEVKITPRO=$HOME/devkitpro_r24
mkdir -p $DEVKITPRO

I recommend versioning this release at the devkitpro directory level. Release r24 contains some breaking changes and by having the possibility to change between r23 and r24 you may save yourself some headaches. At the very least, it will ensure you don't mix versions, which would be a big no-no.

Download the devkitARM r24 files from the sf.net project page
- devkitARM_r24-i686-linux.tar.bz2
- libnds-src-1.3.1.tar.bz2
- dswifi-src-0.3.5.tar.bz2
- maxmod-src-1.0.1.tar.bz2
- default_arm7-src-20081210.tar.bz2
- nds-examples-20081210.tar.bz2
- libfat-src-1.0.3.tar.bz2
I saved them all in $HOME/Downloads, and that's what I've put in this post. Change that path as you need.

Install the devkitARM toolchain. This provides the compiler and C/C++ standard libraries. I usually install it in a versioned file, but if you already have named your devkitpro directory "devkitpro_r24" then there's no need.
```
cd $DEVKITPRO
tar xvf ~/Downloads/devkitARM_r24-1686-linux.tar.bz2
mv devkitARM devkitARM_r24
```

Set the DEVKITARM environment variable to point to our installed toolkit
```
export DEVKITARM=$DEVKITPRO/devkitARM_r24
```

Install libnds
!!! CARE !!! these source tars don't have a top level directory, so you need to create them manually. This is the case for all of the devkitPro tar balls, except the main devkitARM_r24 one. They will splurge their files in the current directory, instead of creating their own top level one. Yuck!
```
mkdir libnds-1.3.1
cd libnds-1.3.1
tar xjf ~/Downloads/libnds-src-1.3.1.tar.bz2
```
Now to compile libnds - if you have DEVKITPRO and DEVKITARM set correctly then this should compile the library succesfully.
```
make -j 3 install
```

libfat is optional, but recommended. Open up the tar, compile and install it too, if you like. libfat for the nds depends on having libnds installed, so you'll have to do step 4 first.
```
mkdir libfat-1.0.2
cd libfat-1.0.2
tar xvf ~/Downloads/libfat-src-1.0.2.tar.bz2
make nds-install
```

Now install dswifi 0.3.5 in a similar way - untar the release files, compile and install. dswifi depends on libnds, so you have to do the previous steps before this one.
```
mkdir dswifi-0.3.5
cd dswifi-0.3.5
tar xvf ~/Downloads/dswifi-src-0.3.5.tar.bz2
make -j 3 install
```

Install maxmod. Again, this depends on libnds and won't compile if you have skipped a step.

mkdir maxmod-1.0.1
cd maxmod-1.0.1
tar xvf ~/Downloads/maxmod-src-1.0.1.tar.bz2
make -j 3 install-nds

Install the new default ARM7 core. This requires dswifi and maxmod, if either are missing then you will get compile or link errors.
```
mkdir default_arm7-20081210
cd default_arm7-20081210
tar xvf ~/Downloads/default_arm7-src-20081210.tar.bz2
make install
```

Try out the nds examples! Now that everything is installed, you can compile and run some examples.

mkdir nds-examples-20081210
cd nds-examples-20081210
tar xvf ~/Downloads/nds-examples-20081210.tar.bz2
cd audio/maxmod
make
# test the nds files on your DS!

Write your own program :-) This part is a bit trickier!

Now once I had all this installed, I needed to update Bunjalloo to use the new code (Elite DS coming soon!). The first step was to see what would compile without drastic changes. The list of changes required in my code was:

Remove irqInit() calls - this is now done by libnds before your main is called
Register name changes: DISPLAY_CR -> REG_DISPCNT, BG0_X0 -> REG_BG0HOFS, BG0_CR -> REG_BG0CNT (and their SUB equivalents, where applicable)
Function name changes: powerON, powerOFF -> powerOn, powerOff, touchReadXY() -> touchRead(touch_structure)
Deprecated header: nds/jtypes.h -> nds/ndstypes.h
Look-up table changes: COS[angle] -> cosLerp(angle), SIN[angle] -> sinLerp(angle)
Some #defines have gone: BG_16_COLOR -> BG_COLOR_16, BG_256_COLOR -> BG_COLOR_256, SOUND_8BIT -> SOUND_FORMAT_8BIT
powerOn/Off now expects a PM_Bits enum value, not an int

Nothing major there... the big breaking changes are on the ARM7 side. Here trying to use your own hand-coded arm7 causes plenty of problems - most of the old inter-processor communications code has been removed from libnds, or has changed completely. Unfortunately, that means it's quite tricky to migrate to the new libnds and use your own ARM7 core. The easiest thing to do here is ignore your ARM7 code completely and use the new default_arm7. There's no one-stop solution, since most IPC code has been built in an ad hoc way, but here are some pointers.

For wifi code, your old wifi init code should be replaced by a single call to Wifi_InitDefault(true/false) - no need to faff about setting up the interrupts and timers. Then you can either now use sockets (if you passed "true", which means connect using the firmware settings) or connect using a detected AP first - all this is documented in the dswifi headers and by the new examples. Pretty neat, and cuts down on maintenance for everyone.

If you have a debug console, then consoleInitDefault() is a bit trickier to update. It has disappeared, pretty much. In most cases consoleDemoInit() will probably suffice. If you are doing anything much more complicated, then you will have to get to grips with the PrintConsole structure. This seems to be quite a powerful new feature, for example it allows printing to both screens from within the same program, but will take a while to get used to. I doubt many people used the old consoleInitDefault in "real" programs, and the new API looks like it might be possible to use the printf stuff on something other than a black/white screen.

Sound has seen a rather large shakeup in this release. The new MaxMod sound engine seems to fill a huge gap in the homebrewer's library arsenal. It isn't without its drawbacks though, if you did just basic NDS sound effects. Previously you could play individual samples by sending the raw data to the playGenericSound() function, after having previously set the sample rate, volume, panning and format via the setGenericSound() function. That has now been completely removed, replaced by new MaxMod sound engine functions. This is a flexible sound and music system, but requires that the sound effects are in a special format. A tool (mmutil) is provided to convert wav and mod file formats to the expected static structure. Alternatively, a more complex streaming system can be used. This latter does not require input sounds to be converted, and is the only way to play sound from a file, but is slightly trickier to use than the straight forward playGenericSound() function.

The sound streaming approach requires you the coder to implement an audio-filling callback and to call the mmUpdateStream() function often enough to keep the sound buffer full. Fortunately there are some good examples of all the MaxMod library's usage in the nds-examples pack, and the new system is a great addition. Besides music, you can do looped samples, proper panning, mixing, etc, etc. Bah, I'm just miffed because I'll have to rewrite some of my SDL sound code ;-)

Needless to say, on the ARM7 side of things the sound handling stuff has completely changed - this alone is a good enough reason to abandon any hand-rolled arm7 code. Added to the new sleep function - no more will-it-wont-it wondering when you close the lid - and all in all I think this is a great release.

Things you never wanted to know about assembler but were too afraid to ask

2008-12-01T21:41:00.003+01:00

When you write homebrew for the Nintendo DS, or the Game Boy Advance, you can choose to use a friendly language like C or C++. This makes life a bit easier. It's comfortable, familiar. But at the back of your mind there's this nagging doubt... Shouldn't I be writing some of this in assembler? After all, if I'm writing C code, I may as well code for a PC and use SDL or Allegro. But then you think... Assembler, that sounds a bit tricky, I wouldn't know where to begin!

And that's where this post comes in.

Getting started with assembler is probably the hardest part. There's aren't many articles on best practices, what not to do, which idioms to use, and so on, as there are for other languages. Many of the ideas you have about C can't be applied to assembler, or at leas that's what you might think at first, so you probably brush it off as impossible. This is a bit sad, because these Nintendo consoles use an ARM processor and ARM assembler has a nice syntax. Unlike assembler for Intel processors, say, the ARM syntax is fairly small, has few real surprises and can be learned quite easily.

Now, I don't want to claim that I'm some assembler guru. I've written about 6,000 lines of ARM for Pocket Beeb and Elite AGB, plus some in a current "top secret" project I'm working on. It's not really that much, especially compared to the millions of lines of C and C++ I've probably written. But it is a good enough amount to get a feel for how to code in this way, I think.

Before we dive in, let's see at how we go from C to "the magic" that runs on a DS. The first step is to write a function in C. We then use a compiler, which converts this to a file in the ELF format. This binary file is joined together with all the other compiled modules in your program, and the output is a "pure" binary blob with some header information. So that's C to object file, then object files to NDS "ROM" file.

Let's rewind to those first steps. The C function is compiled to an ELF file, which typically ends in ".o", for "object file". This step actually skips out quite a lot. You can see the intermediate files involved by passing --save-temps to your compilation step. Lets try this. Here's a useless function that does some pointless thing. We pass in a couple of variables, and add or subtract them depending on their relative values.


int my_function(int x, int y) {
  if (x < y) {
    return x + y;
  } else {
    return x - y;
  }
}

If we compile this as so: "arm-eabi-gcc --save-temps -c -o my_function.o my_function.c" then the output will be 3 files. These are my_function.o (as expected), my_function.i and my_function.s. The .i file is the post-processed output - what happens after running the C preprocessor on the file. This finds-and-replaces any #defines, pastes in any #include'd files and adds in compiler-specific information. The .s file is what we are after. This is the output of the C code converted to ARM assembler. It contains a lot of "cruft" that you wouldn't write if you were to write this function by hand, but some of the features are important:


	.text
	.align  2
	.global my_function
	.type   my_function, %function
my_function:

The lines that start with a dot are called directives. They are psuedo-instructions that tell the assembler to do extra things.

The .text directive adds headers to let the linker know where abouts the code should be placed. It's not too important on the DS, more so on the GBA, where if you omit it then the code may be placed in RAM. You can also force code to go into fast iwram on the GBA by using the directive .section .iwram,"ax",%progbits. Don't ask what it means, just use the voodoo.

The .align directive ensures that our code is padded to 4 byte boundaries. The "2" means the number of bits that must be zero in the location counter at this point. So 2 bits signifies aligned by 4 bytes. Since ARM assembler instructions are 32 bits big (4 bytes) we need to align the code to 4-byte boundaries, or Very Bad Things can happen.

The .global directive makes the symbol name (my_function) visible outside the compilation unit, so you can use it in other parts of your program. Without .global, the symbol would be the equivalent of using "static" in a C definition.

The .type directive is pretty pointless, but I tend to use it to mark what are functions and what are just helper "scraps" of assembler with a named label. It's use is supposed to be for interoperability with other assemblers, but we always use GCC from devkitPro so its a moot point.

Finally, "my_function:" is a label - this is like a goto label in C and is where the program jumps to when we call this function.

The rest of the ARM code spit out by GCC is as good as could be for this example, if you compile with -O2. Normally, the advantages of assembler tend to be minimal for small functions, they are better when you do things that can't be easily done in C. Such as unrolling loops, keeping often-used memory addresses in a single register, that kind of thing. Also, hand coded ARM assembler can make better use of the registers in some cases, more on registers in a minute, but GCC assembler tends to make more use of the stack. This tends to manifest itself only in more complex functions. Anyway, lets just say that it'd complicate matters needlessly to copy paste the rest of the code here, and for this example GCC does a good job.

Registers

I mentioned the registers back there. The ARM processor in the NDS and GBA has 16 registers - usually named r0, r1, r2... up to r12, then sp, lr, pc instead of r13, r14 and r15. I say usually, because there is an alternate naming scheme that some documents use. Here certain registers are given special names:


  std | alt
 -----+-----
  r0  | a1
  r1  | a2
  r2  | a3
  r3  | a4
  r4  | v1
  r5  | v2
  r6  | v3
  r7  | v4
  r8  | v5
  r9  | v6
  r10 | v7 or sl
  r11 | v8 or fp
  r12 | ip
  r13 | sp
  r14 | lr
  r15 | pc

It's probably best to use the r0-r12,sp,lr,pc naming scheme as that is what the ARM documentation uses. The trick is to pick one scheme and stick to it. sp is a mnemonic for "stack pointer" and points to, you guessed it, the stack. lr means "link register" and is used as the return address for function calls. pc is the program counter and is the address of the current instruction (i.e. where we are in the program).

The stack is an area of RAM. Initially it points to the end of the RAM area and grows downwards as programs allocate memory (on the stack). When writing assembler I find it's not that usual to use the stack - generally I only use it for storing register values that I may need later, such as the lr when calling functions recursively.

Code set up

Remember how gcc spat out a .s file? Well, that means "pure assembler code". The common way to store ARM code is in a file ending in ".S". This is interpreted by gcc as a source file containing assembler and pre-processor directives. So here we can use #defines and #includes if we really need to. It also distinguishes hand coded assembler files from any --save-temps left-overs.

Lets try writing our my_function in assembler. First I fire up an $EDITOR and write out the preamble to my_function.S. Now lets think about what we want. This function has the following C prototype:

int my_function(int, int);

So that means it accepts 2 input values and returns a value. The input values in ARM are in the first 4 registers - any more than 4 inputs are stored on the stack. The original function wanted an addition if x was less than y, else it subtracted y from x. Here's my final code to do this:


	.text
	.align  2
	.global my_function
	.type   my_function, %function
my_function:
	cmp	r0, r1
	addlt	r0, r0, r1
	subge	r0, r0, r1
	bx	lr

The header is as we saw before. The my_function label is the start of the function. The r0 and r1 registers contain the input values.

The cmp instruction means "compare r0 to r1 and set the flags accordingly"... now I don't really want to copy out the whole ARM assembler guide here, but suffice to say that this sets some flags based on the mathematical result of performing "r0 - r1". These tests are in the form of those "lt", "ge" that follow the end of the other instructions. This is one of the neat parts about ARM - every instruction is conditional!

The default condition is "al" for "always", but may be omitted for obvious reasons. We don't want to write al after every instruction, right? So here we say, if r0 is less than (lt) r1, then add r0 to r1 and store the result in r0. A normal add would just be "add", but here we write "addlt". This makes the processor skip over the instruction if the "less than" state is not set. It's important to note that conditional instructions like this still have some overhead, the processor still has to read the operation in some way, so if you have more than say 3 or 4 conditional instructions with the same condition, it may be faster to use a branch to skip the code completely.

If the result was greather than or equal (ge) then the subtraction is done and the result stored in r0, followed by a return. The bx instruction is used to return from a function - it means "branch to the address in the register lr". As you see, we use the link register to return to the calling code.

Finally, the result is always stored in r0 in this example. This is part of the ARM binary interface - results are returned in registers r0 and r1. This convention allows C and ARM code to interoperate sensibly. If you are returning to your own ARM function, then you can invent any convention you like - return values in r4, r7 and r11 if you want - but sticking to the "C way" means that you can use the function from C later, if that becomes necessary.

You may have noticed that addition and subtraction seem to work "backwards" compared to C convention. Instead of reading left-to-right and the result being stored at the end, the result is actually stored in the first register given. This is fairly common in other flavours of assembler code too, and once you get used to it, it isn't so bad.

We can write a simple test program to run this code:

#include <nds.h>
#include <stdio.h>

int my_function(int x, int y);

int main(void) {
 consoleDemoInit();
 int x = 1;
 int y = 2;
 iprintf("my_function(%d, %d) returns %d\n", x, y, my_function(x, y));
 x = 100;
 y = 20;
 iprintf("my_function(%d, %d) returns %d\n", x, y, my_function(x, y));

 while(1) { swiWaitForVBlank(); }
 return 0;
}

This prints out the results on the DS screen. Nothing very exciting, but you can see that combining C and ARM assembler in a project is not as difficult as you first thought!

Tips and tricks

There are quite a few traps for the unwary - apart from the shift in thought process needed to code assembler, of course. The first of these is that loading a value into a register requires some thought. You can only load values with "mov" that are shifted 8 bit values - 0xff0, 0x1c00, but not 0x101, for example. There is a psuedo opcode to get around this easily - "ldr r1, =0x101" for example - but it may bite you if you didn't know this. The error would look like this "Error: invalid constant (101) after fixup", just in case you were wondering.

When you write ARM, always try to maximise what each instruction does! Shifts can be added onto the end of instructions, so often instead of a shift then logical orr, you can do the lot in one go:

orr r3, r0, r1, lsl r2

This is the same as r3 = r0 | (r1 << r2) all in one instruction! Similarly you can load data from structures using offsets with ldr, with "ldr r0, [r1, r2, lsl #2]". That means r0 = r1[r2*4] - useful for loading 32-bit values from structures into registers. Remember that in assembler pointer arithmetic works as if all pointers were to char, there are no data types. Adding 1 to a pointer really just adds 1 to it, not like in C where it can increase by the size of the data type.

Generally, make use of all the registers you have available and avoid pushing/popping to and from the stack. You can use r0-r12 however you want, make the most of these and you'll find some nice shortcuts that provide speedy, optimised code. Hopefully. If you need to push multiple registers to the stack, then the idiom to use is the stmfd/ldmfd one (store/load multiple full descending). This stores a comma seprated list and/or range of registers to the stack, and decrements/increments the stack by the right amount. It is written as follows, to store here registers r0 to r7 and lr (r14):


	stmfd sp!,{r0-r7, lr}
... do stuff with registers r0-r7 and lr ...
	ldmfd sp!,{r0-r7, lr}

Always try and reduce duplication - not only is it a maintenance problem, but each copied line is a wasted cycle. Try and reduce code to the minimum number of instructions, that's the name of the game here. Ah, but remember: get a working version first, then optimise heavily. No point having a fast bit of code that doesn't do what you want :-)

I'd recommend this PDF cheat sheet for a quick guide to the ARM instruction set. Keep in mind that the DS doesn't support all the instructions on there - rbit, bfc, and some others - but they are generally the more exotic operations that are used less. If in doubt, compile a quick test file. You'll get the error "Error: selected processor does not support..." if the instruction is not supported by the CPU.

Conclusions.

So why would you want to do all this? Isn't this just for head cases?

I think learning how to write code at this level leads to a better understanding of why things are done in certain ways in high level languages. You'll certainly grok pointers, if you haven't already. Learning new coding styles - not just using C-like languages - will make you a better coder too. Heck, maybe a better person even ;-)

Recovering history with git reflog

2008-11-07T20:14:00.006+01:00

I use git-svn quite a bit at work. One of the nice things about this is that I can create perfect patch sets - instead of commit a whole raft of random chunks, I can polish a change until it works. Quite often I'll reset the HEAD to a few commits back, to get rid of changes that add crap like "printf" or were just bug-fix dead ends. You know, the usual cargo cult "I'll change this to see what happens", that once you understand the problem realise had nothing really to do with the root cause of the problem.

One of the side effects of this is that, after a while I tend to end up with lots of "dead" commits. While using gitk I can see these as little commit balls that aren't joined to anything, like the headless branchy thing in the image. If I close gitk, or click the "Reload" option then the commit tree is cleaned up to show only commits that have a head.

Occasionally, once I've finished a change, squashed all my commits and committed them to subversion, it turns out that one of those lost commits really should have been part of the patch set. Oops! I've closed gitk by now, so all those lost commits can't be seen. How can I get them back? Here's where git reflog comes to the rescue.

I didn't discover the wonders of git reflog until quite recently. Its man page is not too helpful in describing what it does: "Manage reflog information". What it actually does is print out a list of commits that are in a branch, or are in limbo. It doesn't distinguish between the two. So you can use it, together with gitk, to show your lost commits. For example I use this:

gitk --all `git reflog | cut -c1-7`

That's it. It shows all branches (--all) and all the commits in the reflog, so all those lost commits are reachable, cherry-pickable and branchable again. I just wanted to share that with the internet, because I know I'll forget it by next week :-)

Alternatives to Make, Part II

2008-10-13T20:22:00.006+02:00

Last time I described how somebody insane enough could use CMake for their cross compiling needs. This entry I'll consider the next entry on the list of interesting Make replacements: SCons.

SCons

When using SCons as your build system you have 2 options: install scons on the system or include a version in your source tree. The easiest way is to install a version of scons on your system, and mark your build scripts that they require that version or newer of scons to run. This does put a burden on end users as they must appropriate their own copy of SCons, but it means your repository of code isn't full of 3rd party stuff. Additionally, the machine doing the compiling must have Python installed. On Linux this is no problem, but on Windows it's Yet Another Dependency. This is because SCons, unlike CMake, uses an already-existing programming language to describe the build process. In this case, Python. The build scripts are called "SConstruct" and, apart from some oddities, work like normal Python modules.

So on to the cross compiling issue - compiling our C or C++ code into a DS program. Unlike CMake, there's no set way to handle cross compilation. In fact, there are no real "best working practices" outlined anywhere in the (quite extensive) SCons documentation - you're free to do what you like. My recommendation here is to take a leaf out of CMake's book and separate out as much as possible into a toolchain file.

As with the CMake example, let's start off with the ansi_console example from devkitPro. This has a source sub directory and a main.c file. First, create a SConstruct file that will read in a toolchain called "arm":

env = Environment(tools='arm', toolpath='.')

The toolpath means "look for the file in the current directory". The toolchain file is a normal python module. This means a file called "arm.py". In order to make it a true SCons tool file we need to add 2 functions, "generate" and "exists". So that would be:

def generate(env, **kwargs):
   pass

def exists(env):
 return 1

I get the feeling that exists() is not actually used - it certainly isn't in SCons 1.0.1, but the documentation says it is required. The generate function is called when we create the Environment, passing the Environment into the generate function so we can mess about with the innards. So lets do the usual setup, which is to check DEVKITARM and DEVKITPRO are set in the user's environment. SCons by default doesn't pull all the environment variables into its own Environment object. This is good, as we don't really want a build to depend "randomly" on some rogue variable. But! we do want to use some of them. Anyway, on to the code:

from os import environ
from os.path import pathsep,join
def check_devkit(env):
   ENV = env['ENV']
   for var in ('DEVKITARM', 'DEVKITPRO'):
       if var not in environ:
           print 'Please set %s. export %s=/path/to/%s'%(var, var, var)
           env.Exit(1)
       ENV[var] = environ[var]
   ENV['PATH'] = ENV['PATH']+pathsep+join(environ['DEVKITARM'], 'bin')
   if not find_devkitarm(env):
       print 'DevkitARM was not found'
       env.Exit(1)

def generate(env, **kwargs):
   check_devkit(env)

That is pretty straight forward - it checks the environ(ment) and adds $DEVKITARM/bin to the path. Now we need to find out if we have a suitable compiler. Here things get a bit icky. Because SCons isn't a priori designed for cross compilation, it assumes that your GNU-based compiler is called just "gcc". This means that, in order to X-compile, you'll need to install gcc, since we use the gcc tool detection and environment set up and overwrite parts with our arm-eabi-gcc. A bit of an irritating flaw, and one which CMake has understood and implemented correctly. The alternative, of course, is to copy paste the entire SCons gcc tools and replace gcc with arm-eabi-gcc - or suitable prefix. In fact, this alternative approach may well work out better... anyway. For now we'll use the approach that assumes vanilla gcc and overwrites with arm-eabi.

def setup_tools(env):
   gnu_tools = ['gcc', 'g++', 'gnulink', 'ar', 'gas']
   for tool in gnu_tools:
       env.Tool(tool)
   env['CC'] = prefix+'gcc'
   env['CXX'] = prefix+'g++'
   env['AR'] = prefix+'ar'
   env['AR'] = prefix+'as'
   env['OBJCOPY'] = prefix+'objcopy'
   env['PROGSUFFIX'] = '.elf'

def generate(env, **kwargs):
   check_devkit(env)
   setup_tools(env)

So that sets up the environment for compiling, then overwrites the tool names with the arm-eabi equivalent. We also set the PROGSUFFIX (program suffix) to .elf - this makes life easier for the objcopy step. Now we need the "magic flags" that cause our Nintendo DS program to compile.

def add_flags(env):
   # add arm flags
   env.Append(CCFLAGS='-march=armv5te -mtune=arm946e-s'.split())
   env.Append(CPPDEFINES='ARM9')
   env.Append(LIBS='nds9')
   env.Append(LINKFLAGS=['-specs=ds_arm9.specs'])
   # add libnds
   libnds = join(environ['DEVKITPRO'], 'libnds')
   env.Append(LIBPATH=[ join(libnds, 'lib')])
   env.Append(CPPPATH=[ join(libnds, 'include')])

def generate(env, **kwargs):
   check_devkit(env)
   setup_tools(env)
   add_flags(env)

These flags are the usual ARM9 flags and libnds include path for compiling C code, and libnds9 and the specs flag when linking. Without these devkitArm complains (this is as expected, it is a multi-platform ARM compiler, so you need to tell it the exact platform you want to use). There's one more step here - we need a way to build the nds file from an .elf file - but first lets go back to the SConstruct. If you recall, at the top of the post there, we had just created an Environment. Now we have added a load of ARM stuff to that Environment. The next step here is to create a program. In SCons we can do this as follows:

env.Program('ansi_console', os.path.join('source','main.c'))

That's it! Except, no it isn't. This produces ansi_console.elf, which needs to be objcopy'd and ndstool'd. To do that we can go back to our arm.py tool file. Here we add in new "Builders" to do the work. A Builder is like the Program method we saw in the SConstruct - it takes the names of the source files and produces the outputs... so we add in the builders to our arm.py file:

def add_builders(env):
   def generate_arm(source, target, env, for_signature):
       return '$OBJCOPY -O binary %s %s'%(source[0], target[0])
   def generate_nds(source, target, env, for_signature):
       if len(source) == 2:
           return "ndstool -c %s -7 %s -9 %s"%(target[0], source[0], source[1])
       else:
           return "ndstool -c %s -9 %s"%(target[0], source[0])
   env.Append(BUILDERS={
       'Ndstool': SCons.Builder.Builder(
                   generator=generate_nds,
                   suffix='.nds',
                   src_suffix='.arm'),
       'Objcopy': SCons.Builder.Builder(
                   generator=generate_arm,
                   suffix='.arm',
                   src_suffix='.elf')})
def generate(env, **kwargs):
   [f(env) for f in (check_devkit, setup_tools, add_flags, add_builders)]

That's a fair amount of typing! What does it do? Well, the "generate_arm" function is called when we want to generate an .arm file from an .elf file. It returns a SCons-y string that will be executed to do the actual work. Here it is our old OBJCOPY string - the $OBJCOPY is replaced automagically by SCons with the equivalent Environment variable. The "generate_nds" function is called when we want to generate an .nds from an .arm, it too returns the command line that will be executed. There's a bit of a trick there that checks if we need to combine an ARM7 and ARM9 core, or just use the default ARM7, but apart from that it is straightforward. The "env.Append(BUILDERS=...)" bit creates new functions called Ndstool and Objcopy that can be used like Program. Passing in a generator function means we use functions - you could use a fixed string and pass it as an action too.

Armed with our new methods, lets go back to the SConstruct. We can objcopy and ndstool the Program/elf file as follows:

env.Ndstool(env.Objcopy(env.Program('ansi_console', join('source','main.c'))))

That's it. There are a couple of things we can do to make this better. For example, rather than splurge the build artifacts all over the source tree, we can use a build directory. To do this we need to use SCons recursively. The Recursive file is called a "SConscript", and all we have to remember is that to pass objects (like the env created in the top level SConstruct) down to the other SConscripts, we have to use an Export/Import mechanism. A bit confusing, but the code's easy enough:

#in SConstruct
env = Environment(tools=['arm'], toolpath=['.'])
SConscript(join('source','SConscript'),
           variant_dir='build',
           duplicate=0,
           exports='env')

# in source/SConscript
Import('env')
env.Ndstool(env.Objcopy(env.Program('ansi_console', 'main.c')))

Passing exports to the SConscript file exports the named variables. Using Import imports them into the local namespace. Magical! There's also a Return() function to return variables from sub scripts. That's usefull when tracking local library dependencies.

OK, so what about "dual core" programs? It turns out that it is a bit of effort. More even than CMake, I'd say. We can either create a second Environment and set up the variables for ARM9 or ARM7 according to the core in question, or use a single Environment instance, set flags that should be used for ARM9 or ARM7, and use these appropriately for each Program call that we make. The first approach ends up a bit messy, with ifs for processor type in several places. The second approach is cleaner, but means there is a bit more code used when calling env.Program. I use the latter approach in Elite DS, so that's what I'll outline here. You have to set the compiler flags for arm9 and arm7 in the env variable. This can be done as follows, modifying the add_flags() function of our arm.py tool:

THUMB_FLAGS = ' -mthumb -mthumb-interwork '
PROCESSOR_CFLAGS = {
 '9': ' -march=armv5te -mtune=arm946e-s',
 '7': ' -mcpu=arm7tdmi -mtune=arm7tdmi'
}

PROCESSOR_LDFLAGS = THUMB_FLAGS + \
       ' -specs=ds_arm%c.specs -g -mno-fpu -Wl,-Map,${TARGET.base}.map  -Wl,-gc-sections'
EXTRA_FLAGS = ' -Wno-strict-aliasing -fomit-frame-pointer -ffast-math '

def add_flags(env):
   ccflags = ' '.join([EXTRA_FLAGS, THUMB_FLAGS])
   CCFLAGS_ARM9 = ' '.join([ccflags, PROCESSOR_CFLAGS['9']])
   CCFLAGS_ARM7 = ' '.join([ccflags, PROCESSOR_CFLAGS['7']])
   CPPDEFINES_ARM9 = 'ARM9'
   CPPDEFINES_ARM7 = 'ARM7'
   LIBS_ARM9 = ['fat', 'nds9']
   LIBS_ARM7 = ['nds7']
   LINKFLAGS_ARM9 = PROCESSOR_LDFLAGS%'9'
   LINKFLAGS_ARM7 = PROCESSOR_LDFLAGS%'7'

   env.Append(CCFLAGS_ARM9=CCFLAGS_ARM9)
   env.Append(CCFLAGS_ARM7=CCFLAGS_ARM7)
   env.Append(CPPDEFINES_ARM9=CPPDEFINES_ARM9)
   env.Append(CPPDEFINES_ARM7=CPPDEFINES_ARM7)
   env.Append(LIBS_ARM9=LIBS_ARM9)
   env.Append(LIBS_ARM7=LIBS_ARM7)
   env.Append(LINKFLAGS_ARM9=LINKFLAGS_ARM9)
   env.Append(LINKFLAGS_ARM7=LINKFLAGS_ARM7)

   # add libnds
   libnds = join(environ['DEVKITPRO'], 'libnds')
   env.Append(LIBPATH=[ join(libnds, 'lib')])
   env.Append(CPPPATH=[ join(libnds, 'include')])

def generate(env, **kwargs):
   [f(env) for f in (check_devkit, setup_tools, add_flags, add_builders)]

Most of those flags are taken from the devkitPro examples and should be pretty familiar. In order to actually use them, we can override the flags used for each individual program. So in the SConscript, the env.Program line would become:

env.Program('ansi_console', os.path.join('source','main.c'),
       CCFLAGS=env['CCFLAGS_ARM9'],
       CPPDEFINES=env['CPPDEFINES_ARM9'],
       LIBS=env['LIBS_ARM9'],
       LINKFLAGS=env['LINKFLAGS_ARM9'])

More typing, but there's no need to pass extra Environments about. The equivalent for an ARM7 program would obviously replace the 9's for 7's.

As before with my CMake entry, here is a quick summary to help you decide whether or not to use SCons as your build system for cross compiling NDS programs.

The Bad

Scalability: I haven't mentioned this yet, but the biggest problem SCons has is that when you reach several hundred source files, the time that it takes to parse the SConscripts and generate dependencies becomes considerable. If you also use SCons autoconf-like Configure functions, then the configuration step is, by default, run each and every time you compile. The results are cached, but it takes several seconds to go through the tests. This became an issue for me on Bunjalloo, which used to use SCons. Before compiling anything SCons would typically sit around in silence for 15 seconds contemplating the build. I've seen builds that do "nothing" for minutes at a time that, when changed to CMake or whatever, did similar thinking time in seconds. KDE and OpenMoko are two famous SCons "deserters" due to performance problems. On the other hand for Elite DS, which only has around a hundred source files, the do-nothing time is negligible and SCons works great.

Not really cross-compiling friendly: Unlike CMake, SCons is not built with cross compiling in mind. This is demonstrated by the hard-coding of "gcc", etc, in the gcc.py SCons-tools. This means you probably will have to install native tools as well as the cross tools in order to compile anything. (Note: the new 1.1.0 may have fixed this, it was released as I was writing this post!)

Syntax can get complicated: When writing scripts that use multiple local libraries, the Return() and Import/Export() mechanisms that SCons uses can get a bit unwieldy. You end up with lots of global variable names that you have to Import() and Export() across SConscripts, or else Return() everything to the parent file and let it sort out the mess.

Python can look intimidating: Unlike CMake, which still looks a bit Makefile-like, SCons scripts can have any old Python code in it. Without discipline, this can result in build scripts that are difficult to follow, especially if Python is not one of your strong languages.

Dependencies: As with CMake, you still need to install stuff - namely the SCons build system itself. As more projects use scons this will become less of a problem, but at the moment it can be annoying for users ("oh no! where is my Makefile?")

The Good

Python code: If you are familiar with Python already, you don't need to learn yet another language. There are no hacks to get loops working. Proper data structures can be used to describe the build. Any of the Python libraries can be imported and used. This is a very powerful advantage, as well as being multi-platform (providing you stick to cross platform safe python code, of course).

Stable API: SCons is backwards compatible with the practically stone-age Python 1.5.2 and its APIs for building things changes infrequently, first deprecating functions and rarely removing anything. This makes changing from an older version to a newer one fairly painless. A full suite of tests means that regressions on new versions are pretty rare - if something goes wrong, it is likely to be due to a bug in your SConscript, rather than a bug in SCons.

Great documentation: There is loads of good documentation on the SCons website. The manual is particularly well done, with an easy step-by-step guide. Once installed, a very complete man page is installed (on Linux, FreeBSD, etc at least) that contains examples as well as the full API.

Conclusion

As with CMake, you probably need a good reason to not use a bog-standard Makefile. For me, being able to code the build logic in Python is the deciding factor. I think SCons is pretty good, and use it for Elite DS. I'd probably use SCons for other projects I do too, depending on their size (and my mood). The Bunjalloo build became a bit too slow with SCons, but I'll talk more about that next time when I discuss Waf. There seems to be quite an uptake of SCons for new OSS projects (Google's Chrome browser for one), especially in favour of autotools, and hopefully we'll see some improvement in the speed because of this (the latest release, 1.1.0, boasts improved memory use, but I have yet to see any benchmarks). Until then, you could probably do a lot worse than download the latest release and have a mess around to see what it's all about.

DSi and thoughts on the future

2008-10-06T20:25:00.003+02:00

Nintendo have announced a new version of the DS, calling it the DSi. No idea what the i means, a nod at the good work Apple are doing presumably. But what does this mean for Bunjalloo development?

I think it is time to call it a day.

The thing is, this new console includes a web browser and, I assume, it will kitted out with a lot more RAM than the current-gen DS. Pure speculation, but you can't do much with 4 megabytes nowadays. Especially on the ever-expanding internet. While the actual DS can also run a full-blown browser, if you bought Opera, having one by default is the killer. Especially if it is free (not Free, but free "as in beer") and doesn't need a RAM expansion pack. Additionally, the rumour mill is already speculating about the security upgrades on the new DSi, like it'll have a software black list to disallow homebrew carts, upgradable firmware like the PSP and so on. I don't think anyone actually buys DS games anymore. Everyone I've seen IRL with a DS has a flash cartridge thingy of some sort. It just makes economic sense, really. So Nintendo will want to end this rampant piracy with the next gen console, and at the same time make it pretty tricky for normal folk to run homebrew games for a while. At least until a new hack comes out and the cartridge manufacturers take it mainstream. Oh, and region locking a handheld? Grrr...

Anyhow, that doesn't mean I'll give up on Bunjalloo completely, but it will enter Maintainance Mode, where only bug fixes will go in with no new stuff. See? You should have sent me patches to add those cool features, duh! There's a new release out now, 0.7.1, that fixes some of the bugs in 0.7, just to show I'm still hacking around.

Alternatives to Make, Part 1

2008-09-27T00:05:00.004+02:00

In the Nintendo DS scene the build tool that everyone uses is plain ol' Make. There are several alternatives for the brave developer, however. I'm going to write a few posts describing how you can use CMake, SCons or Waf to compile your code. Yes, I'm a bit of a build system junkie. This week viewers we'll look at CMake.

CMake

CMake is a build system that generates files for use with the build tool on your platform. You write a configuration file, called CMakeLists.txt, run the cmake program to generate a load of Makefiles or project files, then run make (or whatever) to compile your project. On Windows, CMake generates project files for use with Visual Studio. It can also generate project files for CodeBlocks, Eclipse CDT and KDevelop.

A hefty obstacle for using CMake with the DS is that currently CMake doesn't support building multiple target architectures in one build tree. So compiling ARM7 and ARM9 cores and combining the result into the final .nds file requires a bit of fiddling. More on that later, as it can be done.

The first thing you will need is the latest version of CMake. The features needed to compile on the DS were only added quite recently (version 2.6 onwards).

After you've installed CMake, you can start hacking away. First we need to create a Toolchain File that CMake uses to understand your target platform. Normally CMake autodetects the local platform type and configures the compiler, linker, compilation flags and so on, to the correct values. When cross compiling for the DS, you have to tell CMake which compiler it should use. We don't want it to try and guess based on any native compiler installed.

As described in the CMake wiki we first have to turn off the auto detection. This is done by setting the variable CMAKE_SYSTEM_NAME to "Generic". Here we come across one of the oddities that takes a bit of getting used to. Instead of the usual BNF variable = value style syntax for assignment, CMake uses a function. So we have set(variable value). Whatever. The other 2 variables set here are optional:

set(CMAKE_SYSTEM_NAME Generic)
set(CMAKE_SYSTEM_VERSION 1)
set(CMAKE_SYSTEM_PROCESSOR arm-eabi)

A note on style here. Commands like if, message, endif, set and so on can be written either in ALL CAPS or in lower case. The keywords inside commands have to be in UPPER CASE. I prefer to write the commands in lowercase and everything else in upper case.

So, after those first set() lines we begin to describe which compiler we'll use. The approach I'm going to use doesn't stray from the standard DEVKITARM and DEVKITPRO environment variables, which should point to where you have the devkitARM toolchain and libnds installed. We set 2 CMake variables based on environment variables as follows:

set(DEVKITARM $ENV{DEVKITARM})
set(DEVKITPRO $ENV{DEVKITPRO})

Actually this is quite good - it means that CMake by default doesn't drag the whole environment into the namespace. This means there is a better chance of our build being repeatable without having to set up a million environment variables. After this, we can check if they are set with these lines of code:

if(NOT DEVKITARM)
 message(FATAL_ERROR "Please set DEVKITARM in your environment")
endif(NOT DEVKITARM)

And similarly for DEVKITPRO. So now we need to check if the compiler is installed. This is done by setting the CMAKE_C(XX)_COMPILER variables as follows. Also we set the CMAKE_FIND_ROOT_PATH, which lets us use objcopy, ar, ranlib and friends. The other flags tell CMake to just use the compiler headers and libraries, to not search DEVKITARM for other libs and headers. Really we could create an install base for the DS, with lib, include and so on, but that is not the usual way to do things. It would probably break a lot of other hand-coded Makefiles that people use, so we'll stick with the $DEVKITPRO/libnds convention. Anyway, here is the code for setting up the compilers:

set(CMAKE_C_COMPILER ${DEVKITARM}/bin/arm-eabi-gcc)
set(CMAKE_CXX_COMPILER ${DEVKITARM}/bin/arm-eabi-g++)
set(CMAKE_FIND_ROOT_PATH ${DEVKITARM})
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)

Here if DEVKITARM points somewhere that doesn't really have the arm-eabi compiler installed, then CMake will throw up an error message when we run it:

CMake Error: your C compiler: "/path/to/devkitARM/bin/arm-eabi-gcc" was not found.   Please set CMAKE_C_COMPILER to a valid compiler path or name.
CMake Error: your CXX compiler: "/path/to/devkitARM/bin/arm-eabi-g++" was not found.   Please set CMAKE_CXX_COMPILER to a valid compiler path or name.

Apart from the compiler, we also need to indicate where libnds lives. This is done by telling CMake how to construct a library name - prepending lib, and adding .a at the end - and where to search for the library and headers:

set(CMAKE_FIND_LIBRARY_PREFIXES lib)
set(CMAKE_FIND_LIBRARY_SUFFIXES .a)
include_directories(${DEVKITPRO}/libnds/include)
link_directories(${DEVKITPRO}/libnds/lib)
find_library(NDS9 nds9)
find_library(NDS7 nds7)

OK, that's it. The rest is now "vanilla" CMake. When we run the cmake program we need to tell it to use cross compiling. This is done with the CMAKE_TOOLCHAIN_FILE variable. We can set it with a gcc-like "-D" flag:

cmake -DCMAKE_TOOLCHAIN_FILE=/full/path/to/devkitArm.cmake .

So now we have this, we can take an example from the nds-examples that are distributed with devkitArm and try to have it compile using CMake. Let's use the ansi_console example. Unzip the examples, and copy ansi_console to some place - it's in the Graphics/2D directory. Now create a file ansi_console/CMakeLists.txt and start editing it. The first thing to do here is to give the project a name:

project(NDS-EXAMPLES)

In order to compile correctly when using libnds, we need to set a C define ARM9. For this, we use the add_definition macro. I'll also create a variable EXE_NAME that we can use instead of copying ansi_console everywhere. This way if we want to change the name of the binary later it won't be too much hassle.

add_definitions(-DARM9)
set(EXE_NAME ansi_console)

Now we can describe how to build the "executable". The next line tells CMake to create an executable from the file source/main.c

add_executable(${EXE_NAME} source/main.c )

That alone isn't enough, as we also need to link with libnds for the ARM9 and pass in the required -specs flag.

target_link_libraries(${EXE_NAME} nds9)
set_target_properties(${EXE_NAME}
 PROPERTIES
 LINK_FLAGS -specs=ds_arm9.specs
 COMPILER_FLAGS "-mthumb -mthumb-interwork")

The set_target_properties macro allows us to fiddle with the compilation and link flags. A bit later, if you decide to use CMake for your NDS projects, you could make a wrapper around this to set up the usual flags for a NDS binary, rather than copying the above lines everywhere.

So to here this would normally be enough on a platform that has an operating system and knows how to load the executable file format. We could run cmake, passing in the devkitArm toolchain file and the path to the ansi_console directory. The DS doesn't have an OS however and we need to strip out all the "elfyness" - the ELF headers and whatnot -, and add the special DS header. Converting the ELF to a pure binary is done with objcopy. Adding the header is done with ndstool, provided as part of the devkitArm distribution. We can write macros to help with these steps, and that allows me to write this kind of thing to finish off the process:

objcopy_file(${EXE_NAME})
ndstool_file(${EXE_NAME})

So where do these come from? Well, we have 2 choices. Either add the macro inline in the CMakeLists.txt file or create a module. The first is easier, but the second approach lets us reuse the module elsewhere. So in a new file called ndsmacros.cmake I'll add the following lines to define the macros OBJCOPY_FILE and NDSTOOL_FILE

macro(OBJCOPY_FILE EXE_NAME)
 set(FO ${CMAKE_CURRENT_BINARY_DIR}/${EXE_NAME}.bin)
 set(FI ${CMAKE_CURRENT_BINARY_DIR}/${EXE_NAME})
 message(STATUS ${FO})
 add_custom_command(
  OUTPUT "${FO}"
  COMMAND ${CMAKE_OBJCOPY}
  ARGS -O binary ${FI} ${FO}
  DEPENDS ${FI})
 get_filename_component(TGT "${EXE_NAME}" NAME)
 add_custom_target("TargetObjCopy_${TGT}" ALL DEPENDS ${FO} VERBATIM)
 get_directory_property(extra_clean_files ADDITIONAL_MAKE_CLEAN_FILES)
 set_directory_properties(
  PROPERTIES
  ADDITIONAL_MAKE_CLEAN_FILES "${extra_clean_files};${FO}")
 set_source_files_properties("${FO}" PROPERTIES GENERATED TRUE)
endmacro(OBJCOPY_FILE)

if(NOT NDSTOOL_EXE)
 message(STATUS "Looking for arm-eabi-objcopy")
 find_program(NDSTOOL_EXE ndstool ${DEVKITARM}/bin)
 if(NDSTOOL_EXE)
  message(STATUS "Looking for arm-eabi-objcopy -- ${NDSTOOL_EXE}")
 endif(NDSTOOL_EXE)
endif(NOT NDSTOOL_EXE)

if(NDSTOOL_EXE)
 macro(NDSTOOL_FILE EXE_NAME)
  set(FO ${CMAKE_CURRENT_BINARY_DIR}/${EXE_NAME}.nds)
  set(I9 ${CMAKE_CURRENT_BINARY_DIR}/${EXE_NAME}.bin)
  add_custom_command(
   OUTPUT ${FO}
   COMMAND ${NDSTOOL_EXE}
   ARGS -c ${FO} -9 ${I9}
   MAIN_DEPENDENCY ${I9}
   )
  get_filename_component(TGT "${EXE_NAME}" NAME)
  add_custom_target("Target9_${TGT}" ALL DEPENDS ${FO} VERBATIM)
  get_directory_property(extra_clean_files ADDITIONAL_MAKE_CLEAN_FILES)
  set_directory_properties(
   PROPERTIES
   ADDITIONAL_MAKE_CLEAN_FILES "${extra_clean_files};${FO}")
  set_source_files_properties(${FO} PROPERTIES GENERATED TRUE)
 endmacro(NDSTOOL_FILE)
endif(NDSTOOL_EXE)

The first macro that defines OBJCOPY_FILE uses the built-in CMake command "add_custom_command". This has lots of options, but the ones used here say that cmake should run the CMAKE_OBJCOPY command passing in the given arguments (ARGS). CMAKE_OBJCOPY is defined automagically from our CMAKE_FIND_ROOT_PATH in the devkitArm toolchain file, it corresponds to arm-eabi-objcopy in the devkitArm installation. The "add_custom_target" isn't really needed here, but we will need it later for combined arm7/9 cores. It adds a new top level target called TargetObjCopy_[name of exe] that depends on the out file to the "all" target, so when we run "make all" it will build the binary file, if needed. Whew! That was all a bit complicated. The rest of the macro adds the output file to the clean target and marks the output as a generated file.

The "if" block in the middle is a bit like the autoconf functionality from the GNU Build System - it tries to find the ndstool program, and raises an error if it cannot be found. Otherwise we get a varible NDSTOOL_EXE that we can use to run the program. The NDSTOOL_FILE macro uses the ndstool exe to create a "single core" binary. Actually, it uses a default arm7 core, which is why we don't have to provide one. Again, we add a custom command to make sure the thing gets built when we run "make all" and add the nds file to the list of things to get cleaned.

So we have ndsmacros.cmake. To include it there are 2 ways that vary subtly. The best approach is to set the CMAKE_MODULE_PATH variable to the directory containing our macro file and then include the macro as a module:

set(CMAKE_MODULE_PATH ${CMAKE_SOURCE_DIR})
include(ndsmacros)

That assumes you have the macro file at the top of the ansi_console directory, which for a small example like this is fine. On a bigger project, if you want lots of extra tools, you would probably park them off in a sub-directory to avoid clutter.

So that's more or less it. Let's run this and see what happens. Ah! first I should just mention that we will compile in a separate build tree, keeping the source tree free from object files and other build artifacts. This is just good practice.

$ cd ansi_console
$ mkdir build
$ cd build
$ cmake -DCMAKE_TOOLCHAIN_FILE=$(pwd)/../devkitArm.cmake ..
-- The C compiler identification is GNU
-- The CXX compiler identification is GNU
-- Check for working C compiler: /path/to/devkitARM/bin/arm-eabi-gcc
-- Check for working C compiler: /path/to/devkitARM/bin/arm-eabi-gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /path/to/devkitARM/bin/arm-eabi-g++
-- Check for working CXX compiler: /path/to/devkitARM/bin/arm-eabi-g++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Looking for grit
-- Looking for grit -- /home/rich/src/ndsdev/devkitARM/bin/grit
-- Looking for arm-eabi-objcopy
-- Looking for arm-eabi-objcopy -- /path/to/devkitARM/bin/ndstool
-- In data
-- Configuring done
-- Generating done
-- Build files have been written to: /path/to/examples-cmake/ansi_console/build

Now if we look at what has been produced we see that there's a normal Makefile in the build directory. So lets run it...

$ make
Scanning dependencies of target ansi_console
[ 25%] Building C object ansi_console/CMakeFiles/ansi_console.dir/source/main.c.obj
Linking C executable ansi_console
[ 25%] Built target ansi_console
Scanning dependencies of target Target9_ansi_console
[ 25%] Generating ansi_console.bin
[ 25%] Generating ansi_console.nds
Nintendo DS rom tool 1.38 - May 14 2008
by Rafael Vuijk, Dave Murphy, Alexei Karpenko
[ 75%] Built target Target9_ansi_console
Scanning dependencies of target TargetObjCopy_ansi_console
[100%] Built target TargetObjCopy_ansi_console

Nice coloured output there. And we have our ansi_console.nds. That's it, job done. As you can see, the Makefile tracks dependencies. For this simple example it isn't too bad - everything really only depends on the main.c file. But for larger projects, not having to think about generating the dependencies manually is a big plus. Sadly we are limited to timestamp checks for updates to dependencies, the make legacy showing through. We can do better than that with hashes. I'll mention that when I discuss SCons next time.

There are still things missing from this, but armed with the basics we can improve the build without too much hassle. For example, we'd need a tool to convert images to data using grit. There are more flags that we need to pass in for C++ code to turn off exceptions and RTTI, for example. And we still have the dual code problem. The grit tool would be pretty much the same as our objcopy_file example, except it'd change a png file to a c source file. A bit of a limitation that, CMake doesn't compile assembler into executables, only C/C++.

Dual core is not too bad - we already have the basics. If we follow the above but for the "combined" example, then the main CMakeLists.txt would have the following lines:

add_subdirectory(arm9)
add_subdirectory(arm7)

We wouldn't actually use the ADD_EXECUTABLE macro here. In the arm7/CMakeLists.txt file, we'd have something like this:

include_directories(${CMAKE_CURRENT_SOURCE_DIR})
add_definitions(-DARM7)
add_executable(combined_arm7 arm7.c)
target_link_libraries(combined_arm7 nds7)
set_target_properties(combined_arm7
 PROPERTIES
 LINK_FLAGS -specs=ds_arm7.specs)
objcopy_file(combined_arm7)

This layout is mandatory because the ADD_DEFINITIONS macro applies the -D flag in the directory where it is used and for all sub directories. I tried using REMOVE_DEFINITIONS to work around the problem, but that does not work:

add_definitions(-DARM7)
add_executable(combined_arm7 arm7/arm7.c)
remove_definitions(-DARM7)
add_definitions(-DARM9)
add_executable(combined_arm9 arm9/arm9.c)

CMake uses the final set of definitions for all the executables. So we'd end up with the -DARM9 flag for the ARM7 binary too, which is not what we want. Ok, so the arm9/CMakeLists.txt file would have similar content to the arm7/CMakeLists.txt one, just switching all "7"s for "9"s. At the end we'd use the following macro to combine the 2 binaries into the nds file, in combined/CMakeLists.txt:

ndstool_files(arm7/combined_arm7 arm9/combined_arm9 combined)

You have probably guessed that this macro is added to our ndsmacros.cmake file and would take 3 arguments:

macro(NDSTOOL_FILES arm7_NAME arm9_NAME exe_NAME)
set(FO ${CMAKE_CURRENT_BINARY_DIR}/${exe_NAME}.nds)
set(I9 ${CMAKE_CURRENT_BINARY_DIR}/${arm9_NAME}.bin)
set(I7 ${CMAKE_CURRENT_BINARY_DIR}/${arm7_NAME}.bin)
add_custom_command(
 OUTPUT ${FO}
 COMMAND ${NDSTOOL_EXE}
 ARGS -c ${FO} -9 ${I9} -7 ${I7})
get_filename_component(TGT "${exe_NAME}" NAME)
get_filename_component(TGT7 "${arm7_NAME}" NAME)
get_filename_component(TGT9 "${arm9_NAME}" NAME)
add_custom_target("Target97_${TGT}" ALL DEPENDS ${FO} VERBATIM)
add_dependencies("Target97_${TGT}"
 "TargetObjCopy_${TGT7}"
 "TargetObjCopy_${TGT9}")
get_directory_property(extra_clean_files ADDITIONAL_MAKE_CLEAN_FILES)
set_directory_properties(
 PROPERTIES
 ADDITIONAL_MAKE_CLEAN_FILES "${extra_clean_files};${FO}")
endmacro(NDSTOOL_FILES)

This is a bit more hacky than I would have liked, that add_dependencies line shouldn't be needed. But if we use DEPENDS in the add_custom_command macro as the documentation suggests, then we only see errors like "No rule to make target `combined/arm7/combined_arm7.bin'" - I imagine it is because of the use of sub directories. We need sub directories though, otherwise we cannot set flags per ARM core, so we'll have to live with the rather hacky solution here. Unless someone knows better and posts a comment! ;-)

That's the lot! Now we can compile either single or dual core nds files and know how to check for the installed tools and libraries. So should you use CMake? Here is my round up to help you decide!

The Bad

Documentation: CMake sometimes requires what feels like cargo cult coding to get the desired result. It took me a lot longer than I would have liked to get this working. Part of the problem as to why I couldn't figure this out was the lack of good documentation. The CMake wiki is, like most wikis, not very well organised. Most help ends up pointing you towards the main site's "documentation" page, which only tells you how to buy a CMake book. The cynical side of me believes that this is a deliberate strategy - why have good free docs when you can sell books?

Syntax: I think the whole mixed case thing is a bit yucky. ALL IN CAPS makes things tricky to read, but at least you know you'll get it right. Mixing case is easier on the eye, but has the potential for cock ups. Meh.

Make's limitations: As we've seen here, there are problems when we push CMake to do things it possibly wasn't meant to do. Often the lowest-common-denominator build tool underneath leaks through the abstraction CMake provides. When I was trying to get the nds file to depend on the 2 binaries built in the sub directories, all the errors came from make - the cmake script looked correct. And those time stamped dependency checks... are you from the past?

Needs installing: Before we can do anything we have to install CMake itself, as well as the bog-standard make (or platform equivalent). This is extra bootstrapping that is not always necessary - waf manages to get round this on Linux, at least.

The Good

Cross platform: we can generate build scripts native to the host platform. Cool!

Configure checks: We can check that the building machine has everything we need to compile our project. If not, we can provide usefull info on how to obtian the code. This is better than having the build explode with "error: foo.h not found" followed by a billion other errors.

Compact syntax: The syntax of CMake is streamlined for building stuff. It's less wordy than make alone and more to-the-point than the other competitors in this field.

Starting to gain critical mass: I think this is important. There's no point using something if you're going to be the only one that uses it. CMake is now gaining enough popularity that it is fairly well tested, has a stable API and enough features to make it worthwhile.

Conclusion

Nobody is going to move away from regular Makefiles to compile the standard "Hello, World!" example. But when your project starts to grow in size and you introduce more sub-libraries, or unit tests, or want to have better control of the way the code gets built, then it's nice to know there are better options out there. I think CMake is a worthy candidate for consideration. The learning curve is quite shallow, especially if you already are familiar with Make or bash scripting, and it is something that is gradually gaining more mainstream recognition. Hey, at least it isn't as difficult to use as autohell ;-)

GMail in new Bunjalloo version

2008-08-26T23:51:00.010+02:00

I've made a new release of Bunjalloo. This version adds inline images, amongst other things.

What I really want to discuss though is this comment that gronfeldt posted on the Bunjalloo wiki/discussion thing:

Comment by gronfeldt, Aug 07, 2008

The browser is great! It's great that the project is still active. I'm mostly
stoked about being able to access GMail, but after hours of trying everything I
can think of I'm still not having any luck. I found a blog somewhere that said:

"Luckily there is a work around - once logged in to the Google collective, you
can navigate to http://www.google.com/xhtml and from there to the Mobile GMail
portal. That works fine and has a better layout for the DS to boot."

The trouble is I can't get anywhere near a page that would allow me to log on
without having to wait a very long time only to receive a message saying
"Unable to load:........".

If someone who has been able to access GMail has a minute I would love it if
they could add a comment with the steps to getting access to GMail.

Thanks so much Richard for putting all this work into something for the
community, very admirable of you.

No problem gronfeldt! I hope this problem hasn't put you off. It's not too good that he couldn't access GMail, as it definitely works (for me, at least). A couple of things make it more difficult than it should be I suppose, so here are the instructions based on version 0.7.0.

First of all, navigate to https://mail.google.com. All ok so far.

Once the page loads, you will need to allow Google to set cookies. By default not all domains are allowed to set them, as this is a privacy problem and, on a limited memory device, a memory problem too. So select the options icon, then the cookie add icon. This is explained in the Bunjalloo wiki in more detail, just in case.

Allow the google.com top-level domain by selecting the radio button and clicking Accept. This means mail.google.com and www.google.com will both be able to do the cookie thing, which is required.

Now enter your user name and password as usual. Once entered, click the sign on button.

Now comes the tricky/buggy bit. There's a bug *somewhere* in Bunjalloo which causes the sign-on process to stop. You'll reach a redirect page and nothing seems to be happening (no more page loads), and it is because of the bug. I'm not sure why, but redirects screw up occasionally. Hitting refresh, or the "moved here" link fixes the problem though, and the sign on carries on... until it stops again :-)

Here it is due to the lack of javascript. So click the "no javascript" or "mobile device" link to finally log in to gmail

That's it! Wasn't it easy? ;-) Next time cookies will be allowed, so you will only have to log-on/refresh/click. I'll keep trying to improve this as it is currently rather annoying, but hey! it (mostly) works. Shame SSL is so slow :-(

There are 2 places with room for improvement here. First, cookies are currently only partially implemented. They should be stored to disk as well as in memory, obeying the correct caching and expiry rules. This would mean that your log in session would be remembered between reboots. There's a lot of work needed there. Secondly, the redirection logic is broken. This is tricky to debug and has been plaguing Bunjalloo pretty much since I started so I would like to fix it "soon".

Google Code, Git and Subversion II

2008-05-28T23:38:00.004+02:00

There have been a couple of posts on the Google Open Source Blog. The second one is an in depth article about mirroring git repositories to Google Code's (GC) subversion repo. Having tried this out in the past, and helped a blog-commenter through the steps to mirror his git repo in svn, I can't help but feel that the post causes more problems than it solves.

First off, it's way too complicated! I'm quite experienced at using both subversion and git now, but I really cringe at that last set of commands to push to the svn mirror. Compared to "git push origin master" when using a native git repo, it makes me feel queasy. If I were the cynical type, I'd say that Google are just trying to make git look overly complicated compare to svn by forcing git to jump through hoops. Compare the command-fu there to a normal everyday "add, commit" work cycle and no wonder people have a bad impression of git.

Subversion is a lossy repository format. Author information is lost, date information is munged. The svn repo just has the author who ran dcommit, not the guy wrote the patch. Nor will it preserve the correct date, just the time when you ran dcommit. The admin user may change subversion revprops (date, message, author) but that's more manual steps and more opportunity for messing up.

If anything goes wrong while pushing to svn - and it will, the Google svn servers fail every now and again, they're only mechanical - then your push-to-svn git branch gets hosed. Like "git reset git-svn" hosed, so you can just send one big patch. If you have to sync more than a few commits, it's a bit risky. The blog hints at this problem ("commit to a local repo and use svnsync"), but the fact is that it's all a bit slow, sucky and error prone.

However, there's an easier way - cut Subversion out of the loop. On your GC page, go to Administer > Tabs and fill in the Source field. Add a wiki file named Source, for example, and in it place instructions on how to clone your proper git repository from github.com, repo.or.cz, or wherever. Then just use svn for the wiki, GC for the downloads and the issue tracker.

Git works OK on Windows too, apart from some strange CRLF problems. Yes, CRLF problems. In 2008. I know, it's crap.

"Why not widen the audience?" the GOSB asks us. And I ask, why not enlighten the audience as to the benefits of moving on to a next gen VCS? Open up the minds of the Subversion-users, whose patch sets will be both easier for them to create and easier for you to integrate via Git.

Not that anyone ever sends me patches, grumble, grumble...

Forthcoming changes in Bunjalloo

2008-03-16T17:57:00.003+01:00

An update on the status of Bunjalloo. I'm fiddling about with options and how to present them at the moment. Last release saw a change in the way I deal with configuration. Up until 0.5.4, the release zip would overwrite the config.ini, search.cfg and allowed cookies file. It was pretty unlikely that many people actually changed these files, since the configurable things have been a bit limited and changing them required a bit of effort, but as options are added it is more and more likely. Well, now it will be possible to change the language and default download location from a configuration screen. The design is typically lo-fi, but seems to work okay.

Next up is to make the list of pages allowed to set cookies editable. The default behaviour of all popular browsers - allow anyone to set any cookie - is not a very wise choice from a privacy point of view. So I've gone the other way and made the list of allowed sites whitelist-based. If you're not on the list, you're not setting cookies. However, in order to add a site to the list one has to use an external editor, which isn't too user friendly. A way to do this from a settings screen is on the list of things to do.

If the release doesn't drag on, I also want to add an automatic updater, as suggested by Sarvesh. This will probably be semi-automatic - i.e. a button with "check for updates" - since the DS network is not the fastest and over zealous checking would be most irritating. This requires adding (at least) unzip support, so would be quite a big addition. It would also mean that zip files could be opened up from within Bunjalloo, which might be interesting.

If I manage all that, which is not likely, I also want to add configurable quick searches. These are searches that are triggered by typing a single letter followed by the search term into the address entry text box. Currently these are hard wired in the search.cfg file (g, y and w for Gogle, Yahoo and Wikipedia), but a system similar to Firefox's quick search wouldn't be too tough to add.

Of course if you would like to see any other features, or help speed up the addition of the above ones, patches are always welcome. I'd be interested to hear from anyone who has tried to compile the source code, and how it went.

My top 6 git commands.

2008-03-13T21:27:00.004+01:00

I've been using git for about 6 months now with my Google Code projects but I am by no means an expert. There's a lot to learn. While the documentation is complete, it can be overwhelming seeing all those options and commands. So here are my 6 git commands that I use quite often when working with subversion repositories.

1. git status

My most commonly used command by a fairly large margin. For a subversion user it can look a bit confusing at first. Why have two classes of changed files? Why not just commit all the changes? Hey, at least it can be in colour! But now that I understand the index and when to stage changes, it's absence in svn is sorely missed.

Don't forget to add this alias - it saves a lot of typing.

git config --global alias.st status

Now you can use simply git st.

2. git gui

This is a powerful mix of the status, diff, add and reset commands. By showing the changed files and letting you mix and match which ones to commit, creating meaningful patches and a logical check-in history is a lot easier. It also makes it harder to accidentally check in changes that are just for testing - though not impossible. Did I mention how much I really miss the whole staging area idea when I use subversion?

3. gitk --all

Having branches is nice, but if you can't visualize them it is pointless. This tool makes it dead simple to see what branches you have and, important for git-svn users and those who push to proper git repos, where you can see at a glance where the HEAD is compared to your remote branches.

4. git svn

Usually in the form of git-svn dcommit. I've aliased that one to svnci:

git config --global alias.svnci "svn dcommit"

This makes working with SVN repositories a much nicer experience. Being able to see all the history at a glance, commit changes instantly and "rewrite history" are especially handy.

5. git merge

When I wrote my HOWTO for using git with a Google Code repository, I mentioned that merging between branches was a bad idea. Well, that's not strictly true. At the time I was ignorant to the fact that the merge performs a "fast forward merge" if there are no other changes on the merged-to branch.

Eh?

Basically, it meant that with git-svn, if you merge a branch into master, and master had not moved on since the other branch was created, it could cause master to lose its upstream svn remote trunk tracking. The solution is to use the --no-ff flag when merging.

git checkout master
git merge --no-ff mytopic

This creates an extra commit on master corresponding to the merge, which can now be dcommit'ed to subversion. Ideally the other branch would also be in subversion, otherwise the merge commit just says "merged branch 'mytopic'", with no real way of knowing what commits went into it.

Of course should you lose your git repository there would be no way to know which revisions of which branches are merged into master (trunk), even if the branches are stored in the svn server. This is more a limitation due to svn than git, as svn has no built in merge tracking. One solution would be to push the git repository to a separate server too, just in case.

6. git rebase

So you've created a local branch that you'd like to "publish" in the subversion repository. Here's how you could go about it using git-rebase and a couple of other tricks. First find the svn revision that the git branch is based on:

sha1 = $( git rev-list --boundary $branch...master | grep ^- | cut -c2- )

That should hopefully give you the commit sha1 of the branch point. Convert it to a svn revision number:

revision = $( git svn find-rev $sha1 )

Now create a branch in the subversion repository from trunk at that revision:

svn cp -r $revision $SVNREPO/trunk $SVNREPO/branches/$branch

If you run git svn rebase --all at this point, you'll have a new remote branch at the same point as the base of your git branch. Something like the image here. The hiccup now is that [branch] is not associated to the remote branch in any way. Here's where we need to rebase:

git rebase --onto remotes/$branch master $branch

This "moves" our local branch on top of the remote branch, keeping master as the parent of both. Running git-svn dcommit now will "push" the local changes on [branch] into the remote subversion branch. Ok, so maybe I don't use this command that often, but it is handy to know what you can use it for.

Bunjalloo Update

2008-02-25T22:29:00.002+01:00

Lately I've read some positive comments about my home-brewed web browser on a few different forums and news sites. It's nice to know that people find the software useful! I've also received some suggestions via email on how to make the browser even better for people. One of the interesting things about having a homebrew browser is that it can do things that the official browser can't, like download files to the memory card or provide translations for languages other than the official 5.

So with this in mind, I'm now working on improving the way Bunjalloo downloads files. Currently it's not very user friendly and requires you to download the file, then save it. This is pretty unintuitive - it'd be better to sniff out the MIME-type of file that the user clicks on, then either download and show it for supported image and text MIME types, or offer to save it to disk for "exotic" MIME types. Y'know, a bit like everyone else does it :-)

As for translations, there's not much user interface to translate at the moment, but what is there has now been pulled out into resource files, defined per language. Currently only the languages supported internally by the DS are selectable, and the language displayed can only be altered by changing the DS's language option from the system settings. But this is only a beginning, if there is enough interest I can extend language support to those not built in to the DS via a configuration option. The actual translations into other languages need doing, so if you want to help out let me know. I have English and Spanish done, Italian, French and German are needed, as is Japanese. Sadly Japanese won't work without changing the font, the one I have at the moment doesn't include the full range of universal characters.

Speaking of helping out, keep that feedback coming. Star issues that you think are most important, report bugs, even write patches ;-) There's a new mailing list/forum for discussing Bunjalloo if mailing me is too much hassle. I look forward to hearing from you!

Another Elite AGB/DS Update

2007-12-31T16:01:00.000+01:00

A hectic few weeks in RL (I'm now a dad) have meant that I have not been able to dedicate much time to coding. However, I've ironed out the last few DS specific bugs in my port of Elite AGB and it is now playable. The only major issue left to resolve is the sound. The XM player that I wrote for the GBA version is incompatible with the DS by design, due to the hardware differences. I have integrated libntxm but it has a problem that causes notes to be cut short. The GBA sound effects use the GB sound hardware, which can be emulated on the DS using its PSG sound system. I got this working without too much hassle, but I get the feeling that it doesn't play nicely with libntxm either. The sound effects work when libntxm is not activated in the build, but adding it in causes the sounds to break.

I think the best option at this point is to remove XM playback - remove the crappy music ;-) - until I port my own XM player to the DS. This way I can make a release soon-ish before I get really sick of the whole thing.

Blame Jeff for the delay...

2007-12-14T20:21:00.000+01:00

There's a reason I've not released anything yet - POWDER. Jeff Lait's devilishly addictive Rogue-like has me hooked. I played it on the GBA a couple of years back, but I've recently rediscovered it in the DS version, with its bigger screen and slightly nicer controls (X+Y are more comfortable than the Start + Select)

Unlike Nethack and friends, Powder was designed for the GBA's d-pad controls - there are no diagonally moving monsters apart from the weak grid-bug - and this really helps the flow of the game. It is very fast. There's also a high wheat to chaff rate - interesting items are abound. On the first few levels you'll have loads of great wands of zapping, potions of damage or healing, and swords of slicing. To decrease the frustration factor, your inventory is huge and you aren't penalised for carrying lots of items. Not very realistic, but immensely gratifying. There's nothing worse that having your movements hampered because your Diogenes syndrome kicks in and you can't bare to throw any of your junk away in case it's useful later.

The dungeon layouts are good too, with the random generator spitting out some interesting set piece rooms every now and again. Woe betide the newbie adventurer who comes across an island of ghosts as shallow as the 4th level! The layout is optimised for the GBA's 512x512 tile mode and the wrap around is a cute idea that makes the levels seem bigger than they really are. The graphics are decent for a homebrew game, perhaps the only quibble is the contrast of the text on the items screen (white on grey) as it can be difficult to make out at times. The main character is novel in the way that your on screen player avatar shows what he is wearing and clutching in his mitts at all times. There's an air of Monty Python absurdity to it each time a new game starts off with the hero stood there, bright pink and in his Y-fronts.

The gods of the game - in all there are 7 - play a crucial part to the way you decide to explore the dungeon. I don't mean just praying for food when you feel a bit peckish either a la Nethack, it is a bit deeper than that. Each of the 7 gods expects a specific behaviour from its desciples and failure to fulfil a god's wishes can end in tears. For example the god of the barbarians, H'ruth, expects plenty of killing and no magic use. If you are a wizard in training then it is very likely you'll piss H'ruth off. Luckily the wizard leader Belweir will protect you, so while H'ruth may poison the player, Belweir will cure the poison and provide gifts of magical items.

The god system has a slight flaw in that the fighter god Klaskov is very easily pleased, so rampaging through the game casting a few damage spells is usually enough to get in his good books. I have no idea how to please the pacifist Pax - casting cure seems to work, but killing any innocent rodents usually ends up with a dead player. These aren't even defects really, indeed they could even be considered extra challenges - "easy" mode would be to play as a fighter, while "hard" would be trying to get through as a cleric. Though quite what the god ><0|V| expects is anyone's guess.

Food plays an interesting role too. There's a saying "you are what you eat" and Powder covers this pretty well. Eating a char grilled fire snake will give you the ability to avoid fire attacks, while eating grid bug bodies leaves you immune to spark attacks. Unlike other rogue-likes the enemy corpses tend not to go rotten, so unless a baddie is obviously poisonous it is usually safe to eat it after defeating it. But don't go overboard if you plan on being a cleric - killing innocent Kiwis and scoffing them down is not very nice. Despite the lower starvation rate than other similar games, lately I've been getting snuffed out because in my hungry state my health wasn't regenerating...

Some spoilers you say? OK, stop reading now if you want to discover these things for yourself!

There's a whole dressing up sub-game hidden within the game. I've so far discovered that to dress as a Wizard you require the pointed shoes, plain robe and floppy hat. In your hands you can carry either nothing (not recommended) or a staff. The advantages... I'm not sure yet. It obviously pleases the gods. To dress as a Ranger, you need a leather tunic or the studded variety, hiking boots, a leather cap and either a bow or a torch in your hands. This get up seems to give you better searching abilities and may also help your ranged weapon attack damage... but it's difficult to say. The powder web site also mentions dressing as a Fighter. I've not discovered this one yet though, but I bet it requires a hefty warhammer and some sturdy iron boots.

Anyway, let's hope I get over this addiction soon. And meanwhile, I really recommend you checking out the latest version. It's one of the unsung classics of the homebrew scene.

WIP update - Elite DS

2007-11-25T17:34:00.000+01:00

I got side tracked with grit, adding the ability to share a palette between various images. Now that this problem is solved (Cearn sent me an interesting program to merge image palettes after I sent him a patch against grit to do something similar) Elite AGB compiles for the DS. All that's left is to get it working! :-)

At the minute the NDS file just shows 2 white screens; a familiar situation for anyone who has coded anything on the DS. So now to start debugging...

EDIT: heh. here's a screenshot of the first build running in desmume. Yes, it is a straight GBA port.

Another reason for GBA -> DS ports

2007-11-19T22:38:00.000+01:00

So I've started working on Elite DS. First thing I did was to compile my old GBA code with the latest devkitArm r21. No problems there, fixed a buffer overflow and now it runs fine... on VBA.

My homebrew hardware for the GBA has gone the way of the dodo.

Well, not quite, but my new laptop doesn't seem to have a parallel port! It has a plastic blob in the shape of a parallel port, but there's no way to pry the thing off - luckily they've remembered to etch the universal symbol for "printer" above it, to mock me. This makes the F2A linker thingy rather useless. One more reason to port all my GBA projects over to the DS, eh readers?

Future efforts?

2007-11-12T22:01:00.000+01:00

First, a rant.

Even though I understand where he's coming from, I don't agree with a lot of Dragon Minded's comments. His latest about wannabe plugin authors, that "they just wanted to do something easy to get their name on the project" seems a bit off - as far as I can see, DSO goes against the spirit of the GPL and actively discourages contributions. How can you do that on one hand and then expect people to use your plugin framework on the other? Or maybe he should just stop reading certain forums - people out for a free ride, downloading Nintendo games, are not going to make useful contributions to very much at all.

Now the rest of the post.

Bunjalloo has been open from day 1, and what help people have offered me has been gratefully received, so I don't want to abandon it. But the fact is that debugging Bunjalloo is most of the time a thankless task. The web is full of terrible HTML and hacky connection methods - witness the simple google log on process. Web pages are increasingly reliant on Javascript and Flash. Things that we take for granted on a PC browser are a painful nightmare on the DS. It can only get worse as web standards are further extended for more powerful devices and more interactive content. A kick in the nuts is that even mobile phones have better web browsers. This is mostly due to the tiny memory the DS has - but also because writing these things nowadays is more than a one man project. And I'm fed up!

So, with your permission, I think I'll take a break from the web browser for a while. Let's leave the apps to the PC and concentrate on the games. After all it's what the DS was designed for, and I have some outstanding issues to resolve...

GMail now sort of works

2007-11-11T18:52:00.000+01:00

I've now managed to read and send gmails from Bunjalloo :-)

Sadly the log in process to a gmail account is a bit strange. It uses lots of server side 302 redirects, meta content refreshes, cache-control:no-caching and some other voodoo that I haven't yet figured out. It breaks w3m as well, so I'm not alone in my frustration.

Luckily there is a work around - once logged in to the Google collective, you can navigate to http://www.google.com/xhtml and from there to the Mobile GMail portal. That works fine and has a better layout for the DS to boot.

The issue page on googlecode doesn't work yet - so you can't post an issue from within Bunjalloo. That's next, meanwhile I'm going to put out a new release.

HTTPS on the way..

2007-11-05T23:11:00.000+01:00

I've made some advances with HTTPS. I only started today and already Bunjalloo can show the GMail log on page! There is an issue that you might like to star to see when HTTPS finally gets added properly. Put the champagne down for the moment, because it seems my naïve approach to cookies doesn't stand up to the Google log in process. Currently I get told that my browser's cookie functionality is switched off :-/

Logging on to read my mail is a #1 priority here. Closely followed by posting issues to the aforementioned issue tracker from Bunjalloo itself. I can already browse the subversion code, but I need to add issue posting to really start eating my own dog food. (Yuck.)

The SSL library I'm using is MatrixSSL. It took me just a few hours to add and get running on the PC, plus a few more to debug for the NDS. I expected it to take days not hours, so I was pleasantly surprised. Now I just have to remove all the damn printfs.