Expert C Programming: Deep C Secrets

I read a friend's copy of "Expert C Programming" many years ago, and ended up buying a copy for later reference. I didn't quite remember why I would purchase a book I had read already. Recently, I cracked open my copy for the first time: it still had the new book smell.

Oh, now I remember.

This is the most enjoyable programming book I have ever read.

The book is about corner cases and implementation-quirks of C, with a smattering of C++ thrown in. The entire book is divided into themes, and each theme is self-contained in a chapter. Each chapter talks about a quirk of the language, something that good C programmers should be aware of.  For example, one chapter talks about the differences between Kernighan&Ritchie C and ANSI C. Each idea is crisply presented, with source code and clear explanation. The chapters are littered with relevant anecdotes, historical background, and tantalizing puzzles.

Programming books can be dry and just about the facts. Recent technologies usually come with books that focus on how to get things done. While this is useful, it does not improve the appreciation of the technology. Expert C Programming was written 20 years after the development of the language. Perhaps time leads to some clarity and a light-hearted view of things.

Programming is more than just a profession: it is a creative process. It helps to have such books in your shelf: so you can them pick up and remind yourself how fun computing can be.

(Image courtesy: Amazon)

Android debugging using the framework source

When writing Android apps, it is helpful to be able to read the implementation of the framework to clarify a point.  This might be when the documentation is incomplete, or if you want to learn from canonical classes like ListView.

Nexus devices contain this framework code unmodified. This allows you to trace your application flow down to the framework level, either to learn how the platform works or to find a bug. Today, the cheapest Nexus device is $199, so having an additional Nexus device is an excellent tool for Android developers, even if you do most of your development for non-Nexus devices.

It is easy to get the source code from the Android Open Source Project (AOSP). You need the following:

1. A good Internet connection. The full AOSP tree is well over 8 gigabytes of data. If you have more than one Android developer, you could download the tree for a single developer, and then mirror it locally.
2. A fast computer. AOSP is a lot of code and Eclipse chews a lot of CPU and RAM with large projects. A computer with two CPUs and about 4GB of RAM should do perfectly, you might get away with lesser RAM.
3. Disk space. If you choose to build the code, you're look at 30 gigabytes of disk space.
4. Eclipse, some recent version and Sun's Java 1.6.
5. Linux or Mac. The instructions work for Ubuntu 10.04/11.10/12.04 and some Mac version.

Here are all the steps:

1. Set up your computer: This can be done on all developer computers simultaneously. This will install all the development tools. At the end of this, you have all the tools but no AOSP code.
2. Download all the AOSP source: This uses your internet connection to download all the source. The AOSP source has everything going back to the earliest releases of Android. You get all source history, the framework, the open source applications, the kernel, all open drivers, everything. There are instructions on that page to set up a local mirror. Follow those steps to download the source once and then share it over your local network. This conserves bandwidth, and saves time. You might want to start this download over a weekend, depending on your network connection and other users' needs. I will assume that you are putting the source in /usr/local/aosp.
3. Build the source. Here, you have two options. You could follow the AOSP instructions. However, if you just want to include the source in an eclipse project, those instructions are overkill. You can get a much smaller project if you follow these alternate instructions. If you have trouble with java versions, read the bottom of this post.

   $ cd /usr/local/aosp
   $ source build/envsetup.sh
   $ lunch full-eng
   $ # The following step takes time. -j<num jobs> increases parallelism.
   $ make -k -j2 sdk
   

4. Create the Eclipse classpath. By default, we can create a project with all the Android source code. This is beneficial if you want to have access to all the AOSP code for reference. If you want just the framework, you can reduce the size of the project significantly. To do this, first create a file in the directory containing the following:

   $ cd /usr/local/aosp
   $ cat excluded-paths 
   ^external/.*
   ^packages/.*
   ^cts/.*
   ^dalvik/.*
   ^development/.*
   ^prebuilts/.*
   ^out/.*
   ^tools/.*
   ^sdk/.*
   ^libcore/.*
   ^gdk/.*
   ^hardware/.*
   ^device/.*
   

   This file allows you to reduce the size of the Eclipse project, which improves Eclipse's performance significantly. Now, we can generate the Eclipse classpath as follows:

   $ cd /usr/local/aosp
   $ ./development/tools/idegen/idegen.sh 
   Read excludes: 3ms
   Traversed tree: 781ms
   $ ls -l .classpath
   -rw-rw-r-- 1 user group 16938 Sep 18 21:50 .classpath
   

5. Create the project in Eclipse. First you need to start Eclipse with increased heap size and virtual memory:

   $ eclipse -vmargs -Xms128m -Xmx512m -XX:MaxPermSize=256m
   

   Now, create a new Eclipse project (File -> New Java Project -> Next). In the dialog, under the "Libraries" tab, click the "Add Library" button -> Add System Library -> Add JRE system library. This will help resolve the references to core libraries like Integer and String. Adding the system library is not required, but it reduces the number of syntax error Eclipse finds with the framework. Click "Finish" when done.
   
6. Done! Try your setup by searching (Ctrl-Shift-T) for FragmentManager. You should be able to see its source code, and navigate through its code. Some handy commands are: Ctrl-Shift-G to look for references of a class, and F3 to look for a method's implementation.

JDK version pain: You might encounter a problem with JDK versions. You need Sun Java 1.6 to compile the AOSP source code, while your Eclipse setup might require a different version (OpenJDK?). One solution is to use sun-java only to compile the AOSP, and switch back to the previous version after the compilation has finished. On Ubuntu, this is done with the commands shown below. Select the number that corresponds to sun-java before the compilation, and run these commands after running idegen.sh to switch back to your previous version of jdk.

$ sudo update-alternatives --config javac
There are 2 choices for the alternative javac (providing /usr/bin/javac).

  Selection    Path                                         Priority   Status
------------------------------------------------------------
  0            /usr/lib/jvm/java-6-openjdk-amd64/bin/javac   1061      auto mode
  1            /usr/lib/jvm/java-6-openjdk-amd64/bin/javac   1061      manual mode
* 2            /usr/local/sun-java-1.6/jdk/bin/javac         1         manual mode

Press enter to keep the current choice[*], or type selection number: 1
update-alternatives: using /usr/lib/jvm/java-6-openjdk-amd64/bin/javac to provide /usr/bin/javac (javac) in manual mode.
$ sudo update-alternatives --config java
There are 2 choices for the alternative java (providing /usr/bin/java).

  Selection    Path                                            Priority   Status
------------------------------------------------------------
  0            /usr/lib/jvm/java-6-openjdk-amd64/jre/bin/java   1061      auto mode
  1            /usr/lib/jvm/java-6-openjdk-amd64/jre/bin/java   1061      manual mode
* 2            /usr/local/sun-java-1.6/jdk/bin/java             1         manual mode

Press enter to keep the current choice[*], or type selection number: 2

Xmodmap: a quick guide

If you use a variety of Linux/Unix machines and are confused between their different keyboard layouts, it might help to learn the xmodmap utility.

Xmodmap allows you to read the keyboard and mouse pointer bindings and to set them. You don't need root permission, and you can modify these without restarting the X session. This is very useful if you use different keyboards, or want to use the CAPS LOCK key for something useful.

The easiest way to use the xmodmap command is to print out its current config. Run

$ xmodmap -pke

This prints out all the keycodes that are assigned. Here is a sample line:
keycode  24 = q Q U094C U0914

This says that 

1. Keyboard code 24 corresponds to the 'q' key. Pressing the key by itself without any modifiers produces the 'q' key.
2. If you hold shift while pressing it, it produces the 'Q' key.
3. If you hold the mode_switch key while pressing it, it produces the Unicode 094C character (in hex)
4. With the mode_switch and the shift, it produces the Unicode 0914 character (in hex).

The mode_switch key is used on non-US keyboards to produce non-English text. If you don't know what it is, you don't need it.

But you do use the shift key. How do you know which key that is? Run:

$ xmodmap -pm

That command tells you what the modifier keys are. Looking for the 'shift' modifier, you can see that it is mapped to two different keys:

shift       Shift_L (0x32),  Shift_R (0x3e)

They have hex keycodes 0x32 and 0x3e. 0x32 is 32 in hex which is 50 in decimal. If you search for keycode 50 in the output of 'xmodmap -pke', you will see the following line:

keycode  50 = Shift_L ISO_Prev_Group Shift_L ISO_Prev_Group

So that's the left shift key.

All these keys can be remapped. The most common use is to remove a key you do not use, like the CAPS LOCK key. That is the 'lock' modifier. On my keyboard, the lock modifier is:

lock          Caps_Lock(0x42)

It is assigned to the Caps_Lock key. Let's remove this binding. To remove bindings, you can write commands as arguments to xmodmap -e, and you can supply many commands in succession, as follows:

$ xmodmap -e "command1" -e "command2"

To remove the Caps_Lock binding, we will remove the 'lock' modifier from it.

$ xmodmap -e "remove lock = Caps_Lock"

If that didn't produce any output, it was successful. Validate by running "xmodmap -pm" again. Now, the lock line should look like

lock

Now Caps_Lock doesn't produce a shift lock anymore. You can also verify by opening a text window and pressing the caps lock key. It will not do anything.

Now, let us put the key to good use: how about assigning it to control? Run:

$ xmodmap -e "add Control = Caps_Lock"

Great, now your caps lock is an additional control. We can combine these two commands:

$ xmodmap -e "remove lock = Caps_Lock" -e "add Control = Caps_Lock"

Easy remapping, and it works independent of the window manager, the keyboard type, and the version of Linux.

In case you want to find what key code a specific key produces, run the 'xev' program. It tells you in great detail which keystrokes are pressed and released, and also all mouse events.  Here is a sample line:

KeyPress event, serial 36, synthetic NO, window 0x3e00001,

    root 0x15d, subw 0x0, time 1355682, (152,6), root:(783,52),

    state 0x4, keycode 38 (keysym 0x61, a), same_screen YES,

    XLookupString gives 1 bytes: (01) " "

    XmbLookupString gives 1 bytes: (01) " "

    XFilterEvent returns: False

I pressed a key which produces keycode 38. Using this method you can narrow down on an unused key and put it to good use.

STM32 with Linux: Device support. Part 1

I recently got a STM32F0-Discovery board. This is a development board that contains a versatile ARM microcontroller and a supporting chip to program and debug the microcontroller. This is Part 1 of a short series to get you up and running with these devices.

0. Motivation
Microcontrollers are computers that contain a processor, memory and storage all in one tiny package. Such computers are used inside devices that need a small amount of processing power: in televisions, printers, or electronic music players. These computers are easy to program, they utilize little power, and are rugged enough to last many decades of continuous use. I had previously posted a set of tutorials showing how to program ARM processors using assembly. These tiny microcontrollers can be programmed with a subset of the ARM instruction set, making them easy to learn.

The STMF0 discovery board is a low cost development board intended to demonstrate the advantages of the platform. It looks very compelling: it has many Input/Output lines, it is low cost ($7.50 each), and it uses ARM assembly. Mike Szczys recently wrote a link program that makes this device programmable within Linux. When I received my device this week, I decided to give the Linux interface a try. In case you have one of these devices and an Ubuntu machine: here are the steps you need to follow to get your device working inside Linux.

You will need:

1. An STM32F0 Development board. At this page, go to Orderable products -> Distributor Availability to find a retailer near you to buy from. 
2. A Linux computer with root access.

1. Install Development packages.
You need a few packages as root:

sudo apt-get install pkg-config libusb-1.0.0-dev git

2. Installing ST-Link
This is code written by Mike Szczys that allows you to communicate with the discovery board in Linux.

mkdir stm_discovery && cd stm_discovery
git clone  https://github.com/texane/stlink.git
cd stlink
./autogen.sh && configure

There shouldn't be any errors in the output to configure. If there are errors, they usually point to missing development tools, so fix them before continuing. Otherwise, proceed to the next step:

make

Checkpoint: You should have two executable files: st-util and st-flash in your directory. They won't do anything yet, for that we'll need device support.

3. Device support.
If you have the STM32 device plugged in, unplug it. If you have a v1 device, you should unplug any USB storage disks like flash disks as well. This is because the v1 device fakes a USB disk, and we will need to change the USB kernel module to get it working. So, unplug usb disks now. Then, run the following commands:

sudo cp 49-stlinkv* /etc/udev/rules.d   # Allow any user to access device.
sudo modprobe -r usb-storage            # Remove the module
sudo cp stlink_v1.modprobe.conf /etc/modprobe.d   # Set device specific policy
sudo modprobe usb-storage               # Add the module again

This sets up the device so any user can play with it.  There is no security issue with allowing users to write to these devices, so the above steps are safe.

Checkpoint: You should have an executable file called st-util, and running it should produce the following output:

$./st-util
2012-06-10T21:23:29 INFO src/stlink-usb.c: -- exit_dfu_mode
2012-06-10T21:23:29 INFO src/stlink-common.c: Loading device parameters....
2012-06-10T21:23:29 INFO src/stlink-common.c: Device connected is: F0 device, id 0x20006440
2012-06-10T21:23:29 INFO src/stlink-common.c: SRAM size: 0x2000 bytes (8 KiB), Flash: 0x10000 bytes (64 KiB) in pages of 1024 bytes
Chip ID is 00000440, Core ID is  0bb11477.
KARL - should read back as 0x03, not 60 02 00 00
init watchpoints
Listening at *:4242...

4. Get ARM development tools
At this point, you have support for the device, but you don't have a compiler or debugger for this board. Your computer is most likely an x86 device so you need a cross compiler to compile ARM code for it. The cross compiler is special: it generates code for a device with no operating system.

Code Sourcery is a free toolchain that compiles code for such devices. You can download it for this specific board and install it as a user. You don't need to be root to install it. This is a 100Mb file, so depending on your download speed, this might take some time. You can install it anywhere, the location isn't special. The installer asks you if you want to modify the PATH environment variable. Choose yes.

This isn't the only way to get the ARM development tools. If you want, you can compile gcc and gdb from source. Adam Kunen has a long article that describes how to do this.

Checkpoint: Start a new terminal. (This is important because your PATH needs to be updated.) You should have the programs arm-none-eabi-gcc and arm-none-eabi-gdb in your path. You can test them by invoking them:
arm-none-eabi-gdb

$ arm-none-eabi-gdb
GNU gdb (Sourcery CodeBench Lite 2011.09-69) 7.2.50.20100908-cvs
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=i686-pc-linux-gnu --target=arm-none-eabi".
For bug reporting instructions, please see:
.
(gdb) quit
$ arm-none-eabi-gcc
arm-none-eabi-gcc: fatal error: no input files
compilation terminated.

Now, you should have full support to program these devices in Linux and to debug them. Part 2 will show a sample program.

(Image courtesy: ST Microelectronics)

Impressive gaming: World of Goo

In order to fight off murderous tendencies while doing taxes, I play video games. This year, I played one game on my Android phone: The World of Goo.

Wikipedia claims that this is a physics-based puzzle game, but that is as accurate as describing chicken tikka as a dead bird. The World of Goo is a brilliant adventure game, told through many levels that involve tiny balls of Goo. The balls start out sticking together, and new levels add interesting capabilities. There is a single objective in every mission, usually collecting a fixed number of Goo balls towards a pipe. (To be fair, Wikipedia has a scary long page about the game. What they lack in humor, they make for in actual content and facts.)

A few years ago, I was saddened by the lack of inventiveness in new games. It seemed that every game was about the same boring plot: there were certain genres and everyone stuck to tried and tested mechanics. Walking through the aisles of a computer store, you could quickly categorize each game into a few genres: RPG, strategy, platform, with little that set the individual games apart. Compared to this, the early days of gaming were filled with creativity as programmers experimented with computers to create something fun and unique.

That wasn't the only problem. I use Linux and many games do not work on my system. Even when had the luxury to install Windows on one partition, games required registration codes and other fiddling just to get working.

In such unhappy times, I came across the demo for World of Goo. There was a Linux version, and I played one level or two and loved the idea behind the game. The game was refreshingly new, with goo balls sticking to one another and making cute sounds when they reached the destination.

For one reason or another, I put off the purchase and then promptly forgot about it. A few weeks ago, I finally bought the first Indie bundle on Android and downloaded my copy of World of Goo. It was as refreshing and enticing as I remembered it, and so I installed it on my Android phone.

Tax week was a whirlwind of administrative paperwork and an ill baby. In between copying numbers from one dull form to another, I was holding my child as he slept clinging to me. My hands were free, but I wasn't coherent enough to do anything productive. My phone was nearby, and the opportunity was perfect for the World of Goo. After a few hours over many days, I finally finished the last level. And I cried because I had no more goo world to conquer. That's how good the game is.

The game has exceptional level design. Most game developers fall in the trap of coming up with a single idea for a game: developing the engine and then churning out levels one after another. In such games, the fun dries up quickly after the first few levels. The game gets devilishly hard and the levels just get tougher. New levels are more challenging, but no more rewarding. Level two is level one, but with less time and fewer resources. That's not fun, that's a chore. In Goo, you go through different scenarios, changing weather and different kinds of goo balls. The game mechanic changes as you tackle new levels. This reminds me of Soul Bubbles for Nintendo DS, which had the same change of mechanic as the game proceeded. I'm glad the developers resisted the urge to copy-paste levels. I'm glad they spent as much time in innovative levels and level mechanics as they did on the creativity of the engine. The levels are inventive, they keep you eager to see what happens next.

As the game proceeds, the story of the goo balls unravels and takes the player through its strange and  humorous turns. World of Goo does a lot of things well, and is an instructive lesson to the cesspool of current-day gaming.

1. Creativity counts. Come up with something unique. Computer worlds are limitless, so don't package the same crap all over again.
2. Entertain in every level. Each level should be fun to play. Doing the same thing twice isn't fun.
3. Leave no player behind. Goo allows you to skip baffling levels rather than requiring your user to fight through tough levels. They can always come back for levels later. This makes the game more fun and accessible.
4. All the fun ideas, and nothing else. The game is small, but it is fun all along. That is far better than padding a game with mediocre content to increase the length.

The game is half a decade old, but is well worth playing.

What is inside electronic devices?

Not many people see the insides of electronic devices. Here is a picture of a Linksys E-1200 router:

If you compare this with electronic boards made a decade ago, the emptiness of the board would strike you immediately. Most of the board is empty space and two black chips dominate the real estate. The remaining components are minor: resistors and capacitors.

There are two big black chips in the center of the board. In order of appearance from left to right are:

1. Winbond W9425G6JH: That's memory, storage, or RAM.
2. Broadcom BCM5357B0: That's everything else: CPU, wireless, ethernet, router. Everything

The fat black things in the bottom look like chips but they are not. The FPE 2020 are tranformers for electrical isolation of the ethernet signal.

Everyone knows that computers getting faster, smaller, more durable and more cost efficient. The silent revolution is in the ubiquity of computers. Tiny computers are everywhere. The desktop was once the only computer in the house. Then came along the laptop, then the smart phone. Then your television set-top-box. Routers, ebook readers, the telephone box to give you cheap international calling. Deep inside, all these devices are empty boards like the one above. One powerful chip containing a computer to do everything.

The other important fact is that these are a full, general-purpose computers. You can run browsers, games, and email programs on these computers. Manufacturers make a board with a general computer and then write router software for it. This method is cheaper than creating a specialized router-only computer. A router is very similar to a television set-top box: they both contain roughly the same parts. The only difference is in the software.

We are getting better at manufacturing these full computer chips: they cost less, last long, and fail less frequently. It is not unusual to run these computers constantly for years. This astonishing reliability is one reason why the repair industry is dying. The other reason is that there isn't much left to repair. Boards contain few parts, and if a CPU or RAM chip goes bad due to overheating or electrical surges, it is nearly impossible to repair it.  And capacitors and connectors fail much more often than CPU and RAM chips. Since there is nothing mechanical left to fail, and the failure rate of silicon components is low, electronic boards provide decades of service.

Another trend to notice is that many of these devices run Linux. Linux has many benefits: it is free, it has support for a wide variety of these new computers, and it is easy to modify.

(Image courtesy Wikia.com)