No project is more important than its users

Let’s begin by looking at some quotes from ELCE 2011 kernel panel:

Linus Torvalds:

The biggest thing any program can do is not the technical details of the program itself; it’s how useful the program is to users.

So any time any program (like the kernel or any other project), breaks the user experience, to me, that’s the absolute worst failure that a software project can make.

No project is more important than the users of the project.

Alan Cox:

If you want to understand the importance of not suddenly changing your users’ experience.
I would go and take a look at GNOME 3.0.
Irrespective of whether it is a good user interface design or not.
It’s a demonstrattion of why you don’t suddenly change everything on people who rely on what you were doing.

If you look at it from the point of view of the people producing the hardware and technology. Then there’s always a pressure from them to get every new feature and every new thing enabled.

But if you talk to the users of these systems, most of them are primarily concerned that what used to work continues to work and in the same way.

You can see the whole video to get a sense of how important it is for Linux developers not to break user experience. Also, how to don’t break the API while still moving forward, and not increasing code complexity. Or breaking API without users noticing.

This should be common sense, but some people don’t agree and there are various arguments I have heard over the years.

Developers vs. Users

According to some people, it doesn’t matter what the users want, only what developers want, because at the end of the day, it’s the developers that will implement it. This mentality even lead to the term Free Software User Entitlement Syndrome; the idea that somehow users have a say in what happens in the project is dismissed with passion by some people.

This kind of thinking ignores a very basic fact; there’s no such thing as a user/developer divide. We were not branded at the moment of birth; “you shall be a developer”. No, developers were mere users at some point in time. Maybe they were not so good at programming when they started, but they eventually learned all the tricks. Or maybe they already were good programmers, but didn’t want to contribute right away until they saw the software was actually useful, saw an area of opportunity, and had enough time.

Moreover, there are all kinds of developers; the ones that contribute only one patch, the ones that contribute a series of patches they did in one weekend, the ones that contribute sporadically, the ones that contribute often (but are not part of the core team), and the core team. There is no natural division either; it’s a continuum, and people often move back and forth in it as time and motivation permits.

In fact, this continuum of developers follows a power law distribution, also referred to as the long tail. It is hard to see in the following image, but the total area from the right (yellow) is the same as the left (green), that’s why the term long tail is used to point out that one should not dismiss certain people that might be labeled as outsiders, or not part of the “core”:

Here is a graph of all the contributors to the git project ordered by the number of commits, which is plotted on the y axis:

It’s not easy to see, but there’s a long tail there. Here it’s plotted in logarithmic scale, and with a graph of a pareto function:

This means that 70% of the people contributing to git have only provided less than 6 patches, and 37% only one patch. Sure, one patch might not look like a lot, but as Clay Shirky points out; what if it’s a patch for a buffer overflow? what if it’s the difference between a malevolent hacker being able to exploit your system or not? Traditional companies can’t make use of people like Zack Brown who contribute only one patch to the project, but open source organizations can, so lets make use of them.

And lets not forget that this continuum is not static; people often move up (and down) in the ladder. Good projects would exploit this long tail by not dividing groups of developers, and making things easy for newcomers. Junio C Hamano, the maintainer of git, points out:

During the entire history leading to 1.6.6, 710 authors contributed code
and documentation. The changes since 1.6.5 were made by 99 authors, among
which 17 are the new contributors to the project.


And of course, this long tail graph doesn’t even take into account the thousands of users, many which are potential contributors. Plus you have people like Ingo Molnar, who is a very active Linux kernel developer, but has only committed 6 patches to the git project, does that mean he is not a “developer”?

Lets get rid of this idea that there is a division between users and developers; in open source projects, this division just exists in people’s imagination.

Lack of resources

As Linus and other developers pointed out in the video above, you can find ways to provide both the old, and the new behavior. Often this is not difficult at all, but requires some imagination.

Unfortunately, some people disagree and just drop features without any way to use the previous behavior and they rationalize the decision using the “lack of resources” card. Sometimes they don’t even try it, but sometimes it’s true that it’s difficult to implement something while keeping the old behavior. However, something being difficult is not an excuse not to do it; it might take more time, but it’s still doable.

This is the point that is often ignored; they don’t want to spend more time.

But there’s another important point; perhaps the lack of developers is due to the amount of features dropped. Think about it. As I explained, all developers were users at some point, and dropping features means dropping users (current and potential), which inevitably means dropping potential developers. The harder you try to keep you users (through keeping features), the more confidence these users would have that they will be using your project in the future, and the more they will see contributing as an investment.

This is one of the reasons Linux is so popular (as I explained in another post; it’s the most important software project in history) everybody that has been using it for years knows (s)he will still be using it in the years to come, as they are not going to suddenly break everything. It’s a contract between users and developers that builds trust. With this trust it’s easy to gather more and more users, and thus, more and more developers.

This also applies the other way; not only about not dropping features, but adding unpopular ones. It is possible to add features not everybody will use, and still keep them isolated from the main experience–although this requires time, and resources.

Imagine a developer that uses project X, which works pretty well from him, but he wants to add a feature. He might be reluctant to do so because a) there’s no guarantee that they will not drop it in the future, and b) they might reject it right away because they don’t want it in their main experience. Until there is a project like Linux that incorporates all the needs of all the people, this developer would have to either fork project X, or stay in apathy.

So, don’t drop any feature, and incorporate all the features; it’s difficult, but this this is the only sure way of being as successful as Linux, and avoid forks.

Users don’t know what they want

While there is some truth to this claim, it’s a very crude oversimplification of reality. For example, users didn’t know they wanted a touch screen before the iPhone, that is true, but after that, they knew they wanted it, and if you were say Nokia, you couldn’t say “oh, users don’t know what they want” and stay with non-touch interfaces; sometimes they do know. Plus, what they definitely know is what they don’t want; like an application that crashes constantly.

This argument is often used as a rationalization when features are removed and users complain. Some developers bring this argument in a patronized way, implying that they know better. Sometimes they do know better, but this should not be used as an excuse to stop listening the complaints.


So, there you have it, it’s really simple; try really hard not to break your user’s experience, because unlike what some people like to think; it’s the users that make a project successful. The user-base is from where all the contributors come from, and the publicity. If you do break user experience, try to do it in a way that can be reverted if your users complain, or even better, provide both an old and new way of doing things; eventually the new way would be as good as the old, and everybody would move eventually.

Appreciate your users, listen to them, and never threat them as expendable, or even worst; a nuisance. Basic stuff that many big projects forget.


Looking for job

If you know me or follow my blog, it should be no surprise that I didn’t like Nokia’s move for WP7, and therefore I don’t see much point in staying.

So now I’m looking for job. I’m interested in working on the linux kernel side rather than user-space, or in the border, but I’m really open to anything.

I would appreciate any recommendations for where I should apply, or sharing my CV with somebody you know; here it is.

I will continue working on gst-dsp regardless of what happens, I still believe MeeGo has a better approach than Android, and I will continue trying to improve the device that is cooking.

Update: It looks like there was some confusion: I do think MeeGo has a better approach than Android, but for now Android is the clear winner. I wouldn’t mind working with Android, in fact, it’s so pervasive that I’m expecting that ๐Ÿ™‚

Why Linux is the most important software project in history

Here’s another post that for some people is obvious, but there are other (e.g. high level managers) that might not necessarily see the importance of Linux, in fact, I have been surprised by many open source developers who don’t seem to be familiar with how Linux works (they think it’s just something that works?). The fact of the matter is that Linux is light years ahead of any other software project, open or closed, I’ll try to explain why.

BTW. By “Linux”, I mean the kernel, not the ecosystem. Which is the name of the project.


First of all, Linux runs everywhere; desktops, smartphones, routers, web servers, supercomputers, TVs, refrigerators, tablets, even on the stock market (London, NY, Johannesburg, etc.).

Not only it runs anywhere, but in many areas it’s the undisputed #1. In smartphones, Android already has the most market share, and grabbing more and more. In supercomputers, Linux has 92% of the TOP500. On servers, it’s 64%.

There’s a lot of benefits to having a single kernel that runs on all kinds of hardware, I’ll mention two examples.

One is the improvements in power consumption that all the embedded people have been pushing for not only benefit laptops and desktops, but even servers. Also, features like dynamic power management that works really well on embedded influence the desktop and server hardware.

Another one is the VFS scalability patches. Nick Piggin found some issues with 64 CPUs machines that require some reorganization of VFS. The problem is that it’s tricky to test issues with these patches, however, since also have a real-time community (they have their own patches, but eventually will get merged), they could find issues on these patches more easily.

Here’s a nice interview with Jim Zemlin from Linux Foundation that explains where Linux comes from, where it is, how it might very well become the building block of all devices in the future.


There’s many open projects, but not many where everyone is involved and working together. The Linux Foundation issues a yearly report of how the kernel is being developed, and you can see competing companies working together, such as Red Hat (12.4%), Novell (7.0%), IBM (6.9%), Intel (5.8%), Oracle (2.3%), Renesas (1.4.%), SGI (1.3%), Fujitsu (1.2%), Nokia (1.0%), HP (1.0%), Google (0.8%), AMD (0.8%), etc.

This is true synergy, not the management bullshit kind; nobody alone (Microsoft) can compete with what everyone together can produce (Linux).


The traffic of the main mailing list (LKML) is astronomical; 250 messages per day, but that’s only one mailing list, there are around 200 subsystem lists. Many of these lists have a lot of traffic as well, one I’m subscribed to is linux-media, which has around 30 messages per day.

And there’s a reason why so many people can follow so much traffic without it becoming a total mess. All kernel mailing lists follow common guidelines; you don’t have to be subscribed!, don’t do reply-to munging, encourage cross-posting, cc the right people, trim unnecessary context, and don’t top post.

Also very important is to send patches through the mailing list. You don’t even have to think about it, just type ‘git send-email’ with a proper –cc-cmd, and Linux’s script would find the right maintainers and contributors to cc, and the proper mailing lists to send the patch to. Again, no need to be subscribed. More next.


I have explained before why sending patches through the mailing list is superior to bugzilla. But it can be summarized as; you don’t need to be subscribed, you don’t need to login anywhere, you don’t need to search for the right component, etc.

As an example I put myself. I am paid by Nokia to work on GStreamer stuff, yet, even though I have a bugzilla account and everything, it’s easier for me to submit (and get merged) patches to Linux (mostly on my free time); it’s just one command. It’s not only easier for me to submit patches, but also to review them; just click reply. I’m not even going to mention closed source, which is horrible in this area.

It is also rewarding that usually that the response to patches is immediate, however, sometimes there are so many comments that patch series have 3, 5, 10, even 30 revisions before they are accepted. This is great for quality reasons, not only for for the project, the developers involved also learn a lot.

Using a mailing list also means that it’s easy to switch from reviewing a patch to a new discussion, based on the patch, thus helping communication.


It is not surprising that such a fine tuned process the results of producing stable releases each three months like clockwork, introducing major features, and from 5 to 6 patches per hour.

I can’t really explain how many things are going on in Linux, but Jonathan Corbet does in his Kernel Report.


So, Linux is an incredibly massive endeavor, easy and fun to work with, an unstoppable behemoth in the software industry, and IMO companies trying to stand in its way are going to realize their mistake in a painful way.

MeeGo scales, because Linux scales

To me, and a lot of people, it’s obvious why MeeGo scales to a wide variety of devices, but apparently that’s not clear to other people, so I’ll try to explain why that’s the case.

First, let’s divide the operating system:

  1. Kernel
  2. Drivers
  3. Adaptation
  4. System Frameworks
  5. Application Framework
  6. Applications

“Linux” can mean many things, in the case of Android, Linux means mostly the Kernel (which is heavily modified), and in some cases the Drivers (although sometimes they have to be written from scratch), but all the layers above are specific to Android.

On Maemo, MeeGo, Moblin, and LiMo, “Linux” means an upstream Kernel (no drastic changes), upstream Drivers (which means they can be shared with other upstream players as they are), but also means “Linux ecosystem”; D-Bus,, GStreamer, GTK+/Qt/EFL, etc. Which means they take advantage of already existing System and Application Frameworks. And all they have to do, is build the Applications, which is not an easy task, but certainly easier than having to do all the previous ones.

Now, the problem when creating MeeGo, is that for reasons I won’t (can’t?) explain here, Maemo and Moblin were forced to switch from GTK+ to Qt. This might have been the right move in the long term, but it means rewriting two very big layers of the operating system, in fact, the two layers that differentiate the various mobile platforms for the most part. And this of course means letting go of a lot of talent that helped build both Maemo and Moblin.

For better or worse, the decision was made, and all we could do is ride along with it. And maturizing MeeGo, essentially means maturizing these two new layers being written not entirely from scratch (as Qt was already there), but pretty much (as you have to add new features to it, and build on top).

Now, did MeeGo fail? Well, I don’t know when this UI can be considered mature enough, but sooner or later, it will be (I do think it will be soon). The timeframe depends also on your definition of “mature”, but regardless of that, it will happen. After that, MeeGo will be ready to ship on all kinds of devices. All the hardware platform vendors have to do, is write the drivers, and the adaptation, and they already do anyway for other sw platforms.

Needless to say, the UI is irrelevant to the hardware platform.

So, here’s the proof that the lower layers are more than ready:

Just after a few months of announcing MeeGo IVI, these guys were able to write a very impressive application thanks to QML, and ignore the official UI.

The OMAP4 guys went for the full MeeGO UI. No problems.

Even though Freescale is probably not that committed to MeeGo, it’s easier to create demo using it (Qt; Nomovok) rather than other platforms. It’s even hardware accelerated.

Renesas also chose the Nomovok demo to show their hardware capabilities.

MeeGo 1.1 running on HTC’s HD2

One guy; yes, one guy. Decides to run MeeGo on his HTC, and succeeds. Of course, he uses the work already done by Ubuntu for HD2, but since MeeGo is close to upstream, the same kernel can be used. Sure, it’s slow (no hardware acceleration), and there’s many things missing, but for a short amount of time spent by hobbyists, that’s pretty great already.

This is one is not so impressive, but also shows the work of one guy porting MeeGo to Nexus S

And running on Archos 9. Not very impressive UI, but the point is that it runs on this hw.


So, as you can see MeeGo is already supported in many hardware platforms; not because the relevant companies made a deal with Nokia or Intel; they don’t have to. The only thing they have to do is support Linux; Linux is what allows them to run MeeGo, and Linux is what allows MeeGo to run on any hardware platform.

This is impossible with WP7 for numerous reasons; it’s closed source, it’s proprietary, it’s Microsoft, etc. It’s not so impossible to do the same with Android, but it’s more difficult than with MeeGo because they don’t share anything with a typical linux ecosystem; they are on a far away island on their own.

Nokia; from a burning platform, to a sinking platform

I’ve been thinking a lot about this decision to use WP7 from Nokia, as I’m sure many people have, but I’ve wanted to wait for the dust to settle down before blogging, so here’s what I think; it doesn’t make any sense from any point of view.

Technically, there is nothing that can compare to the linux kernel, which works on everything; supercomputers, mobile phones, TVs, routers, web servers, desktops, refrigerators, etc. Not only does it work, but it works well, much better than everything else. As an example, the work that has been done to scale linux’s vfs to many processors (64) does benefit embedded, because some operations are more granular. Or the work on power management lead by embedded helps web servers, where decreasing power consumption is also very much wanted. This creates a environment of synergy never seen before, where even competitors work together. Linux won the kernel race, and its use would only increase; the ones that try to fight against it would only fail miserably.
Continue reading

Mercurial vs Git; it’s all in the branches

I have been thinking on comparing Git to other SCM’s. I’ve analyzed several of them and so far I’ve only done the comparison with Monotone (here). Nowadays it’s pretty clear that Git is the most widely used DSCM out there, but Mercurial is the second. So, here is a comparison between Git and Mercurial.

Note: I’m a long-time advocate of Git, and a contributor to the project, so I am biased (but I hope my conclusions are not).

Google’s analysis

So, let’s start with Google’s analysis where they compared Git with Mercurial in order to choose what to use in Google Code. Here I’ll tackle what Google considered to be Mercurial advantages.

Learning Curve. Git has a steeper learning curve than Mercurial due to a number of factors. Git has more commands and options, the volume of which can be intimidating to new users. Mercurial’s documentation tends to be more complete and easier for novices to read. Mercurial’s terminology and commands are also a closer to Subversion and CVS, making it familiar to people migrating from those systems.

That is a value judgement and while that seems to be the consensus, I don’t think Git is much more difficult than Mercurial. In the last Git user survey, people answered the question ‘Have you found Git easy to learn?’ mostly as ‘Reasonably easy’ and few as ‘Hard’ or ‘Very Hard’. Plus, the documentation is not bad, as people answered ‘How useful have you found the following forms of Git documentation?‘ very positively. However, it’s worth noting that the best documentation is online (not the official one) (according to the survey).

Regarding the amount of commands, most of them can be ignored. If you type ‘git’ only the important ones would be listed.

The last point is that Mercurial is easier for people migrating from Subversion and CVS; that is true, but personally I don’t consider that’s a good thing. IMO it’s perpetuating bad habits and mental models. For people that have not been tainted by CVS/Subversion, it might be easier to learn Git.

Windows Support. Git has a strong Linux heritage, and the official way to run it under Windows is to use cygwin, which is far from ideal from the perspective of a Windows user. A MinGw based port of Git is gaining popularity, but Windows still remains a “second class citizen” in the world of Git. Based on limited testing, the MinGW port appeared to be completely functional, but a little sluggish. Operations that normally felt instantaneous on Linux or Mac OS X took several tenths of a second on Windows. Mercurial is Python based, and the official distribution runs cleanly under Windows (as well as Linux, Mac OS X, etc).

That was probably true at the time, but nowadays msysGit works perfectly fine. There has been a lot of work to make Git more portable, and the results have been positive.

Maintenance. Git requires periodic maintenance of repositories (i.e. git-gc), Mercurial does not require such maintenance. Note, however, that Mercurial is also a lot less sophisticated with respect to managing the clients disk space (see Client Storage Management above).

Not any more.

History is Sacred. Git is extremely powerful, and will do almost anything you ask it to. Unfortunately, this also means that Git is perfectly happy to lose history. For example, git-push –force can result in revisions becoming lost in the remote repository. Mercurial’s repository is structured more as an ever-growing collection of immutable objects. Although in some cases (such as rebase), rewriting history can be an advantage, it can also be dangerous and lead to unexpected results. It should be noted, however, that a custom Git server could be written to disallow the loss of data, so this advantage is minimal.

This was an invalid argument from the beginning. Whether history is sacred or not depends on the project, many Git projects have such policy, and they don’t allow rebases of already published branches. You don’t need your SCM to be designed specifically to disallow your developers to do something (in fact rebases are also possible in Mercurial); this should be handled as a policy. If you really want to prevent your developers from doing this, it’s easy to do that with a Git hook. So really, Mercurial doesn’t have any advantage here.

In terms of implementation effort, Mercurial has a clear advantage due to its efficient HTTP transport protocol.

Git has had smart HTTP support since quite some time now.

So, of all the supposed advantages of Mercurial, only the learning curve stands, and I already explained, it’s not that strong of an argument.

IMO Google made a bad decision; it was a matter of time before Git resolved the issues they listed, and in fact Google could have helped to achieve them. Now they are thinking on adding Git support; “We’re listening, we promise. This is one of the most starred issues in our whole bugtracker.” (Update: already done). My recommendation to other people facing similar decisions is to choose the project that has a brighter future, chances are the “disadvantages” you see in the present would be resolved soon enough.

stackoverflow’s comparison

The best comparison I’ve seen is the one on stackoverflow’s question Git and Mercurial – Compare and Contrast, the answer from Jakub Narebski is simply superb.

Here’s the summary:

  • Repository structure: Mercurial doesn’t allow octopus merges (with more than two parents), nor tagging non-commit objects.
  • Tags: Mercurial uses versioned .hgtags file with special rules for per-repository tags, and has also support for local tags in .hg/localtags; in Git tags are refs residing in refs/tags/ namespace, and by default are autofollowed on fetching and require explicit pushing.
  • Branches: In Mercurial basic workflow is based on anonymous heads; Git uses lightweight named branches, and has special kind of branches (remote-tracking branches) that follow branches in remote repository.
  • Revision naming and ranges: Mercurial provides revision numbers, local to repository, and bases relative revisions (counting from tip, i.e. current branch) and revision ranges on this local numbering; Git provides a way to refer to revision relative to branch tip, and revision ranges are topological (based on graph of revisions)
  • Mercurial uses rename tracking, while Git uses rename detection to deal with file renames
  • Network: Mercurial supports SSH and HTTP “smart” protocols; modern Git supports SSH, HTTP and GIT “smart” protocols, and HTTP(S) “dumb” protocol. Both have support for bundles files for off-line transport.
  • Mercurial uses extensions (plugins) and established API; Git has scriptability and established formats.

If you read the whole answer you would see that most of the differences between Mercurial and Git are very subtle, and most people wouldn’t even notice them, however, there’s a fundamental difference that I’ll tackle in the next section: branches.

It’s all in the branches

Update: here’s a new blog post that goes deeper into the branching model differences.

Let’s go directly for an example. Say I have a colleague called Bob, and he is working on a new feature, and create a temporary branch called ‘do-test’, I want to merge his changes to my master branch, however, the branch is so simple that I would prefer it to be hidden from the history.

In Git I can do that in the following way:
git checkout -b tmp-do-test bob/do-test
git rebase master
git mergetool # resolve conflicts
git rebase --continue
git mergetool # resolve conflicts
git rebase --continue
git checkout master
git merge tmp-do-test
git branch -D tmp-do-test

(Of course this is a simplified case where a single ‘git cherry-pick’ would have done the trick)

Voilรก. First I created a temporary branch ‘tmp-do-test’ that is equal to Bob’s ‘do-test’, then I rebase it on top of my master branch and resolve the conflicts, Git is smart enough to notice that by not picking any line of Bob’s last commit, the commit should be dropped. Then I go to the master branch and merge this temporary branch, and finally remove that temporary branch. I do variations of this flow quite a lot.

This is much more tricky in Mercurial (if possible). Why?

hg branch != git branch

In Git, a branch is merely one of the many kinds of ‘refs’, and a ‘ref’ is simply a pointer to a commit. This means that there’s nothing fundamentally different between ‘bob/do-test’ or ‘tmp-do-test’, or ‘master’, they are all pointers to commits, and these pointers can be easily be deleted, renamed, fetched, pushed, etc. IOW you can do pretty much whatever you want with them.

In Mercurial, a branch is embedded in a commit; a commit done in the ‘do-test’ branch will always remain in such a branch. This means you cannot delete, or rename branches, because you would be changing the history of the commits on those branches. You can ‘close’ branches though. As Jakub points out, these “named branches” can be better thought as “commit labels”.


However, there’s an extension (now part of the core) that is relatively similar to Git branches, the bookmarks. These are like Git ‘refs’, and although they were originally intended to be kept local, since Mercurial 1.6 they can be shared, but of course, both sides would need this extension enabled. Even then, there is no namespace to delimit say, my ‘do-test’ bookmark from Bob’s ‘do-test’ bookmark, like Git automatically does.


In the future perhaps Mercurial would add namespaces to bookmarks, and would make the bookmarks and rebase extensions part of the core. At this point it would be clear that traditional Mercurial branches didn’t make much sense anyway, and they might be dropped. And then of course there would not be much practical difference between Mercurial and Git, but in the meantime Git branches are simply better.


The conclusion shouldn’t be that surprising; Git wins. The way Git deals with branches is incredibly simple, and yet so powerful, that I think that’s more than enough reason to crush Mercurial in respect to functionality. Of course, for simple use-cases they are the same, and perhaps Mercurial would be easier for some people, but as soon as you start doing some slightly complicated handling of branches, the “complexity” of Git translates into very simple commands that do exactly what you want. So, every workflow is supported in Git in a straight-forward way. No wonder most popular projects have chosen Git.

Projects using Git: Linux Kernel, Android, Clutter, Compiz Fusion, Drupal, Fedora, FFmpeg (and libav), (X.Org, Cairo, D-Bus, Mesa3D), GCC, GNOME, GStreamer, KDE, LibreOffice, Perl5, PulseAudio, Qt, Ruby on Rails, Samba, Wine, VLC

Update: Eclipse is also using Git for a bunch of projects.

Projects using Mercurial: Adium, Go Programming Language, Mozilla, OpenJDK, OpenSolaris, Pidgin, Python, Vim

My ARM development notes

These are my notes to get useful cross-compilation, even with autotools, and GStreamer stuff.


The convention is to have ‘arm-linux-gcc‘ and so on, so that you can compile with ‘make CROSS_COMPILE=arm-linux-‘, the kernel and many other projects assume this is the default.

First, you would need ‘~/bin‘ to be on your path, so make sure you have it on ‘~/.bash_profile‘ (export PATH="$HOME/bin:$PATH") or whatever your favorite shell uses.

I use CodeSourcery (GNU/Linux 2009q3), you can fetch it from here.

cd ~/bin
for x in $toolchain/bin/arm-none-linux-gnueabi-*
ln -s $x arm-linux-${x#$toolchain/bin/arm-none-linux-gnueabi-}


This is needed for sb2 in order to kind of emulate an ARM system.

git clone git://
cd qemu
git checkout -b stable v0.12.5
./configure --prefix=/opt/qemu --target-list=arm-linux-user
make install


This is needed to avoid most of the pain caused by autotools (thank you GNU… not!).

git clone git://
cd scratchbox2
git checkout -b stable 2.1
./ --prefix=/opt/sb2
make install

Add sb2 to the PATH:
export PATH=/opt/sb2/bin:$PATH

sb2 target

Now it’s time to configure a target.

cd /opt/arm-2009q3/arm-none-linux-gnueabi/libc/
sb2-init -c /opt/qemu/bin/qemu-arm armv7 /opt/arm-2009q3/bin/arm-none-linux-gnueabi-gcc

You can check that it works with:
sb2 gcc --version


We are going to install everything into ‘/opt/arm/gst‘, so:

export PKG_CONFIG_PATH=/opt/arm/gst/lib/pkgconfig

You can skip the steps here and go directly to deployment if you download and extract this tarball on your target.


This is needed by GLib’s gio (which cannot be configured out).

wget -c
tar -xf zlib-1.2.5.tar.gz
cd zlib-1.2.5
sb2 ./configure --prefix=/opt/arm/gst
sb2 make install


GLib has bugs (623473, 630910) detecting zlib (thank you Mattias… not!). So either apply my patches, or do the C_INCLUDE_PATH/LDFLAGS hacks below:

export C_INCLUDE_PATH='/opt/arm/gst/include' LDFLAGS='-L/opt/arm/gst/lib'

git clone git://
cd glib
git checkout -b stable 2.24.1
./ --noconfigure
sb2 ./configure --prefix=/opt/arm/gst --disable-static --with-html-dir=/tmp/dump
sb2 make install


git clone git://
cd gstreamer
git checkout -b stable RELEASE-0.10.29
./ --noconfigure
sb2 ./configure --prefix=/opt/arm/gst --disable-nls --disable-static --disable-loadsave --with-html-dir=/tmp/dump
sb2 make install


Needed by many GStreamer components.

git clone git://
cd liboil
git checkout -b stable liboil-0.3.17
./ --noconfigure
sb2 ./configure --prefix=/opt/arm/gst --disable-static --with-html-dir=/tmp/dump
sb2 make install


git clone git://
cd gst-plugins-base
git checkout -b stable RELEASE-0.10.29
./ --noconfigure
sb2 ./configure --prefix=/opt/arm/gst --disable-nls --disable-static --with-html-dir=/tmp/dump
sb2 make install


git clone git://
cd gst-plugins-good
git checkout -b stable RELEASE-0.10.23
./ --noconfigure
sb2 ./configure --prefix=/opt/arm/gst --disable-nls --disable-static --with-html-dir=/tmp/dump
sb2 make install


So now we have everything installed in ‘/opt/arm/gst‘, but how to run on the target? Just copy the exact same files into the target on the exact same location, and then:

export PATH=/opt/arm/gst/bin:$PATH

That’s it, you can run gst-launch, gst-inspect, and so on.


Ok, it should be clear how to do development from the previous steps, but in case it wasn’t clear, here’s how to:


Each time you want to cross-compile, you need to tell pkg-config where to find the packages:

export PKG_CONFIG_PATH=/opt/arm/gst/lib/pkgconfig

git clone git://
cd gst-dsp
git checkout -b stable v0.8.0

Note that gst-dsp doesn’t use autotools, so sb2 is not needed.

Now, once you have the plugin (, copy to ‘/opt/arm/gst/lib/gstreamer-0.10‘ on the target.

And finally, you can run real gst-launch pipelines:
gst-launch playbin2 uri=file://$PWD/file.avi

Note: If you are missing some elements, play around with flags (flags=65 for native video-only)

Do some more development, type make, copy, repeat ๐Ÿ™‚

Enjoy ๐Ÿ˜‰