Getting proxy support on GNOME; for real (libproxy-simple)

If you open gnome-network-properties and setup your proxy accordingly you would notice that most network applications don’t really work. There are several reasons for that.

Since as long as I can remember I have looked for a way to add proxy support on my applications. There has been a number of network libraries, none of which have ever fitted the bill for me. Finally, came GIO, now part of GLib, which has many advantages (although IMO not ideal; it works perfectly). Unfortunately, no proxy support.

There’s also libproxy, which one might think has proxy support, but it doesn’t; it merely provides a mechanism to find proxy configuration. This is good, but not the full picture.

But since GIO has plug-ins, it might be possible to provide proxy support transparently, with the help of libproxy… right? That’s where GLib networking comes. Great! So we have everything we need.

Not really. Continue reading

Pidgin picking the wrong DVCS again; Mercurial

I have long criticized Pidgin’s move to Monotone. I have tried to analyse this rather marginal DVCS tool, I wrote my own mtn->git conversion tool, and I helped validate and improve Monotone’s official tool afterwards. I have spent countless hours identifying Pidgin’s contributors, finding their real names, email address, etc. I have also manually dug through old commits to properly identify the real authors of a patch (as opposed to the committer). Finally, I have also wrote scripts that automatically create a nice, clean, git repo.

pidgin-git-import

Now that Pidgin is thinking on switching away from Monotone (which was my recommendation long time ago), they are considering Mercurial, and I’ve also helped on their conversion scripts.

pidgin-mtn-conv-files

However, I feel they are making yet another mistake, because they haven’t actually analysed their decision. In order to demonstrate the double standards, cognitive dissonance, and lack of follow up, I’m posting chunks of the recorded history on the mailing list, with links.

Continue reading

msn-pecan 0.1.2 released; critical bug-fix

Hi,

This is an important maintenance release, everybody should update. Apparently Microsoft shut down the Nexus servers that were used for authentication, which meant msn-pecan stopped working completely. Fortunately I found a trick to circumvent the problem without requiring an update the protocol used; use Passport 3.0 authentication. Had I known the fix would have been so easy I would have tried earlier. Sorry for the long delay.

Plus:

  • Fix authorization; Passport 3.0 instead of nexus
  • Fix offline message reception
  • Improve reconnection/disconnection detection
  • Ignore reverse-list; the server returns wrong info
  • Show correct alias in chat window

Enjoy :)

Demétrio (1):
     Ignore reverse presence

Felipe Contreras (19):
     plugin: show proper alias in the chat window
     contactlist: fix for existing "null" group
     Improve disconnections
     ns: add time out detection
     Ignore reverse list completely
     Update libmspack to 0.2 alpha
     libmspack: fix compilation warnings
     libsiren: fix compilation warnings
     Fix some compilation warnings
     build: check for more warnings
     Fix authentication
     Improve auth parsing
     auth: reorganize to have a callback
     auth: add private header
     oim: use generic auth stuff
     oim: improve and fix message parsing
     Get rid of pn_auth_start()
     hack: mingw32 workarounds
     win32: tag 0.1.2

Download from the (usual place).

Why Linux is the most important software project in history

Here’s another post that for some people is obvious, but there are other (e.g. high level managers) that might not necessarily see the importance of Linux, in fact, I have been surprised by many open source developers who don’t seem to be familiar with how Linux works (they think it’s just something that works?). The fact of the matter is that Linux is light years ahead of any other software project, open or closed, I’ll try to explain why.

BTW. By “Linux”, I mean the kernel, not the ecosystem. Which is the name of the project.

Everywhere

First of all, Linux runs everywhere; desktops, smartphones, routers, web servers, supercomputers, TVs, refrigerators, tablets, even on the stock market (London, NY, Johannesburg, etc.).

Not only it runs anywhere, but in many areas it’s the undisputed #1. In smartphones, Android already has the most market share, and grabbing more and more. In supercomputers, Linux has 92% of the TOP500. On servers, it’s 64%.

There’s a lot of benefits to having a single kernel that runs on all kinds of hardware, I’ll mention two examples.

One is the improvements in power consumption that all the embedded people have been pushing for not only benefit laptops and desktops, but even servers. Also, features like dynamic power management that works really well on embedded influence the desktop and server hardware.

Another one is the VFS scalability patches. Nick Piggin found some issues with 64 CPUs machines that require some reorganization of VFS. The problem is that it’s tricky to test issues with these patches, however, since also have a real-time community (they have their own patches, but eventually will get merged), they could find issues on these patches more easily.

Here’s a nice interview with Jim Zemlin from Linux Foundation that explains where Linux comes from, where it is, how it might very well become the building block of all devices in the future.

Collaboration

There’s many open projects, but not many where everyone is involved and working together. The Linux Foundation issues a yearly report of how the kernel is being developed, and you can see competing companies working together, such as Red Hat (12.4%), Novell (7.0%), IBM (6.9%), Intel (5.8%), Oracle (2.3%), Renesas (1.4.%), SGI (1.3%), Fujitsu (1.2%), Nokia (1.0%), HP (1.0%), Google (0.8%), AMD (0.8%), etc.

This is true synergy, not the management bullshit kind; nobody alone (Microsoft) can compete with what everyone together can produce (Linux).

Communication

The traffic of the main mailing list (LKML) is astronomical; 250 messages per day, but that’s only one mailing list, there are around 200 subsystem lists. Many of these lists have a lot of traffic as well, one I’m subscribed to is linux-media, which has around 30 messages per day.

And there’s a reason why so many people can follow so much traffic without it becoming a total mess. All kernel mailing lists follow common guidelines; you don’t have to be subscribed!, don’t do reply-to munging, encourage cross-posting, cc the right people, trim unnecessary context, and don’t top post.

Also very important is to send patches through the mailing list. You don’t even have to think about it, just type ‘git send-email’ with a proper –cc-cmd, and Linux’s get_maintainer.pl script would find the right maintainers and contributors to cc, and the proper mailing lists to send the patch to. Again, no need to be subscribed. More next.

Patches

I have explained before why sending patches through the mailing list is superior to bugzilla. But it can be summarized as; you don’t need to be subscribed, you don’t need to login anywhere, you don’t need to search for the right component, etc.

As an example I put myself. I am paid by Nokia to work on GStreamer stuff, yet, even though I have a bugzilla account and everything, it’s easier for me to submit (and get merged) patches to Linux (mostly on my free time); it’s just one command. It’s not only easier for me to submit patches, but also to review them; just click reply. I’m not even going to mention closed source, which is horrible in this area.

It is also rewarding that usually that the response to patches is immediate, however, sometimes there are so many comments that patch series have 3, 5, 10, even 30 revisions before they are accepted. This is great for quality reasons, not only for for the project, the developers involved also learn a lot.

Using a mailing list also means that it’s easy to switch from reviewing a patch to a new discussion, based on the patch, thus helping communication.

Speed

It is not surprising that such a fine tuned process the results of producing stable releases each three months like clockwork, introducing major features, and from 5 to 6 patches per hour.

I can’t really explain how many things are going on in Linux, but Jonathan Corbet does in his Kernel Report.

Conclusion

So, Linux is an incredibly massive endeavor, easy and fun to work with, an unstoppable behemoth in the software industry, and IMO companies trying to stand in its way are going to realize their mistake in a painful way.

MeeGo scales, because Linux scales

To me, and a lot of people, it’s obvious why MeeGo scales to a wide variety of devices, but apparently that’s not clear to other people, so I’ll try to explain why that’s the case.

First, let’s divide the operating system:

  1. Kernel
  2. Drivers
  3. Adaptation
  4. System Frameworks
  5. Application Framework
  6. Applications

“Linux” can mean many things, in the case of Android, Linux means mostly the Kernel (which is heavily modified), and in some cases the Drivers (although sometimes they have to be written from scratch), but all the layers above are specific to Android.

On Maemo, MeeGo, Moblin, and LiMo, “Linux” means an upstream Kernel (no drastic changes), upstream Drivers (which means they can be shared with other upstream players as they are), but also means “Linux ecosystem”; D-Bus, X.org, GStreamer, GTK+/Qt/EFL, etc. Which means they take advantage of already existing System and Application Frameworks. And all they have to do, is build the Applications, which is not an easy task, but certainly easier than having to do all the previous ones.

Now, the problem when creating MeeGo, is that for reasons I won’t (can’t?) explain here, Maemo and Moblin were forced to switch from GTK+ to Qt. This might have been the right move in the long term, but it means rewriting two very big layers of the operating system, in fact, the two layers that differentiate the various mobile platforms for the most part. And this of course means letting go of a lot of talent that helped build both Maemo and Moblin.

For better or worse, the decision was made, and all we could do is ride along with it. And maturizing MeeGo, essentially means maturizing these two new layers being written not entirely from scratch (as Qt was already there), but pretty much (as you have to add new features to it, and build on top).

Now, did MeeGo fail? Well, I don’t know when this UI can be considered mature enough, but sooner or later, it will be (I do think it will be soon). The timeframe depends also on your definition of “mature”, but regardless of that, it will happen. After that, MeeGo will be ready to ship on all kinds of devices. All the hardware platform vendors have to do, is write the drivers, and the adaptation, and they already do anyway for other sw platforms.

Needless to say, the UI is irrelevant to the hardware platform.

So, here’s the proof that the lower layers are more than ready:

Just after a few months of announcing MeeGo IVI, these guys were able to write a very impressive application thanks to QML, and ignore the official UI.

The OMAP4 guys went for the full MeeGO UI. No problems.

Even though Freescale is probably not that committed to MeeGo, it’s easier to create demo using it (Qt; Nomovok) rather than other platforms. It’s even hardware accelerated.

Renesas also chose the Nomovok demo to show their hardware capabilities.

MeeGo 1.1 running on HTC’s HD2

One guy; yes, one guy. Decides to run MeeGo on his HTC, and succeeds. Of course, he uses the work already done by Ubuntu for HD2, but since MeeGo is close to upstream, the same kernel can be used. Sure, it’s slow (no hardware acceleration), and there’s many things missing, but for a short amount of time spent by hobbyists, that’s pretty great already.

This is one is not so impressive, but also shows the work of one guy porting MeeGo to Nexus S

And running on Archos 9. Not very impressive UI, but the point is that it runs on this hw.

Conclusion

So, as you can see MeeGo is already supported in many hardware platforms; not because the relevant companies made a deal with Nokia or Intel; they don’t have to. The only thing they have to do is support Linux; Linux is what allows them to run MeeGo, and Linux is what allows MeeGo to run on any hardware platform.

This is impossible with WP7 for numerous reasons; it’s closed source, it’s proprietary, it’s Microsoft, etc. It’s not so impossible to do the same with Android, but it’s more difficult than with MeeGo because they don’t share anything with a typical linux ecosystem; they are on a far away island on their own.

Mercurial vs Git; it’s all in the branches

I have been thinking on comparing Git to other SCM’s. I’ve analyzed several of them and so far I’ve only done the comparison with Monotone (here). Nowadays it’s pretty clear that Git is the most widely used DSCM out there, but Mercurial is the second. So, here is a comparison between Git and Mercurial.

Note: I’m a long-time advocate of Git, and a contributor to the project, so I am biased (but I hope my conclusions are not).

Google’s analysis

So, let’s start with Google’s analysis where they compared Git with Mercurial in order to choose what to use in Google Code. Here I’ll tackle what Google considered to be Mercurial advantages.

Learning Curve. Git has a steeper learning curve than Mercurial due to a number of factors. Git has more commands and options, the volume of which can be intimidating to new users. Mercurial’s documentation tends to be more complete and easier for novices to read. Mercurial’s terminology and commands are also a closer to Subversion and CVS, making it familiar to people migrating from those systems.

That is a value judgement and while that seems to be the consensus, I don’t think Git is much more difficult than Mercurial. In the last Git user survey, people answered the question ‘Have you found Git easy to learn?’ mostly as ‘Reasonably easy’ and few as ‘Hard’ or ‘Very Hard’. Plus, the documentation is not bad, as people answered ‘How useful have you found the following forms of Git documentation?‘ very positively. However, it’s worth noting that the best documentation is online (not the official one) (according to the survey).

Regarding the amount of commands, most of them can be ignored. If you type ‘git’ only the important ones would be listed.

The last point is that Mercurial is easier for people migrating from Subversion and CVS; that is true, but personally I don’t consider that’s a good thing. IMO it’s perpetuating bad habits and mental models. For people that have not been tainted by CVS/Subversion, it might be easier to learn Git.

Windows Support. Git has a strong Linux heritage, and the official way to run it under Windows is to use cygwin, which is far from ideal from the perspective of a Windows user. A MinGw based port of Git is gaining popularity, but Windows still remains a “second class citizen” in the world of Git. Based on limited testing, the MinGW port appeared to be completely functional, but a little sluggish. Operations that normally felt instantaneous on Linux or Mac OS X took several tenths of a second on Windows. Mercurial is Python based, and the official distribution runs cleanly under Windows (as well as Linux, Mac OS X, etc).

That was probably true at the time, but nowadays msysGit works perfectly fine. There has been a lot of work to make Git more portable, and the results have been positive.

Maintenance. Git requires periodic maintenance of repositories (i.e. git-gc), Mercurial does not require such maintenance. Note, however, that Mercurial is also a lot less sophisticated with respect to managing the clients disk space (see Client Storage Management above).

Not any more.

History is Sacred. Git is extremely powerful, and will do almost anything you ask it to. Unfortunately, this also means that Git is perfectly happy to lose history. For example, git-push –force can result in revisions becoming lost in the remote repository. Mercurial’s repository is structured more as an ever-growing collection of immutable objects. Although in some cases (such as rebase), rewriting history can be an advantage, it can also be dangerous and lead to unexpected results. It should be noted, however, that a custom Git server could be written to disallow the loss of data, so this advantage is minimal.

This was an invalid argument from the beginning. Whether history is sacred or not depends on the project, many Git projects have such policy, and they don’t allow rebases of already published branches. You don’t need your SCM to be designed specifically to disallow your developers to do something (in fact rebases are also possible in Mercurial); this should be handled as a policy. If you really want to prevent your developers from doing this, it’s easy to do that with a Git hook. So really, Mercurial doesn’t have any advantage here.

In terms of implementation effort, Mercurial has a clear advantage due to its efficient HTTP transport protocol.

Git has had smart HTTP support since quite some time now.

So, of all the supposed advantages of Mercurial, only the learning curve stands, and I already explained, it’s not that strong of an argument.

IMO Google made a bad decision; it was a matter of time before Git resolved the issues they listed, and in fact Google could have helped to achieve them. Now they are thinking on adding Git support; “We’re listening, we promise. This is one of the most starred issues in our whole bugtracker.” (Update: already done). My recommendation to other people facing similar decisions is to choose the project that has a brighter future, chances are the “disadvantages” you see in the present would be resolved soon enough.

stackoverflow’s comparison

The best comparison I’ve seen is the one on stackoverflow’s question Git and Mercurial – Compare and Contrast, the answer from Jakub Narebski is simply superb.

Here’s the summary:

  • Repository structure: Mercurial doesn’t allow octopus merges (with more than two parents), nor tagging non-commit objects.
  • Tags: Mercurial uses versioned .hgtags file with special rules for per-repository tags, and has also support for local tags in .hg/localtags; in Git tags are refs residing in refs/tags/ namespace, and by default are autofollowed on fetching and require explicit pushing.
  • Branches: In Mercurial basic workflow is based on anonymous heads; Git uses lightweight named branches, and has special kind of branches (remote-tracking branches) that follow branches in remote repository.
  • Revision naming and ranges: Mercurial provides revision numbers, local to repository, and bases relative revisions (counting from tip, i.e. current branch) and revision ranges on this local numbering; Git provides a way to refer to revision relative to branch tip, and revision ranges are topological (based on graph of revisions)
  • Mercurial uses rename tracking, while Git uses rename detection to deal with file renames
  • Network: Mercurial supports SSH and HTTP “smart” protocols; modern Git supports SSH, HTTP and GIT “smart” protocols, and HTTP(S) “dumb” protocol. Both have support for bundles files for off-line transport.
  • Mercurial uses extensions (plugins) and established API; Git has scriptability and established formats.

If you read the whole answer you would see that most of the differences between Mercurial and Git are very subtle, and most people wouldn’t even notice them, however, there’s a fundamental difference that I’ll tackle in the next section: branches.

It’s all in the branches

Update: here’s a new blog post that goes deeper into the branching model differences.

Let’s go directly for an example. Say I have a colleague called Bob, and he is working on a new feature, and create a temporary branch called ‘do-test’, I want to merge his changes to my master branch, however, the branch is so simple that I would prefer it to be hidden from the history.

In Git I can do that in the following way:
git checkout -b tmp-do-test bob/do-test
git rebase master
git mergetool # resolve conflicts
git rebase --continue
git mergetool # resolve conflicts
git rebase --continue
git checkout master
git merge tmp-do-test
git branch -D tmp-do-test

(Of course this is a simplified case where a single ‘git cherry-pick’ would have done the trick)

Voilá. First I created a temporary branch ‘tmp-do-test’ that is equal to Bob’s ‘do-test’, then I rebase it on top of my master branch and resolve the conflicts, Git is smart enough to notice that by not picking any line of Bob’s last commit, the commit should be dropped. Then I go to the master branch and merge this temporary branch, and finally remove that temporary branch. I do variations of this flow quite a lot.

This is much more tricky in Mercurial (if possible). Why?

hg branch != git branch

In Git, a branch is merely one of the many kinds of ‘refs’, and a ‘ref’ is simply a pointer to a commit. This means that there’s nothing fundamentally different between ‘bob/do-test’ or ‘tmp-do-test’, or ‘master’, they are all pointers to commits, and these pointers can be easily be deleted, renamed, fetched, pushed, etc. IOW you can do pretty much whatever you want with them.

In Mercurial, a branch is embedded in a commit; a commit done in the ‘do-test’ branch will always remain in such a branch. This means you cannot delete, or rename branches, because you would be changing the history of the commits on those branches. You can ‘close’ branches though. As Jakub points out, these “named branches” can be better thought as “commit labels”.

Bookmarks

However, there’s an extension (now part of the core) that is relatively similar to Git branches, the bookmarks. These are like Git ‘refs’, and although they were originally intended to be kept local, since Mercurial 1.6 they can be shared, but of course, both sides would need this extension enabled. Even then, there is no namespace to delimit say, my ‘do-test’ bookmark from Bob’s ‘do-test’ bookmark, like Git automatically does.

Future

In the future perhaps Mercurial would add namespaces to bookmarks, and would make the bookmarks and rebase extensions part of the core. At this point it would be clear that traditional Mercurial branches didn’t make much sense anyway, and they might be dropped. And then of course there would not be much practical difference between Mercurial and Git, but in the meantime Git branches are simply better.

Conclusion

The conclusion shouldn’t be that surprising; Git wins. The way Git deals with branches is incredibly simple, and yet so powerful, that I think that’s more than enough reason to crush Mercurial in respect to functionality. Of course, for simple use-cases they are the same, and perhaps Mercurial would be easier for some people, but as soon as you start doing some slightly complicated handling of branches, the “complexity” of Git translates into very simple commands that do exactly what you want. So, every workflow is supported in Git in a straight-forward way. No wonder most popular projects have chosen Git.

Projects using Git: Linux Kernel, Android, Clutter, Compiz Fusion, Drupal, Fedora, FFmpeg (and libav), Freedesktop.org (X.Org, Cairo, D-Bus, Mesa3D), GCC, GNOME, GStreamer, KDE, LibreOffice, Perl5, PulseAudio, Qt, Ruby on Rails, Samba, Wine, VLC

Update: Eclipse is also using Git for a bunch of projects.

Projects using Mercurial: Adium, Go Programming Language, Mozilla, OpenJDK, OpenSolaris, Pidgin, Python, Vim

My ARM development notes

These are my notes to get useful cross-compilation, even with autotools, and GStreamer stuff.

toolchain

The convention is to have ‘arm-linux-gcc‘ and so on, so that you can compile with ‘make CROSS_COMPILE=arm-linux-’, the kernel and many other projects assume this is the default.

First, you would need ‘~/bin‘ to be on your path, so make sure you have it on ‘~/.bash_profile‘ (export PATH="$HOME/bin:$PATH") or whatever your favorite shell uses.

I use CodeSourcery (GNU/Linux 2009q3), you can fetch it from here.

cd ~/bin
toolchain=/opt/arm-2009q3
for x in $toolchain/bin/arm-none-linux-gnueabi-*
do
ln -s $x arm-linux-${x#$toolchain/bin/arm-none-linux-gnueabi-}
done

QEMU

This is needed for sb2 in order to kind of emulate an ARM system.

git clone git://git.savannah.nongnu.org/qemu.git
cd qemu
git checkout -b stable v0.12.5
./configure --prefix=/opt/qemu --target-list=arm-linux-user
make install

sbox2

This is needed to avoid most of the pain caused by autotools (thank you GNU… not!).

git clone git://gitorious.org/scratchbox2/scratchbox2.git
cd scratchbox2
git checkout -b stable 2.1
./autogen.sh --prefix=/opt/sb2
make install

Add sb2 to the PATH:
export PATH=/opt/sb2/bin:$PATH

sb2 target

Now it’s time to configure a target.

cd /opt/arm-2009q3/arm-none-linux-gnueabi/libc/
sb2-init -c /opt/qemu/bin/qemu-arm armv7 /opt/arm-2009q3/bin/arm-none-linux-gnueabi-gcc

You can check that it works with:
sb2 gcc --version

GStreamer

We are going to install everything into ‘/opt/arm/gst‘, so:

export PKG_CONFIG_PATH=/opt/arm/gst/lib/pkgconfig

You can skip the steps here and go directly to deployment if you download and extract this tarball on your target.

zlib

This is needed by GLib’s gio (which cannot be configured out).

wget -c http://zlib.net/zlib-1.2.5.tar.gz
tar -xf zlib-1.2.5.tar.gz
cd zlib-1.2.5
sb2 ./configure --prefix=/opt/arm/gst
sb2 make install

glib

GLib has bugs (623473, 630910) detecting zlib (thank you Mattias… not!). So either apply my patches, or do the C_INCLUDE_PATH/LDFLAGS hacks below:

export C_INCLUDE_PATH='/opt/arm/gst/include' LDFLAGS='-L/opt/arm/gst/lib'

git clone git://git.gnome.org/glib
cd glib
git checkout -b stable 2.24.1
./autogen.sh --noconfigure
sb2 ./configure --prefix=/opt/arm/gst --disable-static --with-html-dir=/tmp/dump
sb2 make install

gstreamer

git clone git://anongit.freedesktop.org/gstreamer/gstreamer
cd gstreamer
git checkout -b stable RELEASE-0.10.29
./autogen.sh --noconfigure
sb2 ./configure --prefix=/opt/arm/gst --disable-nls --disable-static --disable-loadsave --with-html-dir=/tmp/dump
sb2 make install

liboil

Needed by many GStreamer components.

git clone git://anongit.freedesktop.org/liboil
cd liboil
git checkout -b stable liboil-0.3.17
./autogen.sh --noconfigure
sb2 ./configure --prefix=/opt/arm/gst --disable-static --with-html-dir=/tmp/dump
sb2 make install

gst-plugins-base

git clone git://anongit.freedesktop.org/gstreamer/gst-plugins-base
cd gst-plugins-base
git checkout -b stable RELEASE-0.10.29
./autogen.sh --noconfigure
sb2 ./configure --prefix=/opt/arm/gst --disable-nls --disable-static --with-html-dir=/tmp/dump
sb2 make install

gst-plugins-good

git clone git://anongit.freedesktop.org/gstreamer/gst-plugins-good
cd gst-plugins-good
git checkout -b stable RELEASE-0.10.23
./autogen.sh --noconfigure
sb2 ./configure --prefix=/opt/arm/gst --disable-nls --disable-static --with-html-dir=/tmp/dump
sb2 make install

Deployment

So now we have everything installed in ‘/opt/arm/gst‘, but how to run on the target? Just copy the exact same files into the target on the exact same location, and then:

export PATH=/opt/arm/gst/bin:$PATH

That’s it, you can run gst-launch, gst-inspect, and so on.

Development

Ok, it should be clear how to do development from the previous steps, but in case it wasn’t clear, here’s how to:

gst-dsp

Each time you want to cross-compile, you need to tell pkg-config where to find the packages:

export PKG_CONFIG_PATH=/opt/arm/gst/lib/pkgconfig

git clone git://github.com/felipec/gst-dsp.git
cd gst-dsp
git checkout -b stable v0.8.0
make

Note that gst-dsp doesn’t use autotools, so sb2 is not needed.

Now, once you have the plugin (libgstdsp.so), copy to ‘/opt/arm/gst/lib/gstreamer-0.10‘ on the target.

And finally, you can run real gst-launch pipelines:
gst-launch playbin2 uri=file://$PWD/file.avi

Note: If you are missing some elements, play around with flags (flags=65 for native video-only)

Do some more development, type make, copy, repeat :)

Enjoy ;)

GStreamer, embedded, and low latency are a bad combination

This has been a known fact inside Nokia (MeeGo) for quite a long time due to various performance issues we’ve had to workaround, but for some reason it wasn’t acknowledged as an issue when it was brought up in the mailing list.

So, in order to proof beyond reasonable doubt that there is indeed an issue, I wrote this test. It is very minimal, there’s essentially nothing of a typical GStreamer pipeline, just an element and an app that pushes buffers to it, that’s it. But then, optionally, a queue (typical element in a GStreamer pipeline) is added in the middle, which is a thread-boundary, and then the fun begins:

Graph for x86
Graph for arm

The buffer size legends corresponds to exponentiation (5 => 2 ^ 5 = 32), and the CPU time is returned by the system (getrusage) in ms. You can see that in ARM systems not only more CPU time is wasted, but adding a queue makes things worst at a faster rate.

Note that this test is doing nothing, just pushing buffers around, all the CPU is wasted doing GStreamer operations. In a real scenario the situation is much worst because there isn’t only one, but multiple threads, and many elements involved, so this wasted CPU time I measured has to be multiplied many times.

Now, this has been profiled before, and everything points out to pthread_mutex_lock which is only a problem when there’s contention, which happens more often in GStreamer when buffers are small, then the futex syscall is issued, is very bad in ARM, although it probably depends on which specific system you are using.

Fortunately for me, I don’t need good latency, so I can just push one second buffers and forget about GStreamer performance issues, if you are experiencing the same, and can afford high latency, just increase the buffer sizes, if not, then you are screwed :p

Hopefully this answers Wim’s question of what a “small buffer” means, how it’s not good, and when it’s a problem.

Update

Ok, so the discussion about this continued in the mailing list, and it was pointed out that that the scale is logarithmic, so the exponential result was expected. While that is true, the logarithmic scale matches what people experience; how else would you plot the range from 10ms to 1s? Certainly not linearly.

But there’s a valid point; the results should not be surprising. We can take the logarithmic scale out of the equation by dividing the total CPU time by the number of buffers actually pushed, as Robert Swain did in the comments, that should give a constant number, which is the CPU time it took to do a push. The results indeed converge to a constant number:

queue: 0.078, direct: 0.011

This means that in a realistic use case of pushing one buffer each 10ms through a queue, the CPU usage on this particular processor (800mhz) is 0.78%.

Also, there’s this related old bug that recently got some attention and a new patch from Wim, so I gave it a try (I had to compile GStreamer myself so the results are not comparable with the previous runs).

Before:
queue: 0.074, direct: 0.011

After:
queue: 0.065, direct: 0.007

So the improvement for the queue case is around 12%, while the direct case is 31%.

Not bad at all, but the conclusion is still the same. If you use GStreamer, try to avoid as many elements as possible, specially queues, and try to have the biggest buffer size you can afford, which means that having good performance and low latency is tricky.

Update 2

Stefan Kost suggested to use ‘queue’ instead of ‘queue2′, and I got a pandaboard, so here are the results with OMAP4.

pandaboard (2 core, 1GHz):
queue: 0.017, direct: 0.004

N900:
queue: 0.087, direct: 0.021

i386 (2 core, 1.83GHz):
queue: 0.0059, direct: 0.0015

So, either futex got better on Cortex A9, or OMAP4 is so powerful it can’t be considered embedded :p

gst-av 0.4; better performance for flac, vorbis and mp3 (part 2)

This is a continuation of my previous post. Based on the feedback I decided to do two things; investigate the strange FLAC high CPU usage with FFmpeg, and get more accurate measurements.

GStreamer sucks

It turns out that GStreamer flac parser uses four times more CPU than FFmpeg’s decoder. Thanks to perf, I was able to quickly figure out the biggest offenders: GStreamer’s horrible bitstream reader (GST_BIT_READER_READ_BITS) was by far the worst.

53.03% libgstbase-0.10.so.0.26.0
24.78% libavcodec.so.52.72.2
17.35% libgstxiph.so
1.52% libc-2.12.1.so

This is on my laptop just running the parser (filesrc ! flacparse ! fakesink), in total it was taking 2.67s.

After reading the code and trying different things, I decided to go for something similar to what FFmpeg is doing, and I also borrowed pieces of the architecture-specific optimizations, now it even looks ok:

72.68% libavcodec.so.52.72.2
14.20% libgstxiph.so
4.00% libc-2.12.1.so

And it takes 0.81s.

But how much would this affect battery life on the N900?

Smart battery script

I tried different ideas, and after refreshing myself on statistics I wrote this script in Ruby that runs all the tests, gathers the battery capacity in a separate thread, and finally generates a report per test. Much easier than before.

Since I’m already working on FLAC, I decided to also apply some patches that split the decoder from the parser, and optimizations from Måns Rullgård (good thing I grabbed them because he seems to have left the project and deleted his repos).

Battery life graph

Battery life


Battery drain graph

Battery drain

So, yeah, much better now ;)

But how credible are these results? Well, judge by yourself, listed below are the raw measurements, the samples are the differences in capacity (mAh) measured each 10 minutes, from which the drain and battery life are calculated.

== baseline ==
samples: 3, 3, 3, 3, 4, 5
drain: 21.00±1.87mA
life: 65.39±4.77h
== av flac ==
samples: 9, 8, 8, 8, 7, 8, 7
drain: 47.14±1.45mA
life: 28.19±0.87h
== flac ==
samples: 11, 11, 11, 11, 11, 11
drain: 66.00±0.00mA
life: 20.00±0.00h
== av mp3 ==
samples: 11, 11, 11, 11, 11, 10
drain: 65.00±0.91mA
life: 20.33±0.30h
== nokiamp3 ==
samples: 12, 12, 12, 12, 12, 12
drain: 72.00±0.00mA
life: 18.33±0.00h
== av vorbis ==
samples: 10, 11, 11, 10, 11, 11
drain: 64.00±1.15mA
life: 20.67±0.38h
== vorbis ==
samples: 19, 18, 18, 19, 18, 19
drain: 111.00±1.22mA
life: 11.90±0.13h

If you are interested in the code: gst-av, gst-maemo-xiph. Enjoy ;)

gst-av 0.3; better performance for vorbis and mp3

So, I’ve been working on gst-av, a GStreamer plug-in to use FFmpeg codecs (only audio for now), in order to get it in good shape for ogg support. First, I had to fix oggdemux and flacparse to be compatible with tagreadbin, it seems I managed to do it (with the help of a patch from Sreerenj Balachandran), so now the custom tracker extractors are not needed any more.

Then, with a bit of work I managed to get not only vorbis, but flac, and mp3 working.

That was good, but was it really worth it? Tuomas Kulve did a nice comparison of gst-av vs the default vorbisdec, and I wanted to do something similar, however, running a series of tests each taking 20 hours to complete wasn’t so appealing.

So I asked in #meego and #maemo IRC channels for a simple way to measure battery drain reliably, and automatically. It seems powertop can do that on some platforms, but Maemo’s powertop is a very different beast. Fortunately, the folks at #maemo seem to have been busy trying to get all possible information from the battery, and they pointed me to a very nice powerscript. However, I got some tips to get even better results (from ShadowJK, DocScrutinizer, and SpeedEvil), and the result is this maemo-battery script (needs i2c-tools, and root permissions), which essentially prints the current charge of the battery each 10 minutes.

With this I was ready, but just to be clear how to properly measure battery draw; make sure you are in offline mode, plug your headphones (otherwise pulse-audio would run extra algorithms), and immediately blank the screen.

Here are the results (units in hours of battery life):

These results show that vorbis with FFmpeg is massively better than libvorbis, so my work wasn’t in vain :). But it’s also interesting that FFmpeg’s mp3 decoder is slightly better than Nokia’s proprietary one. Also, FFmpeg still needs some work to complete with libflac. My guess is that these decoders can’t be optimized much further; now the bottlenecks would have to be pulseaudio and gstreamer.

This is the raw data (in mA); I ran my script for one hour for each test, and some I ran multiple times just to verify; the results seem to vary ±1 mA.

current -- mp3: 63, vorbis: 110, flac: 62
gst-av -- mp3: 61, vorbis: 62, flac: 69

gst-ffmpeg

Why not use gst-ffmpeg? You might ask. Initially that’s what I tried, but it doesn’t support vorbis, nor flac, which seems to fit GStreamer’s tradition of getting away from FFmpeg as much as possible. Then when I read the code it was clear to me that it was overly complicated; I’m familiar with FFmpeg’s API (it’s unbelievably simple), so I decided to play around, and see if I could get something working; I did, and the result was incredibly simple, and oh so sweet :) As a comparison, gst-ffmpeg is 16357 lines of code, gst-av is 563 (sure, gst-av does much less; just what is needed). Another reason that goes hand-in-hand with this, is the ability to tweak it; my goal is to get the absolutely best performance, and for that I want to be able to understand what the code is doing. And finally, gst-ffmpeg is using deprecated API.

What about performance?

The difference is not that big: ~1.6h of battery life, but it’s something.

current: 63, gst-av: 61, gst-ffmpeg: 66

What now?

Now we need to package FFmpeg; probably just include the codecs we need, and then ogg support might include these instead. Any volunteers?

Update

It turns out the issue was flacparse which is total crap: it’s using 4 times more CPU time than FFmpeg’s decoder just for parsing. After fixing it now it takes only 20%. I’m trying to get new measurements in a more automated and precise way now. I’ve pushed the code to my repo already.