Felipe Contreras

It took me a lot more than I expected, but I finally managed to get the beagleboard booting and happy with the latest linux kernel (2.6.32-rc3), DSS2, and dsp-bridge driver.

And then I could run gst-dsp: a native GStreamer plug-in to access Texas Instruments’ DSP algorithms for OMAP3 platforms. Which marks the time for making a public release.

Here’s the video

The video playback is running on the beagleboard with gst-dsp and gst-omapfb (no X) with TI public DSP binaries; it’s a WVGA (800×480) MPEG-4 video. It’s not running as smoothly as I wanted; it seems the public binaries are a bit buggy, and there’s some problem with the dsp-bridge driver writing directly to the framebuffer, but at least it somewhat works

The video recording is done with an N900, official system (which uses gst-dsp ), and the resulting video is ; 848×480 MPEG-4.

You can find the demo image for the beagleboard, along with instructions, here.

The kernel code is on gitorious; the important tag is v2.6.32-felipec1.

And gst-dsp is hosted on Google Code, although the git repository is actually on github.

The code wasn’t written from scratch, TI’s projects: tiopenmax and libbridge, helped a lot.

And of course many other people made the first release possible (see the shortlog at the end).

Enjoy

Andriy Shevchenko (1):
      base: fix a crash on send codec data

Felipe Contreras (180):
      Initial commit
      Register dsp node
      Add README
      Fix and update copyrights
      Add ALLOCATE_HEAP and ALLOCATE_SN to dsp_bridge
      Add handy dsp_send_message
      dummy: use dsp_send_message
      Rename gstdsp.* to plugin.*
      Makefile: cleanup
      dummy: trivial clanups
      Add log utility
      Use log utility
      dmm_buffer: size_t improvements
      dmm_buffer: always unmap when freeing
      dmm_buffer: use getpagesize()
      dmm_buffer: alignment improvements
      dmm_buffer: add user_data field
      Add MPEG-4 video decoder
      README: update
      mp4vdec: trivial cleanup
      mp4vdec: send signal to output_loop
      mp4vdec: flush output buffers too
      mp4vdec: reset output port
      mp4vdec: extra check for null buffer
      mp4vdec: use atomic operations for status
      mp4vdec: use more atomic operations for status
      mp4vdec: send stop signal before
      mp4vdec: re-use comm buffers
      dmm_buffer: reorganize a bit
      dmm_buffer: add dmm_buffer_reserve
      dmm_buffer: allow to re-reserve memory
      dmm_buffer: allow re-mapping
      mp4vdec: trivial cleanup
      dmm_buffer: unmap before unreserving
      mp4vdec: re-use mappings for output buffers
      mp4vdec: convert flush condition to semaphore
      Remove cond.h
      Rename mp4vdec to vdec
      vdec: trivial cleanup
      vdec: trivial reorganization
      vdec: prepare for multiple algos
      vdec: move create_node to dsp_start
      vdec: start dsp node after getting the caps
      vdec: initial support for H.264
      vdec: add Juha to authors list
      README: update
      vdec: cleanup
      vdec: make dsp_thread static
      vdec: reorganize a bit
      New base class
      Add new video encoder
      base: handle more commands
      base: reorganize got_message a bit
      venc: improve jpeg args
      venc: send jpeg dynamic params
      base: cleanup setup_output_buffers
      base: remove unused buffer_count
      base: reorganize a bit
      base: add use_pad_alloc option
      base: free mapped buffers on dsp_stop()
      base: be more verbose on get_slot()
      README: update
      Makefile: check for missing symbols
      New utility gstdsp_register()
      base: detect dsp errors
      base: properly handle dsp errors
      base: post error in the bus
      base: extra check for status in outout_loop()
      base: free events array
      base: reinitialize state on NULL->READY
      base: use circular buffer for timestamps
      base: increase ts_array
      base: increase mapping cache
      dummy: reorganize map_buffer
      dummy: input buffers don't need alignment
      dummy: cleanup
      dummy: don't map buffers
      venc: increase framesize limit for jpeg
      base: add gstdsp_post_error()
      venc: allocate a buffer when framesize is unaligned
      base: decrease wait for events timeout
      base: more error messages
      base: re-initialize on READY->PAUSED
      base: don't panic on wrong status
      base: destroy node at the right time
      base: catch playback completed message
      base: possible memleak fixes
      vdec: send codec data for MPEG-4
      base: make map cache optional
      plugin: set more proper ranks
      vdec: add framerate workaround
      vdec: remove gstdsp_send_buffer()
      base: add create_node() vmethod
      base: add parsing facilities
      Add h263 parser
      parse: update framesize only when unset
      Random cleanups
      base: add support for stream params
      venc: add H.263
      venc: use h263 by default
      Reorganize encoders
      base: send codec data for all the codecs
      base: keep trying if parse func fails
      base: trivial cleanup
      Trivial cleanups
      log: don't display info level
      log: decrease log level for buffer allocs
      log: add pr_test
      base: rename array to cache
      base: rename 'buffer' to 'comm'
      base: event cleanup
      base: reorganize a bit
      base: assume output buffer is always there
      base: remove out_buffer, use port buffer
      base: store input buffer
      base: trivial cleanup
      base: flush ports on stop
      base: plug some possible leaks
      base: make map_buffer() more conservative
      base: trigger semaphore after buffer modifications
      base: re-use input buffer
      base: add port index field
      Add async queue
      base: allow multiple buffers
      base: allow child elements to configure the ports
      vdec: increase the number of buffers to 2
      venc: trivial fixes
      log: add missing include
      base: re-enable queues properly
      venc: decrease input buffer size
      base: wait for eos
      base: possible fix
      Initial MPEG-4 video encoder support
      gstdspvenc.h: preemptively add H.264 to the list
      base: add send_codec_data() helper
      vdec: use send_codec_data()
      vdec: extra checks
      Add skip hack
      Revert "venc: forcing mpeg4 I frame each i_frame_interval"
      venc: reorganize stream/dynamic params
      base: trivial cleanup
      base: properly set param virt addr
      Add param argument to buffer callbacks
      Add buffer argument to buffer callbacks
      base: add buffer recv_cb
      base: add check for end addr alignment
      vdec: fix extra unref for codec-data
      base: trivial cleanups
      Rename dmm_buffer_flush() to dmm_buffer_clean()
      base: fix memory read
      dmm_buffer: clean instead of flush
      dmm_buffer: add cache 'flush' function back
      Use more proper cache functions
      base: handle bad node termination
      base: make EOS alignment an option
      jpegenc: enable eos align
      venc: improve integer framerate calculation
      venc: fix bitrate calculation
      venc: cleanup bitrate calculation
      venc: remove jpeg from bitrate calculation
      venc: tweak bitrate calculation
      venc: trivial cleanups
      venc: add 'quality' field
      venc: calculate smaller buffer sizes
      Fix some static analysis warnings
      log: avoid pr_info when gst debugging is off
      base: remove use_map_cache
      Trivial cleanups
      Cleanup type registrations
      base: improve some compiler hints
      dmm-buffer: check cache flush size
      base: properly free node resources
      Create custom dsp_node_t
      dsp-bridge: store node heap ourselves
      dsp-bridge: store node msgbuf ourselves
      dsp-bridge: cleanup node_free
      base: copy buffers when appropriate
      base: remove unnecessary cache flushing
      venc: set rate-control to variable
      base: post critical error mesages to the bus

Hoseok Chang (1):
      venc: tune mp4v parms for better performance

Juha Alanen (5):
      vdec: set profile based on the frame size
      vdec: improve H.263 args
      vdec: initial support for WMV9
      venc: set profile correctly for H.263 and MPEG4
      venc: disable single scan output for JPEG encoder

Marco Ballesio (8):
      vdec: fix srcpad setup
      venc: rename mp4venc_stream_params
      venc: add mp4venc_out_stream_params
      base: use proper buffer length
      venc: forcing mpeg4 I frame each i_frame_interval
      venc: reordered mp4venc_args initialization
      venc: added bitrate computation formula
      venc: propagate keyframes properly

Mark Nauwelaerts (2):
      base: safer buffer allocation and freeing
      base: fix element ref leak

Miguel Verdu (2):
      venc: tune MPEG-4 parameters
      venc: tune MPEG-4 parameters for quality

René Stadler (3):
      base: fix thread leak
      base: advance timestamp pointer for empty output buffers
      base: don't use DSP flushing

Tim-Philipp Müller (1):
      base: unref unused output buffer when skipping output

I finally managed some time to make another libmtag release. This is mostly tiding up the code, cleanups, new codestyle, building improvements and moved to github

Also some handy features:

• strip tag: now you can remove say id3v1 tags while keeping id3v2 intact
• get specific tag: similarly, you can retrieve only id3v2 information
• get all: this function might is useful for some UIs, specify some callback, and get all the tags

I wonder why not so many people are using this, is there a better tagging library?

Any packagers interested?

Grab it while it’s hot

I read Christian Schaller’s post which says essentially that Google is evil because it’s using FFmpeg in Chromium and arguing the legality of it by using an unusual interpretation of LGPL. because Google is using an unusual interpretation of LGPL.

Let’s step back for a second and see this through the point of view of FFmpeg developers. Probably they, just like any other FOSS developer, just want to code kick-ass software, and the license (LGPL) is just a tool to make sure their software is not stolen, so if a company chooses to use FFmpeg, they must contribute the changes back.

Google is contributing back their changes, they are publicly available in their repo. So how could FFmpeg developers loose? Their code will be used in an amazing product, that they will probably use too, it’s a win-win situation. This subject was brought up on the mailing list and nobody complained; Chromium is not listed in FFmpeg’s hall of shame.

Now, is it legal? The only way to know is to bring the issue to court and see the resolution, but who will bring this to court? FFmpeg developers? No. Perhaps H.264 patent holders, but only in countries with wicked patent laws (USA), and who cares about them? If Google looses in court against H.264 patent holders, I wouldn’t consider it evil, quite the contrary.

And finally, Google lawyers said that what the FSF thinks about this movement is irrelevant… damn right! They wrote the license, so? FFmpeg choose the license, and FFmpeg can choose another license if they so wish, in fact, they can re-license to Google in any other license they see fit because they have the copyright. Of course there’s no need for that because LGPL works fine, and the explanation of Google lawyers is completely logical to me.

Personally I cheer the Google Chromium team for such a bold movement, and congratulate FFmpeg developers for making kick-ass software that might soon be even more widely used.

It took some time but finally tiopenmax 0.3.5 was released. It’s essentially 0.3 plus DSP binaries that actually work.

I verified with gst-openmax (git omap branch) and they work just fine Thanks Daniel Díaz!

So people with OMAP3 hardware (beagleboard) can already try D1 MPEG-4 decoding using less than 15% of CPU. If you missed the demo, here it is:

Update: the latest information is actually here.

All the information was available in my previous post. One minor update is that I’ve made a tag (v2.6.28-tidspbridge) to the linux-omap tree on my github repo to make it extra easy for people to compile a stable kernel with the dsp-bridge driver. There’s many DSP fixes available and some performance improvements which are not in this tag, but I’ll make sure they are once 2.6.29-omap1 is tagged.

Here is a video showing off gst-openmax on the beagleboard.

It’s a 720×400 MPEG-4 video at 24 fps. Essentially gst-openmax is using TI’s OpenMAX IL implementation, which is using the DSP on the OMAP 3530 to decode the video.

The CPU usage on the ARM side is about 10% which leaves plenty of room to decode audio or anything else. When debugging is turned off on the dsp-bridge kernel driver, the CPU usage is less, but it’s more unstable.

So, if you want to try this out on your beagleboard, just follow these instructions. If you don’t have one, what are you waiting for?!

Everything is open source, ready for eager hackers, except the DSP software, which is provided as binaries and you need to run an installer that extracts them after you agree with the “free for non-commerical”-use license. Texas Instruments is also providing all the tools to write DSP software publicly, so you can write your own open source codecs if you want

There’s one minor “but”. Texas Instruments uploaded the wrong DSP socket nodes for MPEG-4 video, so it doesn’t really work out of the box, but they are working on that.

This is what we are going to use on the Maemo 5 devices so if you want to get your hands dirty early on, you just need a beagleboard, which is not expensive. This is a lot of improvement since previous Maemo releases, where Nokia developed custom DSP software which unfortunately wasn’t picked up by other parties, so it was pretty hard to develop on top of it, even if was more open.

Or you can try on x86 with STMicro’s OpenMAX IL implementation, bellagio, which can use FFmpeg, as well as other open-source codec libraries.

Since bellagio is pretty extensible, that’s what will be used on Maemo 5 to run the audio codecs. More details on that later.

OpenMAX IL on x86 is not so much fun yet because you’ll not get much advantage, unless your graphic card somehow provides hardware acceleration through the OpenMAX IL interface (hint to ATI and NVIDIA).

You can find more information about gst-openmax on the freedesktop wiki. My personal repo is at github, where there’s the latest stuff and branches for things to come like tunneling support that NXP developed in collaboration with us.

So there’s many things you can help with. If you only have x86 you can help improve gst-openmax, or the core of bellagio. If you have an OMAP3 device then you can help with TI’s openmax, or dsp-bridge which needs a lot of cleanups before its merged into the Linux kernel.

If you happen to like git you can find my personal repos for all these technologies at github:

• gst-openmax
• libomxil-bellagio
• libdspbridge
• libomxil-ti

I even have an example DSP socket node (with instructions) that you can use for reference, or to test the dsp-bridge. The patches for the dsp-bridge are on a branch on the linux-omap tree here.

Needles to say, comments and patches are welcome

codeswarm.rb is a rewrite of Michael Ogawa’s code_swarm in ruby using cairo.

For an example see:

The format of the events xml file is compatible with code_swarm’s one,
and the physics engine is basically the same.

The code is less than 500 lines of code, so it should be fairly hackable.

The code is there:
http://github.com/felipec/codeswarm.rb

And here is a picture (youtube is too blurry):

Apparently there’s renewed discussion regarding Nokia’s support for Ogg Vorbis on the next platform. I can only comment on the technical side of things.

We will be using the OMAP3 chip, which has an ARM Cortex A8 processor, which has NEON technology. This allows for many optimizations of multimedia software, so audio decoders can run on the ARM processor without draining the battery, in fact that’s what we are aiming at in our architecture.

That means people can start optimizing Ogg Vorbis software using NEON technology and when the next device comes, Vorbis won’t be a second-class citizen; it won’t drain the battery (if properly optimized). GCC optimizations and Orc runtime compiler can be used, or the traditional manual assembly.

Now, if that doesn’t seem enough; Texas Instruments is opening more stuff for DSP development. So it would be easier to develop a DSP Ogg Vorbis decoder.

In fact, Nokia is moving a way from the home-made dsp-gateway, and will use TI’s dsp-bridge, which is being merged upstream. That means third-party DSP nodes would be able to run unmodified in other devices, not just Nokia Maemo devices.

You can already get your hands dirty with the next technology by using the Beagle Board, whatever works there, will work on the next device.

I understand the frustration people feel regarding Ogg Vorbis, but from the trenches the only thing we can do is push for better third-party codec support, for which there is progress.

A while ago I wrote the “official” video transcoding for Maemo how-to based on the wiki page and some heuristics.

The interesting thing here is the intelligent script that automatically finds the right parameters to use for the encoding.

Some of the considerations include:

• As higher framerate as possible. For smooth videos.
• Keep aspect-ratio. So the videos don’t look weird.
• No cropping. You might miss something.
• Single pass. Bigger files, but constant CPU usage.

Is this thing really smart? Let’s try it.

Crappy clip from YouTube

transcode.rb -v -i youtube.flv
* Input: 320x240 [4:3], 29.917 fps, 0 kbps.
* Output: 320x240 [4:3], 29.917 fps, 400 kbps.
mencoder youtube.flv -o youtube_it.avi -srate 44100 -oac mp3lame -lameopts vbr=0:br=128 -af volnorm -ovc lavc -lavcopts vcodec=mpeg4:vbitrate=400 -ofps 29.917000

Nothing special, the script maintained all the parameters, so there shouldn’t be any problems. Also, the aspect-ratio is not so different from the one of the device. This one is OK.

DVD

transcode.rb -v -i dvd://1
* Input: 720x480 [3:2], 29.97 fps, 7500000 kbps.
* Output: 400x240 [5:3], 29.97 fps, 400 kbps.
mencoder dvd://1 -o 1_it.avi -srate 44100 -oac mp3lame -lameopts vbr=0:br=128 -af volnorm -ovc lavc -lavcopts vcodec=mpeg4:vbitrate=400 -ofps 29.970000 -vf-add scale=400:240

In this case the aspect-ratio is slightly modified (less than 10%), but the target aspect-ratio is exactly the same as the one of the device, so, it will look good.

So far the script does seem intelligent

High Definition

Now let’s try a non-standard resolution, high bitrate, and H.264 format.

transcode.rb -v -i starcraft.divx -f h264 -q 8
* Input: 1024x436 [256:109], 23.99 fps, 1917920 kbps.
* Output: 352x144 [22:9], 23.99 fps, 1500 kbps.
mencoder starcraft.divx -o starcraft_it.avi -srate 44100 -oac mp3lame -lameopts vbr=0:br=128 -af volnorm -ovc x264 -x264encopts bitrate=1500:nocabac -ofps 23.990000 -vf-add scale=352:144

The aspect ratio now is about 4% different, that’s good. Unfortunately the aspect-ratio is quite different from the one of the device, but that’s not too bad.

This is how it would look like:

note H.264 videos look pretty good on the device.

Conclusion

So far there hasn’t been a single video where I had to modify the parameters that this algorithm suggested; sometimes the quality is not that good but increasing it with the -q option does the trick.

Wondering if it would find the right parameter for your clips? Why don’t you try it and find out

The script is here. And the garage project here.

Continuing my previous GStreamer introduction this new tutorial will guide you on your first GStreamer application written in C.

For anxious people the code is here.

The whole thing is here.

It’s in bluwiki so if you want to modify it, feel free to do so

I feel a little bit ashamed of posting such simple things, but there doesn’t seem to be anything for the really newbies.

I’ve read similar stories but the actual position in Amazon.com was different. Now the positions are quite good.

Here you can see that the N800 is #1 and the N810 is #5.

I guess we must be doing something good