Handling audio and video

As you know, I have been working on WebcamStudio and ScreenStudio in the last few years.  The main issue I faced while developing those projects was ensuring that audio and video would be synchronized.

You have to keep in mind that you are always limited by the power of the CPU.  A faster machine means that you will be able to capture larger displays and encode into a more compressed format.  Capturing a webcam at 320x240 is actually quite easy to do.  Capturing your whole desktop is another thing.  The video format is also putting a lot of stress on the CPU as you need to encode the video source (and audio) in realtime.

Assuming that the capture is done at 30 fps, you will have less than 33 milliseconds to capture one frame, apply any video effect on the image and encode it into the video stream.  I'm telling you, 33 milliseconds is really not a long time to do all this work.  For each frame, WebcamStudio has to:

• Capture a single frame from all videos sources (Screen, webcam, image, etc...)
• Apply effects on each frame
• Merge all capture frames into a single frame by compositing each image one over the other in the proper order
• Push the final output frame image to the encoder buffer
• Encode the image frames from the buffer into the video file.
• Synchronize captured audio with the video stream in the video file.

The same goes on with audio in WebcamStudio.  Frankly, a lot is happening in a 33 milliseconds timespan.

ScreenStudio has a simpler approach.  It relies on FFMPEG or AVCONV to execute the desktop capture into a video stream.  Where WebcamStudio is doing all the job, ScreenStudio is relying on FFMPEG/AVCONV to do that job.

The biggest issue when capturing audio and video is synchronizing them.  Look around the web, the issue of unsynchronized audio and video is everywhere.  Why is it so hard to do?

At the low level, when you capture a single image frame from a video source or a single audio frame from an audio source, a timestamp is set to the frame.  This timestamp (PTS: Presentation TimeStamp) is used to tell the encoder when to use this frame in the final video stream. The software executing the capture will rely on the driver used to set the proper timestamp on each frame.  Sadly, each driver (video or audio) will provide two kinds of value:

• Zero based:  The first captured frame is 0, the second is 1, and so on...
• Time based: Each captured frame is using the current clock time.

When using multiple capture sources (audio and video), it becomes harder to synchronize them is they are using different timestamp values.  So you might think that we just need to use a conversion to set every input stream to the same type of PTS...  Wrong!  Technically is would work in an ideal world but its not.

For example, when capturing images from a webcam, you are totally dependant on the webcam driver.  The first image provided by the driver can have the right timestamp if it's based on the clock time but by converting this timestamp to a Zero Based, you are loosing the exact time of that frame.  There is no way to know if the first image was captured by the webcam as of now, a few milliseconds ago or a whole second ago.  This is due to the fact that the driver is buffering a few frames before making them available to the capturing software.

Have you noticed that when displaying your webcam with any software (Cheese, PhotoBooth, etc...), there is a slight delay before seeing the first image?  In some case, you can even see a really small delay between what you are doing and what is actually displayed on your computer screen.  Just showing the webcam on the computer screen does not required a lot of processing power.  Capture the image and paint it on the screen.  Easy, right?

Just do this simple test:  Display your webcam on your screen, and count to 5 with your voice and fingers at the same time.  You will see a small delay between what you hear from your voice and what is displayed on your computer screen.  The difference for this live capture is always around 100 milliseconds.  It's not really big and it's almost unnoticeable but it can be a real pain when developing a software that will capture your computer screen, you webcam, the microphone in realtime.

To achieve a perfect synchronization when doing a realtime audio and video capture, many technics can be used.  And frankly, sometimes, it's almost impossible to do...

Hope you enjoyed!

Patrick

ScreenStudio's Youtube Channel : https://www.youtube.com/channel/UCdddEE-W0AFRkhlPq1L1IQA

My own Youtube Channel: https://www.youtube.com/patrickballeux

Coding hard boiled eggs

How would a software coder describe the recipe to make hard boiled eggs?




The procedural coder:

- Take a saucepan
- Put water in the saucepan
- Put the saucepan on the stove
- Turn on the stove to maximum
- Wait for the water to reach boiling temperature
- Put three eggs in the boiling water
- Wait 3 minutes
- Turn off the stove
- Take the saucepan
- Empty boiling water into the sink
- Take the first egg, be careful full it's hot
- Roll the first egg on the kitchen counter to crack the shell gently
- Remove completely the shell from the first egg
- Put the first egg on a serving plate
- Take the second egg, be careful full it's hot
- Roll the second egg on the kitchen counter to crack the shell gently
- Remove completely the shell from the second egg
- Put the second egg on a serving plate
- Take the third egg, be careful full it's hot
- Roll the third egg on the kitchen counter to crack the shell gently
- Remove completely the shell from the third egg
- Put the third egg on a serving plate
- Voilà, enjoy!

The object-oriented coder:

- On a stove, in a saucepan, put water
- Turn on the stove to maximum
- When the water is boiling, on the stove, in the saucepan, in the water, put three eggs
- After 3 minutes, turn off the stove
- Empty the water, from the saucepan, on the stove, into the kitchen's sink
- For each egg, in the saucepan:
- Be careful, it's hot
- Roll it on the kitchen counter
- When the shell, on the egg, is cracked, remove completely
- Put it on a serving plate
- Voilà, enjoy!

The delegate-oriented coder:

- You need a stove, a saucepan, water, a kitchen sink, a kitchen counter, a serving plate and three eggs
- Turn on the thing that can heat to maximum with the thing that was filled with stuff that can boil
- When the event where the result of boiling has been reach, add the items that can be cooked in the stuff that is hot enough to cook them
- After a while, set to 3 minutes, remove the stuff that was boiling and put it in the thing that can dispose of it
- For each item, in a container, roll it on the thing that is hard enough to crack the external stuff of the item. At the same time, be careful as a warning message and it has been set that it can be hot.
- In the event that the item is cracked enough, remove the outer thing completely and put the item in something that can be used to serve for a meal
- In the event that that everything is done, Voilà and at the same time, Enjoy!
- When the items are cooked, turn off the heat source

Which one are you?

Patrick Balleux

Connecting to Twitter, Facebook or Google with Ubuntu

Ubuntu is a great operating system.  One neat feature is being able to connect to your online social network account.  See the tutorial on my Youtube channel:

Have a great day!

OOOOHHH! Twitter Analytics

A brand new tool has arrived in town: Twitter Analytics

It is now available for all Twitter users for free and it looks like an amazing tool to monitor your Twitter account and the activity related to it.  Twitter Analytics arrived just in time for me as I am currently investing more time into my account (@patrickballeux) to promote this blog and my Youtube Channel (www.youtube.com/patrickballeux).

With social networks, it is hard to evaluate the impact of your shared posts.  Using Google Analytics, you can do this monitoring on your blob/website quite easily.  But with your accounts (Facebook, G+, Twitter), it was near impossible.

Now, time to explore and investigate this new tools in my arsenal and start learning a bit more about my social network exposure.

Have a great day!

Monitor your network without IT knowledge!

Ever wished you could find out who is using your home network bandwidth?

There is an easy solution called EtherApe, available from the Ubuntu Software Centre.  Check it out!

Promoting your blog, a few tips...

When sharing content with the world on the web, we are expecting to get a few fans with time. Be it a blog, a Youtube channel or some fan page, the goal is to have followers and having a small community that will have an interest in our projects.

Even if you post and share repeatedly with your friends, it can be hard to reach the whole world.  Here's a few tips and links that will help you getting more exposures for free.

Social Networks

Ensure that you share over the social networks like Facebook, Twitter and G+.  Make your profile public and use appropriate tags (hashtags) to maximize visibility.

Profile image

Show your face!  This is you and not some random cat that is working hard at creating great content.  A good picture, that people will notice, will help you being recognized.  The same goes for your banner.  Make it stand out.  Here's the current inspiration for mine...

This is my current profile image and as you can see, I spoofed the great Dr House M.D. to create a funny profile image, but still being the main character.

Sharing and curating sites

There are some sites will index your links in some specific categories.  You can be surprised how effective they can be just by sharing your content into those website for free.

• www.stumbleupon.com
• www.reddit.com
• scoop.it
• Communities on G+
• Exposing your Twitter on www.twiends.com

In the end...

There are many ways to increase your visibility on the web for maximum exposure.  Those links and tips have been helping me a lot with my blog and Youtube channel in the last year.

Have a great day and good luck!

Watch movies for free with PopcornTime

Ubuntu users are looking for the Netflix killer since Netflix service does not really work well on Ubuntu computers.

Here's an alternative that may answer your prayers: PopcornTime