A few days a go I was tweaking ScreenStudio 2.0. The main idea is to create a dynamic overlay a bit like WebcamStudio without having to use a lot of CPU resources.
WebcamStudio is great but do required a powerful computer to make it work. On the other hand, ScreenStudio is more limited but do have the advantage of being simple to broadcast live your desktop over Twitch and Youtube.
ScreenStudio do support a static overlay created by rendering an HTML file using a JLabel (a label component in Java Swing). The result is pretty good but do lack dynamic content. I've tried several ways without major success. That is until a few days ago.
I knew that FFMpeg is able to read from many sources. I had, in the past, used a TCP/IP connections to get a BufferedImage to FFMpeg. The process is slow and do required a lot of code to handle the connections.
I also tried using the OutputStream for the Process to output the content to FFMpeg but it does not work reliably.
Then it occured to me that I could use a fifo file using the command "mkfifo" in linux to output a BufferedImage directly to FFMpeg.
As you know, I have been working on WebcamStudio and ScreenStudio in the last few years. The main issue I faced while developing those projects was ensuring that audio and video would be synchronized.
You have to keep in mind that you are always limited by the power of the CPU. A faster machine means that you will be able to capture larger displays and encode into a more compressed format. Capturing a webcam at 320x240 is actually quite easy to do. Capturing your whole desktop is another thing. The video format is also putting a lot of stress on the CPU as you need to encode the video source (and audio) in realtime.
Assuming that the capture is done at 30 fps, you will have less than 33 milliseconds to capture one frame, apply any video effect on the image and encode it into the video stream. I'm telling you, 33 milliseconds is really not a long time to do all this work. For each frame, WebcamStudio has to:
Capture a single frame from all videos sources (Screen, webcam, image, etc...)
Apply effects on each frame
Merge all capture frames into a single frame by compositing each image one over the other in the proper order
Push the final output frame image to the encoder buffer
Encode the image frames from the buffer into the video file.
Synchronize captured audio with the video stream in the video file.
The same goes on with audio in WebcamStudio. Frankly, a lot is happening in a 33 milliseconds timespan.
ScreenStudio has a simpler approach. It relies on FFMPEG or AVCONV to execute the desktop capture into a video stream. Where WebcamStudio is doing all the job, ScreenStudio is relying on FFMPEG/AVCONV to do that job.
The biggest issue when capturing audio and video is synchronizing them. Look around the web, the issue of unsynchronized audio and video is everywhere. Why is it so hard to do?
At the low level, when you capture a single image frame from a video source or a single audio frame from an audio source, a timestamp is set to the frame. This timestamp (PTS: Presentation TimeStamp) is used to tell the encoder when to use this frame in the final video stream. The software executing the capture will rely on the driver used to set the proper timestamp on each frame. Sadly, each driver (video or audio) will provide two kinds of value:
Zero based: The first captured frame is 0, the second is 1, and so on...
Time based: Each captured frame is using the current clock time.
When using multiple capture sources (audio and video), it becomes harder to synchronize them is they are using different timestamp values. So you might think that we just need to use a conversion to set every input stream to the same type of PTS... Wrong! Technically is would work in an ideal world but its not.
For example, when capturing images from a webcam, you are totally dependant on the webcam driver. The first image provided by the driver can have the right timestamp if it's based on the clock time but by converting this timestamp to a Zero Based, you are loosing the exact time of that frame. There is no way to know if the first image was captured by the webcam as of now, a few milliseconds ago or a whole second ago. This is due to the fact that the driver is buffering a few frames before making them available to the capturing software.
Have you noticed that when displaying your webcam with any software (Cheese, PhotoBooth, etc...), there is a slight delay before seeing the first image? In some case, you can even see a really small delay between what you are doing and what is actually displayed on your computer screen. Just showing the webcam on the computer screen does not required a lot of processing power. Capture the image and paint it on the screen. Easy, right?
Just do this simple test: Display your webcam on your screen, and count to 5 with your voice and fingers at the same time. You will see a small delay between what you hear from your voice and what is displayed on your computer screen. The difference for this live capture is always around 100 milliseconds. It's not really big and it's almost unnoticeable but it can be a real pain when developing a software that will capture your computer screen, you webcam, the microphone in realtime.
To achieve a perfect synchronization when doing a realtime audio and video capture, many technics can be used. And frankly, sometimes, it's almost impossible to do...
As a software developer, I can understand anyone wanting to protect their work from piracy. In the past, I have also developed a few sharewares that I wanted to spread around the world and make a bit of money from them.
Over the years, I've realized that the thing that I liked the most was learning from those projects. Supporting them, commercially, was taking over my free personal time. In the end, it was not fun anymore.
I still wanted to create new ideas, explore new concepts and learn new technologies. At work, we are always limited by all kinds of constraints like time, budget and many other things. It's a fact and there is nothing wrong with it. It is the world of business where productivity also means profitability.
But deep down inside, developers are full of imagination, curiosity and ambition. We are creative and resourceful in many ways. We are the aspiring new talents of the Technollywood!
Personally, I am more interested in creating and sharing than trying hard to push my software as a marketing product. Those of you who know me from my previous projects will agree that I spend more time talking about my project, trying to build a community around it and having some crazy fun ideas to promote it on the web. I'm a star cruising the streets of Technollywood. And I like it!
Of course, a new project will take time before getting some followers so you better have a good idea to put on the table. And then, one day, people will notice your idea and share them with their friends.
The Open Source licensing is a great way to share your work and ideas. Some people will be interested in the final usable software while others will follow the development and contribute with their own ideas. The final result is a community effort to create a great software and you are leading that team. This is a fun way to learn working with customers, team members and reviewers.
An Open Source project is much more than a free software, it's the result of minds collaborating in a common goal.
I was looking at my Youtube videos and went back to the first ones where I was testing the idea of recording my computer display using scripting and different solutions.
This was before ScreenStudio, before WebcamStudio. To be exact, the first posted video was recorded on August 25th, 2008. Almost 6 years ago...
Recording was blurry, laggy, low-res... But this is where it started. Eventually I made another one:
Not much better, but I was going somewhere...
The fist official demo of WebcamStudio! September 8, 2008...
For me, this is where the big adventure of streaming and videos started. Before that, I was working on another project called NW-E00x Mp3 Manager that was quite successful also.
I want to thank all my fans and followers that have provided my with their help, their time and their support.
If you a developer, there is a good chance that you dreamed about creating the next Facebook or the next Candy Crush Saga. For sure, it's fun to think that maybe your next app will create a buzz around the next.
In real life, the chance of becoming the next superstar of the software industry is quite slim. Does it means you have to forget about your dreams? Not at all.
Every project starts with an idea. Whatever is the source of your inspiration, it doesn't matter. Try it anyway, believe in yourself and who knows, you may create something that a few thousand people will appreciate.
Even if your project does not get any attention, you will learn a lot of things in the process. It's not just about being famous, it's about learning
and becoming better overtime. The more you learn, the more you have a chance of succeeding. In my personal history, I've started many projects on the web, just for the fun of being creative:
mcpe.crombz.com (Exploring the idea of sharing your online games)
ScreenStudio (Going back to stream/record your display in an easy way)
Some projects never took off. I learned a great deal in the process. Other projects became famous to a point I would never expect (WebcamStudio, NWE-00X Mp3 Manager)
Overall, I learned so much with these projects that I am proud to list them in my resume. They all prove one thing: I am a project creator!
... And sometimes, I get it right...
Go ahead, try, test, explore! You can be a creator too!
While creating an event for the release of ScreenStudio 1.0, Karl Ellis and I got together and has a great chat about the history of WebcamStudio. (Hosted by DirtyPenguins)
