How to capture desktop in Java: Fast method!

If you are developing software in Java, you may need to capture the desktop at some point.  There is the Robot API that does provide a simple way to capture the desktop, but it's not very efficient.

I've found a better way to capture the desktop, or the webcam in Java, relying on FFMpeg.  There is no need for JNI bindings or other obscure libraries dependencies.  All you need is an executable FFMpeg binary in your path.

Why not use Robot?

The Robot API is really easy to use but it has a major drawback: Speed.  On some computers, you may be able to capture up to 30 fps but on other, you'll struggle to capture images at 3 fps.  It seems that the operating system and the video driver are sensitive areas for the Robot API.

FFMpeg?

FFMpeg is a console application that can capture, transcode and even stream video and audio file.  It's a swiss army knife for multimedia content.  It is available on almost all Linux distros, OS X and Windows.

If not already installed, you can download it here: https://ffmpeg.org/

How to capture video?

Once FFMpeg is installed on your system, you can invoke the binary using the Process class as any other executable.

The trick is to retrieve the captured data directly from FFMpeg using the Process class.  Once the raw data is captured, use a BufferedImage to store those raw data and you'll end up with an Image available for whatever you want to do with it.

Here's the code snippet:

BufferedImage buffer = new BufferedImage(720, 480, BufferedImage.TYPE_3BYTE_BGR);
 
String command = "ffmpeg -nostats -loglevel 0 -f x11grab -r 30 -video_size "+ buffer.getWidth() +"x" + buffer.getHeight() + "-i :0.0 -f rawvideo -pix_fmt bgr24 -";

Process p = Runtime.getRuntime().exec(command);

java.io.DataInputStream in = new java.io.DataInputStream(p.getInputStream());
 
byte[] data = ((DataBufferByte) buffer.getRaster().getDataBuffer()).getData(); 

while (!stopMe) {

in.readFully(buffer);

}

in.close();

p.destroy();

 This code will capture at the highest frame rate that the computer can support.  The speed of the computer will be the limit of your capture processing speed.

The thing you need to focus is the output format of FFMpeg.  The video encoding must be "rawvideo" with a pixel format matching the BufferedImage.  In this example, a 24 bit/pixel is used in BGR format (Blue, Green, Red).  The data is them exported directly to the console (notice the "-" at the end of the command line...) instead of a file.  That means that using the InputStream from the Process class, Java is able to read the data as fast as it can.

Execute this capture process in a Thread making the BufferedImage available for other parts of your software. 

See the whole thing running...

Using this method, I have built ScreenStudio to capture the desktop into a virtual video mixer built entirely in Java.  If you want to see it running live, download the source code of ScreenStudio at http://screenstudio.crombz.com


Leave you questions in the comments or tweet me at http://twitter.com/patrickballeux


Have fun!