Question

The follwing problem should watch the screen, record an event (a measuring textbox turns green) and record all the events leading up to it, producing a “film” of the events leading up to it. Unfortunately the whole screen needs to be recorded. I have so far done the part where the recognition takes part. However i barely get two frames per second. I would like to have around 25 to 30 fps.

My Idea was to do the writing and reading in two seperate threads. Because the writing event is rare and can run in the background, the recording event can take up more time and run faster. Unfortunately the whole thing seems to be too slow. I would like to be able to write on disk the screen the 10 to 20 seconds before the event occurred.

Edit:
If possible i would like to stay as platform-independent as possible.

Edit 2:
there seems to be a platform independent jar file for Xuggler. Unfortunately i don’t really get how i will be able to use it for my purpose: recording the 20 seconds leading up to the point where isarecord is triggered.

Here is what i have done so far:

package fragrecord;

import java.awt.AWTException;
import java.awt.Color;
import java.awt.Dimension;
import java.awt.Rectangle;
import java.awt.Robot;
import java.awt.image.BufferedImage;
import java.io.File;
import java.io.IOException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;

import javax.imageio.ImageIO;

public class Main {
    public static void main(String[] args) {
        //The numbers are just silly tune parameters. Refer to the API.
        //The important thing is, we are passing a bounded queue.
        ExecutorService consumer = new ThreadPoolExecutor(1,4,30,TimeUnit.SECONDS,new LinkedBlockingQueue<Runnable>(100));
        System.out.println("starting");
        //No need to bound the queue for this executor.
        //Use utility method instead of the complicated Constructor.
        ExecutorService producer = Executors.newSingleThreadExecutor();

        Runnable produce = new Produce(consumer);
        producer.submit(produce);  
        try {
            producer.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
        } catch (InterruptedException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
        consumer.shutdown();
        try {
            consumer.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
        } catch (InterruptedException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }

    }
}

class Produce implements Runnable {
    private final ExecutorService consumer;

    public Produce(ExecutorService consumer) {
        this.consumer = consumer;
    }
    boolean isarecord(BufferedImage image){
        int x=10, y = 10;
        Color c = new Color(image.getRGB(x,y));
        int red = c.getRed();
        int green = c.getGreen();
        int blue = c.getBlue();
        // Determine whether to start recording
        return false;

    }


    @Override
    public void run() {

        Robot robot = null;
        try {
            robot = new Robot();
        } catch (AWTException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
        //
        // Capture screen from the top left to bottom right
        //
        int i = 0;
        while(true) {

            i++;
        BufferedImage bufferedImage = robot.createScreenCapture(
                new Rectangle(new Dimension(1024, 798)));

        Runnable consume = new Consume(bufferedImage,i);
        consumer.submit(consume);
        }

    }
}

class Consume implements Runnable {
    private final BufferedImage bufferedImage;
    private final Integer picnr;
    public Consume(BufferedImage bufferedImage, Integer picnr){
        this.bufferedImage = bufferedImage;
        this.picnr = picnr;
    }

    @Override
    public void run() {
        File imageFile = new File("screenshot"+picnr+".png");
        try {
            System.out.println("screenshot"+picnr+".png");
            ImageIO.write(bufferedImage, "png", imageFile);
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
    }
}

Answer 1

for a variety of reasons using executors, rewriting to nio and similar things are not going to help.

here are a bunch of things you should consider:

1. image capture is SLOW in java, nothing you can do about it without JNI dependencies
2. use jpeg instead of png, much faster
3. any image compression is SLOW using ImageIO. you could use the old-school, proprietary JpegEncoder classes (in the com.sun package) or the TurboJPEG java library (JNI as well) but imho it’s not the bottleneck
4. disk i/o is DEFINITELY not the problem (you’re writing <5mb/s, so don’t worry)
5. writing many images in parallel will in fact slow down your application, not speed it up (unless you have an ssd)
6. consider parallelizing the capture/analysis thread instead (e.g. do the analysis only every 20th frame) (*)
7. i bet you would finish writing this app twice for each platform using a native language before you can optimize the java app enough so it runs at 25fps (using 100% cpu xD)
8. seriously consider hybrid solutions as well; e.g. record everything to a compressed movie with an available tool, then do the analysis later (**)

in one word: java sucks at what you want to do, really badly.

nevertheless, i’ve written my own version of this tool.
http://pastebin.com/5h285fQw

one thing it does is allow to record a smaller rectangle following the mouse.
at 500×500 it gets me easily to 25fps with images being written in the background (image compress+write takes 5-10ms for me, so it writes much faster than it records)

---

(*) you don’t talk about how you analyze the images, but this seems to be the primary source of your performance problem. a few ideas:

• only look at every N-th frame
• take a capture of only a subportion of the screen and move that subportion around over time (later reasemble the images; this will cause terrible tearing but maybe it doesn’t matter for your purpose)
• capturing itself should be “not too slow” (maybe 10-20fps for fullscreen); use serial capture but parallelize the analysis

(**) on macosx you can use the built-in quicktime x to record to hdd quite efficiently; on windows i heard playclaw (http://www.playclaw.com/) is quite good and maybe worth the money (think what you’d make in the time wasted optimizing the java beast 🙂 )