How to properly refresh a SurfaceImage object

[plarser48]

Newcomer here ...

I have an JPEG image that I'd like to project onto a WorldWind layer. The JPEG gets refreshed periodically by an external process.

Projeting the JPEG wasn't very diffucult with SurfaceImage.java demo file as guidance. My problem comes in when refreshing. I've tried:

(1) Repeating the same code every time the JPEG file is updated. For example:
Instantiated at start once:

RenderableLayer layer = new RenderableLayer();

Runs every time file is updated:

SurfaceImage si1 = new SurfaceImage(JPEG_FILE_NAME, new ArrayList<LatLon>(Arrays.asList( LAT_LONG_CORNERS )));
layer.addRenderable(si1);
(update)

But this never seems to refresh the image, only displays the first one. Then I tried,

(2) Reading the file into a BufferedImage object first, and then loading the BufferedImage object into SurfaceImage, i.e.

BufferedImage imgbuff = ImageIO.read( new File( JPEG_FILE_NAME) );
SurfaceImage si1 = new SurfaceImage(imgbuff, new ArrayList<LatLon>(Arrays.asList( LAT_LONG_CORNERS )));

Method #2 works, but I get a Java Out-of-memory-error after about 7-10 refreshes. I tried performing a imgbuff.flush() between iterations with no luck.



So to summarize, whats the ideal way to use and then refresh a SurfaceImage object derived from a JPEG file?


Thanks.

[nlneilson]

It looks like you have the JPEG image OK.

When your image is updated just call: wwd.redraw();
Do a search for "redraw", there are several ways doing it. I think Pat had a post on this and the preferred way of doing it.

Adding to many "new RenderableLayer();"s will overflow the java heap space.
Why your "imgbuff" is not being flushed and why the new image does not remove and replace the old rather than take up memory for both is a good question.

When you debug what is shown for "imgbuff" and it's size?

[plarser48]

With your feedback, I decided to investigate into the issues with method #2 (BufferredImage into a SurfaceLayer). I'm using this application in conjuction with Matlab (communicating image file locations across UDP, etc). So I think the problem has to do with running both based on my findings below. Note the image I am loading is a 1200x900 pixel grid.

To narrow down the problem down, I followed these tasks:

(a) Tried commenting out all layer rendering and worldwind refreshes.i.e. Just load the imgbuff
Result: Worked

(b) Commented out all but layer.addRenderable call, Ran with matlab
Result: Out of memory error

(c) Tried running WorldWind app independently from Matlab in a for-loop, reimaging the same file on every loop.
Result: Worked for N=100,1000 (trials).

(d) Tried running Matlab alongside as usual, but loading only a 10x10-pixel image file constantly refreshing.
Result: Worked

(e) Tried (d), but with a non-changing 1200x900-pixel image
Result: Out-of-memory error


Looking at the state of the layer's renderables and the imgbuff in debug, they both in fact get zero'd out on every image refresh call. That is using:

        [ get the sLayer, a renderable layer]
        try {
            img = ImageIO.read(new File(imgFile));
        } catch (IOException e) {
        }
        si1 = new SurfaceImage(img, new ArrayList<LatLon>(Arrays.asList(
               [lat/long array]
               )));
        img.flush();
        sLayer.removeAllRenderables();
        sLayer.addRenderable(si1);
        wwd.redraw();

So I think the issue is possibly Matlab stealing memory? It works with small-pixel images when running with Matlab. And it works with large pixel-images when running without Matlab.


For records, here is the java error I get as well:

Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: Java heap space
        at java.awt.image.DataBufferByte.<init>(DataBufferByte.java:58)
        at java.awt.image.ComponentSampleModel.createDataBuffer(ComponentSampleModel.java:397)
        at java.awt.image.Raster.createWritableRaster(Raster.java:938)
        at javax.imageio.ImageTypeSpecifier.createBufferedImage(ImageTypeSpecifier.java:1056)
        at javax.imageio.ImageReader.getDestination(ImageReader.java:2879)
        at com.sun.imageio.plugins.jpeg.JPEGImageReader.readInternal(JPEGImageReader.java:913)
        at com.sun.imageio.plugins.jpeg.JPEGImageReader.read(JPEGImageReader.java:885)
        at javax.imageio.ImageIO.read(ImageIO.java:1422)
        at javax.imageio.ImageIO.read(ImageIO.java:1282)
        at gui.mainGUI.mapMessageEventOccurred(mainGUI.java:2656)
        at gui.NetworkReceiverThread.fireMapMessageEvent(NetworkReceiverThread.java:266)
        at gui.NetworkReceiverThread.access$400(NetworkReceiverThread.java:45)
        at gui.NetworkReceiverThread$5.run(NetworkReceiverThread.java:178)
        at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:209)
        at java.awt.EventQueue.dispatchEvent(EventQueue.java:597)
        at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:269)
        at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:184)
        at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:174)
        at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:169)
        at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:161)
        at java.awt.EventDispatchThread.run(EventDispatchThread.java:122)

I guess I'll try debugging more by looking into ways to manage Java memory (allocate more?) or tame Matlab's usage a bit?

[plarser48]

A clarification on the above. I'm running matlab and WWJ independently, not as Matlab calling WWJ As described in another post

[nlneilson]

It's good to know from (c) that WWJ works without MatLab.

There have been several posts so just do a search for "MatLab". There may be something there regarding excessive memory usage. I tinkered with that a few years ago but have not since.

I have used sockets and serial ports to interact with WWJ apps but have not tried the serial UDP (User Datagram Protocol). If the WWJ and MatLab are on the same computer (they must have been for your tests) try a socket. If not try serial without the UDP.

The Win Task Manager under Processes will show the CPU and Memory usage for each process. Your WWJ app will probably be javaw.exe

[plarser48]

Turns out I can't rule out the WWJ app just yet ...

I adjusted the program so that a simple button exists on the WWJ display. When pressed, it attempts to refresh the image by calling a function "testLoad()".

(Code trimmed)

public void testLoad() {
        sLayer = getSurfaceLayer();
        SurfaceImage si1;
        BufferedImage img = null;
        sLayer.removeAllRenderables();  // Clear layer
        // Load image into buffer
        try {
            img = ImageIO.read(new File(imgFile)); // imgFile is a fixed filename (jpg)
        } catch (IOException e) {
        }
        // Store image in a surfaceimage
        si1 = new SurfaceImage(img, new ArrayList<LatLon>(Arrays.asList(...       )));
        sLayer.addRenderable(si1);  // Add to layer
        wwd.redraw();  // Refresh
}

Now I run the application without matlab at all (not even open in background).

Each time I press the button, I see my available memory drop about 3MB on the the ImageIO call. This is as expected, as the file is a 1200x900 jpeg (assuming 24 bits).

BUT, even though I call img.flush(). The memory is still being used. I even try to force a garbage collect with System.gc() with no luck.

The next time I press the button, an approx. additional 3MB is lost. This continues until it dies with a java heap error (memory leak)


At first I though perhaps I was pushing the button too quick, loading images in before the garbage collector had a chance to react to img.flush(). But I left large gaps of time between each and it still leaks.

When I load the BufferredImage into the SurfaceImage object, what is the proper way to clear the SurfaceImage object completely (and remove from layer) to get the memory back?

Thanks again.

[plarser48]

whoops, missed a line above. After the "si1 = ..." line, I do have a "img.flush();"

[nlneilson]

Originally Posted by plarser48

Looking at the state of the layer's renderables and the imgbuff in debug, they both in fact get zero'd out on every image refresh call.

You need to find the difference in your code so your imgbuff gets zero'd out.
You do have just ONE imgbuff rather than a new one each time, right?
That is your layer's "renderable" so it should be zero'd out, unless I am mistaken.

C++ is less dependent on garbage collection and depends more on the programmer to free memory.
Java should have a method to do that also and be less dependent on gc.

edit: If you have just one imgbuff (and one renderable layer to display it) replacing the 3MB in the imgbuff with another 3MB that should take no more than 3MB of memory if your layer is not holding the previous 3MB. I don't think you have a memory leak. You are adding 3MB each time somewhere.

matlab has it's place but the math functions in Java, C++ or Python will usually get most jobs done with less overhead.

[plarser48]

I placed

System.out.println(Runtime.getRuntime().freeMemory() + "\t" Runtime.getRuntime().totalMemory())

before and after the ImageIO.read() call to see the free memory before/after the file is loaded. Also, I noticed the imgbuff I showed above was recreated (not as new) on every call to the function. I figured it would flush and then destroy upon function exit. But just in case I pulled it out as a private class variable, initialized once during init (same with surfacelayer). The same happened. I can't post an attachment of my similified test file since I think I am still awaiting activation by mods on this board.

Then I noticed that my total heap space displayed was 64MB (the JVM default?) as I ran in Netbeans. Is this too small for WWJ? I saw several posts on this board of folks using 256MB default, often 512MB, and sometimes higher.

So I tried bumping up my heap to 128MB. It died, but not as quickly. Then up to 256MB. At 256, it would die no matter how quickly I would push the image refresh button. I still worry that even at 256, my memory usage still sinks down to <10% before the garbage collect kicks in. (Is this normal?)

Is it recommended to have a min of 256MB with WWJ?

[nlneilson]

You can try 512MB but that will not solve your problem.

When debugging see if you can get the memory size of the imgbuff and also that of your layer. I have not done that in WWJ.

From post #3 where "Result: Worked for N=100,1000 (trials)." may be a good place to go to and just make a few changes at a time to see where the problem starts.

One other possibility is the buffer that receives the new 3MB images. Check the memory usage for that also and see to it that there is only one buffer for that.

For my WWJ app the memory usage without a bunch of layers on even though another layer is hiding it, controlling the redraw and culling data that is not in the view took time.
Nice when you get the bugs worked out.