Debug/Symbol Builds for gluegen-rt.dll, jogl.dll, jogl_awt.dll, jogl_cg.dll?

[buzz3791]

We're battling a "committed virtual memory" leak. This value is reported by jconsole on the "VM Summary" tab. This value is also equivalent to the Windows Task Manager "Virtual Memory Size" (Task Manager->View->Select Columns->Memory Commit Size->Enable Checkbox); on Windows 7 it is called "Commit - Size"). After 13 days our application, jconsole reported the committed virtual memory as 1.6GB and eventually crashed with EXCEPTION_ACCESS_VIOLATION (0xc0000005) emitting hs_err_pid3104.log. In case you're wondering, we've already fixed a few Java heap memory leaks but now we're down to this C/C++ leak.

We're still trying to isolate the source of the leak. I'm guessing the leak is caused by our Java code forgetting to release a resource. Since this leak shows up in the C/C++ side of the Java Virtual Machine, debug/symbol build version of all the involved C/C++ code would be very helpful. Does the World Wind development team have debug/symbol builds for the following DLLs for say Windows XP 32-bit (really any flavor of Windows, we support most of them after XP)? Such builds would let me isolate the source using Rational Purify etc.

1. gluegen-rt.dll
2. jogl.dll
3. jogl_awt.dll
4. jogl_cg.dll

I looked on the World Wind wiki and searched through lots of posts but haven't found anyone asking this question.

Does anyone have tips on locating C/C++ leaks when code is running on a world wind "stack" (java.exe/jogl/videodriver)? I found this link fairly helpful http://www.oracle.com/technetwork/ja...499.html#gbyvk

I've attached the hs_err_pid3104.log to this post. Anyone have an idea what might be allocating the 16MB chunks shown at bottom of the cpp-heap-detail-day10.png attachment?

[nlneilson]

Originally Posted by buzz3791

After 13 days ... 1.6GB and eventually crashed

C/C++ leak.

Since this leak shows up in the C/C++ side of the Java Virtual Machine,

2. jogl.dll
3. jogl_awt.dll

How many machines have you tried this on?

13 days is a long time to run an app.
Is it a consistent increase in memory usage each day?

jogl.dll and jogl_awt.dll are the only two that I find necessary.

Something as simple as polylines if you don't get rid of them can
can build up over time or too many will do it in a hurry.

What is your CPU usage?

I use WWJ as the display and C/C++ apps to interact with it.
In many instances the CPU is < 10%.
It took a considerable amount of time optimizing the code but
that kept the memory usage down also.

Without a clue to what layers you are using, the number of polylines (or paths), icons, etc.. it is hard to guess.

[buzz3791]

Originally Posted by nlneilson

How many machines have you tried this on?

7 machines. So far I've only reproduced the issue on a Dell OptiPlex 320 running Windows 7, 32-bit with an ATI Radeon Xpress 200 motherboard-based video chip running an ATI driver version 8.501.0.0.

In a few days, I'll know if we can reproduce this issue on another system.

Originally Posted by nlneilson

13 days is a long time to run an app.
Is it a consistent increase in memory usage each day?

On this system, over the last 6 days it was leaking about 50-80MB per day.

Originally Posted by nlneilson

Something as simple as polylines if you don't get rid of them can
can build up over time or too many will do it in a hurry.

Thanks for the tip. I'll look into our polyline usage.

Originally Posted by nlneilson

What is your CPU usage?

Our client systems typically have 2 CPU cores and most of them sit around 60% usage.

Originally Posted by nlneilson

Without a clue to what layers you are using[...]

The layers we're using are

1. Blue Marble 2004
2. i-cubed LandSat
3. Earth Elevation Model (USA 10m, WordWide 30m, Ocean 900m)
4. Place names
5. MS Virtual Earth Roads
6. MS Virtual Earth Aerial
7. Political Boundaries
8. A custom KML layer. This layer is made up of
8a. gov.nasa.worldwind.layers.RenderableLaye r
8b. gov.nasa.worldwind.layers.AnnotationLaye r
8c. gov.nasa.worldwind.layers.IconLayer.

Originally Posted by nlneilson

Without a clue to [...], the number of polylines (or paths), icons, etc.. it is hard to guess.

The issue occurred on a system with about 60 UserFacingIcon instances and 160 gov.nasa.worldwind.render.Polyline instances.
The Polyline instances consist of a number of segments calculated using this code...

//compute the number of segments we need for this line based on distance
//The default is 10 and it covers pretty much... but looks very crappy
//when these lines span the globe (which will never happen, of course)
//So let's try 1 segment per 100,000 units in legth, which is about 1.3
//lat and long degrees, plus 10 to compensate as a min for shorter distances.

setNumSubsegments((int) (getLength(globe)/100000.0) + 10);

[nlneilson]

Originally Posted by buzz3791

1. In a few days, I'll know if we can reproduce this issue on another system.

2. Thanks for the tip. I'll look into our polyline usage.

3. Our client systems typically have 2 CPU cores and most of them sit around 60% usage.

8. A custom KML layer. This layer is made up of
8a. gov.nasa.worldwind.layers.RenderableLaye r
8b. gov.nasa.worldwind.layers.AnnotationLaye r
8c. gov.nasa.worldwind.layers.IconLayer.

4. The issue occurred on a system with about 60 UserFacingIcon instances and 160 gov.nasa.worldwind.render.Polyline instances.

5. The Polyline instances consist of a number of segments calculated using this code...

1. It will be interesting to know if the other systems are effected the same way or if it is more or less of a problem.

2. A very good example to follow for polylines is \src\gov\nasa\worldwind\layers\LatLonGra ticuleLayer.java
How the lines that are not in view are culled is important and the altitude they are displayed.

I have a file with 25,000 lines where only 10 may be in view at a time but I
sort that in C++ before it is sent to WWJ. Also limit the altitude when they are displayed. I did that in Java before.

3. Unless you have a lot of motion and redraws you should be able to get lower than 60% on dual cores.

Your layers look OK. I tried turning off the layers that were hidden like the Marble and I3 when something like the VE layer was on, didn't make too much difference except when downloading new data.

The KML layer should be OK just cull the Annotations and Icons that are not in view the same as polylines.

4. 60 icons and 160 lines shouldn't be a problem.

5. So far I have not had a need to use a formula to add intermediate points to lines. .setNumSubsegments(3); is the highest I have used AFAIR.
If your lines can be on the terrain .setFollowTerrain(true); works good.
If they need to be above the terrain I thought at one time there was an example that drew lines like the grids above the terrain.
Most of my lines that are not on the terrain the points are the distance a plane can travel in one second so linear works fine and specifically call a redraw only after all tracks are updated.

You may be surprised at how the CPU (and memory) will drop with a bit of effort.

Culling or not "drawing" things not in the view is often the key.

edit: post #5 has some links to images of what I am doing. Note that with 50 tracks the CPU is < 10% http://forum.worldwindcentral.com/sh...ad.php?t=29345

Neil

[buzz3791]

Hi Neil,

Thanks for another response.

Originally Posted by nlneilson

3. Unless you have a lot of motion and redraws you should be able to get lower than 60% on dual cores.

4. 60 icons and 160 lines shouldn't be a problem.

I should've mentioned that our system has lots of movement. The nodes are air and ground vehicles, so they are constantly moving. The lines between the nodes represent communication channels between the moving nodes, so the lines are moving as well. In the test environments, they are polling at least every 10 seconds for changes (encoded as KML updates). Our KML layer then processes the changes and updates the position of the icons and lines.

By the way, I added some attachments to the original post which contain pictures of the leak.