Layer opacity via shader

[BuM]

Hi there,

we want to bring the field AbstractLayer.opacity to good use by having its value multiplied to each of its renderables. This way we would have easy control over the visibility of a whole layer, no matter the kinds of its renderables.

We thought an easy approach would be a shader assigned to the layer which just multiplies the alpha value to each fragment created by the layer. As we are new to shaders, our solution is rather clumsy:

uniform sampler2D testTexture;

// alpha value to use
uniform float alpha;

// to decide where the color of the fragment comes from
uniform int textured = 0;

void main(void) {
 if (gl_TexCoord[0].s < 0.0 || gl_TexCoord[0].s > 1.0) discard;
 if (gl_TexCoord[0].t < 0.0 || gl_TexCoord[0].t > 1.0) discard;

 if (textured > 0) {
  gl_FragColor = texture2D(testTexture, gl_TexCoord[0].xy);
  gl_FragColor = gl_FragColor * alpha;
 } else {
  gl_FragColor = gl_Color * alpha;
 }
}

As there seem to be two sources for the color (textured and gl_Color, depending on the renderable) and we do not know how to automatically tell which one to take, we have this extra field "textured" which we have to set manually for each (overridden) renderable class.

1. Is there a way to automatically identify the correct source of the fragment color?
2. Is there even an easier way to accomplish our goal?

Thanks a lot!

[BuM]

A problem with this approach is that it is not enough to activate the shader at the start of the AbstractLayer.doRender() method because not all renderables are rendered at this phase:
1. OrderedRenderables are not rendered, but added to DrawContextImpl.orderedRenderables to be drawn later.
2. Same with surface objects which are drawn at another phase.

So for both types of renderables the shader (or just the opacity value) has to be set inside the renderable class, which takes away much of the convenience of the shader solution.

Another idea of ours is to iterate through each of the layer's renderables at the preRender phase of the layer to set their opacity value. Unfortunately, not all renderables do have such an opacity value (like UserFacingIcon, and possibly others).

So now we have different solutions for different types of renderables, which is rather unsatisfying and not easily adaptable to new situations.
Any ideas for improvement are welcome!