A bit more clarification:
Every layer can't have all channels accessible.
Now layer have
RGBA + Specular (4 mono channels at all)
I am probably missing something here, but it seems like what I wrote would be compatible with what you have explained. No additional per layer channels are envisioned. I do see now there is only a single RGBA value for the layer, which changes things a little bit, making it simpler.
I was trying to describe a way for the user to control where the existing RGBA data for each layer would be accumulated, accumulating the RGBA components of each layer to one of the final output textures. The layer would only be able to assign one destination for the RGBA.
I am probably missing something, but it seems like what I am describing is just how the user would select the destination textures using mutually exclusive buttons on the layer's row in the panel rather than menu items. The implementation seems very similar to what you said here:
Other way is to keep blend modes as it was but add additional option how to treat layer - as color + specular, as emissive or as colored specular. But it is more complex (in terms of UI and understanding of user) and not sure that it is really better than previous case.
That's pretty much what my UI sample is intended to demonstrate - how to treat the layer in a user friendly manner. The UI seems like it would be very straightfoward - just click on the icons visible on the currently selected layer row for which one of the layers summary that the RGBA should go to. Since there would be no new per-layer data, other than flags controlling which target buffer it should send data to, this would make it possible to have the additional output buffers (reflection, etc) with the only increase in memory being the final output texture and perhaps a secondary buffer used during accumulation. The shader used to present these textures could do whatever they wanted with the data they are provided.
Here's an update of the sample that works with the single RGBA channel. Only one button would be able to be selected at a time, and it would control if it would be Color+Specular (the original 3DC behavior), ColorSpec RGB + Gloss A, Emissive RGB + Luminance A (used for glow effect shaders), and Reflection RGB + Refraction scalar. Its much simpler now that I understand there is only a single RGBA color to work with, and it seems to fit the standard set of texture data assets might want to use. Note: The X shown over the normal would also be shown over all but one of the icons.
I really appreciate being able to make these suggestions, even if they turn out to not make sense to do for whatever reason.
Thanks for reading!