Add support for using BC5 and EAC_RG compressed textures as normal maps.

[castano]

Normal maps are often compressed using the BC5 and EAC_RG formats. This is currently not supported by three.js as the shaders expect the sampled normal map values to contain RGB components representing the packed XYZ coordinate.

To address this issue I propose we add a projectZ attribute to the NormalMapNode object. When this attribute is true, the generated shader code computes the Z component of the normal by projecting the XY components over the hemisphere.

Texture loaders are expected to set the projectZ attribute of the texture, which in turns activates the code generation in the corresponding NormalMapNode.

At the moment there are no texture loaders doing this, but this is something that the existing KTX and KTX2 loaders could benefit from. Additionally, I'd like to use this feature in texture loaders that use the spark.js encoders.

This contribution is funded by Ludicon

[github-actions]

📦 Bundle size

Full ESM build, minified and gzipped.

	Before	After	Diff
WebGL	338.24 79.14	338.29 79.15	+48 B +13 B
WebGPU	579.91 160.21	580.5 160.47	+588 B +255 B
WebGPU Nodes	578.52 159.97	579.1 160.23	+584 B +255 B

🌳 Bundle size after tree-shaking

Minimal build including a renderer, camera, empty scene, and dependencies.

	Before	After	Diff
WebGL	469.88 113.85	469.88 113.85	+0 B +0 B
WebGPU	649.62 175.88	650.04 176.06	+423 B +174 B
WebGPU Nodes	603.73 165.04	604.15 165.21	+423 B +171 B

[Mugen87]

We should not rely on monkey-patched code here.

How about you introduce the property in the CompressedTexture class? Or is it also relevant for Texture?

Alternatively, move the property into the userData field of normalMap for now.

[Mugen87]

When thinking about this: Why not checking for specific RG texture formats here and only then set the node property to true? In this way we need no additional texture property and it would not be necessary to update loaders.

[castano]

This is most useful in CompressedTexture, but in my particular application I'm using this with ExternalTexture and it can also be useful with uncompressed textures, to reduce their size by storing only the RG components.

I agree this should be done in some other way. A potential problem with this PR is that the attribute won't be cloned with the texture and that may cause issues. Adding it to the userData field as you propose would solve that problem.

I'm not sure enabling the Z reconstruction is the right thing to do for all RG texture formats. RG textures can be used for other purposes in addition to normal maps (heat distortion maps, 2d vector fields, chrominance values, etc). It may not cause any harm to just compute the Z component this way, but this won't extend to ASTC normal maps if we ever want to support that.

[castano]

I went with the userData approach. Let me know if that looks good to you.

[Mugen87]

I'm not sure enabling the Z reconstruction is the right thing to do for all RG texture formats.

My idea was to only do this for specific map types. So for a normal map, a format like bc5-rg-snorm (or SIGNED_RED_GREEN_RGTC2_Format) must hold packed RG data, otherwise it would be unusable for a normal map. This is something that could be implemented in the node classes so loaders don't have to worry about this detail.

But it's true a separate property allows more flexibility than deriving from texture formats.

It would be interesting to hear the opinion of others about this PR. Would you be okay if we label this PR as a "Draft PR" for now? I have the feeling we need to discuss the API a bit more in the team/community before making a decision.

[castano]

Yeah, now that I think about it, it could make sense to do it based on the format. This happens inside the NormalMapNode, so I think it's safe to assume the texture being sampled is indeed a normal map.

The explicit unpack mode allows supporting ASTC LA normal maps or DXT5nm and in these cases it's not possible to infer the packing mode from the format alone.

I don't see an option to convert this into a Draft PR, but(Found the link) I'd be happy to hear other people's feedback.

[Mugen87]

The explicit unpack mode allows supporting ASTC LA normal maps or DXT5nm and in these cases it's not possible to infer the packing mode from the format alone.

@donmccurdy I'll ping you directly since you have done so much work in context of texture compression.

Similar to Texture.colorSpace, what do you think about introducing a new property Texture.packing that describes the packing strategy of a texture. The default would be NoPacking (as a pendant to NoColorSpace).

Packing strategies usually apply on certain texture types so probably most of the new constants for Texture.packing would be quite specific (like RGNormalPacking or GANormalPacking). But that is a similar situation like with Texture.colorSpace where certain constants only apply to color textures.

So...what do you think about this approach? Do you have maybe alternatives in mind?

[donmccurdy]

I wouldn't be opposed to adding a Texture#packing property, I think @bhouston had proposed something similar once? A few other thoughts:

1. If we see texture.format = THREE.RGFormat (or similar) assigned to a normal map slot, possibly that is enough in this case?

2. In the newer TSL/NodeMaterial APIs with WebGPURenderer, is a property on THREE.Texture sure to be accessible? Or could the Texture instance have been wrapped in other nodes, obscuring the underlying Texture? If so, we could handle this by a new node to handle the unpacking, or a material property similar to material.normalMapType, instead.

[castano]

OK, I've been looking at the possibility of enabling the RGNormalPacking mode automatically based on the texture format. It could be something along these lines:

if ( material.normalMap.format == THREE.RGFormat || material.normalMap.format == RED_GREEN_RGTC2_Format ) {
	node.unpackNormal = NormalRGPacking;
}

Let me know if it's worth adding this. The default behavior could still be overridden with the user data attribute: normalMap.userData.unpackNormal.

Note, this only handles RG textures and BC5. I think there's no identifier for EAC_RG11 in three.js. If support for it is added in the future, this could be handled here as well.

We could also add support for signed RG textures and eliminate the 2*x-1 unpacking in the normal map node, but barely anybody uses the signed formats, so I think it's not worth it.

I'm ambivalent about adding a Texture#packing property. I can experiment with that if you think that's the way to go.

[donmccurdy]

I suspect checking .format would make sense for WebGLRenderer but not for TSL in WebGPURenderer. Since the focus of this PR is WebGPURenderer, we can skip that for now, but I wouldn't be opposed to adding this support in WebGLRenderer later.

[castano]

This could use the colorToDirection function in Packing.js

That said, in the case of normal maps it may make more sense to use:

let normalMap = this.node.mul( 2.0 ).sub( 254.0 / 255.0 );

As that prevents lighting discontinuities on flat normals along uv seams where the tangents do not match exactly.

[castano]

To add a bit more motivation, here's a detail from SciFiHelmet showing the normals encoded as RGB using BC7, RG using BC5 (this PR) and RGB uncompressed:

[castano]

I've put together a demo that uses the changes I'm proposing here. You can check it out here: https://ludicon.com/sparkjs/gltf-demo/

[castano]

Does anybody have any more thoughts about this?

I'd like to release the source code of the spark.js gltf-demo once the changes in this PR are included in three.js.

[Mugen87]

I'm not happy with the name yet since it is quite unspecific. I would prefer to integrate the normal term somehow in the name. I also like the pack/unpack notation.

How about using unpackNormal()? The idea is to have a function that converts a packed, two channel normal into a three channel (unpacked) normal.