Unity3D: Unlit Alpha Mask Shader

17 Replies

Over the past several years working in Unity many of my projects have involved some kind of mixed reality, augmented reality, transparent MovieTexture, or something of the sort. One custom shader that I have reused a lot for these effects is an unlit alpha mask shader. Alpha masking is simply setting the opacity of a texture’s pixels, usually with a separate texture map.

I’m certainly no Unity shader expert, but creating this shader is relatively straightforward if we start with the right foundation. In this case we’ll be modifying the closest thing to what we want – Unity’s own unlit alpha shader. We’ll start with the completed version and then explain how it was modified from the source.

Unity Shader

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
// Unlit alpha-blended shader.
// - no lighting
// - no lightmap support
// - no per-material color

Shader "Unlit/AlphaMask" {
Properties {
    _MainTex ("Base (RGB)", 2D) = "white" {}
    _AlphaTex ("Alpha mask (R)", 2D) = "white" {}
}

SubShader {
    Tags {"Queue"="Transparent" "IgnoreProjector"="True" "RenderType"="Transparent"}
    LOD 100
   
    ZWrite Off
    Blend SrcAlpha OneMinusSrcAlpha
   
    Pass {  
        CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag
           
            #include "UnityCG.cginc"

            struct appdata_t {
                float4 vertex : POSITION;
                float2 texcoord : TEXCOORD0;
            };

            struct v2f {
                float4 vertex : SV_POSITION;
                half2 texcoord : TEXCOORD0;
            };

            sampler2D _MainTex;
            sampler2D _AlphaTex;
           
            float4 _MainTex_ST;
           
            v2f vert (appdata_t v)
            {
                v2f o;
                o.vertex = mul(UNITY_MATRIX_MVP, v.vertex);
                o.texcoord = TRANSFORM_TEX(v.texcoord, _MainTex);
                return o;
            }
           
            fixed4 frag (v2f i) : SV_Target
            {
                fixed4 col = tex2D(_MainTex, i.texcoord);
                fixed4 col2 = tex2D(_AlphaTex, i.texcoord);
               
                return fixed4(col.r, col.g, col.b, col2.r);
            }
        ENDCG
    }
}

}

Important: This shader will only work for Unity 4.5 or newer. Earlier versions will be similar but usually output COLOR from the fragment shader instead of SV_Target.

What we do here is:
  1. Start with the built-in Unity unlit alpha shader (version 4.5.1)
  2. Add a texture sampler for an alpha mask
  3. Apply the color channel from the mask texture to the alpha of the main texture

Since we’re using an unlit shader as our base, our texture’s color will be unaffected by scene lighting since it is ignored. In addition the shader will be more efficient than a normal lit shader since it doesn’t have to calculate light or shadows. And as an alpha shader we can assume it already contains the tags, blending, etc. to render transparency properly.

Note: You can find an archive of Unity’s built-in shaders here.

Unity Shader

1
Shader "Unlit/AlphaMask" {

This is the actual name of the shader that Unity will recognize preceded by its folder path which is used for organization. Regardless of the name of a .shader file, once imported it can be added as the shader of a material via the drop down menu as Folder/Name.

Unity Shader

1
_AlphaTex ("Alpha mask (A)", 2D) = "white" {}

Here we expose a new alpha channel texture that defaults to a solid white. This is what allows us to dynamically set a texture in the Unity editor like so:

Alpha mask properties

Unity Shader

1
sampler2D _AlphaTex;

This defines a new sampler2D variable linked to our alpha texture property above. It is used to read a pixel color of the texture, given a UV coordinate.

Unity Shader

1
2
3
fixed4 col2 = tex2D(_AlphaTex, i.texcoord);
               
return fixed4(col.r, col.g, col.b, col2.r);

Here in the fragment shader is where all the magic happens. By this point variable col contains the rgba color channel data from our main texture. We simply added a second variable col2 to store the color data from our alpha texture. Lastly we return a 3rd new color combining the first two, using the rgb from col and the r from col2 as the alpha. This is then returned by the shader as the pixel color drawn to the screen!

Note: We pull from the r channel of our alpha texture because ideally you want your textures to be as small as possible. Usually a full color texture with alpha is 32-bit rgba (8 bits for each color channel). Although, since only one color channel is needed for our alpha mask, we can use an 8-bit single channel gray-scale texture which, when imported into Unity as a default texture type, lives in the r channel.

Note: The second parameter of tex2D() we are passing a pixel’s UV coordinates. In the example shader we are using the same UV’s as the main texture. If you look up into struct v2f you will see it defined there as TEXCOORD0. If you wanted to use a second UV set for the mask you can define a new variable for it there as well as TEXCOORD1 and then pass that to tex2d() instead.

Concluding..

Hopefully you learned a little something about tweaking shaders. This shader is perhaps the most useful because you can use it to add alpha to a MovieTexture, switch the two and have an animated alpha channel, or even use a MovieTexture for both! There are certainly other uses for it and shaders like it though, like sharing pixel data between multiple images or dynamically modifying a texture’s alpha.

Enjoy!

Ben

17 thoughts on “Unity3D: Unlit Alpha Mask Shader”

  1. ernie

    Copied and pasted code into new shader but errors:
    Shader error in ‘Custom/alpha2’: Program ‘frag’, unknown semantics “SV_Target” specified for “frag” at line 49

  2. Ben Post author

    Hi Ernie. Sounds like you’re using a Unity version pre 4.5. Like I need mentioned above, try replacing ‘SV_Target’ with ‘ COLOR’.

  3. Nick

    Hey Ben,

    Thank you very much for this small tutorial. i have used the same script for my own project.
    Only i have used 2 movie files. i have a movie from someone in front of a green screen and made separate files for rgb and alpha. imported them at the base RGB and Alpha Mask in unity and it works as a charm.
    the only thing is. the RGB start playing but the Alpha doesn’t. is there a piece of script that i am missing to make the alpha movie also play?
    thank you.
    Nick.

  4. Nick

    Hoi Ben,

    A little suplement on my last message. The shader is now set on Unlit.
    is there a way to change that so that lights are effected and shadows will occur?
    thank you.
    Nick.

  5. Ben Post author

    Hi Nick, you just need a small script on the object with the shader that plays both videos. You get the secondary texture through GetTexture() by passing the name that you used for the alpha texture in the shader, in this case “_AlphaTex”.

    C#

    1
    2
    3
    4
    5
    void Start ()
    {
      (renderer.material.mainTexture as MovieTexture).Play();
      (renderer.material.GetTexture("_AlphaTex") as MovieTexture).Play();
    }
  6. Ben Post author

    As far as doing the same effect with a lit shader, I recommend going through the same steps that the tutorial does explaining how the unlit shader was derived from Unity’s default Unlit-Alpha.shader with the default Alpha-Diffuse.shader. The changes should be about the same.

  7. Thom

    I made some small changes to this shader to allow it to be faded in and out.

    I added a new property:

    1
     _AlphaVal ("AlphaVal", Range (0,1) ) = 1.0

    Then in the pass added this:

    1
    float _AlphaVal;

    and altered the return line to this:

    1
    return fixed4(main.r, main.g, main.b, (main.a*alph.r*_AlphaVal));

    By changing the _AlphaVal, I can control the overall opacity.

    Now a question: I’d also like to be able to change the offsets of the MainTex and AlphaTex independently, but the way it’s currently written the AlphaTex moves with the MainTex. Any ideas how to achieve this? I’m new to writing shaders. Thanks!

  8. Thom

    As so often happens, I answered my own question after asking it. I tried something like this unsuccessfully earlier, but it “clicked” this time after re-reading your post. The solution:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    // Unlit alpha-blended shader.
    // - no lighting
    // - no lightmap support
    // - no per-material color

    Shader "Unlit/AlphaMask" {
    Properties {
        _AlphaVal ("AlphaVal", Range (0,1) ) = 1.0
        _MainTex ("MainTex (Sprite)", 2D) = "white" {}
        _AlphaTex ("AlphaTex (R)", 2D) = "white" {}
    }

    SubShader {
        Tags {"Queue"="Transparent" "IgnoreProjector"="True" "RenderType"="Transparent"}
        LOD 100
       
        ZWrite Off
        Blend SrcAlpha OneMinusSrcAlpha
       
        Pass {  
            CGPROGRAM
                #pragma vertex vert
                #pragma fragment frag
               
                #include "UnityCG.cginc"

                struct appdata_t {
                    float4 vertex : POSITION;
                    float2 texcoord : TEXCOORD0;
                    float2 texcoordA : TEXCOORD1; // alpha uv
                };

                struct v2f {
                    float4 vertex : SV_POSITION;
                    half2 texcoord : TEXCOORD0;
                    half2 texcoordA : TEXCOORD1; // alpha uv
                };

                sampler2D _MainTex;
                sampler2D _AlphaTex;
                float _AlphaVal;
               
                float4 _MainTex_ST;
                float4 _AlphaTex_ST; // for alpha uv
               
                v2f vert (appdata_t v)
                {
                    v2f o;
                    o.vertex = mul(UNITY_MATRIX_MVP, v.vertex);
                    o.texcoord = TRANSFORM_TEX(v.texcoord, _MainTex);
                    o.texcoordA = TRANSFORM_TEX(v.texcoordA, _AlphaTex); // note texcoordA
                    return o;
                }
               
                fixed4 frag (v2f i) : SV_Target
                {
                    fixed4 main = tex2D(_MainTex, i.texcoord);
                    fixed4 alph = tex2D(_AlphaTex, i.texcoordA);
                   
                    return fixed4(main.r, main.g, main.b, (main.a*alph.r*_AlphaVal));
                }
            ENDCG
        }
    }

    }
  9. Chris

    Note: The second parameter of tex2D() we are passing a pixel’s UV coordinates. In the example shader we are using the same UV’s as the main texture. If you look up into struct v2f you will see it defined there as TEXCOORD0. If you wanted to use a second UV set for the mask you can define a new variable for it there as well as TEXCOORD1 and then pass that to tex2d() instead.

    Could you possibly show how to do this? i.e. keeping the mask co-ordinates from the mask texture, rather than taking them from the main texture? I’ve tried it a few ways based on the above but having no luck.

    Many thanks
    Chris

  10. Jacob

    Thanks for sharing this shader Ben. Looks like it’s working, however, I have 2 different prefabs which have different MovieTextures (and different alpha masks), but it seems whenever I change the BASE and ALPHA MASK values on one prefab, it changes it for the other prefab as well; so the mask is off for one or the other. Any thoughts? I originally created the first prefab from a duplicate of the GAMEOBJECT with the alpha shader… would that matter?

  11. Ethan

    That’s very cool stuff!
    I’m new in unity3d. Please give a code example for creating a second UV set for the mask.
    Thanks

  12. Martin Chan

    This is awesome.
    I personally don’t know much on how to code shader. This really help me out a lot.

    Thanks for the share.

  14. Reena

    New
    Hi,
    I have tried all the shader about mention,
    The masking thing appears in scene view but not in game view.

  16. Ran

    Hello Ben, thanks for this wonderful shader – I tweaked the render a little bit for my own needs, but it is indeed both simple and awesome.

    But there is one major problem : everything works perfectly fine in the editor, but in the final build (Unity 2017.1.1f1), every Image that had a material with this shader attached becomes invisible. Do you maybe know a workaround? I would be sad to have to let go the very nice result I had with this technology…

    My problem is better explained here with pictures :
    https://forum.unity.com/threads/shader-and-or-material-disabled-on-bluid.668899/

  17. Ran

    (follow up of last comment)

    All right, I found what was missing, it was just :
    ZTest [unity_GUIZTestMode]
    And probably later on :
    #include “UnityUI.cginc”

    Thanks again !

Leave a Reply

Your email address will not be published. Required fields are marked *