Godot Color Mask Tutorial

Pre-tutorial note

This tutorial uses Markdeep! Markdeep renders to HTML locally on your web browser and this may take a second, so please be patient! 🙂

**Godot Color Mask Shader Tutorial**

# Godot Color Mask Shader Tutorial





In this tutorial, we'll be looking at how to make a simple color masking shader in Godot, and a few ways it can be used in both 2D and 3D.

Through the course of this tutorial, we'll be covering the following:

* How to write custom Godot shaders
* How color masking works
* A look at how to achieve a few different effects with color masking.

!!! NOTE: Note
    This tutorial is written to be beginner friendly, however some game development experience with Godot is recommended.
    If you have any questions or feedback, let me know in the comments!
    
    **This tutorial was written using Godot 3.1.1**


# Getting setup

The first thing you'll need to do is create a new Godot project.

!!! TIP: Tip
    A download is provided at the end of this tutorial with the finished project used in this tutorial!

Once you have your project created, create a new Node2D scene. Then add a ColoredRect node, or a sprite with a white texture. Finally add a sprite and assign a texture to it, I'll be using the default Godot project icon, and position it so it is overlapping a portion of the ColoredRect node.

Your scene should now look something like this:





Next we need to give the Sprite node a custom material and shader.

Select the Sprite node, and in the inspector scroll down to the Material section. Expand it, and then select the dropdown beside the Material field. Select the New Shader Material option from the dropdown:





Then click the newly created material to expand its properties. Then in the Shader field, choose the New Shader option:





Finally, all that is left is clicking the newly created shader. This will bring up the shader editor at the bottom of the Godot editor:





Now we are ready to write the color masking shader!



# Creating the Godot color mask shader

Before we start writing the color mask shader, let's quickly cover a very high level view of what a shader is and what programming language Godot uses for shaders.

## Quick look at what a shader is

I should start this off by quickly mentioning that I am primarily a game developer with skills akin to a jack-of-all trades. I know some about OpenGL and graphics programming, but **I am by no means an expert on computer graphics!** What I have written below is based off my own experience and some research I have done on the topic.

A shader is, for the sake of this tutorial, code that is executed on the GPU that is responsible for the drawing of anything on the screen.
What a shader does is it takes information passed in from the CPU, and does calculations to set pixels on the screen using the correct position, color, and more.

How this is done varies a bit based on the graphics language/library used to communicate with the GPU. As of Godot 3.1.1, Godot uses OpenGL as the rendering library to communicate with the GPU and to draw to the screen. OpenGL uses GLSL (Graphics Language Shader Language) as the programming language, which gets compiled to run on the GPU. GLSL is a powerful language that has been around for quite awhile, leading to there being many examples and documentation to use as reference. However, OpenGL is rather old at this point, and doesn't have some of the rich and powerful APIs present in other, newer, graphics libraries.

Despite this, OpenGL is still used today for lots of projects and is supported on the majority of computer hardware on the market. If you are interested in learning more about OpenGL and GLSL, I would highly suggest checking out [LearnOpenGL.com](https://learnopengl.com/)! It has great tutorials teaching everything from the basics to more advance topics. Definitely worth a look at if you are at all interested in graphics programming and OpenGL.

**Disclaimer**: *I am not at all affiliated or associated with LearnOpenGL, I just think it is a great resource!*

However, Godot does not allow for writing GLSL code directly. Instead, you write code in Godot's custom shading language. This allows Godot to target both OpenGL 3.3, OpenGL ES 3, OpenGL 2, and OpenGL ES 2 without the user needing to write custom shaders for each version. The trade off is that there are a few minor differences when comparing to GLSL, but a big bonus is that Godot handles the low level stuff for you automatically.

!!! NOTE: Note
    You can read more about Godot and its shader language at the following links:
    [Godot Shaders](http://docs.godotengine.org/en/3.1/tutorials/shading/shading_reference/shaders.html) and [Godot Shading language](http://docs.godotengine.org/en/3.1/tutorials/shading/shading_reference/shading_language.html) documentation pages.

!!! TIP: Tip
    The majority of what a shader does is *shade* the data given as input, hence the name "shader"!

## Quick look at shaders in Godot

Both Godot's shading language and GLSL are similar to the C programming language. The shader language is statically typed, semicolons are needed at the end of each line, and certain conditions must be met for the shader code to execute correctly.

In GLSL there are several operations/things you can do, and Godot allows for the majority of these through exposing certain functions. In Godot, there are three 'main' functions exposed to the Godot shader language: The vertex function, the fragment function, and the light function. These three functions handle the majority of usecases for writing custom shaders, and each function handles a specific and important part of the process.

Let's start with the vertex function first. The vertex function is designed to perform operations on each individual vertex in the mesh/object/thing passed into the shader. The vertex function is called for each vertex, with the vertex being operated on as an argument passed in to the function. Depending on which shader type you are using in Godot, you can access different functions and variables. You can find a list built-in variables/functions for the vertex shader here: [CanvasItem](http://docs.godotengine.org/en/3.1/tutorials/shading/shading_reference/canvas_item_shader.html#vertex-built-ins), [Spatial](http://docs.godotengine.org/en/3.1/tutorials/shading/shading_reference/spatial_shader.html#vertex-built-ins), [Particle](http://docs.godotengine.org/en/3.1/tutorials/shading/shading_reference/particle_shader.html#vertex-built-ins).

The fragment function is called for every visible pixel that makes up each face of the mesh/object/thing passed into the shader. This is where the color is applied, along with other data like normals, roughness, ambient occlusion, and other texturing and shading information. The fragment shader is where the majority of the visual work in shaders is done, as it deals with the individual pixels. You can find a list built-in variables/functions for the fragment shader here: [CanvasItem](http://docs.godotengine.org/en/3.1/tutorials/shading/shading_reference/canvas_item_shader.html#fragment-built-ins), [Spatial](http://docs.godotengine.org/en/3.1/tutorials/shading/shading_reference/spatial_shader.html#fragment-built-ins).

Finally, the light function handles how the lighting data effects the inputted mesh/object/thing passed into the shader. This function is called on a per light basis, with the data for each individual light being passed in as an argument. To be honest, I haven't used the light function very much, so I only have a limited idea of what it can do and how it's used. You can find a list built-in variables/functions for the fragment shader here: [CanvasItem](http://docs.godotengine.org/en/3.1/tutorials/shading/shading_reference/canvas_item_shader.html#light-built-ins), [Spatial](http://docs.godotengine.org/en/3.1/tutorials/shading/shading_reference/spatial_shader.html#light-built-ins).

For this tutorial, we will only be using the fragment function, as all of the operations we are doing will be on a per pixel basis.

While there is a lot more that can be said about shaders, I think at this point we are ready to write the shader. If you are interested in learning more about how shaders work, I would highly suggest taking a look at GLSL (OpenGL) and HLSL (DirectX) shader tutorials, as shader writing is quite interesting and it allows you to do all sorts of cool things that otherwise would be impossible or difficult.

But enough rambling, we have a shader to write!


## Write a color mask shader

Let's get back to our Godot project. There are many different types of color masking shaders, but for this tutorial we are going to use a simple technique that is efficient and allows for some interesting effects depending on how it is used.

With the shader editor open from where we left off, input the following code:

Let's go over how this code works!

First, we need to tell Godot what type of object this shader will be operating on. For Node2D and Control based nodes, we need to define the shader_type as canvas_item, which is the base class both nodes extend. The first line of code tells Godot we will be writing a shader for a node/object that extends the CanvasItem class. This is required so Godot sends the proper data when the shader is used.

The next two lines of code are defining a couple uniforms. Put simply, uniforms are variables we expose so we can change them on a per material basis. This allows these variables to be changed either through the editor or through code. The first uniform, masking_color is a Vector4 that we will use to define the color we want to use as a mask. The second variable, masking_range is a float that will define how strong the masking effect will be.

Next we define the fragment function. Remember, this function will be called for each pixel rendered, with the data we need for calculating each pixel passed in by Godot/GLSL. The fragment function does not return any data, so its return type is set to void. The reason behind this is we will modify the actual data itself, instead of returning anything.

In the next line of code we are defining a new variable, a Vector4, called world_pixel. This variable will store the world/screen pixel underneath the pixel that is about to be drawn. How we get the pixel underneath the pixel about to be drawn is we use the texture function, and we passed in the SCREEN_TEXTURE variable (which is given to us by Godot/GLSL) as the texture we want to get a pixel from, and for the position we passed in SCREEN_UV (given by Godot/GLSL), which is the coordinates of the pixel we are about to draw relative to the game window. This gives us the pixel underneath the pixel are about to draw, which is exactly what we are looking for.

The next line of code is an if check. First we get the difference of world_pixel and masking_color. However, because we don't know which pixel is stronger/higher than the other, we need to get the absolute difference using the abs function so we are not working with negative numbers. Finally, we get the length of the absolute difference we just calculated and check to see if it is MORE than the value in the masking_range uniform.

If the length of the absolute difference between colors is more than the masking_range uniform, then we simply write discard;. discard does as the name implies, it simply doesn't draw the pixel to the screen at all. This is great for our shader, because it makes it where only pixels on the inputted color, masking_color that fall within mask_range will be drawn.

______

Once you put the shader code in, you might notice that your sprite is no longer drawing! Don't panic, you likely wrote everything correctly. To make the sprite visible again, we just need to change the masking color uniform in the Godot editor to whatever color we want. For the picture below, I have assigned the masking color uniform to white.





Once you have the color set, you'll see that the sprite is only drawn on the color set in the masking color uniform!


!!! NOTE: Note
    There is a slight performance cost when using discard, but so long as you are not calling it a few hundred thousand times per frame, it is probably not going to be an issue. If it does become an issue for your game, a *slight* performance boost would be to modify the alpha of the texture to 0 instead of calling discard.

And with a simple change, you can have the opposite effect! In the Godot shader, change the following line of code:

All that changed is instead of checking of the length of the absolute difference between colors is more than masking_range, instead it is now checking to see if the length is less than masking_range. This will make the sprite draw when it is NOT on the mask, instead of drawing when it is on the mask.





Honestly, that is all there is to it! The code where all of the color masking happens is (length(abs(masking_color - world_pixel)), which gets the difference of the color underneath the pixel about to be drawn. That tiny bit of code is the backbone to the entire shader.



# What can you do with this?

You might be wondering why this is needed at all. After all, can't the Node2D/Control just use it's Z-Index to achieve this functionality?

It is true that for the example above, there really isn't much point. However, using this effect you can achieve an effect akin to the indie game [INK](https://store.steampowered.com/app/385710/INK/):





*This is from a small protoype game I made based on INK! It was one of my first Godot 2 projects, that I later ported to Godot 3.*


Here's an overview of how the prototype above works:

* The background and the platforms are *almost* exactly the same color, with a difference of roughly 0.02.
* All of the cirlces are drawn using a Node2D that takes advantage of Godot's draw function to draw all of the circles in a single pass.
* The 'Ink' system has the draw on mask shader above, with masking_range set to 0.01 and the color set to the same color as the platforms. This makes the circles drawn by the 'Ink' system to only render on the platforms, given the appearance of 'painting' them! This isn't how the indie game INK does it, but it gives a very similar appearance. Because the masking is all done on the GPU and the circles are drawn using a single _draw function call, there is very little performance cost!

So as you can hopefully see, while the color mask shader is simple, it can be used to create interesting game play!

Another thing you can do with the masking shader is instead of reading the pixels from the screen, you can instead read the pixels from a texture. This can give interesting effects:





The green pixels in the picture above is from using a noise texture as the color mask and coloring the icon based on whether the pixel is over the mask or not. There are interesting things you can do when you use a texture as the input, as you can see later in this tutorial.

**You can find the code for the picture above in the download link at the bottom of this tutorial.** The included download explains how the shaders in the project work (INK prototype not included)


# Color masking in 3D

This shader can also be used in 3D, and it works almost entirely the same way! However, for 3D, let's try something a little different. Let's use a texture as the mask, but have the texture be from a Viewport! This will allow us to do some interesting 3D color masking effects.

To do this, we'll need a few things. The first thing we will need is a Viewport to render the color mask. For my project, I've just created a Viewport node with a Camera node as it's only child. The camera under this Viewport node renders only a few visual layers, while the 'normal' camera (the one rendering what we'll see) has those layers disabled so we cannot see what the color mask Viewport can see.

We'll also need to have objects in the visual layers for the Color mask Viewport to render. For this tutorial, I will just be using three cubes of different sizes and rotation with a simple white shadeless SpatialMaterial.

Another thing we will need is to keep the 'normal' camera and the color mask Viewport in sync so any movement in the 'normal' camera is applied to the color mask Viewport camera. To do this, I'm just added a simple script that takes the global transform of the 'normal' camera, and applies it to the Viewport node camera.

!!! NOTE: Note
    You might be able to use the RemoteTransform node to achieve this functionality without needing to write any code, however I have found that the RemoteTransform node sometimes acts strangely when parented to a Viewport outside of the root Viewport.

Finally, the last thing we will need to do is assign the texture in the shader to the Viewport texture. I did this through a small bit of code. The code exposes the NodePath to the Viewport, and then sets the material uniform of the shader to the Viewport texture using the set_shader_param function in GDScript.

!!! TIP: Tip
    Remember, a download is provided at the end that shows how these shaders work, so don't worry if you don't know how to set this all up. The included download has a 3D scene setup that you can use as a reference, with all of the scripts and shaders included.

Once we have all that setup, we can start creating the 3D color masking shader. Let's start with something simple, and create a shader that changes the color based on whether the pixel is masked or not.



## Simple color masking shader

Let's look at the shader code that we'll be using:

It should be quickly noted that the majority of the code above is from Godot's SpatialMaterial to ShaderMaterial conversion. I trimmed the shader down as much as I could for the purposes of this tutorial. **I am only going to cover the color masking code, not the Godot generated code.** If there is interest I can make a general, broader Godot shader tutorial, however for this tutorial we are only going to cover the color masking portion.

Let's break down what this code does.

First, we define a new uniform called mask_texture. The mask_texture uniform is a Sampler2D, an image, that we will be using as the color mask. For this tutorial, I will be using the Viewport texture, but in theory it will work fine with other inputs and textures.

Next, after some Godot generated code, we define a new variable called mask_pixel in the fragment shader. mask_pixel is a Vector4 that we are retrieving using the texture function, passing the mask_texture uniform as the texture, and SCREEN_UV as the position. This will give us the pixel in the inputted mask_texture that is at the same position as the pixel that the shader is about to draw on the screen. This is exactly the same as the 2D example code above, but instead of using SCREEN_TEXTURE we are using mask_texture.

Next the shader checks to see if the pixel stored in mask_pixel is over 0.9 on the red color channel. The reason we're doing a check like this, instead of getting the length of the absolute difference of two colors, is because in my example project I know everything will be in a monochrome color scheme, meaning I can just check a single color channel (minus alpha) to get the value of the pixel at that position.

Finally, if the value stored in mask_pixel is over 0.9, then the shader sets ALBEDO to vec3(1, 0, 0), which will equal the color red.

If everything is setup and we run the project, you'll find the following result:





If you look in the top right, you can see what the color mask we are passing to the shader looks like.

What is cool about doing something like this in 3D is that you can kinda the cubes *in* the sphere, despite the fact that all we did was change the color. Especially in motion, this is a very cool effect.



## Some other examples

There are other things we can do to achieve different effects. For example, if we instead discard the pixels instead of changing the albedo to red, we get the following result:





Which gives a cool cutout look that might be visually interesting for certain projects.

Another cool thing we can do is cull if the pixels are *not* in the mask. With a little shader magic, you get a picture like you saw at the beginning (shown below in *Final Notes* section).

If you are wondering what this could be used for, the answer is all sorts of things! For example, I am using color masking in my 3D painting game project:


Worked more on the Godot paint project. Added support for moving objects and color mask support so decals on moving objects wouldn't look strange when interacting with static objects.#GodotEngine #gamedev #indiedev #screenshotsaturday #WIP pic.twitter.com/tqeYOD5rqE
— RandomMomentania (@TwistedTwigleg) July 7, 2019



The code I'm using is a mixture of the 3D cutout shader and the 2D color masking based on the color. This allows me to define which of the 'paint' decals are placed where without them overlapping onto each other, giving a much nicer visual experience on moving objects. In the video above, you can see the color mask in the top right corner. It works exactly the same as the code shown in this tutorial, it just takes what I needed from both shaders and puts them together.

**A download link is provided at the bottom of this tutorial that contains all of the shaders shown so far** with comments in the shader code explaining how it all works!

(The download does not include the INK prototype and the project shown in the tweet)


# Final Notes





Now you know how to write custom Godot shaders, how to make a color masking shader, and some ways you can use it in your games!

In the future, I make make more tutorials on custom Godot shaders. If you want to keep up to date with what I'm doing, please consider following me on [Twitter](https://twitter.com/TwistedTwigleg), where I post regularly! You can also join the [Godot community forums](https://godotforums.org/) if you want to join an awesome community of Godot developers.

**Disclaimer:**: *I one of the admins on the Godot community forums.*


!!! WARNING: Warning
    If you ever get lost, be sure to read over the code again!
    
    **You can access the finished project for this tutorial right [HERE](https://drive.google.com/open?id=1Vw5CqP1lWv7y08VsyyNW_bdEZLH3GPyy)!** It includes all of the examples shown above and contains helpful comments explaining how everything works.
    
    Please read LICENSE.html or LICENSE.pdf for details on attribution. **You must include attributions to use some/all of the assets provided**. See the LICENSE files for details.


!!! TIP: Thanks
    
    Thanks for reading this tutorial!

    If you like what we do, **please consider buying one of [our paid games](https://randommomentania.com/paid-games/) or [products](https://randommomentania.com/other-products/) to help support RandomMomentania!**

    Have a suggestion for a future tutorial? Let us know either in the comments section below, or by following the instructions on our [Contact and FAQ](https://randommomentania.com/contact/) page!

    



© RandomMomentania 2019