The alien grasshopper - Complex layer masking using channel data

In this tutorial I will show how to create a layer mask based on the image data itself, and how to select given portions of an image seamlessly and with little effort.

To illustrate this technique, I will take as example the grasshopper image that I posted around some time ago, and show how I isolated this nice insect from the flower and the rest of the background. To show the technique more clearly, in this post I will turn the grasshopper into an alien insect by applying a selective hue shift, like you see below:

Isolating the insect manually, either by drawing with a brush or by using a freehand selection tool, would be very time consuming and boring, and probably not as precise as the technique shown here.
Instead, I will show you here how you can use the image data itself to create a convenient grasshopper mask, all that in few mouse clicks. The idea is to find image channels in which the grasshopper is well separated from the flower, and then add a curves adjustment to the channel data so that the grasshopper appears completely white and the flower completely black in the final mask (with some smooth variation in the transition regions). The first step is to identify which channels are more promising. You can choose between the individual Red, Green and Blue channels of the RGB colorspace and the L, a and b channels of the Lab colorspace.

The grasshopper differs a lot from the flower in terms of color, but not too much in terms of luminosity, therefore we can already guess that the "L" channel will not be very useful... the other channels are all potentially promising, and we need to look at them directly to see which are best.

To visualize the individual channels, do the following: create a "Clone" layer (which I called "channel selector", select the layer you want to inspect, and choose the channel from the corresponding drop-down list. The individual channels of the grasshopper image are shown below (move the mouse over the caption items to see the corresponding image). The contrast of the "a" and "b" channels has been increased to better see the tonal variations.

Click type to see: Red - Green - Blue - Lab "a" - Lab "b"

As expected, the "a" and "b" channels look the most promising ones... Not surprising, since the grasshopper is searated from the flower mostly in terms of color. The "a" and "b" channels might be difficult to interpret if you are not familiar with the Lab colorspace, so I'll spend a few words to explain why they look like that.
The "a" and "b" channels encode the color information of the image, independently of the luminosity, in a quite peculiar way: a neutral grey corresponds to "a=50%, b=50%", while values above and below 50% encode complementary colors. For example, "a<50%" corresponds to green and "a>50%" to magenta, while "b<50%" corresponds to blue and "b>50%" to yellow. The higher the distance from 50%, the higher is the saturation of the corresponding color component.

With that in mind, we can already expect that the flower and the grasshopper will have quite different values in the "a" channel (the grasshopper is greenish and therefore "a<50%", while the flower is pinkish and so we can expect that "a>50%"). The "b" channel is less obvious (the grasshopper is certainly yellowish with "b>50%", but the b values in the flower are not so evident...) but a direct look to it shows that there is a quite strong separation between grasshopper and flower in this channel as well.

For this image I ended up using a combination of the "a"and "b" channels to isolate the grasshopper, and the Red channel to mask the out-of-focus background.

Now that the masking strategy is ready, I made the "channel selector" layer invisible (but kept it there in case I wanted to inspect again the individual channels later on). Invisible layers do not consume any memory and are simply skipped by the processing pipeline, so you can have as many as you like without any negative impact on the processing performance...
I added a "Hue/Saturation" adjustment layer on top of the invisible channel selector. I will change the Hue of the layer until the green color of the insect will turn into electric blue, but first I will work on the layer mask to isolate the grasshopper from the rest. For that, I double-clicked on the icon corresponding to the "opacity mask" (the one at the extreme right right of the layer row) to open the layer stack of the mask itself (initially empty).
First, I added a new layer group (I've called it "a channel mask") and then added a "Clone" layer inside this group. I selected the "a" channel of the background layer as source, and clicked on the "show active layer" button below the preview window to activate the visualization of the mask itself. The preview window now looks like this:

Next, I've added a "Curves" layer on top of the "a" clone and applied a "threshold-like" curve that turns the mask into pure white above the grasshopper, and pure black above the flower (with some smooth transitions).
Here is how I did it:

I repeated then the above steps for the "b" channel (created a group, inserted a clone layer into the group and then a "Curves" adjustment on top of the cloned channel), obtaining this result:

The two masks look similar, but not identical... actually, the best would be a combination of the two. Combining the two masks is quite easy: I simply changed the blending mode of the top group (with the "b" channel mask) to "Lighten" mode, to obtain this:

At this point, the masking of the grasshopper looks quite ok (we could remove a bit of remaining red areas by hand), but I still need to isolate the out-of-focus background. For that, the "Red" channel looked like the most appropriate. The technique is always the same: I added a layer group, inserted a clone layer inside the group and a curves adjustment on top of it. I ended up with this red channel edit:

The red channel mask needs now to be combined with the "a+b" one: for that, I have applied a slight blur (5 pixels of radius) and then changed the blending mode the the red channel group to "Darken", to obtain an almost final result:

There are still a couple of unmasked spots in the background, but those are easily corrected by drawing directly on the mask with a large black pencil. Therefore, my last masking step is to add a "Draw" layer on top of the "red channel mask" group, set the background color to white and the pen color to black, and change the blending mode of the "Draw" layer to "Darken". Now I could draw with a large pencil (I've set the size to 100 pixels) to completely remove the unmasked regions. The final result looks like this:

As you can see, I've ended up with a quite complex mask edit. However, I can guarantee that the procedure is much longer to describe than to realize. With a little practice, it is easy to identify the channels that are likely to be good for masking, and the technique is rather simple: create a group, clone a channel inside the group, and add a threshold-like curve adjustment to create the mask.

The nice thing is that all steps in the mask creation are non-destructive, and you can go back and tweak any setting whenever you like, even after saving and reopening the file later on.
Moreover, despite the relatively large number of layers and blendings, the mask is computed quite fast and does not slow down significantly the preview of the edited result.

Finally, it's time to turn the color of the grasshopper into electric blue. First I switched back to the main "Layers" panel to edited the "Hue/Saturation" adjustment. I double-clicked on the corresponding layer name to open the configuration dialog, and shifted the hue by 150 degrees in the positive direction... and there it is, a weird alien has taken the place of the nice insect!

Of course, the same technique can be applied to more "traditional" edits... I've used the same mask to increase a bit the contrast and give some more "pop" to the grasshopper. That's the final result (mouse over to see original):