Ive been going through painful moments in order to understand and do a decent bumpmapped texture with a reflected mask. I thought I had the answer before, but then tested it, and it didnt work. Now I have the answer, and it has been proven!!!
This guide assumes you read the Valve wiki on normal mapping, and reflective materials:
http://developer.valvesoftware.com/wiki/Reflective_Materials
http://developer.valvesoftware.com/wiki/Creating_Normal_Maps
Quick guide to masking a reflection on a normal map:
1)The targa file that is the _normal file must be normal mapped first.
2)After the normal map, copy/paste the image of the normal map to another canvas of the same dimensions. (duplicate option) The copied version must now be in greyscale. White is full reflection, black is no reflection.
3)Now after you screw with the white/black, go back to the original normal map. Create a new channel (alpha)
4)Now go to the greyscale. Leave it in grey mode and ctrl A (select all) paste it to the alpha channel of the normal map.
5)Save in 32 bit format, compile it to a VTF.
6)Now, the regular VTF (not the _normal version) must have a vmt. In the vmt make sure to put "$normalmapalphaenvmapmask" 1 along with all the other parameters needed.
BIG helper!!!: Download the VMT plugin for photoshop to reopen compiled VMT's and screw with it until you get the handle of it. (instead of having to create more and more textures to test)
Download here: http://nemesis.thewavelength.net/index.php?p=39
VTF edit (GUI) is very good too: DL here:
http://nemesis.thewavelength.net/index.php?p=40