Repeatmasking will not remove genes to my knowledge. For that, you'll need to reannotate the masked file (or did you do this?).
Generally, repeat masking will annotate repetitive DNA regions, and either soft-mask (replace with lower case letters by some conventions, eg atgc instead of ATGC) or hard-mask (replace ATGC with NNNN).
Try running Gmap with a transcript set on your original and hard-masked outputs if you want to do a comparison. I wouldn't expect the number of genes/transcripts to change much, since the intergenic DNA is more likely to be repeat-masked.