Your stated goal:
I want to find the sweet spot where my reads from species A align to the index at a very high rate, but reads from species B do not
do not match the method you want to use:
I want to decrease the stringency of bowtie2 when mapping. How can I increase the overall alignment rate?
If you want reads from the "correct" species to map, but reads from the "incorrect" species to not map, you want to increase alignment stringency. For that, an easy way is to use the
--end-to-end --very-fast settings.
Alternatively, there is a plethora of parameters to adjust, such as
-R (Effort options),
-i and other options (Alignment options), and several alignment scoring options, such as
--score-min (see Scoring options and Setting function options). The settings found for one pair of species wouldn't necessarily work for other species, and if the species are very close, it may be impossible to separate a large proportion of reads.
Do you have a reference genome for species B?