Reference genome file is a description of the genome sequence. And annotation file is a description of where genetic element(intron, exon) located in the genome, in the form begin and end coordinate. Reference genome file are mostly in .fasta format and annotation are mostly in .gff or .bed format. Another format .genbank sometime contain both reference and annotation information. Google each format for details.
For human, the best way to download that file is http://genome.ucsc.edu/. You can also download it from ncbi.