You might want to take a look at Ensembl BioMart. GO related data is in the "EXTERNAL" section. Select all GO options if you need all the details of the associated GO term. The data would be available gene wise, i.e. multiple lines per gene for every GO term associated with it. Do not forget to select the "Associated gene name" option from the "GENE" section for getting the gene symbol, else you will only get the Ensembl transcript and gene IDs along with the GO terms, which I
I would have ideally recommended DAVID knowledgebase but it seems that the knowledgebase link is not available currently for some reasons. Please do try and check it out some time if it is still available. It gives all the genes-GO data in every possible combination, sorted by Biological Process,Molecular Function and Cellular Component properly in separate files.
After obtaining the files, you can count the total number of genes associated with the desired GO terms using a simple script.
I tried Biomart and used ensembl gene 72 database followed by yeast database (i am working on cervesiae). then selected what you said and found 31 genes associated with the GO term which i was looking but i crossed check with amigo (GOC) and found that there are 255 genes annotated to the term. I do not know why there is a difference?