You can do for example:
draw.pairwise.venn(100,50, 20, cex=0)
This gives a letter size of 0, the letters won't appear. You can also do label.col="white" if your background is white.
Alternatively, you can save the diagram as an SVG then delete the number boxes using Inkscape or Photoshop.