According to central limit theorem, t-test can be used for non-normally distributed sample. Beside, RNA-Seq fits better to negative binomial distribution which doesn't significantly differ from normal distribution. So why can't we just use t-test for DE estimation?
I have a feeling you're actually after the answer to a slightly different question, which could be along the lines of: "Why do we need a statistical package at all to process RNA-seq count data?"
As genomax and Devon have correctly pointed out: a t-test can be used in the realm of DE analysis, but you should absolutely, never ever apply it on the raw counts, no matter how many samples you have, because raw counts are never absolute measures of expression for a specific gene within a given sample. The actual number of reads per gene depends on the efficiency of the library prep including RNA extraction and cDNA synthesis and the amount of contamination from non-coding transcripts (e.g. rRNA, tRNA) and, of course, the actual sequencing depth, i.e. the number of reads per sample, also strongly influences the final value. All of these issues need to be taken into account before any statistical test, and this is where the packages have contributed a lot, too -- in addition to establishing ways of estimating variances from as little as 2-3 replicates per condition.
You can use a T-test, that's what limma is doing (though after passing counts through
voom and then using some empirical bayes methods). The reason no one uses a simple T-test for RNA-seq is the same reason no one did it for array data, namely that you rarely have sufficient sample numbers to accurately estimate variance without pooling information across genes (see the original limma paper for a nice discussion of this).
As a trivial example, a t-test comparing (1,2,3) to (4,5,6) returns the same answer as one comparing (10,20,30) to (40,50,60) But if those are raw counts, you can't say that the likelihood of the two groups being different is the same between the two sets, because the lower count one is so much more prone to be way off due to sampling errors.