I have ~9.5 TB of data files. Each file contains a network in the format for Gephi. It is numerical data in ASCII format. I want to be able to work on compressed files to do some quantitative surveys on each file and some other analyses.
I would like to get the data down to below 4TB so it can be stored on a single HD at least. I would also like something that is fast , as I will be working with the data continuously. So far I have found lzop (deafult compression) to be the best. Anyone know anything better for this ? Or any advice for working with data like this ?
Thanks , R