Question

Working with big network data

0

Entering edit mode

7.9 years ago

moranr ▴ 290

Hi,

I have ~9.5 TB of data files. Each file contains a network in the format for Gephi. It is numerical data in ASCII format. I want to be able to work on compressed files to do some quantitative surveys on each file and some other analyses.
I would like to get the data down to below 4TB so it can be stored on a single HD at least. I would also like something that is fast , as I will be working with the data continuously. So far I have found lzop (deafult compression) to be the best. Anyone know anything better for this ? Or any advice for working with data like this ?

Thanks , R

networks Big Data • 1.6k views

ADD COMMENT • link updated 7.9 years ago by Giovanni M Dall'Olio 28k • written 7.9 years ago by moranr ▴ 290

score 3 · Accepted Answer · 2016-06-13

3

Entering edit mode

7.9 years ago

Giovanni M Dall'Olio 28k

A couple of suggestions:

remove any unnecessary metadata
use a graph database, e.g. neo4j (which by the way works quite well with Gephi)

ADD COMMENT • link 7.9 years ago by Giovanni M Dall'Olio 28k

0

Entering edit mode

Never knew about graph databases. Amazing, thank you.

ADD REPLY • link 7.9 years ago by moranr ▴ 290

1

Entering edit mode

One of the limitations of neo4j is that you only have one database per installation. The common practice for when you have multiple graphs is to put them all together, and have a flag or property to differentiate them.

ADD REPLY • link 7.9 years ago by Giovanni M Dall'Olio 28k

0

Entering edit mode

so just something simple like concatenate the node defs together and then the edgelists together and combine the two ? could a flag be adding an extra column to each network file with the file ID or something ? Or does neo4j have a way of setting flags ?

ADD REPLY • link 7.9 years ago by moranr ▴ 290