I have to store some amount of data for analytical purposes.
- The data source produces 2TB data per month.
- Data is collected on a monthly basis (not real-time).
- Data is fully structured.
- There are 100+ different columns of data.
- Availability of SQL is important.
- Engineer/developer resources are limited.
I planned to use Postgres (probably with column-oriented extension), however, it would not be feasible for such data amounts (more than 20TB per year). I also made a research on Hadoop/Spark, however, it looks like a bit massive solution (considering, that the data is fully structured). I don't consider cloud-based solutions, as well as expensive ones (preferably, free-licence) .
Would you be so kind to suggest, which data storage to use for big amounts of structured data for analytical purposes?
Asked by Leeloo
(111 rep)
Dec 24, 2020, 12:23 PM
Last activity: Dec 24, 2020, 02:18 PM
Last activity: Dec 24, 2020, 02:18 PM