Sample Header Ad - 728x90

Data management pattern for temporarily storing/retrieving excessive quantities of data

0 votes
0 answers
37 views
I have been tasked with creating an application to ingest raw binary data from a software-defined radio (SDR) and display it in a web frontend. One of the purposes of this app is to allow a user to take a "capture" from a radio, allowing them to download a short historical period of raw data. I am struggling with the best way to store and process this data. The goal of this app is to be able to demonstrate frequencies well into the megahertz, requiring sample rates that could approach the gigahertz range. Given that a single complex sample requires eight bytes, the rate of ingest could easily surpass 1 GB/s. This will stretch the memory of my program. I believe it is far too great of a data rate to write to simply write to disk. I need some advice on database systems to manage this high-volume data pattern. The important part is that no data analysis is necessary (all data is strictly binary), and some delay in data recall can be tolerated. **Edit:** I realized that I should clarify better how I am thinking of storing the data. Here's the preconditions and my thoughts: * Binary data is received incrementally from the radio. * Data needs to be able to be queried in some way related to time. However, this does not *necessarily* mean that timestamps need to be stored. If there is little to no data loss, then the size of a required return payload can be deduced based on the number of samples requested (since each sample is known to be eight bytes). * Rather than exact timing, the important part is that data is stored in a continuous, non-lossy manner, to produce accurate waveforms. For example, supposing that data is stored in 1GB incremental records, the system can approximate how many records are needed for one request, and then trim the data server-side. * The data can be regularly flushed once a certain capacity is reached, if this is beneficial to performance. Captures need not be more than 10GB each. * Data will be associated relationally with a radio, whether this be through a table-based structure or simply an ID. (this is the easy part)
Asked by moltarze (101 rep)
May 21, 2024, 06:12 PM
Last activity: May 22, 2024, 03:56 PM