How to count the number of bytes in a very large file, grouping the same bytes?
1
vote
4
answers
1569
views
I am searching for a way to get a statistics on a very large (multiple times larger than the available RAM) the outputs what byte values in the files are present and how often:
A0 01 00 FF 77 01 77 01 A0
I need to know how many A0 bytes there are in this file, how many 01, and so on. The result could be:
A0: 2
01: 3
00: 1
FF: 1
77: 2
Therefore this question is very close to the question [How to count the number of bytes in a file, grouping the same bytes?](https://unix.stackexchange.com/questions/527521/how-to-count-the-number-of-bytes-in-a-file-grouping-the-same-bytes) but non of the existing answers works for larger files. From my understanding all answers require a minimum RAM equal to the size of the file to be tested (up to multiple times).
Hence the answers don't work on systems with small RAM, e.g. a Raspberry for processing a multi-GB file.
Is there a simple solution that works on any file size even if we have for example only 512MB RAM available?
Asked by Robert
(163 rep)
Mar 30, 2020, 12:31 PM
Last activity: Mar 31, 2020, 12:51 AM
Last activity: Mar 31, 2020, 12:51 AM