I want to use
dc
to handle some base 16 numbers with hexadecimal points but I'm running into precision problems. For example, below I'm multiplying F423F.FD
by 100
, both hex. The expected answer is F423FFD
, instead it is giving F423FFA.E1
, close but not accurate enough even after rounding.
$ dc
16 d i o F423F.FD 100 * p
F423FFA.E1
I read that dc
was an unlimited precision calculator, and this is not a large number by any means. Is there something I'm doing wrong?
Thank for your answers. Given the issues with dc
, I bit the bullet and wrote my own parser for real numbers in other bases. If anyone is interested in the code, I can post it here.
Asked by Yimin Rong
(953 rep)
Jul 17, 2018, 04:26 PM
Last activity: Jul 19, 2018, 07:35 PM
Last activity: Jul 19, 2018, 07:35 PM