Sample Header Ad - 728x90

Losing precision with dc

13 votes
3 answers
2258 views
I want to use dc to handle some base 16 numbers with hexadecimal points but I'm running into precision problems. For example, below I'm multiplying F423F.FD by 100, both hex. The expected answer is F423FFD, instead it is giving F423FFA.E1, close but not accurate enough even after rounding. $ dc 16 d i o F423F.FD 100 * p F423FFA.E1 I read that dc was an unlimited precision calculator, and this is not a large number by any means. Is there something I'm doing wrong? Thank for your answers. Given the issues with dc, I bit the bullet and wrote my own parser for real numbers in other bases. If anyone is interested in the code, I can post it here.
Asked by Yimin Rong (953 rep)
Jul 17, 2018, 04:26 PM
Last activity: Jul 19, 2018, 07:35 PM