sng problem, fbc, gcc

General FreeBASIC programming questions.
Post Reply
dafhi
Posts: 1641
Joined: Jun 04, 2005 9:51

sng problem, fbc, gcc

Post by dafhi »

num / (num+1) supposed to be less than 1

on my machine i am seeing 1

Code: Select all

dim as ulong a = 4294967282
dim as single s = a / 2^32
? s, a; " "; culng(-1)
MrSwiss
Posts: 3910
Joined: Jun 02, 2013 9:27
Location: Switzerland

Re: sng problem, fbc, gcc

Post by MrSwiss »

dafhi wrote:num / (num+1) supposed to be less than 1 ... on my machine i am seeing 1
That result is OK, since Single doesn't posses the necessary amount of precision, for such small differences.
Use a Double instead ...
dafhi
Posts: 1641
Joined: Jun 04, 2005 9:51

Re: sng problem, fbc, gcc

Post by dafhi »

okay. this seems to be the way to fix it for sng

[updated]

Code: Select all

dim as single s = a / (2^32 + 128)
Post Reply