Rounding of Single

albert · Post by **albert** » Dec 26, 2019 22:23

@Coders

Problem with single precision numbers... the rounding of .5

( where a .5 rounds down to 0 , and a 7.5 rounds up to 8 )

In some cases a .5 rounds up .5 and sometimes it rounds down .5

Could you guys fix the rounding bug???
Either make it so it always rounds up , or always rounds down...

Code: Select all


screen 19

dim as single s1
dim as longint v1

for a as longint = 1 to 20
    
    s1 = a / 2
    
    v1 = s1
    
    print a , a / 2 , s1 , v1
    
next

sleep
end

jj2007 · Post by **jj2007** » Dec 26, 2019 22:59

You want to see something like this:

Code: Select all

1       0.5000000000000000001   1
2       1.000000000000000000    1
3       1.500000000000000000    2
4       2.000000000000000000    2
5       2.500000000000000000    3
6       3.000000000000000000    3
7       3.500000000000000000    4
8       4.000000000000000000    4
9       4.500000000000000000    5
10      5.000000000000000000    5
11      5.500000000000000000    6
12      6.000000000000000000    6
13      6.500000000000000000    7
14      7.000000000000000000    7
15      7.500000000000000000    8
16      8.000000000000000001    8
17      8.500000000000000001    9
18      9.000000000000000001    9
19      9.500000000000000001    10
20      10.000000000000000001   10

The problem is the inherent imprecision of floating point numbers. It's not a bug. If you really need this result, use the following but be aware of the problems:

Code: Select all

screen 19

dim as single s1
dim as longint v1
dim as double two=1.9999998

for a as longint = 1 to 20
   
    s1 = a / two
   
    v1 = s1
   
    print a , a / 2 , s1 , v1
   
next

sleep
end

Stonemonkey · Post by **Stonemonkey** » Dec 27, 2019 0:35

No, not a bug, it's rounding to nearest but in the case of x.5 the nearest is neither round up or down so it rounds to even. It's possible to change the FPU rounding mode on x86 but I'm not sure if FB on x64 makes use of the FPU or SSE units for floating point ops, I'll have a look to see what's going on when I get home tomorrow.

jj2007 · Post by **jj2007** » Dec 27, 2019 8:44

Interesting what it does in 32-bit mode:

Code: Select all

dim as single s1
dim as longint v1, v2
dim as double two=1.9999998

for a as longint = 1 to 20
   asm int 3
    s1 = a / 2
   asm nop
    v1 = s1
   asm nop
   v2=int(a/2)
   asm nop
   asm nop
   asm nop

    print a , a / 2 , s1 , v1, v2
   
next

sleep
end

Code: Select all

 1             0.5           0.5           0             0
 2             1             1             1             1
 3             1.5           1.5           2             1
 4             2             2             2             2
 5             2.5           2.5           2             2
 6             3             3             3             3
 7             3.5           3.5           4             3
 8             4             4             4             4
 9             4.5           4.5           4             4
 10            5             5             5             5
 11            5.5           5.5           6             5
 12            6             6             6             6
 13            6.5           6.5           6             6
 14            7             7             7             7
 15            7.5           7.5           8             7
 16            8             8             8             8
 17            8.5           8.5           8             8
 18            9             9             9             9
 19            9.5           9.5           10            9
 20            10            10            10            10

Under the hood:

Code: Select all

Address     Hex dump               Command                             Comments
004015F9    ³> ÚCC                 Úint3                               ; ³
004015FA    ³. ³DF6D E0            ³fild qword ptr [ebp-20]            ; ³s=a/2
004015FD    ³. ³DC35 10604100      ³fdiv qword ptr [416010]            ; ³float 2.000000000000000
00401603    ³. ³D95D F8            ³fstp dword ptr [ebp-8]             ; ³
00401606    ³. ³90                 ³nop                                ; ³
00401607    ³. ³D945 F8            ³fld dword ptr [ebp-8]              ; ³v1=s1
0040160A    ³. ³DF7D F0            ³fistp qword ptr [ebp-10]           ; ³
0040160D    ³. ³90                 ³nop                                ; ³
0040160E    ³. ³DF6D E0            ³fild qword ptr [ebp-20]            ; ³v1=int(a/2)
00401611    ³. ³DC35 10604100      ³fdiv qword ptr [416010]            ; ³float 2.000000000000000
00401617    ³. ³83EC 04            ³sub esp, 4                         ; ³
0040161A    ³. ³D93C24             ³fstcw [esp]                        ; ³save current flags
0040161D    ³. ³8B0424             ³mov eax, [esp]                     ; ³get current flags
00401620    ³. ³25 FFF30000        ³and eax, 0000F3FF                  ; ³modify to get
00401625    ³. ³0D 00040000        ³or eax, 00000400                   ; ³DOWN mode
0040162A    ³. ³50                 ³push eax                           ; ³on stack
0040162B    ³. ³D92C24             ³fldcw [esp]                        ; ³load flags: set FPU to DOWN 64
0040162E    ³. ³83C4 04            ³add esp, 4                         ; ³
00401631    ³. ³D9FC               ³frndint                            ; ³do the rounding
00401633    ³. ³D92C24             ³fldcw [esp]                        ; ³restore flags
00401636    ³. ³83C4 04            ³add esp, 4                         ; ³
00401639    ³. ³DF7D F0            ³fistp qword ptr [ebp-10]           ; ³save the value

Setting DOWN 64 looks incorrect to me.

Same but in 64-bit mode:

Code: Select all

64-bit:
0000000000405A1D   | CC                        | int3                                  |
0000000000405A1E   | F2 41 0F 59 F1            | mulsd xmm6,xmm9                       |
0000000000405A23   | 66 0F EF D2               | pxor xmm2,xmm2                        |
0000000000405A27   | F2 0F 5A D6               | cvtsd2ss xmm2,xmm6                    |
0000000000405A2B   | F3 0F 11 54 24 2C         | movss dword ptr ss:[rsp+2C],xmm2      |
0000000000405A31   | 90                        | nop                                   |
0000000000405A32   | F3 0F 10 44 24 2C         | movss xmm0,dword ptr ss:[rsp+2C]      |
0000000000405A38   | E8 23 FE FF FF            | call <sub_405860>                     |
0000000000405A3D   | 44 0F 28 C0               | movaps xmm8,xmm0                      |
0000000000405A41   | 90                        | nop                                   |
0000000000405A42   | 90                        | nop                                   |
0000000000405A43   | 90                        | nop                                   |
0000000000405A44   | 90                        | nop                                   |
...
0000000000405860 < | F3 0F 11 44 24 F4         | movss dword ptr ss:[rsp-C],xmm0       |
0000000000405866   | D9 44 24 F4               | fld dword ptr ss:[rsp-C]              |
000000000040586A   | 50                        | push rax                              |
000000000040586B   | 51                        | push rcx                              |
000000000040586C   | D9 3C 24                  | fnstcw word ptr ss:[rsp]              | save current flags
000000000040586F   | 48 8B 04 24               | mov rax,qword ptr ss:[rsp]            | [rsp]:sub_4059A0+9D
0000000000405873   | 48 83 C8 20               | or rax,20                             |
0000000000405877   | 48 89 44 24 08            | mov qword ptr ss:[rsp+8],rax          |
000000000040587C   | D9 6C 24 08               | fldcw word ptr ss:[rsp+8]             |
0000000000405880   | D9 FC                     | frndint                               |
0000000000405882   | 9B                        | wait                                  |
0000000000405883   | DB E2                     | fnclex                                |
0000000000405885   | D9 2C 24                  | fldcw word ptr ss:[rsp]               |
0000000000405888   | 59                        | pop rcx                               |
0000000000405889   | 58                        | pop rax                               |
000000000040588A   | D9 5C 24 F4               | fstp dword ptr ss:[rsp-C]             |
000000000040588E   | F3 0F 10 44 24 F4         | movss xmm0,dword ptr ss:[rsp-C]       |
0000000000405894   | C3                        | ret                                   |

Surprise, surprise: It does use the FPU, though in a hilariously complicated and inefficient way!

dodicat · Post by **dodicat** » Dec 29, 2019 20:38

You could try changing the rounding mode by fpu settings this way:

Code: Select all


enum
    default
    rounddown
    roundup
    truncicate
end enum
sub setFPU(byval mode as integer=0)
    if mode<0 or mode>3 then exit sub
    dim as integer n:asm fstcw  word ptr[n]
   mode=895+mode shl 10:asm fldcw word ptr[mode] 
end sub

setfpu(rounddown)

screen 19

dim as single s1
dim as longint v1

for a as longint = 1 to 20
   
    s1 = a / 2
   
    v1 = s1
   
    print a , a / 2 , s1 , v1
   
next

sleep
end

Only tested on windows.

SARG · Post by **SARG** » Dec 29, 2019 22:34

@jj2007

This is the code generated by gas64. What do you think about it ?
#Ox are tags put by the optimization process.
I have just added '#====' to separate the interesting parts and commented 'int3' to get an executable.

Code: Select all

.intel_syntax noprefix
.file "test.bas"
.section .text

   .text
   .globl main
main:
   push rbp
   mov  rbp,rsp
   sub rsp, 192
   call __main
   mov DWORD PTR 16[rbp], ecx
   mov QWORD PTR 24[rbp], rdx
#O4lea r11, -116[rbp]
   #O4mov DWORD PTR [r11], 0
   mov DWORD PTR -116[rbp], 0 #Optim 4
   mov ecx, 16[rbp]
   mov rdx, 24[rbp]
   xor r8d, r8d
   call fb_Init
   .Lt_0002:
#O4lea r11, -120[rbp]
   #O4mov DWORD PTR [r11], 0
   mov DWORD PTR -120[rbp], 0 #Optim 4
#O4lea r11, -128[rbp]
   #O4mov QWORD PTR [r11], 0
   mov QWORD PTR -128[rbp], 0 #Optim 4
#O4lea r11, -136[rbp]
   #O4mov QWORD PTR [r11], 0
   mov QWORD PTR -136[rbp], 0 #Optim 4
   mov rax, 0x3FFFFFFFCA501ACB # DBL=1.9999998
   mov QWORD PTR -144[rbp], rax
   mov QWORD PTR -152[rbp], 1
   .Lt_0007:
   # int 3 
   #===============================
   pxor xmm0,xmm0
   cvtsi2sd xmm0, QWORD PTR -152[rbp]
   movq r11, xmm0
   #O1 movq xmm0, r11
   mov rax, 0x4000000000000000 # DBL=2
   movq xmm1, rax
   divsd xmm0, xmm1
#O3movq r10, xmm0
   #O3movq xmm1, r10
   movq xmm1, xmm0 #Optim 3
   cvtsd2ss xmm0, xmm1
#O3movd r11d, xmm0
   #O3mov -120[rbp], r11d
   movss -120[rbp], xmm0 #Optim 3
   #================================
   nop 
   #=================================
   movss xmm0, -120[rbp]
   roundss xmm0,xmm0,4
   cvttss2si rax, xmm0
#O2mov r11, rax
   #O2mov -128[rbp], r11
   mov -128[rbp], rax #Optim 2
   #==================================
   nop
   #==================================
   pxor xmm0,xmm0
   cvtsi2sd xmm0, QWORD PTR -152[rbp]
   movq r11, xmm0
   #O1 movq xmm0, r11
   mov rax, 0x4000000000000000 # DBL=2
   movq xmm1, rax
   divsd xmm0, xmm1
   movq r10, xmm0
   #O1 movq xmm0, r10
   call floor
   movq r10, xmm0
   #O1 movq xmm0, r10
   roundsd xmm0,xmm0,4
   cvttsd2si rax, xmm0
#O2mov r11, rax
   #O2mov -136[rbp], r11
   mov -136[rbp], rax #Optim 2
   #===============================
   nop
   nop
   nop
   xor ecx, ecx
   mov rdx, -152[rbp]
   mov r8d, 2
   call fb_PrintLongint
   pxor xmm0,xmm0
   cvtsi2sd xmm0, QWORD PTR -152[rbp]
   movq r11, xmm0
   #O1 movq xmm0, r11
   mov rax, 0x4000000000000000 # DBL=2
   movq xmm1, rax
   divsd xmm0, xmm1
   movq r10, xmm0
   xor ecx, ecx
   movq xmm1, r10
   mov r8d, 2
   call fb_PrintDouble
   xor ecx, ecx
   movss xmm1, -120[rbp]
   mov r8d, 2
   call fb_PrintSingle
   xor ecx, ecx
   mov rdx, -128[rbp]
   mov r8d, 2
   call fb_PrintLongint
   xor ecx, ecx
   mov rdx, -136[rbp]
   mov r8d, 1
   call fb_PrintLongint
   .Lt_0005:
   inc QWORD PTR -152[rbp]
   .Lt_0004:
   cmp QWORD PTR -152[rbp], 20
   jle .Lt_0007
   .Lt_0006:
   mov ecx, -1
   call fb_Sleep
   xor ecx, ecx
   call fb_End
   .Lt_0003:
   xor ecx, ecx
   call fb_End
   mov eax, -116[rbp]

   mov rsp,rbp
   pop rbp
   ret

jj2007 · Post by **jj2007** » Dec 30, 2019 19:22

SARG wrote:This is the code generated by gas64. What do you think about it ?

I think it looks too complicated for my taste ;-)
Take a FB example and insert asm int 3 before the line that does the rounding. With Just-In-Time (JIT) debugging, you can step through the relevant instructions and see where it commits errors.

jj2007 · Post by **jj2007** » Dec 30, 2019 19:26

dodicat wrote:You could try changing the rounding mode by fpu settings this way:

setfpu(roundup) works, surprisingly :thumbsup:

Rounding of Single

Rounding of Single

Re: Rounding of Single

Re: Rounding of Single

Re: Rounding of Single

Re: Rounding of Single

Re: Rounding of Single

Re: Rounding of Single

Re: Rounding of Single