fbcunit - fbc compiler unit testing component

User projects written in or related to FreeBASIC.
dodicat
Posts: 7976
Joined: Jan 10, 2006 20:30
Location: Scotland

Re: fbcunit - fbc compiler unit testing component

Post by dodicat »

I took off my post, I was being unkind, complaining about the use of hex.
I can re-post the code if requested.
Anyway I have been trying out the nextafter (CRT) function to get the "graininess" of double.
this is related to this thread subject matter(as in twice removed perhaps)
it doesn't seem to work properly.
It only returns properly if I use a double and nextafterf.
I hit the three score and ten this year with any luck.

Code: Select all

#include "crt/math.bi"

dim as double d

d= nextafter(.1,100)
print d


d= nextafterf(.1,100)
print d
sleep

   
 
srvaldez
Posts: 3373
Joined: Sep 25, 2005 21:54

Re: fbcunit - fbc compiler unit testing component

Post by srvaldez »

@dodicat
you are right, nextafter seems to have no effect, however this C version does show a tiny difference if you print enough decimals

Code: Select all

#include <stdio.h>
#include <math.h>

int main(void)
{
	double d=0.1;
	printf ("%22.17f\n", d);
	d=nextafter(d,100);
	printf ("%22.17f\n", d);
	d=nextafterf(d,100);
	printf ("%22.17f\n", d);
	return 0;
}

Code: Select all

   0.10000000000000001
   0.10000000000000002
   0.10000000894069672
 
srvaldez
Posts: 3373
Joined: Sep 25, 2005 21:54

Re: fbcunit - fbc compiler unit testing component

Post by srvaldez »

try this

Code: Select all

#include "crt/math.bi"

dim as double d=.1
print "d before    = ";hex(peek(longint,@d))
d= nextafter(d,100.0)
print "d nextafter = ";hex(peek(longint,@d))

Code: Select all

d before    = 3FB999999999999A
d nextafter = 3FB999999999999B
or binary

Code: Select all

d before    = 11111110111001100110011001100110011001100110011001100110011010
d nextafter = 11111110111001100110011001100110011001100110011001100110011011
dodicat
Posts: 7976
Joined: Jan 10, 2006 20:30
Location: Scotland

Re: fbcunit - fbc compiler unit testing component

Post by dodicat »

Thanks srvaldez.
I have tried casting doubles to ulongint (not using nextafter) to get a granularity of sorts

Code: Select all



function isequal(d1 as double,d2 as double) as boolean
    dim as double d3=abs(d1-d2)
    dim as long p=-2*log(d3)/log(2)
    return abs(*(cast(ulongint ptr,@d1))-*(cast(ulongint ptr,@d2)))<p
end function

dim as double d
for n as long=1 to 20
    d+=.1
    var f=n/10
    print f;tab(30);d;tab(60);isequal(f,d)
next
print
print
d=-5
for n as double=-50 to 50 step 1
    dim as double f=n/10.00000000
    print f;tab(30);d;tab(60);isequal(f,d)
     d+=.1
next
print
 
try altering the last place of /10.00000000 to /10.00000001
(BTW it fails at zero)
I'll work on your suggestion of casting nextafter.

The idea of course is to create an isequal function for doubles without using tiny epsilons.
(a subset of this thread, not too far off topic I hope.)
coderJeff
Site Admin
Posts: 4313
Joined: Nov 04, 2005 14:23
Location: Ontario, Canada
Contact:

Re: fbcunit - fbc compiler unit testing component

Post by coderJeff »

@dodicat, heh, your subtle and odd unkindness was lost on me, sorry. ;) Using hex is a convenient way to specify the bit patterns to test the float's bit fields independent of the FP-CPU and math libraries. And by looking at the output in hex/bin, no information is lost in the float->decimal conversion.

As TeeEmCee points out, we are bound to see inconsistencies across platforms and implementations of math functions.

The test-suite is the developer's safety net and floating point math is a significant component of the compiler's capability. I think it's very worthwhile to explore improving the math tests, and understand the failures; where and why the accuracy or precision breaks down. It all looks on-topic to me.
coderJeff
Site Admin
Posts: 4313
Joined: Nov 04, 2005 14:23
Location: Ontario, Canada
Contact:

Re: fbcunit - fbc compiler unit testing component

Post by coderJeff »

I have expanded fbcunit API assertions to include:
CU_ASSERT_SINGLE_EXACT( a, b )
CU_ASSERT_SINGLE_EQUAL( a, b, epsilon )
CU_ASSERT_SINGLE_APPROX( a, b, ulps )
CU_ASSERT_DOUBLE_EXACT( a, b )
CU_ASSERT_DOUBLE_EQUAL( a, b, epsilon )
CU_ASSERT_DOUBLE_APPROX( a, b, ulps )

This is the (simple) floating point test file from fbcunit itself: fbcu_float.bas. Gives an idea how the assertions are used if nothing else.

I have patched it in to my local fbc repo only for now, to review tests in the fbc test-suite and run on different versions of the compiler.
counting_pine
Site Admin
Posts: 6323
Joined: Jul 05, 2005 17:32
Location: Manchester, Lancs

Re: fbcunit - fbc compiler unit testing component

Post by counting_pine »

Sorry to only be watching from afar, but this looks like great work!
The only thing that occurs to me is that the names sngULP/dblULP confused me a little, because I thought that they would calculate the actual "unit in last place", which is (usually) the value of 'x - cvs( cvl(x)-1) )'
coderJeff
Site Admin
Posts: 4313
Joined: Nov 04, 2005 14:23
Location: Ontario, Canada
Contact:

Re: fbcunit - fbc compiler unit testing component

Post by coderJeff »

Thanks. yea, I see, how the naming of the function is misleading, so here's what I did (and pushed changes to my public repo):
- renamed sngULP(a,b)/dblULP(a,b) to sngULPdiff(a,b)/dblULPdiff(a,b) - calculate difference in ULPs
- added sngIsInf(a) & dblIsInf(a) - check for infinity
- added sngULP(a) & dblULP(a) - calculate the ULP for a single & double (using a variant of the formula you posted, thank-you!)

sngULP(a) & dblULP(a) also fails (qNaN) for zero, like dodicat's. I think he has the correct approach though, to attempt an automatic selection of a tolerable error difference.

I have also started rewriting the tests to use the new SUITE() & TEST() framework, I've got as far as the pretest, boolean, & comments directory, then got a little sidetracked finding tests that should be added and investigating related bugs on sf.net. I'm thinking maybe I should create a 'fbc-tests' branch in the repo to push progressive updates.
coderJeff
Site Admin
Posts: 4313
Joined: Nov 04, 2005 14:23
Location: Ontario, Canada
Contact:

Re: fbcunit - fbc compiler unit testing component

Post by coderJeff »

I've completed rewriting about a third of the tests. Efforts are pushed to https://github.com/jayrm/fbc/tree/fbc-tests

Attempted to provide a consistent style for tests using SUITE()/TEST() from fbcunit. Without making major changes, there appear to be 4 styles for writing tests, using straightforward translation:
pretest/style_simple.bas - common
pretest/style_namespace.bas - common
pretest/style_module.bas - only if needed
pretest/style_direct.bas - was only way until now

- found 2 cases so far where the test was written, but not added, and catches bugs in current compiler. An understandable oversight given how the tests had to be written before
- Tried to name things well, what wasn't always done before. The name you get when running fbc-tests is how you should find it in the test suite.
- Added ENABLE_CHECK_BUGS=1 makefile option. This allows adding and checking known bugs in the test-suite; but only when enabled. Works for unit-tests and log-tests.
- Random thought: if fbc was to be built for a platform where module constructors don't work, what the current test-suite depends on, at least should be able to search for SUITE() and TEST() in the files to build a list of procedures that could be invoked another way.
- might pass on a few files and come back to them later, my eyes start to bug out after 2 days looking at 1000's of uniquely named things...

Hopefully, no more than 2 weekends to complete, then can get back to the floating point stuff...
coderJeff
Site Admin
Posts: 4313
Joined: Nov 04, 2005 14:23
Location: Ontario, Canada
Contact:

Re: fbcunit - fbc compiler unit testing component

Post by coderJeff »

kind of got side tracked away from the floating point stuff, but that's OK, I think recent work is worthwhile also.

I have a pull request cooking: fbc-tests Test Suite update Mar 2018

If anyone, it would affect St_W; I don't think many users ever really look at the test suite -- which is huge btw.

Here's a few of my thoughts, some related to this thread, some not.

What has been working well for me, is to get development from one usable state to the next usable state (with improvements), and having the calendar month be the time frame. It's not perfect, and seemingly never quite finished, however, seems like reasonable time frame to go from one working thing to the next working thing even if some work is going to be left for next update.

For the benefit of those that are not yet following the FreeBASIC project on github (you can create a login, follow projects, and subscribe to get updates sent to your email), the current issue is: I am confident the changes I have staged are good, but Travis-CI, whose purpose is to validate the changes, is not working correctly. Travis-CI is great; I love it. But what to do when it is not working? The "old way" was for a developer to just push changes to the main trunk and let others report the problems. aw, I was just having thoughts about "what did we do before Travis-CI?", well, we just had confidence that our changes were good, and if they weren't, expect other users to help report the problems.
dkl
Site Admin
Posts: 3235
Joined: Jul 28, 2005 14:45
Location: Germany

Re: fbcunit - fbc compiler unit testing component

Post by dkl »

We should merge the changes anyways, because what else can you do when Travis randomly starts failing. I mean, it would most likely fail to build the current master too, with or without the changes. For fbc, I don't think Travis builds passing or reviews on every change can be made mandatory, due to the lack of manpower.
St_W
Posts: 1618
Joined: Feb 11, 2009 14:24
Location: Austria
Contact:

Re: fbcunit - fbc compiler unit testing component

Post by St_W »

I've had a quick look at the test results after I saw your PR had been merged and noted one little point that could be improved: The test(-suite) naming is not fully consistent anymore, e.g. there are "string", "string_" (with a trailing underscore) and "strings". Would it be possible to align those different test-suite names again to a single one?
Otherwise the tests are working fine without any adaptations of the build scripts, which is really great :-)
I'll definitely look into your changes in more detail at a later point, but haven't been able to do so yet.
coderJeff
Site Admin
Posts: 4313
Joined: Nov 04, 2005 14:23
Location: Ontario, Canada
Contact:

Re: fbcunit - fbc compiler unit testing component

Post by coderJeff »

The intent is that the suite/test name = directory/file name, and only suffixing '_' character if there is a keyword collision . Sorry, after 40-50 hours of working on this part, I needed a break from the thing. It will be in the works for the next update. I like the idea suggested by dkl, deriving suite names from filenames. I'm not sure what form it will take to pick up the directory name. So, even if it's not an automatic thing for the next update, I'll make sure it is consistent/unified/aligned in the next update. Thanks, St_W.
St_W
Posts: 1618
Joined: Feb 11, 2009 14:24
Location: Austria
Contact:

Re: fbcunit - fbc compiler unit testing component

Post by St_W »

coderJeff wrote:The intent is that the suite/test name = directory/file name, and only suffixing '_' character if there is a keyword collision . Sorry, after 40-50 hours of working on this part, I needed a break from the thing. It will be in the works for the next update. I like the idea suggested by dkl, deriving suite names from filenames. I'm not sure what form it will take to pick up the directory name. So, even if it's not an automatic thing for the next update, I'll make sure it is consistent/unified/aligned in the next update.
I saw that you're using the suite/test name as namespace now, which imposes restrictions on the naming, of course. Consistent naming is just a cosmetic issue - so it's "nice to have", but not urgent in any way. A quick solution would be to just trim trailing underscores in the (XML) output, but as you've implemented the whole thing you probably know best what a good solution could look like :-)
Thank you for your great work.
coderJeff
Site Admin
Posts: 4313
Joined: Nov 04, 2005 14:23
Location: Ontario, Canada
Contact:

Re: fbcunit - fbc compiler unit testing component

Post by coderJeff »

Trimming the underscore would be no problem. You are the only consumer of this right now, so not concerned about anyone else.

EDIT:

Created a pull request #75 to update suite names. I normalized all the suite names manually. To compare suite names to file names I used grep & sed script (it's not pretty!).

Code: Select all

grep -r -n --include=*.bas --exclude-dir=fbcunit -e '.*SUITE.*(.*)$' . > names.lst 
sed names.lst  -e 's/\.\/\([-A-Za-z0-9_]*\)\/\([-A-Za-z0-9_]*\)\.bas:[0-9]*:SUITE([ ]*fbc_tests\.\([A-Za-z0-9_]*\)\.\([A-Za-z0-9_]*\)[ ]*)/\1:\2:\3:\4:\0/' > names2.lst
sed names2.lst -e 's/-/_/g' > names3.lst
sed names3.lst -e 's/\([A-Za-z0-9_]*\):\([A-Za-z0-9_]*\):\1_*:\2_*:\(.*\)//' > names4.lst
sed names4.lst -n -e '/^..*$/p'
I thought of several ideas for deriving the suite names automatically from filenames. They seem either too convoluted, or we need to add features to fbc. The challenge is making the compiler aware of the directory name.
1) makefile defines global with name of directory
2) an include file in each directory contains a define with the name of the directory
3) string processing macros added to fbc's preprocessor to extract directory name

St_W,
I also modified the xml output to trim the trailing underscores only. Here is a sample, with a simulated test case failure to show the xml before and after:
BEFORE:

Code: Select all

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
    <testsuite name="fbc_tests.wstring_.woct_" errors="0" tests="4" failures="1" time="0">
        <testcase classname="fbc_tests.wstring_.woct_" name="byte_" />
        <testcase classname="fbc_tests.wstring_.woct_" name="short_" />
        <testcase classname="fbc_tests.wstring_.woct_" name="integer_" />
        <testcase classname="fbc_tests.wstring_.woct_" name="long_">
            <failure message="" type="Failure">
                Condition : CU_ASSERT(valulng( "&o" + woct( n ) ) =TEST_VAL-1)
                File      : wstring\woct.bas
                Line      : 12
            </failure>
        </testcase>
    </testsuite>
</testsuites>
AFTER:

Code: Select all

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
    <testsuite name="fbc_tests.wstring.woct" errors="0" tests="4" failures="1" time="0">
        <testcase classname="fbc_tests.wstring.woct" name="byte" />
        <testcase classname="fbc_tests.wstring.woct" name="short" />
        <testcase classname="fbc_tests.wstring.woct" name="integer" />
        <testcase classname="fbc_tests.wstring.woct" name="long">
            <failure message="" type="Failure">
                Condition : CU_ASSERT(valulng( "&o" + woct( n ) ) =TEST_VAL-1)
                File      : wstring\woct.bas
                Line      : 12
            </failure>
        </testcase>
    </testsuite>
</testsuites>
Post Reply