fbcunit - fbc compiler unit testing component

[coderJeff]

I recently wrote 'fbcunit': a library for FreeBASIC - source is published at https://github.com/jayrm/fbcunit

It is intended as a replacement for fbc's current unit testing library which is dependent on libCUnit. I want to add it to fbc as a component. It works with the current fbc-tests test suite. I am **really* close to creating a pull request but I am stuck on licensing. Now, I could just put it in to fbc's repo, but licensing has been a challenge area for me before and I thought I would take the opportunity to work through some of the challenges. I think the license is important. It makes it very clear to user's what is allowed and/or expected.

On my git repo I have published the source code with explicit copyright - no license. For now, I claim ownership of the code. Next step is to add a license that is compatible with fbc. This is where I am stuck.

fbc's readme.txt, sounds like what I want
- GNU GPLv2 or later
- As a special exception, the copyright holders of this library give you permission to link this library ... etc.

For my fbcunit library, I want to follow fbc's runtime library license.
LGPL? Library or Lesser? Does it matter?
GNU LGPL version 2 or later?
GNU LGPL version 2.1 or later?

I have been reading lots of pages about (L)GPL but could really use some help to sort it out.

[Imortis]

I think FB's licence for the rtlib and fbgfx would be appropriate.

[Munair]

I ran into licensing issues too with the Unicode library. Part of it was ported from the Lazarus project. Part of it is my own. So I looked at the FreePascal Modified LGPL to see if it would conflict with FBC and FB libraries. There's a difference between the two. I settled for the MLGPL because I prefer the community to benefit from contributions that would enrich the FreeBasic project. If you're interested, I copied the MLGPL and it is now part of the BasicStudio project. The final paragraph in bold is key.

[coderJeff]

@Imortis, I agree.

@Munair, yes, that is same exception given for fbc's runtime, gfx, and multithreaded libraries.

More info:
Years ago, when I wrote fbcdoc I used GPL, and for libfbdoc I used LPGL. Those licenses made sense to me at the time. For some sources I turned the copyright over to FreeBASIC Development Team, and some sources are still copyrighted by me. I use libfbdoc in my weeide project, but can maintain it through fbc project because I am a developer. Anyway, hopefully what I learn here I can use to make good choices about copyrights and licenses for the parts of FreeBASIC I control and other code I might share.

What doesn't make sense to me is the use of LGPL, does it mean Library or Lesser and is the version 2 or 2.1 important. The fbc development tree references both, and sometimes not specifically as it use LGPL acronym only. I'm not sure if that is intentional or an oversight.

[Munair]

What doesn't make sense to me is the use of LGPL, does it mean Library or Lesser and is the version 2 or 2.1 important. The fbc development tree references both, and sometimes not specifically as it use LGPL acronym only. I'm not sure if that is intentional or an oversight.

History of LGPL:

The license was originally called the GNU Library General Public License and was first published in 1991, and adopted the version number 2 for parity with GPL version 2. The LGPL was revised in minor ways in the 2.1 point release, published in 1999, when it was renamed the GNU Lesser General Public License to reflect the FSF's position that not all libraries should use it. Version 3 of the LGPL was published in 2007 as a list of additional permissions applied to GPL version 3.

[St_W]

It is intended as a replacement for fbc's current unit testing library which is dependent on libCUnit. I want to add it to fbc as a component. It works with the current fbc-tests test suite. I am **really* close to creating a pull request but I am stuck on licensing.

First, I'd like to say that I like the idea to replace the buggy and old cunit library. I'm not really a friend of the GPL, but LGPL and especially LGPL with static linking exception (just like what FB's rtlib uses) is okay for sure.
Regarding the replacement of cunit: Currently I'm using the XML report generated by CUnit to generate a test report on the Jenkins CI server. After a quick look on your code I didn't find such a feature in your code. So I'd like to request the addition of that feature before you merge your changes and replace the use of the original cunit library - thank you!
CUnit supports two different log file formats: its own CUnit format and the JUnit format. The former one is unimportant and not supported by any other software I know and unfortunately the implementation of the latter is quite buggy. I fixed the most critical issues in my cunit fork. For the remaining issues and some additional transformations I'm using a XSL Transformation. You can find examples for the (XML-based) JUnit file structure in the repo https://bitbucket.org/st_w/fb-jenkins/src/ (in xsl-converters and cunit folders), but they are probably a bit outdated and e.g. still contain erroneous failure counters as produced by the original CUnit library.

[coderJeff]

So I'd like to request the addition of that feature before you merge your changes and replace the use of the original cunit library - thank you!

Can you give me a spec for the output you want? Looks like you want (ignoring closing tags):

Code: Select all

<testsuites>
  <testsuite errors="##1##" failures="##2##" tests="##3##" name="##4##">
    <testcase classname="##4##" name="##5##" time="##6##">
      <failure message="" type="Failure">
         Condition: ##7##
         File     : ##8##
         Line     : ##9##

##1## = number of errors? what is this?
##2## = failures ... is this number of tests failed? or number of asserts failed
##3## = number of tests in the suite
##4## = test suite name
##5## = test name
##6## = time? what is this?
##7## = copy of failed line
##8## = path/filename.bas
##9## = line number

The info available currently is: "suite name", "test name", count of asserts total, asserts passed, asserts failed. I didn't compute pass/fail totals per suite or test, but I can.

[St_W]

Can you give me a spec for the output you want? Looks like you want (ignoring closing tags):

Code: Select all

<testsuites>
  <testsuite errors="##1##" failures="##2##" tests="##3##" name="##4##">
    <testcase classname="##4##" name="##5##" time="##6##">
      <failure message="" type="Failure">
         Condition: ##7##
         File     : ##8##
         Line     : ##9##
[...]

The info available currently is: "suite name", "test name", count of asserts total, asserts passed, asserts failed. I didn't compute pass/fail totals per suite or test, but I can.

The primary goal should be to follow the JUnit format. Unfortunately no official standardized specification is available, but this one is very good: http://help.catchsoftware.com/display/ET/JUnit+Format
It contains a XML Schema description (XSD) explaining the structure and attributes in detail. Some may not be appropriate for FB and can just be removed (if they are optional) or set to a default value (if they are required). An error occurs e.g. when the test cannot be executed or ends unexpectedly (e.g. crashes / throws an exception; however I don't think that you've implemented any mechanism to handle that and thus it just will be zero always probably). "time" would be the execution time needed for a test, if available. The contents of the failure (or error) node(s) is not specified and you can put there anything you find useful.

[coderJeff]

History of LGPL:.....

Thanks. I think I have it sorted.

./readme.txt <= LGPLv2 (or later) is what started the confusion for me, though it is a correct statement
./src/rtlib/license.txt <= "GNU Lesser General Public License version 2.1 (or later)"
./src/gfxlib2/license.txt <= "GNU Lesser General Public License version 2.1 (or later)"

[coderJeff]

@St_W,
found this reference http://llg.cubic.org/docs/junit/

The <testsuite> element has attribute that can only be computed after the tests are complete, which means that the tests/suites have to run first, then counted for totals, then the report can be generated.

In the event of an unexpected abort or crash, and to get partial results, maybe it is better to write out results to a data file (can be structured) then process the file afterwards to create the junit.xml file, much like you are doing now.

[coderJeff]

I pushed changes to https://github.com/jayrm/fbcunit that add an junit xml report.

This code example:

Code: Select all

/'
	fbcunit example

		- write XML report
'/

#include once "fbcunit.bi"

SUITE( testpass )

	TEST( pass1 )
		CU_ASSERT( true )
	END_TEST

	TEST( pass2 )
		CU_ASSERT( true )
	END_TEST

END_SUITE

SUITE( testfail )

	TEST( pass1 )
		CU_ASSERT( true )
	END_TEST

	TEST( fail2 )
		CU_ASSERT( false )
	END_TEST

END_SUITE

'' no summary
fbcu.run_tests false

fbcu.write_report_xml "results.xml"

Will generate this output:

Code: Select all

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
    <testsuite name="testfail" errors="0" tests="2" failures="1" time="0">
        <testcase classname="testfail" name="fail2">
            <failure message="" type="Failure">
                Condition : CU_ASSERT(false)
                File      : examples\ex09.bas
                Line      : 28
            </failure>
        </testcase>
        <testcase classname="testfail" name="pass1" />
    </testsuite>
    <testsuite name="testpass" errors="0" tests="2" failures="0" time="0">
        <testcase classname="testpass" name="pass2" />
        <testcase classname="testpass" name="pass1" />
    </testsuite>
</testsuites>

@St_W, Is this something you can work with?

[St_W]

I pushed changes to https://github.com/jayrm/fbcunit that add an junit xml report.
[...]
@St_W, Is this something you can work with?

Thank You very much, it's fine.

I used it as a stand-in replacement for fbcu+cunit and ran the fbc test suite with your tool. Some tests didn't work out-of-the-box and I had to disable them because of various compile-time issues. I didn't take a closer look at the issues as I guess you'll do or even have done that already anyway. Just FYI, the following files didn't compile with your testframework:

tests\compound\for_UDT_counter.bas
tests\dim\common-shared-1.bas
tests\dim\extern-not-allocated-by-local-1.bas
tests\dim\extern-not-allocated-by-local-2.bas
tests\expressions\mul-unsigned.bas
tests\functions\odd-arg.bas
tests\functions\paraminit.bas
tests\gfx\image-expr.bas
tests\namespace\dups.bas
tests\namespace\extimp.bas
tests\namespace\global2.bas
tests\namespace\import_method.bas
tests\namespace\interf_imp.bas
tests\namespace\outside.bas
tests\namespace\redim-3.bas
tests\namespace\reimp1.bas
tests\namespace\reimp2.bas
tests\namespace\reimp3.bas
tests\namespace\sum.bas
tests\numbers\cast_f2ll.bas
tests\pointers\new-delete.bas
tests\pp\typeof.bas
tests\quirk\inline-asm-syntax.bas
tests\quirk\inline-asm.bas
tests\quirk\quirk_arraystatement.bas
tests\string\concat.bas
tests\structs\dynamic-array-fields.bas
tests\swap\intfloat.bas
tests\virtual\rtti.bas
tests\virtual\virtual.bas
tests\wstring\utf_conv.bas

Except for those it works really great and will simplify fbc's build & test process even more.

[coderJeff]

St_W, Thank-you for testing and reporting.

Yeah, the compile problems you got for the test files you listed is name collisions with new fbcunit helper macros. The helper macros can be disabled and the tests compiled as-is. I should be able to create a pull request soon so it's much clearer what I am doing:
1) Add new libfbcunit code at ./tests/fbcunit
2) Some changes to makefiles, otherwise, test-suite should compile & run as-is, no changes to test files, by adding this compatibility header

Code: Select all

'' fbcu.bi
#define FBCU_ENABLE_MACROS 0
#include "fbcunit.bi"
#include once "crt/string.bi"
#ifndef NULL
#define NULL 0
#endif

3) In the individual *.bas tests, change fbcu.bi to new to fbcunit.bi and fix the compile errors, optionally rewriting the test with the new helper macros.
4) remove ./tests/fbcu directory
5) Tested on win 32-bit gas/gcc, 64-bit gcc, now working on testing for lin-64bit.

[St_W]

Thank you for your explanations. Just some thoughts:

2) Some changes to makefiles, otherwise, test-suite should compile & run as-is, no changes to test files, by adding this compatibility header
3) In the individual *.bas tests, change fbcu.bi to new to fbcunit.bi and fix the compile errors, optionally rewriting the test with the new helper macros.

If you create a compatibility header "fbcu.bi", why do you need to change the include for the tests? (or, the other way round: if you change the include statements for the tests, why do you need a compatibility header?) Or are these sequential steps of the transition from the old fbcu+cunit to your fbcunit testing tool? And - you'll probably know that but I'm mentioning anyway - note that the unit tests are detected by looking for the fbcu.bi include file IIRC, so you've to change that too.

btw, why is it called fbcunit? (and not fbunit)

and another note: the old fbcu supported several configuration options (for example, I used "auto=1" to enable the report file generation); would be nice if at least the most important ones of those would be available for the new tool too.

I'm looking forward to testing your integration in the fbc repository. And thanks again for your efforts.

[coderJeff]

fbcunit = FreeBasic Compiler UNIT testing library is what made sense in my mind.

The compatibility header is a transition step. It will get removed. There are new features I want to add that are not compatible with existing tests, but I wanted a step where we keep all the tests exactly the same, minimally change the makefiles, and **only** change the underlying test framework. Once that is proven, I feel more comfortable about changing the makefiles and tests to suite the new framework, searching for and using "fbcunit.bi"

I forgot to mention, that old fbcu defined the MAIN entry point -- in the library, I didn't like that design. In the new test-suite there will be an actual "./tests/fbc-tests.bas" that gets compiled for the main entry point, where we can set options, from command line, etc. I think this approach makes running the test-suite and generating desired output simple to follow.

Next 12 hours or so, I should have the pull request created and hopefully, passes Travis CI with no major troubles.