Freebasic 1.20.0 Development

[fxm]

But if we ensure that our starting buffer is large enough to handle the assignment, here's the expected changes for VARIABLES if string*n and zstring*n and wstring*n passed to byref as string is allowed to be modified:

When the starting buffer is not large enough to handle such a assignment (passing an argument to a byref parameter of a procedure), why not return a truncated string ?
This is the same operating as for a simple assignment:

Code: Select all

Dim As String s = "123456789"
Dim As Zstring * 6 z

z = s
Print z  '' print "12345"

Sleep

[coderJeff]

But if we ensure that our starting buffer is large enough to handle the assignment, here's the expected changes for VARIABLES if string*n and zstring*n and wstring*n passed to byref as string is allowed to be modified:

When the starting buffer is not large enough to handle such a assignment (passing an argument to a byref parameter of a procedure), why not return a truncated string ?

Yes, my mistake, I think the truncation should be what will happen, and if it is not, then we should determine if it is possible to use truncation or note why it will be unsafe. Sorry, too many tests and I am not being methodical about the results.

---
Just running my last tests for now and then going to push the progress of today.
string*n initializing to spaces should work with redim, erase, redim preserve, including arrays of udt's containing string*n fields. I have not tested dynamic arrays within UDT's yet.

[fxm]

... which maybe makes a case for the "new" STRING*n being given a different name, as I suggested when I first commented on this issue.

I will be able to give my opinion when I have been able to update my code of 5000 lines which uses a lot of STRING*N (simple variables and arrays) with files I/O and prints + graphic drawings.
I will come back to this code when bugs #1 and #2 are fixed.

I am happily surprised to have only had to modify around ten lines (out of the 5000) for my code to work well, seen from the outside.
However, the code uses a dozen STRING*N, including around ten dynamic arrays.
I have not tested the operation of any internal tests checking all syntax read in text files at program startup (maybe error tests no longer trigger).

The changed lines all correspond to compare lines where STRING*N's are involved:
I simply surrounded such strings with the RTRIM keyword (which ensures backwards compatibility).

So from my point of view, upgrading the code is simple and therefore does not require keeping the old definition of STRING*N.


[added]
Despite the last change:
- Allow ZSTRING*N and WSTRING*N arguments passed to BYREF AS STRING parameters to be modified (copied back to original variable),
changing all declarations of STRING*N into ZSTRING*N+1 is catastrophic because of countless LEN(s) which should be transformed into SIZEOF(s)-1.

[fxm]

On this occasion, I just noticed that the following code did not trigger the following error message from the compiler:
"error 4: Duplicated definition, ..."
(same behavior with ZSTRING and WSTRING)

Code: Select all

Dim As String * 20 array()

Redim As String * 10 array(0)  '' no error !
Print Len(array(0))
Print Sizeof(array(0))

Sleep

[coderJeff]

On this occasion, I just noticed that the following code did not trigger the following error message from the compiler:

Code: Select all

Dim As String * 20 array()
Redim As String * 10 array(0)  '' no error !

This seems familiar - maybe something to do with passing array arguments? but I can't quite place it. Perhaps we will be making some efforts on arrays.

Just playing around and find this (in fbc 1.10, 1.09, etc): this produces an error, but also fails an internal assert indicating that it is not fully handled.

Code: Select all

sub proc( a() as string * 10 )
end sub

Source comment: How to handle How to handle fixed-length string arrays?

[fxm]

So we would have,

Code: Select all

type T1: f as string*10 = any: end type  '' uninitialized
type T2: f as string*10      : end type  '' initialize to string(10,0)
type T3: f as string*10 = "" : end type  '' initialize to space(10)

After that assignments always pad with spaces.

I would like that.
This way, only the last syntax requires an implicit constructor.

On the other hand, dynamic arrays could never be pre-filled with spaces

After reflection (given the interest in doing so and inducing dynamic arrays never initialized with spaces), I am not in favor of the possibility of initializing a fixed length string with 0's as:
type T2: f as string*10 : end type '' initialize to string(10,0)

I would prefer this:

Code: Select all

Dim s1 As String*N = any         '' uninitialized
Dim s2 As String*N               '' initialize to space(N)
Dim s3 As String*N = any_string  '' initialize to any_string, and right pad with spaces up to Nth character

like the current definition.

It would remain to be decided whether the case 's2' declared in a UDT always generates constructor code (regardless of other member data), like the case 's3'.
But what to do for “UNION” structures ?
So let us leave things as they are.

[fxm]

String trimming is not symmetric for left versus right side

When trimming a var-len string by Ltrim/Rtrim/Trim, null characters are treated the same as spaces but only on the right side:

Code: Select all

Sub stringViewer(Byref text As String, Byref s As String)
    Print text;
    For I As Integer = 0 To Len(s) - 1
        Print Right("   " & s[I], 4);
    Next I
    Print
End Sub

Print "'Dim As String s'"
Dim As String s
Print

Print "'s = String(2, 0) & ""  test  "" & String(2, 0)'"
s = String(2, 0) & "  test  " & String(2, 0)
stringViewer("   s       : ", s)
stringViewer("   Ltrim(s): ", Ltrim(s))
stringViewer("   Rtrim(s): ", Rtrim(s))
stringViewer("   Trim(s) : ", Trim(s))
Print

Print "'s = ""  "" & String(2, 0) & ""test"" & String(2, 0) & ""  ""'"
s = "  " & String(2, 0) & "test" & String(2, 0) & "  "
stringViewer("   s       : ", s)
stringViewer("   Ltrim(s): ", Ltrim(s))
stringViewer("   Rtrim(s): ", Rtrim(s))
stringViewer("   Trim(s) : ", Trim(s))

Sleep

• Code: Select all

  'Dim As String s'
  
  's = String(2, 0) & "  test  " & String(2, 0)'
     s       :    0   0  32  32 116 101 115 116  32  32   0   0
     Ltrim(s):    0   0  32  32 116 101 115 116  32  32   0   0
     Rtrim(s):    0   0  32  32 116 101 115 116
     Trim(s) :    0   0  32  32 116 101 115 116
  
  's = "  " & String(2, 0) & "test" & String(2, 0) & "  "'
     s       :   32  32   0   0 116 101 115 116   0   0  32  32
     Ltrim(s):    0   0 116 101 115 116   0   0  32  32
     Rtrim(s):   32  32   0   0 116 101 115 116
     Trim(s) :    0   0 116 101 115 116

Does anyone know the reason for this ?

Same behavior for the new fix-len string (fbc version 1.20.0).
For the old fix-len string (fbc version 1.10.0), its weird behavior (for initialization/assignment versus null character) does not allow reliable analysis.

[coderJeff]

Does anyone know the reason for this ?

https://github.com/freebasic/fbc/blob/d ... skip.c#L23

Code: Select all

    /* fixed-len's are filled with null's as in PB, strip them too */
    while( (--len >= 0) && (((int)*p == c) || ((int)*p == 0) ) )
		--p;

I think this comment is from 2005. I think something is wrong with this comment because I think PB's string*n are padded with spaces just like we are trying to do now. But I don't know about PB's RTRIM behaviour.

Maybe this predates [L|R]TRIM( text, [any] chars ) variants and was never updated?

[fxm]

String trimming is not symmetric for left versus right side

When trimming a var-len string by Ltrim/Rtrim/Trim, null characters are treated the same as spaces but only on the right side:
.....
.....

Does anyone know the reason for this ?

Same behavior for the new fix-len string (fbc version 1.20.0).
For the old fix-len string (fbc version 1.10.0), its weird behavior (for initialization/assignment versus null character) does not allow reliable analysis.

Perhaps a generalization due to the final padding with null characters on the right side of the old STRING*N (fbc version 1.10.0):
(to be removed in fbc version 1.20.0 ?)

Code: Select all

#macro stringRawViewer(_text_, _string_)  '' String index operator '[]' only applies on string references
    Print _text_;
    For I As Integer = 0 To Len(_string_) - 1
        Print Right("   " & _string_[I], 4);
    Next I
    Print
#endmacro

Sub stringCopyViewer(Byref text As String, Byref s As String)  '' String index operator '[]' does not apply on strings returned by value as for the string functions
    Print text;
    For I As Integer = 0 To Len(s) - 1
        Print Right("   " & s[I], 4);
    Next I
    Print
End Sub

Print "'Dim As String * 12 s'"
Dim As String * 12 s
Print

Print "'s = ""  test  ""'"
s = "  test  "
stringRawViewer ("   s        (raw) : ", s)
stringCopyViewer("   s        (copy): ", s)
stringCopyViewer("   Ltrim(s) (copy): ", Ltrim(s))
stringCopyViewer("   Rtrim(s) (copy): ", Rtrim(s))
stringCopyViewer("   Trim(s)  (copy): ", Trim(s))
Print

Sleep

• - With fbc version 1.10.0:

  Code: Select all

  'Dim As String * 12 s'
  
  's = "  test  "'
     s        (raw) :   32  32 116 101 115 116  32  32   0   0   0   0    <==
     s        (copy):   32  32 116 101 115 116  32  32
     Ltrim(s) (copy):  116 101 115 116  32  32   0   0   0   0            <==
     Rtrim(s) (copy):   32  32 116 101 115 116                            <==
     Trim(s)  (copy):  116 101 115 116
  

  - With fbc version 1.20.0:

  Code: Select all

  'Dim As String * 12 s'
  
  's = "  test  "'
     s        (raw) :   32  32 116 101 115 116  32  32  32  32  32  32
     s        (copy):   32  32 116 101 115 116  32  32  32  32  32  32
     Ltrim(s) (copy):  116 101 115 116  32  32  32  32  32  32
     Rtrim(s) (copy):   32  32 116 101 115 116
     Trim(s)  (copy):  116 101 115 116

[fxm]

Regardless of the fbc version, the built-in functions that return subsets of strings (Dim, Left, Right, Ltrim, Rtrim, Trim, ...) take into account all characters in a fix-len string 's' (declared STRING*N), including the padding characters (null characters for fbc version 1.10.0, and spaces for fbc version 1.20.0) in addition to the user characters.
Otherwise, for the other uses of 's', the fbc version 1.10.0 only provides the user characters while the fbc version 1.20.0 always provides all characters (including padding spaces).

Example:

Code: Select all

#macro stringRawViewer(_text_, _string_)  '' String index operator '[]' only applies on string references
    Print _text_;
    For I As Integer = 0 To Len(_string_) - 1
        Print Right("   " & _string_[I], 4);
    Next I
    Print
#endmacro

Sub stringCopyViewer(Byref text As String, Byref s As String)  '' String index operator '[]' does not apply on strings returned by value as for the string functions
    Print text;
    For I As Integer = 0 To Len(s) - 1
        Print Right("   " & s[I], 4);
    Next I
    Print
End Sub

Print "'Dim As String * 12 s'"
Dim As String * 12 s
Print

Print "'s = ""  test  ""'"
s = "  test  "
stringRawViewer ("   s           (raw) :", s)
stringCopyViewer("   s           (copy):", s)
stringCopyViewer("   Mid(s,1)    (copy):", Mid(s,1))
stringCopyViewer("   Left(s,12)  (copy):", Left(s,12))
stringCopyViewer("   Right(s,12) (copy):", Right(s,12))
stringCopyViewer("   Lcase(s)    (copy):", Lcase(s))
stringCopyViewer("   Ucase(s)    (copy):", Ucase(s))
stringCopyViewer("   Ltrim(s)    (copy):", Ltrim(s))
stringCopyViewer("   Rtrim(s)    (copy):", Rtrim(s))
stringCopyViewer("   Trim(s)     (copy):", Trim(s))
Print

Sleep

• - With fbc version 1.10.0:

  Code: Select all

  'Dim As String * 12 s'
  
  's = "  test  "'
     s           (raw) :  32  32 116 101 115 116  32  32   0   0   0   0
     s           (copy):  32  32 116 101 115 116  32  32                    <==
     Mid(s,1)    (copy):  32  32 116 101 115 116  32  32   0   0   0   0
     Left(s,12)  (copy):  32  32 116 101 115 116  32  32   0   0   0   0
     Right(s,12) (copy):  32  32 116 101 115 116  32  32   0   0   0   0
     Lcase(s)    (copy):  32  32 116 101 115 116  32  32   0   0   0   0
     Ucase(s)    (copy):  32  32  84  69  83  84  32  32   0   0   0   0
     Ltrim(s)    (copy): 116 101 115 116  32  32   0   0   0   0
     Rtrim(s)    (copy):  32  32 116 101 115 116
     Trim(s)     (copy): 116 101 115 116
  

  - With fbc version 1.20.0:

  Code: Select all

  'Dim As String * 12 s'
  
  's = "  test  "'
     s           (raw) :  32  32 116 101 115 116  32  32  32  32  32  32
     s           (copy):  32  32 116 101 115 116  32  32  32  32  32  32
     Mid(s,1)    (copy):  32  32 116 101 115 116  32  32  32  32  32  32
     Left(s,12)  (copy):  32  32 116 101 115 116  32  32  32  32  32  32
     Right(s,12) (copy):  32  32 116 101 115 116  32  32  32  32  32  32
     Lcase(s)    (copy):  32  32 116 101 115 116  32  32  32  32  32  32
     Ucase(s)    (copy):  32  32  84  69  83  84  32  32  32  32  32  32
     Ltrim(s)    (copy): 116 101 115 116  32  32  32  32  32  32
     Rtrim(s)    (copy):  32  32 116 101 115 116
     Trim(s)     (copy): 116 101 115 116

[fxm]

Change : STRING*N occupies N bytes and has no terminating null character

At first glance, list of pages that will need to be updated:

• KeyPgType
  LangQB
  KeyPgString
  KeyPgOpStrptr
  ProPgStringsTypes
  KeyPgSizeof
  KeyPgSwap
  KeyPgDim
  TblVarTypes

• +
  ProPgInitialization
  KeyPgUnion
  KeyPgByref

I think I will start updating the documentation tomorrow.

It is very likely that I miss documentation pages (see list above) that need to be updated.
Let me know it please.

Completed documentation update for the pages I identified above.

[coderJeff]

Perhaps a generalization due to the final padding with null characters on the right side of the old STRING*N (fbc version 1.10.0):
(to be removed in fbc version 1.20.0 ?)

Seems like something could be improved here and need to investigate what is going on in the rtlib:
- rtrim( string )
- rtrim( string, filter )
- rtrim( string, any filter )
Unless my test is wrong, none of these forms are able to trim *only* chr(32) or trim *only* chr(0) while retaining the other chr(0|32).

Code: Select all

function escapeChar( byval ch as ubyte ) as string
	if( ch >= 33 andalso ch <= 126 ) then
		return chr(ch)
	end if
	return str(ch)
end function

sub showChars( byval p as ubyte ptr, byval n as integer )
	print "n="; right( space(2)+str(n), 2); ": "; escapeChar(p[0]);
	for i as integer = 1 to n - 1
		print ",";escapeChar(p[i]);
	next
	print
end sub

sub CheckRTRIM( byref s as string, byref filter as string )
	dim r as string

	print "STRING                     : ";
	showChars( strptr(s), len(s) )
	print "FILTER                     : ";
	showChars( strptr(filter), len(filter) )

	print "RTRIM( string )            : ";
	r = rtrim( s )
	showChars( strptr(r), len(r) )

	print "RTRIM( string, filter )    : ";
	r = rtrim( s, filter )
	showChars( strptr(r), len(r) )

	print "RTRIM( string, any filter ): ";
	r = rtrim( s, any filter )
	showChars( strptr(r), len(r) )

	print
end sub

dim b as string = "abcd"

CheckRTRIM( b + chr(0,0,32,32), chr(0) ) 
CheckRTRIM( b + chr(32,32,0,0), chr(32) )
CheckRTRIM( b + chr(0,0,32,32), chr(0,32) ) 
CheckRTRIM( b + chr(32,32,0,0), chr(0,32) )
CheckRTRIM( b + chr(0,0,32,32), chr(32,0) ) 
CheckRTRIM( b + chr(32,32,0,0), chr(32,0) )

sleep

[coderJeff]

It would remain to be decided whether the case 's2' declared in a UDT always generates constructor code (regardless of other member data), like the case 's3'.

I would like to try keep the optimization for now even though it currently gives a more complicated rule. Ideally, the underlying implementation should not matter to the user - inline initialize memory or call a procedure, the user syntax and declaration should be the same. The current issue is a limitiation on where the user is allowed to use TYPE declarations that require a default constructor or destructor.

So, I also briefly looked at:

Code: Select all

scope  '' or could be a sub/function
	type T
		s as string
	end type
	dim x as T
end scope

Because the constructor and destructor are default only, it would be nice to be able to allow this also. Then for UDT's with STRING or STRING*N and no other user defined member procedures, the TYPE could be used in the same kinds of scopes even if mixed with other data types. However, fbc cannot do this currently because fbc needs to emit the default constructor and destructor procedures upon parsing the "end type" statement which can only be done at module level scope. So we would need to add a facility to parse the "end type" but then delay the emitting of procedures until the parser is back to a module level scope - plus some internal unique naming of the locally scoped TYPE name.

[Lost Zergling]

Simply because of not having said it often enough, but nevertheless, thank you, really thank you, Jeff and fxm for your involvement in the continuity of the FB project.

[srvaldez]

I concur with Lost Zergling
thank you coderJeff and fxm