Weird behavior in fixed length strings

General FreeBASIC programming questions.
angros47
Posts: 2014
Joined: Jun 21, 2005 19:04

Weird behavior in fixed length strings

Post by angros47 »

While a ZSTRING can't contain a null byte, a regular string should be able to contain it, no matter if it is a variable length or a fixed length one. Yet, I found a strange behavior:

Code: Select all

dim s as string*80=chr(0)+"Alpha"+chr(0)+"Beta"
print asc(s)
print s
as expected, the first character is a null character, and the string is usable and printable anyway (a ZSTRING would have been empty).

Now, let's try:

Code: Select all

dim s as string*80
s=chr(0)+"Alpha"+chr(0)+"Beta"
print asc(s)
print s
In theory, it should be perfectly equivalent. But actually, the first character won't be an ASCII 0 but an ASCII 65 (capital "A")
The string is still printed, but the zero bytes are gone. Why does this happen?
SARG
Posts: 1489
Joined: May 27, 2005 7:15
Location: FRANCE

Re: Weird behavior in fixed length strings

Post by SARG »

In the first case, Initialization is done using 3 temporary STRINGs. The last one is then used to create the fixed lenght string.

In the second case, it's a serial concatenation-assignation and I guess that as a fixed lenght string is involved the string chr(0) alone is considered as an empty string.

Doing the same operation with a normal string keeps the chr(0)s.

Should we get the same result ? that's the question. :wink:

Code: Select all

dim s1 as string*80=chr(0)+"Alpha"+chr(0)+"Beta"
print asc(s1)
print s1
print

dim s2 as string*80
s2=chr(0)+"Alpha"+chr(0)+"Beta"
print asc(s2)
print s2
print

dim s3 as string
s3=chr(0)+"Alpha"+chr(0)+"Beta"
print asc(s3)
print s3
sleep
deltarho[1859]
Posts: 3575
Joined: Jan 02, 2017 0:34
Location: UK
Contact:

Re: Weird behavior in fixed length strings

Post by deltarho[1859] »

With 160 views, I am surprised that only SARG has responded.
SARG wrote:Should we get the same result ? that's the question.
To my mind we should get the same result and angros47's second code is failing.
SARG wrote:the string chr(0) alone is considered as an empty string.
In the manual's description of String we have: "Despite the use of the descriptor, an implicit NULL character (Chr(0)) is added to the end of the string, to allow passing them to functions in external libraries without making slow copies. FreeBASIC's internal functions will ignore this character, and not treat it as part of the string."

In which case, angros47's first code is failing. :)

Looking at the asm produced by gas the two asms are different, but I am not an expert at asm. Concatenation is used in both asms but it is not clear to me why the first code is not 'ditching' the Chr(0); or SARG's third code for that matter.

If we have a large string comprising many substrings using Chr(0) as a delimiter, then the second code will not do as we expect it to. Parsing the large string for Chr(0) will fail because they have been ripped out.
SARG
Posts: 1489
Joined: May 27, 2005 7:15
Location: FRANCE

Re: Weird behavior in fixed length strings

Post by SARG »

I looked in the C source code of concatenation.
The concatenation is made using this lenght of the current string as an offset.

For var lenght string the lenght is retrieved in the descriptor so null characters don't matter.

Differently for fixed lenght string the lenght is determined by searching the first null.

So in second angros47's example the first chr(0) is added in position 0, then alpha also in position 0, the second chr(0) in position 5 and beta also in position 5.

As I wrote, the initialization (first example) is done using STRINGs so no problem with nulls. However I agree not consistent.
deltarho[1859]
Posts: 3575
Joined: Jan 02, 2017 0:34
Location: UK
Contact:

Re: Weird behavior in fixed length strings

Post by deltarho[1859] »

SARG wrote:However I agree not consistent.
To be consistent, they should either both preserve Chr(0) or both 'ditch' Chr(0) and not oppose each other.

To my mind, Chr(0) should not be 'ditched' so angros47's second code's interpretation needs looking at.
SARG
Posts: 1489
Joined: May 27, 2005 7:15
Location: FRANCE

Re: Weird behavior in fixed length strings

Post by SARG »

Want to yell :D especially cases 3 and 4

Code: Select all

dim as string *50 fixstrg="alpha"+chr(0)+"beta"
dim as string *50 fixstrg2
dim as STRING varstring="beta"

print fixstrg

fixstrg="alpha"+chr(0)+fixstrg
print fixstrg

fixstrg="beta"
fixstrg="alpha"+chr(0)+fixstrg
print fixstrg

fixstrg="beta"
fixstrg2="alpha"+chr(0)+fixstrg
print fixstrg2

fixstrg="alpha"+chr(0)+varstring
print fixstrg

fixstrg="alpha"+chr(0)+"beta"
print fixstrg

sleep
Output

Code: Select all

alpha beta
alpha alpha
alpha beta
alphabeta
alphabeta
alphabeta
deltarho[1859]
Posts: 3575
Joined: Jan 02, 2017 0:34
Location: UK
Contact:

Re: Weird behavior in fixed length strings

Post by deltarho[1859] »

:o
fxm
Moderator
Posts: 11204
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: Weird behavior in fixed length strings

Post by fxm »

SARG wrote: Jul 07, 2022 0:44 For var lenght string the lenght is retrieved in the descriptor so null characters don't matter.

Differently for fixed lenght string the lenght is determined by searching the first null.
Not exactly:
- For Zstring, the length is determined by looking for the first null.
- But for fix-len String, the length is constant, fixed by the declaration.

Code: Select all

Dim As Zstring * 21 z = "Zstring * 21"
Dim As String * 20  s =  "String * 20"

Print """"; Z; """", "Len ="; Len(z), "Sizeof ="; Sizeof(Z)
Print """"; s; """", "Len ="; Len(s), "Sizeof ="; Sizeof(s)

Sleep

Code: Select all

"Zstring * 21"              Len = 12      Sizeof = 21
"String * 20         "      Len = 20      Sizeof = 21
SARG
Posts: 1489
Joined: May 27, 2005 7:15
Location: FRANCE

Re: Weird behavior in fixed length strings

Post by SARG »

@fxm
The whole lenght is fixed but the 'real' lenght (up to first null) of the current string is determined with strlen otherwise it would not be possible to concatenate a string with a fix-len string.

Extracted from str_concatassign.c in case of fixed lenght string :

Code: Select all

dst_len = strlen( (char *)dst );     <----------- dst a pointer to the current string
........
fb_hStrCopy( &(((char *)dst)[dst_len]), src_ptr, src_len );
Lost Zergling
Posts: 522
Joined: Dec 02, 2011 22:51
Location: France

Re: Weird behavior in fixed length strings

Post by Lost Zergling »

I think user should be allowed to put/concatenate/handle Chr(0) into fixed len strings like he can do into var len strings. Further more, talking about zstring, Chr(0) is not filtered, it is just used as 'mark string end'.
I can see the usefullness filtering Chr(0) preventing some bugs or unexpected behaviour.
Other hand I hardly see the conceptual interest filtering Chr(0) into concatenation wich actually leads to other inconsistencies. I did never meet this problem using almost exclusively var len regular strings and zstrings for fixed len strings.
Sounds like the issue is related to fixed len strings..
SARG
Posts: 1489
Joined: May 27, 2005 7:15
Location: FRANCE

Re: Weird behavior in fixed length strings

Post by SARG »

Lost Zergling wrote: Jul 07, 2022 10:12 I think user should be allowed to put/concatenate/handle Chr(0) into fixed len strings
Explain me how you can find the 'real' end of the string ?

Suppose you add a string of 5 chr(0), where is the end of the string when you want to add an other string ?
deltarho[1859]
Posts: 3575
Joined: Jan 02, 2017 0:34
Location: UK
Contact:

Re: Weird behavior in fixed length strings

Post by deltarho[1859] »

More fun and games in the future?

From the docs for String: “Fixed length strings are also terminated with a NULL character, and so they use size + 1 bytes of space. This NULL terminator may be removed in future, to prevent the redundant character complicating data layout in user-defined Types.”

Patient: Doctor, it hurts when I do this.
Doctor: Well, don't do it then.

I also get problems when using Chr(0) as a delimiter in fixed length strings.
Well, don't do it then.

:)
Lost Zergling
Posts: 522
Joined: Dec 02, 2011 22:51
Location: France

Re: Weird behavior in fixed length strings

Post by Lost Zergling »

@Sarg. I think the problem, maybe, is that fixed len strings behaves a little bit as would do a 'zstrings wrapper', meaning they are same time regular.FB strings compliant (descriptor, string len), and they shall be zstrings compliant (reason why the chr(0)). So far, depending on the context the chr(0) induce inconsistency as you.pointed it out. Perhaps it should be.said chr(0) is really NOT supported into fixed len strings data, or perhaps it should be said yes.chr(0) is supported like it is with var len strings, meaning the real end would be systematically determined by the regular len originaly declared (as fxm pointed it out), otherwise it would be Chr(0) if the reference is zstring. The issue is that fixed len strings are sometimes assimilated.for.computation as if they where.zstrings, wich they are not because they're supposed being also var len strings compliant, chr(0) support.
Point is : when a var.is originally a string (even though fixed len) and not a zstring, len should never be evaluated differently than the one in the descriptor.
SARG
Posts: 1489
Joined: May 27, 2005 7:15
Location: FRANCE

Re: Weird behavior in fixed length strings

Post by SARG »

Lost Zergling wrote: Jul 07, 2022 21:53 Point is : when a var.is originally a string (even though fixed len) and not a zstring, len should never be evaluated differently than the one in the descriptor.
There is no descriptor for fixed len string.....
Lost Zergling
Posts: 522
Joined: Dec 02, 2011 22:51
Location: France

Re: Weird behavior in fixed length strings

Post by Lost Zergling »

Flute, c'est ballot :oops: . I understand the size is only known at first allocation time, then it vanish ?
There should be a (preprocessed ?) constant (aliassed reserved names ?) for each fixed size string, maybe.
Or another process to keep sizes in memory, don't know.
Sounds heavy. Perhaps better choice would be to forget that chr(0)... :mrgreen:
Post Reply