[Solved] When/how is a temporary array descriptor created

[shadow008]

Solved: A new descriptor to a static array is created when the static array is passed byref as a function parameter etc. Normal scoping rules /lifetimes apply.

As a part of a reflection/serialization library I've been writing, I've found it very nice to be able to make a copy of a static freebasic array's array descriptor and use that while simply holding an "any ptr" to the start of the array. The code works great, but I'm rather concerned with the hack I used to get there. Consider the following:

Code: Select all

#include "crt/mem.bi"
#include "fbc-int/array.bi"

type thing
	dim x(9) as string
end type

dim savedArrayDesc as FBC.FBARRAY

memcpy( _
	@savedArrayDesc, _
	FBC.ArrayConstDescriptorPtr(cast(thing ptr, 0)->x()), _ 'How bad is this?  Undefined?  Ok but hacky?
	sizeof(savedArrayDesc))

'Never use these members when doing this hack, they'll be garbage
savedArrayDesc.index_ptr = 0
savedArrayDesc.base_ptr = 0

'Demonstrate that things worked just fine
print "dims:   ";savedArrayDesc.dimensions
print "lbound: ";savedArrayDesc.dimTb(0).lbound
print "ubound: ";savedArrayDesc.dimTb(0).ubound

sleep

(Again, this is only used for arrays that were declared with a constant size)

Ultimately, FBC.ArrayConstDescriptorPtr() is calling the rtlib function fb_ArrayGetDesc which looks like this:

Code: Select all

FBCALL FBARRAY *fb_ArrayGetDesc( FBARRAY *array )
{
	return array;
}

So my questions are: What's happening during that dereference of 0? When is the actual array descriptor created? Does the creation of the array descriptor involve, in any part, reading from the variable it's expected to reside in?

My guess is that: The compiler is treating the subsequent parenthesis in a special way that causes it to generate a temporary array descriptor and that array descriptor is being passed to the function. It's then destroyed when the function returns and the value goes out of scope. The dereference is ignored in the same way it would be in, say, a typeof() expression.

Simply put this code works and I want to know if it actually should be working.

[fxm]

Why would you want to create your own copy of the array descriptor?

'FBC.Array[Const]DescriptorPtr()' creates one or refers to an already existing one, and this array descriptor
is or will be permanently accessible from its address returned by 'FBC.Array[Const]DescriptorPtr()'.

Note about the array descriptor creation process:
- Static and dynamic array descriptors are not treated the same way (the dynamic array descriptor is created right at the array declaration level, while the static array descriptor is only created when necessary like array passing to a procedure).
- In addition, a static array included in a UDT induces a particular process for its descriptor (a new array descriptor is created at each time the UDT included array is passed to a procedure or each 'FBC.Array[Const]DescriptorPtr()' call).

Code: Select all

#include "fbc-int/array.bi"

type thing
	dim x(9) as string
end type

var ArrayDescPtr = FBC.ArrayConstDescriptorPtr(cast(thing ptr, 0)->x())  '' only valid for a static array in a UDT
'' (created descriptor corresponds to a fictitious array starting at address 0 : array offset in the UDT)

'Demonstrate that things worked just fine
print "size:       : " & ArrayDescPtr->size
print "element_Len : " & ArrayDescPtr->element_Len
print "dims:       : " & ArrayDescPtr->dimensions
print "elements    : " & ArrayDescPtr->dimTb(0).elements
print "lbound:     : " & ArrayDescPtr->dimTb(0).lbound
print "ubound:     : " & ArrayDescPtr->dimTb(0).ubound

sleep

[shadow008]

Hey my guess was right!

the static array descriptor is only created when necessary like array passing to a procedure

'' (created descriptor corresponds to a fictitious array starting at address 0 : array offset in the UDT)

Together those two statements answer my question.

The reason I'm doing this is for constructing an array of a custom descriptor of type members. These array is then used as a way of describing the UDT in a generic way so that I can use it to construct a serializer/deserializer/reflection library on top of this array of descriptors. I've prioritized generic applicability over specificity for the description, which results in basically a descriptor that holds the offset to that member (offsetof(thing, x(0)), various pointers to overloaded functions that accept the underlying type (ProcPtr(... as typeof(...->x(0))), and, in the case that a static array is specified, a copy of the array descriptor. Because it's static, the lbound/ubound/elem_size will always be the same for any instance of that member, and it works beautifully.
I'm pretty close to finishing this library's first version and will release it eventually after it's been sufficiently dogfooded.

[fxm]

'FBC.Array[Const]DescriptorPtr()' creates one or refers to an already existing one, and this array descriptor
is or will be permanently accessible from its address returned by 'FBC.Array[Const]DescriptorPtr()'.

Note about the array descriptor creation process:
- Static and dynamic array descriptors are not treated the same way (the dynamic array descriptor is created right at the array declaration level, while the static array descriptor is only created when necessary like array passing to a procedure).
- In addition, a static array included in a UDT induces a particular process for its descriptor (a new array descriptor is created at each time the UDT included array is passed to a procedure or each 'FBC.Array[Const]DescriptorPtr()' call).

Code highlighting this behavior:

Code: Select all

#include "fbc-int/array.bi"
Using FBC

Function DescriptorPtr(Byval p As Any Ptr) As Any Ptr
    Return p
End Function
Var ArrayPassDescriptorPtr = Cast(Function(() As Integer) As Any Ptr, @DescriptorPtr)

Dim As Integer array1(10)
Dim As Integer array2()
Type UDT
    Dim As Integer array1(10)
    Dim As Integer array2(Any)
End Type
Dim As UDT u

Print "Successive descriptor address of an isolated static array"
Print "   passed to a procedure : ", ArrayPassDescriptorPtr(array1()), ArrayPassDescriptorPtr(array1()), ArrayPassDescriptorPtr(array1())
Print "   requested by the user : ", ArrayConstDescriptorPtr(array1()), ArrayConstDescriptorPtr(array1()), ArrayConstDescriptorPtr(array1())
Print
Print "Successive descriptor address of an isolated dynamic array"
Print "   passed to a procedure : ", ArrayPassDescriptorPtr(array2()), ArrayPassDescriptorPtr(array2()), ArrayPassDescriptorPtr(array2())
Print "   requested by the user : ", ArrayConstDescriptorPtr(array2()), ArrayConstDescriptorPtr(array2()), ArrayConstDescriptorPtr(array2())
Print
Print "Successive descriptor address of a static array in UDT"
Print "   passed to a procedure : ", ArrayPassDescriptorPtr(u.array1()), ArrayPassDescriptorPtr(u.array1()), ArrayPassDescriptorPtr(u.array1())
Print "   requested by the user : ", ArrayConstDescriptorPtr(u.array1()), ArrayConstDescriptorPtr(u.array1()), ArrayConstDescriptorPtr(u.array1())
Print
Print "Descriptor address of a dynamic array in UDT"
Print "   passed to a procedure : ", ArrayPassDescriptorPtr(u.array2()), ArrayPassDescriptorPtr(u.array2()), ArrayPassDescriptorPtr(u.array2())
Print "   requested by the user : ", ArrayConstDescriptorPtr(u.array2()), ArrayConstDescriptorPtr(u.array2()), ArrayConstDescriptorPtr(u.array2())

Sleep

Output example:

Code: Select all

Successive descriptor address of an isolated static array
   passed to a procedure :  1703504       1703504       1703504
   requested by the user :  1703504       1703504       1703504

Successive descriptor address of an isolated dynamic array
   passed to a procedure :  1703384       1703384       1703384
   requested by the user :  1703384       1703384       1703384

Successive descriptor address of a static array in UDT
   passed to a procedure :  1703240       1703200       1703160
   requested by the user :  1703124       1703088       1703052

Descriptor address of a dynamic array in UDT
   passed to a procedure :  1703348       1703348       1703348
   requested by the user :  1703348       1703348       1703348

[shadow008]

I'd like to focus on that third scenario of static array in a UDT for a minute:

Code: Select all

#include "fbc-int/array.bi"
Using FBC

Function DescriptorPtr(Byval p As Any Ptr) As Any Ptr
	dim as FBARRAY ptr a = p
	print "Base pointer: ";p
	print "    index_ptr: ";a->index_ptr
	print "    base_ptr: ";a->base_ptr
	print "    size: ";a->size
	print "    element_len: ";a->element_len
	print "    dimensions: ";a->dimensions
	print "    flags: ";a->flags
	for i as integer = 0 to FB_MAXDIMENSIONS-1
		print "        elemen: ";a->dimTb(i).elements
		print "        lbound: ";a->dimTb(i).lbound
		print "        ubound: ";a->dimTb(i).ubound
		print ""
	next
	print ""
    Return p
End Function
Var ArrayPassDescriptorPtr = Cast(Function(() As Integer) As Any Ptr, @DescriptorPtr)

Type UDT
    Dim As Integer array1(10)
    Dim As Integer array2(Any)
End Type
Dim As UDT u

Print "Successive descriptor address of a static array in UDT"
Print "passed to a procedure : "
ArrayPassDescriptorPtr(u.array1())
ArrayPassDescriptorPtr(u.array1())
ArrayPassDescriptorPtr(u.array1())

Output:

Code: Select all

Successive descriptor address of a static array in UDT
passed to a procedure :
Base pointer: 6421616
    index_ptr: 6421856
    base_ptr: 6421856
    size: 88
    element_len: 8
    dimensions: 1
    flags: 49
        elemen: 11
        lbound:  0
        ubound:  10

        elemen: 16
        lbound:  6421856
        ubound:  140720225927195

        elemen: 0
        lbound:  140720272847664
        ubound:  2

        elemen: 0
        lbound:  7541728
        ubound:  16

        elemen: 0
        lbound:  1
        ubound:  352

        elemen: 140720253226809
        lbound:  0
        ubound:  352

        elemen: 7538784
        lbound:  7536640
        ubound:  242199300183662

        elemen: 4202521
        lbound:  2
        ubound:  140720253291907


Base pointer: 6421696
    index_ptr: 6421856
    base_ptr: 6421856
    size: 88
    element_len: 8
    dimensions: 1
    flags: 49
        elemen: 11
        lbound:  0
        ubound:  10

        elemen: 1
        lbound:  352
        ubound:  140720253226809

        elemen: 0
        lbound:  352
        ubound:  7538784

        elemen: 7536640
        lbound:  242199300183662
        ubound:  4202521

        elemen: 2
        lbound:  140720253291907
        ubound:  0

        elemen: 0
        lbound:  0
        ubound:  0

        elemen: 0
        lbound:  0
        ubound:  0

        elemen: 0
        lbound:  0
        ubound:  0


Base pointer: 6421776
    index_ptr: 6421856
    base_ptr: 6421856
    size: 88
    element_len: 8
    dimensions: 1
    flags: 49
        elemen: 11
        lbound:  0
        ubound:  10

        elemen: 140720253291907
        lbound:  0
        ubound:  0

        elemen: 0
        lbound:  0
        ubound:  0

        elemen: 0
        lbound:  0
        ubound:  0

        elemen: 0
        lbound:  0
        ubound:  0

        elemen: 0
        lbound:  0
        ubound:  0

        elemen: 8
        lbound:  1
        ubound:  17

        elemen: 0
        lbound:  0
        ubound:  0

As we can see, the most important bits do not change. Namely element_len, dimensions, and the contents of dimTb() array up to # dimensions count. That fact is what's important to me.

But what is that garbage in the rest of dimTb? Is that a security concern? Can the contents be exploited somehow? I'm not knowledgeable enough in that field to say one way or the other but it doesn't look good.

[adeyblue]

But what is that garbage in the rest of dimTb?

That's your code not following the instructions.

fbc-int/array.bi

'' take care with number of dimensions; fbc may allocate
'' a smaller descriptor with fewer than FB_MAXDIMENSIONS
'' in dimTb() if it is known at compile time that they
'' are never needed. Always respect number of
'' dimensions when accessing dimTb()

[shadow008]

fbc-int/array.bi

'' take care with number of dimensions; fbc may allocate
'' a smaller descriptor with fewer than FB_MAXDIMENSIONS
'' in dimTb() if it is known at compile time that they
'' are never needed. Always respect number of
'' dimensions when accessing dimTb()

Dammit I even knew that going into all this... I guess I just blocked it out because in my mental model a type is a type is a type... never "a type is part of a type". Thank you for reminding me.

[fxm]

Indeed, only when an array is first declared as : 'Dim array() As ...', the memory allocated for the descriptor corresponds to the maximum number of dimensions of an array.
Otherwise, when the number of dimensions is explicitly first declared as for example : 'Dim array(Any, ...) ...' or 'Dim array(19, ...) ...', the memory allocated for the descriptor corresponds to the exact number of dimensions of the array.
So accessing to a non allocated part of a descriptor can induce a runtime error.

In your code, limit the loop to 'a->dimensions - 1' instead of 'FB_MAXDIMENSIONS - 1':

Code: Select all

#include "fbc-int/array.bi"
Using FBC

Function DescriptorPtr(Byval p As Any Ptr) As Any Ptr
	dim as FBARRAY ptr a = p
	print "Base pointer: ";p
	print "    index_ptr: ";a->index_ptr
	print "    base_ptr: ";a->base_ptr
	print "    size: ";a->size
	print "    element_len: ";a->element_len
	print "    dimensions: ";a->dimensions
	print "    flags: ";a->flags
	for i as integer = 0 to a->dimensions-1
		print "        elemen: ";a->dimTb(i).elements
		print "        lbound: ";a->dimTb(i).lbound
		print "        ubound: ";a->dimTb(i).ubound
		print ""
	next
	print ""
    Return p
End Function
Var ArrayPassDescriptorPtr = Cast(Function(() As Integer) As Any Ptr, @DescriptorPtr)

Type UDT
    Dim As Integer array1(10)
    Dim As Integer array2(Any)
End Type
Dim As UDT u

Print "Successive descriptor address of a static array in UDT"
Print "passed to a procedure : "
ArrayPassDescriptorPtr(u.array1())
ArrayPassDescriptorPtr(u.array1())
ArrayPassDescriptorPtr(u.array1())

[fxm]

@Jeff,

For a static array included in a UDT, a new array descriptor is created at each time the UDT included array is passed to a procedure or each 'FBC.Array[Const]DescriptorPtr()' call.

Jeff, could you confirm that :
- In case of passing such array to a procedure, the passed descriptor is temporary (the corresponding memory is deallocated at each procedure return).
- In case of repetitive calling 'FBC.Array[Const]DescriptorPtr()' on such array, only the descriptor corresponding to the last call is valid (the memories corresponding to the previous descriptor calls are deallocated).

[coderJeff]

I will check a couple of things, but should be:

- An implicit array descriptor is allocated for each instance where a fixed size array is passed as an argument.
- An implicit array descriptor should be valid until the end of the scope where it is created. Therefore, not just the last call, but it does depend on scope.
- So if the implicit descriptor is created as part of an expression, it may be invalid when the expression ends. Therefore the pointer to the descriptor should be obtained within the scope where it is to be used.
- The underlying data that the descriptor points to will be valid if the underlying data is valid (i.e. static allocation versus local allocation).

Currently fbc is doing:
- fixed length static array variable passed to procedures is allocated an implicit static descriptor
- fixed length static array field (of UDT's) passed to procedures is allocated an implicit local descriptor

[fxm]

- An implicit array descriptor should be valid until the end of the scope where it is created. Therefore, not just the last call, but it does depend on scope.

Indeed, OK:

Code: Select all

#include "fbc-int/array.bi"

Type UDT
    Dim As Integer array1(10)
End Type

Dim As UDT u
Dim as Const FBC.FBARRAY ptr p

p = FBC.ArrayConstDescriptorPtr(u.array1())
Print p
p = FBC.ArrayConstDescriptorPtr(u.array1())
Print p
p = FBC.ArrayConstDescriptorPtr(u.array1())
Print p
p = FBC.ArrayConstDescriptorPtr(u.array1())
Print p
p = FBC.ArrayConstDescriptorPtr(u.array1())
Print p
p = FBC.ArrayConstDescriptorPtr(u.array1())
Print p
p = FBC.ArrayConstDescriptorPtr(u.array1())
Print p
p = FBC.ArrayConstDescriptorPtr(u.array1())
Print p
p = FBC.ArrayConstDescriptorPtr(u.array1())
Print p

Print

For I As Integer = 1 To 3
    p = FBC.ArrayConstDescriptorPtr(u.array1())
    Print p
    p = FBC.ArrayConstDescriptorPtr(u.array1())
    Print p
    p = FBC.ArrayConstDescriptorPtr(u.array1())
    Print p
Next I

Sleep

Code: Select all

1703504
1703468
1703432
1703396
1703360
1703324
1703288
1703252
1703216

1703168
1703132
1703096
1703168
1703132
1703096
1703168
1703132
1703096