Array Descriptor (split from Wiki Improvements)

[fxm]

The size of the array descriptor may depend on the number of dimensions.
If you increase the number of dimensions, the descriptor should be reallocated in memory, which should often change its address, but change the referencing of an array is not possible.

I think that only if you declare the dynamic array as:
{Dim|Static} [Shared] As datatype array()
(without any indication of sizing)
the allocated descriptor is sized for the maximum number of dimensions (8).

[Juergen Kuehlwein]

Yes, that´s true. Fixed sized arrays get allocated a descriptor only for the dimensions needed. Dynamic arrays get always allocated room for 8 dimensions (and i would leave it this way at any rate). Currently the RTL cannot distinguish fixed sized from dynamic arrays. Therefore it doesn´t allow dynamic arrays (once dimmed) to increase their number of dimensions with REDIM(). Once knowing fixed size or not, this issue can be addressed too.


JK

[fxm]

I think that only if you declare the dynamic array as:
{Dim|Static} [Shared] As datatype array()
(without any indication of sizing)
the allocated descriptor is sized for the maximum number of dimensions (8).

But if you declare the dynamic array as:
{Dim|Static} [Shared] As datatype array(Any, Any)
or (1st declaration):
Redim [Shared] As datatype array(1, 2)
the allocated array descriptor is sized for only 2 dimensions.

[Juergen Kuehlwein]

Code: Select all

... As datatype array(Any, Any)

is somehow a hybrid for me. I can tell the number of dimensions beforehand, but i want to set the index bounds later. I suppose the compiler allocates two dimensions. This syntax definitely fixes the number of dimensions, so i wouldn´t call it dynamic.

But for a dynamic array

Code: Select all

... As datatype array ()

it would be nice to be allowed to

Code: Select all

REDIM array(10)
...
REDIM array(10, 10)
...
REDIM array(10, 10, 10)

which currently isn´t possible


JK

[fxm]

For this to be complete, we must also allow in UDTs:
- 'As datatype array()' in UDTs (in this case, the descriptor is allocated for 8 dimensions),
- in addition to 'As datatype array(Any, ...)' (where the allocated descriptor is sized depending on the number of 'Any' declared),
- and also 'Redim As datatype array(...)' (where the allocated descriptor is sized depending on the number of dimensions declared through the bounds).

[dodicat]

With -gen gas you can undefine a variable then re define it again.
For example you could undefine an array with one dimension then redefine it again with two dimensions.
But not with -gen gcc

Same goes for a simple variable.
However you can still undefine a label then re-define it again with -gen gcc.

Code: Select all

print"hello ";
if false then goto label: else print"hello ";

 label:
print
#undef label

print "Goodbye"
if true then goto label: else print "Goodbye"

 label:
print
#undef label


dim as integer i=6
print i
#undef i

dim as integer i=5
print i

sleep


  

(The gcc errors are cryptic!)

[coderJeff]

OK, so we have a number of array related bugs reported on sf.net, plus some new ones reported in this topic, and maybe some others hidden in other topics on the forum.

First step, I think, is to expose the existing array descriptor for developers and advanced users. This will give us a record of our starting point before we attempt to fix bugs and add some features that JK is proposing.

- in github PR #165 I have added a very small API to expose the array descriptor in a relatively stable way. It also kind of establishes the pattern I would like to see if we expand on this idea in future adding other API's that expose fbc internals.
- I added a wiki page DevArrays in the developer section to document array and array descriptor behaviours from the point of view of the developer. It gives us a place to record all the details of arrays that don't fit in the reference or programmer's guide.

So, a very simple example of usage for fbc-int/array.bi

Code: Select all

#include "fbc-int/array.bi"

#macro printArrayDesc( title, array )
scope
	dim ap as FBC.FBARRAY ptr = fb_ArrayGetDesc( array )

	print title
	print , "index_ptr  : " & hex( cint( ap->index_ptr ) )
	print , "base_ptr   : " & hex( cint( ap->base_ptr ) )
	print , "size       : " & ap->size
	print , "element_len: " & ap->element_len
	print , "dimensions : " & ap->dimensions

	for i as integer = 0 to ap->dimensions-1
	print , "dimTb(" & i & ").elements: " & ap->dimTb(i).elements
	print , "dimTb(" & i & ").lbound  : " & ap->dimTb(i).lbound
	print , "dimTb(" & i & ").ubound  : " & ap->dimTb(i).ubound
	next
end scope
#endmacro


dim a() as integer
printArrayDesc( "a() as integer", a() )

dim b(1 to 2, 3 to 5, 7 to 13) as byte
printArrayDesc( "b() as byte", b() )

I've merged the changes in to fbc/master as it won't affect any existing code. I appreciate any comments on the addition.

- in __FB_LANG__="fb" declaring the new symbols in namespace FBC, thus reserving "FBC" for internals. This maybe is a poor choice as we are already using "fbc as fbcctx" in the compiler sources, so we have to rename one of "fbc" symbol names.
- header in a separate directory "./inc/fbc-int" which will be for internal stuff, but exposed to the user.

[MrSwiss]

I like the approach: nicely, step by step

The naming used currently is however, not very clear (ambiguous/misleading).
FBC-int -- C's 32 bit signed integer data type, comes to mind instantly, instead of referencing 'internal'.
(not considering the name of the namespace proposed)
FBintern -- whould clear the issue ... just a idea, up for discussion.

[coderJeff]

FBintern -- whould clear the issue ... just a idea, up for discussion.

Thanks, MrSwiss. Yes, the changes I added immediately create 2 issues that I will lazily defer to later to resolve.

1) namespace FBC, I really would like to reserve the FBC namespace for compiler internals, so, should the need arise, I will rename the 'fbc' variable in fbc compiler sources to 'fbcompiler' to resolve the conflict. It's not needed just now, but if we ever use './inc/fbc-int/*.bi' directly in compiler sources, this is my solution.

2) __FB_LANG__="??". As it stands, the API will have namespace FBC in lang "fb", no namespace in lang "fblite", and fail in lang "qb" due the differences in integer sizes for lang "qb". I'm going to leave the solution open ended for now. The solution probably involves creating a "./inc/fbc-int/types.bi" kind of include. Though, for comparison, "./inc/fbgfx.bi" deals with namespace FB in the same way as current addition for array descriptor API, so basically I don't know what is best long term solution, at the moment.

I like the approach: nicely, step by step

Thank-you. Your comment helps to justify the time I will take to explain each step we take on this development path. Nice to know community is listening. :)

[Juergen Kuehlwein]

@Jeff,

so you decided to expose the descriptor itself!
Ok, no problem - this makes some of my functions obsolete, because you now can take the officially declared FBARRAY type.

As already said above when you expand the FBARRAY type, add one or more "reserved" members. This gives room for future ideas, without breaking compatibility with every new step.

For the additions i wrote, i need at least one bit field as a flag for one of the features. It could of course be an entire member variable to be used as a flag then too.

I appreciate your initiative in this area!


JK

[speedfixer]

Does this mean that all arrays will have descriptors, exposed and available, in this future version?

david

[coderJeff]

As already said above when you expand the FBARRAY type, add one or more "reserved" members. This gives room for future ideas, without breaking compatibility with every new step.

Yes, I read through the code for your proposed additions. From your code and other comments in this topic, the "flags" field is looking like the following (so far):

Code: Select all

enum FBARRAY_FLAGS
	FBARRAY_FLAGS_DIMENSIONS = &h0000000f      '' number of entries allocated in dimTb()
	FBARRAY_FLAGS_STATIC     = &h00000010      '' array points to static/fixed-length memory
	FBARRAY_FLAGS_ATTACHED   = &h00000020      '' array points to memory managed by another descriptor
	FBARRAY_FLAGS_LOCKED     = &h00000040      '' array in use: do not allow redim/erase
	FBARRAY_FLAGS_RESERVED   = &hffffff80      '' reserved, do not use
end enum

FBARRAY_FLAGS_DIMENSIONS:
fbc will allocate the smallest array descriptor possible, if it can determine the dimension count at compile time otherwise it allocates room for FB_MAXDIMENSIONS. Haven't decided on this one as the compiler and runtime are currently set up to try and dimension an array once and then not allow number of dimensions to change. Allowing number of dimensions to change during run time is counter productive to compile time array bounds checks and advice noted in the sources with respect to dimensions. Needs more thought

FBARRAY_STATIC:
if the descriptor points to data with static allocation, then REDIM should be disallowed. ERASE could still be used to clear the array. This is the first flag that I will be focusing on.

FBARRAY_ATTACHED:
this is a feature I saw in your proposed code where an array descriptor can point to arbitrary data, basically providing an array(index, ...) interface, rather than cast( datatype ptr, @array(0) ) + index ... pointer maths. Have to be careful with this one, especially with complex UDT's. It might look like a safe interface, but has all the dangers of pointers under the hood. So may want to think of incorporating some help from the compiler. Most simple syntax would be something like "var byref a() = b()" to create an alias to an existing array, plus other extensions for different dimensions or data types.

FBARRAY_LOCKED:
not sure about this one, based on a comment in this topic, though the intent would be that REDIM should return a run-time error, if array is "locked". Why an array would be locked; yet to be defined.

FBARRAY_RESERVED:
don't use.

[coderJeff]

Does this mean that all arrays will have descriptors, exposed and available, in this future version?

david

Yes, that is the intent. Plus, also trying to create a bit of a structure for other additions, like string descriptor, or file i/o, or whatever. To note, I am trying to keep a little separation from the main language, to give continued development half a chance, rather than immediately locked in for next 10 years.. i.e. the separate ./inc/fbc-int folder and documented in the developer section.

Also, "all arrays will have descriptors" is true only if the array is passed as argument to a procedure. This includes the procedure to get the array descriptor itself. Technically, there are some cases where arrays won't have descriptors because fbc optimizes them out, but if the descriptor is never accessed, the user will never know.

[coderJeff]

The field will have 32 bits available for array descriptor options.

I hope this will be an Integer (32/64 bits) like all other fields, so 32 bits useful?

Where exactly do you intend to add this field in the array descriptor structure?
(perhaps just before the number of dimensions)

yes, new field will be uinteger, just like all the other fields being one of integer, uinteger, or pointer.

For new code it probably doesn't matter exactly where the new "flags" field is placed. In my tests I have placed it just after the dimensions field. Is placing it just before better?

[fxm]

A remark a little off-topic about the syntax to use to designate an array (in its entirety):
- We must use either 'array()' or 'array' (without parentheses) depending on the keyword used.
- This constraint is a nuisance when one wants to define a macro with such a parameter, because it is not known (to my knowledge) how to truncate a macro parameter of its final parentheses (only argument concatenation exists).