Array Descriptor (split from Wiki Improvements)

Forum for discussion about the documentation project.
Post Reply
Juergen Kuehlwein
Posts: 284
Joined: Mar 07, 2018 13:59
Location: Germany

Re: Wiki improvements

Post by Juergen Kuehlwein »

Array descriptor size change makes this a full break if the user lib uses array descriptors
I think the beauty of my proposal is - the size doesn´t change! If we come across an "old" descriptor, we will have no other problem than we have right now (we just can´t tell), but it doesn´t break the code (at least i thinks so).


JK
fxm
Moderator
Posts: 12081
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: Wiki improvements

Post by fxm »

Maybe a cookie (var/fix length array flag) could be stored just before the descriptor structure:
  • - a "hidden" Integer at address = @Cptr(Integer Ptr, @descriptor)[-1]
(similar to the usage of a cookie (nb. of elements) stored by 'New UDT[]' (at "address - 1 Integer") when the UDT has a destuctor)
coderJeff
Site Admin
Posts: 4313
Joined: Nov 04, 2005 14:23
Location: Ontario, Canada
Contact:

Re: Wiki improvements

Post by coderJeff »

My comments so far are only about changing the meaning of the array descriptor structure. We still need to do something with it, generating different code in the compiler and/or implementing different behaviour in the rtlib. So maybe will have other new bugs or crashes not yet realized due to linking new and old libs. I'm not really looking at anymore for now.
Juergen Kuehlwein
Posts: 284
Joined: Mar 07, 2018 13:59
Location: Germany

Re: Wiki improvements

Post by Juergen Kuehlwein »

Don´t worry Jeff, it´s just thinking about and discussing concepts. Feel free to move this topic and corresponding posts elsewhere, because it´s not really about wiki improvments
Maybe a cookie (var/fix length array flag) could be stored just before the descriptor structure:

- a "hidden" Integer at address = @Cptr(Integer Ptr, @descriptor)[-1]
This is better than my proposal."Old" code (e.g. a dll) would definitely get into trouble with my "new" descriptor, even if the size didn´t change, because it would still interpret the bitmask as a value and therefore read wrong values.

Taking a closer look at the RTL i can see that the descriptor isn´t saved to disk when saving an array, it´s only the array´s data. So the only remaining case i can think of when "new" code is confronted with an "old" descriptor, would be a call (an "old" dll calls an exported sub/function of a "new" executable, passing an array) from "old" code (do you agree, or did i miss something?). This could be a problem, because how can we be sure, that the integer before the descriptor is valid memory? And if it is valid, how can we decide, if it´s our hidden flag or something else (random)?

The other way round ("old" code receiving a "new" descriptor) it should work quite well, because the "old" code gets exactly, what it expects. It doesn´t know and it doesn´t care about the hidden flag.


JK
speedfixer
Posts: 606
Joined: Nov 28, 2012 1:27
Location: CA, USA moving to WA, USA
Contact:

Re: Wiki improvements

Post by speedfixer »

@jk and coderjeff:

" ... we would have 28 bits free for other values or flags ... "

If one more bit were used to flag the new format then old code could remain compatible. Otherwise, old code would be miss-interpreted.

This extra 'new format' flag bit would mean the new descriptor code could act like the old when the new format flag bit isn't present. A tiny bit of extra overhead (I hate that), but nothing would break. Good remarks and any maintainer could clean it out later.

Remark the code, and remove that new convention flag and support for the 'old format' in a few versions up the road.

david
coderJeff
Site Admin
Posts: 4313
Joined: Nov 04, 2005 14:23
Location: Ontario, Canada
Contact:

Array Descriptor

Post by coderJeff »

I've tried to extract the array descriptor discussion out of "Wiki Improvements" and put it all here. In theory, both topics, should still make sense.
Juergen Kuehlwein
Posts: 284
Joined: Mar 07, 2018 13:59
Location: Germany

Re: Array Descriptor (split from Wiki Improvements)

Post by Juergen Kuehlwein »

In my new array features, i solved this problem by adding room for one more dimension each time an array is created (fixed size or dynamic). This extra dimension has a size of three integers. I use the first two integers for an identifier ("arrayext") and the last one is used for new flags. So we have 32 bits for flags or values. All new code has this extra dimension, old code doesnt have it. We can determine by reading the identifier, if there is an extra dimension (and extra flag and values) or not. So a mix of old and new binaries doesn´t cause problems.

The only bad thing, which could happen, is trying to read the identifier from invalid memory (in case new code tries to read the extra dimension of an array created by old code. Most of times the memory after the last dimension will be valid but we cannot be sure). For this case the routine reading the identifier installs a signal handler to recover from such a situation. The same routine restores the default handler before it returns.

This method allows for extracting additional information (is array fixed size or dynamic, is array attached or not, etc.) from the descriptor, without breaking anything. And it is compatible with existing binaries.


JK
coderJeff
Site Admin
Posts: 4313
Joined: Nov 04, 2005 14:23
Location: Ontario, Canada
Contact:

Re: Array Descriptor (split from Wiki Improvements)

Post by coderJeff »

Juergen Kuehlwein wrote:In my new array features, i solved this problem by adding room for one more dimension each time an array is created...
The only bad thing, which could happen, is trying to read the identifier from invalid memory ... For this case the routine reading the identifier installs a signal handler to recover from such a situation.
While this may work (I did not test your solution), it's convoluted. Because 1) Extra dimension, used for purpose other than dimension, 2) signal handlers.

Simplest solution is to add information (i.e. a "flags" field) to existing array descriptor and break compatibility. Because this is not the only bug reported that is going to break compatibility to fix, we are going break binary compatibility anyway on next release.
Juergen Kuehlwein
Posts: 284
Joined: Mar 07, 2018 13:59
Location: Germany

Re: Array Descriptor (split from Wiki Improvements)

Post by Juergen Kuehlwein »

What´s wrong with a signal handler? There is a very small chance of running into invalid memory, so why not catch and handle this situation instead of letting the app randomly crash?


JK


PS: if you choose to add a "flags" field and thus break binary compatibility, please add one ore more "reserved" fields for whatever may come in the future. A few bytes more or less don´t matter these times.


Added later:
Adding one or more fields to the descriptor is by far the cleanest solution, i agree. The major downside is, that this definitely will break binary compatibility. My goal was to avoid such a step, but, if it is inevitable anyway, i will be happy with a flag field - makes things easier.
fxm
Moderator
Posts: 12081
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: Array Descriptor (split from Wiki Improvements)

Post by fxm »

coderJeff wrote:Simplest solution is to add information (i.e. a "flags" field) to existing array descriptor and break compatibility. Because this is not the only bug reported that is going to break compatibility to fix, we are going break binary compatibility anyway on next release.
Apart from the static/dynamic array flag, what are the other bugs that need new information in the descriptor?
Juergen Kuehlwein
Posts: 284
Joined: Mar 07, 2018 13:59
Location: Germany

Re: Array Descriptor (split from Wiki Improvements)

Post by Juergen Kuehlwein »

We should reserve one flag bit for indicating the attached state after array(attach, ...). Otherwise i don´t know, but it makes sense to be prepared for more.


JK
Juergen Kuehlwein
Posts: 284
Joined: Mar 07, 2018 13:59
Location: Germany

Re: Array Descriptor (split from Wiki Improvements)

Post by Juergen Kuehlwein »

Just another thought:

Maybe we should add a new "dwsize" member to the new descriptor. Maybe in second place, because currently there is a data pointer, whose value must point to valid memory (> 65000), so if we read a value smaller than that but > 0, we know it´s our new size member and we know it´s the new version of our descriptor. This gives us a chance for handling old binaries without a crash. That is, such a descriptor change then wouldn´t break binary compatibility. I remember MS doing this with some structures for the Windows API.


JK
coderJeff
Site Admin
Posts: 4313
Joined: Nov 04, 2005 14:23
Location: Ontario, Canada
Contact:

Re: Array Descriptor (split from Wiki Improvements)

Post by coderJeff »

fxm wrote:
coderJeff wrote:Simplest solution is to add information (i.e. a "flags" field) to existing array descriptor and break compatibility. Because this is not the only bug reported that is going to break compatibility to fix, we are going break binary compatibility anyway on next release.
Apart from the static/dynamic array flag, what are the other bugs that need new information in the descriptor?
For the current reported bugs, I think only static/dynamic array flag needs to be added to fix the bugs. In brief, the fb run time needs to make decisions based on the array being static/dynamic, which in some cases can't be known at compile time. Honestly, I need to review the bugs for the array stuff, as it has been a couple months I've been working on other things.

A few examples of static/fixed/dynamic descriptors and constants.

Code: Select all

sub proc( a() as integer )
end sub

sub proc1()
	'' no descriptor (provided we never pass as procedure argument), fixed contents
	'' indexing and address of elements is solved at compile time
	'' fixed location of descriptor and elements for duration of program
	static a(1 to 10) as integer
end sub

sub proc2()
	'' descriptor is created and allocated at compile time
	'' because passing to procedure requires that it have
	'' a descriptor.  Otherwise, is same as proc1() above.
	static a(1 to 10) as integer
	proc( a() )
end sub

sub proc3()
	'' static descriptor, allocated at compile time
	'' dynamic contents, allocated at run time
	static a() as integer
	redim a(1 to 3)
end sub

sub proc4()
	'' local descriptor, allocated at run time
	'' fixed contents, allocated at run time
	dim a(1 to 10) as integer
end sub

sub proc5()
	'' local descriptor, allocated at run time
	'' dynamic contents, allocated at run time
	dim a() as integer
	redim a(1 to 3)
end sub

proc1()
proc2()
proc3()
proc4()
proc5()
coderJeff
Site Admin
Posts: 4313
Joined: Nov 04, 2005 14:23
Location: Ontario, Canada
Contact:

Re: Array Descriptor (split from Wiki Improvements)

Post by coderJeff »

Juergen Kuehlwein wrote:Just another thought:
No. :)
I'm not giving the compatibility another thought. fxm's idea of placing data just before the descriptor (I think), would be fairly compatible, though also not 100% in all cases, so my opinion, is to not bother, and go for the simplest solution.

I'm adding a field for flags in the array descriptor. That way don't have to deal with struct packing, integer sizes, endian-ness, bit masks with other fields, magic numbers, extra checks for various descriptor formats, and other portability nonsense. Very simple.

The field will have 32 bits available for array descriptor options. So if we identify other uses in future, like an alternate format, or "attached" arrays as you propose in your array update, there is some room for growth.
fxm
Moderator
Posts: 12081
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: Array Descriptor (split from Wiki Improvements)

Post by fxm »

coderJeff wrote:The field will have 32 bits available for array descriptor options.
I hope this will be an Integer (32/64 bits) like all other fields, so 32 bits useful?

Where exactly do you intend to add this field in the array descriptor structure?
(perhaps just before the number of dimensions)
Post Reply