How to add new features? (syntax, user API)

General discussion for topics related to the FreeBASIC project or its community.
Lost Zergling
Posts: 534
Joined: Dec 02, 2011 22:51
Location: France

Re: How to add new features? (syntax, user API)

Post by Lost Zergling »

@andy. Sorry about my epidermic reaction.. As I said previously UDK should be a true conceptual advance. I shortly went to your link. I m not sure gforth stands out from object programming. Till you can program a kwd using oop, you still fall back into oop and you just dedigned an extension using a macro syntax. So far, the challenge is enormous... I do not like very much Gpl 3 I do not know what it is but doubt it could be qualified as "Public domain". A little bit frustrated about it, sorry again. So yes I have few idea about the "Udk" ideas and problematics, but it just is ghost draft...
paul doe
Moderator
Posts: 1730
Joined: Jul 25, 2017 17:22
Location: Argentina

Re: How to add new features? (syntax, user API)

Post by paul doe »

Juergen Kuehlwein wrote:...
What i propose allows for adding new features at the cost of one (one single, only one, i promise) new key word (which could become a new global key word) for each new topic. All the rest is hidden without even a possibility of a naming conflict. It´s about weighing the pros and cons, what do i get, what must i pay for it. The price is "sacrificing" one new reserved word, the gain is a bunch of new (and expandable) features. Of course the will be users, who will have to adapt their code (mostly run a replace function in their editor, this is inevitable), but there will be a benefit for the whole community.
...
This looks interesting, naturally. Can you elaborate a bit? And, which reserved keyword are you willing to 'sacrifice'?
Juergen Kuehlwein
Posts: 284
Joined: Mar 07, 2018 13:59
Location: Germany

Re: How to add new features? (syntax, user API)

Post by Juergen Kuehlwein »

I gave a basic example of how to do it in the opening post. Using a macro (and some new preprocessor operators) allows for a very flexible syntax without the need for re-writing any compiler code. A new reserved word "ARRAY" would be needed for array features, maybe "AUDIO" for audio extensions (angros47), "VIDEO" for video extensions, "NET" for network extensions. This mimics already existing syntax like e.g. "OPEN", which can do a variety of related things with a quite flexible syntax.

The downside is (and i see Jeff´s point here) that a macro overrides everthing. That is, you cannot use the word "ARRAY" anyhwere else in the same code (therefore you must "sacifice" it for this purpose). This might raise naming conflicts in existing code.

Not each and everything should be added this way, it´s about missing core features. I´m not thinking of hundreds of new key words, i´m thinking of only a few obvious and deliberately choosen words. As already said above: It´s about weighing the pros and cons, what do i get (a nice and easy to understand, BASIC-like syntax) and what must i pay for it (a few new key words).


JK
Lost Zergling
Posts: 534
Joined: Dec 02, 2011 22:51
Location: France

Re: How to add new features? (syntax, user API)

Post by Lost Zergling »

I suppose anyone could use the preprocessor to define new keywords extensions this way. If so, perhaps should be used a common naming convention using aliases, ex : pp_ARRAY (or anything else standing for every pre-processor overrides) ? Because these will actually behaves differently and have differenciated underliying meanings. Code readability and consistency of extension set as it evolves also plead in favor such requirement.
paul doe
Moderator
Posts: 1730
Joined: Jul 25, 2017 17:22
Location: Argentina

Re: How to add new features? (syntax, user API)

Post by paul doe »

Juergen Kuehlwein wrote:...
The downside is (and i see Jeff´s point here) that a macro overrides everthing. That is, you cannot use the word "ARRAY" anyhwere else in the same code (therefore you must "sacifice" it for this purpose). This might raise naming conflicts in existing code.
...
Indeed. I for example would have to make changes to existing codebases (see here), but the overall policy should be 'dialect first'. So, if a new feature is to be added to the language, it should not be hampered by these kind of issues. In this case, all I need to do is simply rename a few abstractions and that's it (an afternoon of work, maybe less). Not a big issue for me: evolve or die.
Juergen Kuehlwein wrote:...
Not each and everything should be added this way, it´s about missing core features. I´m not thinking of hundreds of new key words, i´m thinking of only a few obvious and deliberately choosen words. As already said above: It´s about weighing the pros and cons, what do i get (a nice and easy to understand, BASIC-like syntax) and what must i pay for it (a few new key words).
...
Care must be taken when considering 'obvious and deliberately choosen words', naturally, lest you end with code that resembles that of QB64. But I'd say that you go for it, and release a 'draft' version to see how it all plays out (I also think that FreeBasic is much too barebones as it stands now).
Juergen Kuehlwein
Posts: 284
Joined: Mar 07, 2018 13:59
Location: Germany

Re: How to add new features? (syntax, user API)

Post by Juergen Kuehlwein »

I made a branch containing all preprocessor changes for a more flexible macro syntax i have in mind. You can find it here https://github.com/jklwn4/fbc/tree/macro

In detail these changes are:

Macro parameter check modification:
the compiler checks the parameter count of macros and throws a compile error, if the actual number of parameters doesn´t match the definition. This check has been disabled in case of a variadic macro parameter. That is, a variadic parameter may now not only be missing at all, but also qualify as one or more parameters. A variadic parameter may have commas inside, which in effect makes it more than one parameter. A macro may now also receive less parameters than specified in it´s definition.

New macro syntax:
While a classic macro requires brackets (and therefore is only available in a function-like syntax, even if there is no return value: <macro(param_a, param_b, ...), this addition makes a sub-like macro syntax without brackets possible (<macro> param_a, param_b, ...). This works only under certain conditions: at line start, after a statement separator (":") and after THEN and ELSE in if clauses. In any other case this kind of syntax raises an error.

TYPEOF :
return the type of a variable at run time (previous version worked only in preprocessor i.e. at compile time. Might not be the most elegant solution, a workaround for a missing "variabletype" function. Returning the variable type as string has one major advantage over a "variable type derived from dtype" approach, it allows for keeping apart UDTs by their name. Accepts "()" for array parameters too.

#REDEF :
allows for temporarily re-defining (and thus overriding) an already existing #define (which was forbidden before). This new define is valid only in the current scope and is automatically restored to it´s previous definition when a scope ends. In global scope you may use #UNDEF to restore the previous #define. This allows for otherwise forbidden key words to be used in macros and elsewhere. Care must be taken using this feature!

#? :
new preprocessor operator for implementing quirk words like "any". This operator makes two comma separated parameters out of one parameter. If the original parameter contains a space (outside of a quoted string literal), the parameter is split into two parts. The first part is converted into a number, the ascii codes of the first four characters of the the first part form a LONG. The second part is passed as is.
If there is no space, "0, <parameter> " is returned. In effect leading quirk words are separated from the actual parameter and passed as an extra parameter in form of the Ascii-code of the leading four characters of the quirk word. This allows for different quirk words. Passing it as string, would bias overload resolution. Converting and passing as number, doesn´t raise problems in this regard.

"###" :
new preprocessor operator, which returns the number of arguments (count of commas + 1) in a variadic macro parameter.

"#&#" :
stringize uppercase, that is a macro parameter is returned as uppercase string - same as "#", but uppercase.


I will post examples and more explanations for all of this soon. I could supply Windows executables for beta testing these features. In general it would help, if others tested it in order to find problems or bugs i may have missed.


JK
Last edited by Juergen Kuehlwein on Nov 09, 2019 14:40, edited 1 time in total.
fxm
Moderator
Posts: 12081
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: How to add new features? (syntax, user API)

Post by fxm »

Juergen Kuehlwein wrote: TYPEOF :
return the type of a variable at run time (previous version worked only in preprocessor i.e. at compile time. Might not be the most elegant solution, a workaround for a missing "variabletype" function. Returning the variable type as string has one major advantage over a "variable type derived from dtype" approach, it allows for keeping apart UDTs by their name. Accepts "()" for array parameters too.
In case of sub-type polymorphism (inheritance polymorphism), does TYPEOF return at run-time the compiled type of the reference (its declared type at compile-time) or the real type of the referred object (its real type at run-time)?

Example:

Code: Select all

Type Parent Extends Object
End Type

Type Child Extends Parent
End Type

Dim Byref As Parent u = *New Child

Print Typeof(u)  '' prints "PARENT" or "CHILD" ?

Sleep
Delete @u
Juergen Kuehlwein
Posts: 284
Joined: Mar 07, 2018 13:59
Location: Germany

Re: How to add new features? (syntax, user API)

Post by Juergen Kuehlwein »

@fxm,

it prints "PARENT"
paul doe
Moderator
Posts: 1730
Joined: Jul 25, 2017 17:22
Location: Argentina

Re: How to add new features? (syntax, user API)

Post by paul doe »

Nice. I'd like to comment more extensively on these, but I don't have much time as of lately. Some quick impressions:
Juergen Kuehlwein wrote:...
Macro parameter check modification:
the compiler checks the parameter count of macros and throws a compile error, if the actual number of parameters doesn´t match the definition. This check has been disabled in case of a variadic macro parameter. That is, a variadic parameter may now not only be missing at all, but also qualify as one or more parameters. A variadic parameter may have commas inside, which in effect makes it more than one parameter. A macro may now also receive less parameters than specified in it´s definition.
Will have to test this to comment more extensively. Basically, the variadic parameter stood for 'one or more' before, and now it stands as 'zero or more'. Useful, indeed.
Juergen Kuehlwein wrote:...
New macro syntax:
While a classic macro requires brackets (and therefore is only available in a function-like syntax, even if there is no return value: <macro(param_a, param_b, ...), this addition makes a sub-like macro syntax without brackets possible (<macro> param_a, param_b, ...). This works only under certain conditions: at line start, after a statement separator (":") and after THEN and ELSE in if clauses. In any other case this kind of syntax raises an error.
I can't comment on this really, since I usually call subs/functions with the brackets, even if they have an empty parameter list (C/C++ style).
Juergen Kuehlwein wrote:...
TYPEOF :
return the type of a variable at run time (previous version worked only in preprocessor i.e. at compile time. Might not be the most elegant solution, a workaround for a missing "variabletype" function. Returning the variable type as string has one major advantage over a "variable type derived from dtype" approach, it allows for keeping apart UDTs by their name. Accepts "()" for array parameters too.
This is something that was long missing from the compiler. It might not be elegant, but it's incredibly useful for some efficient dynamic typecasting (and can even be open-ended, to boot!). Very, very nice.
Juergen Kuehlwein wrote:...
#REDEF :
allows for temporarily re-defining (and thus overriding) an already existing #define (which was forbidden before). This new define is valid only in the current scope and is automatically restored to it´s previous definition when a scope ends. In global scope you may use #UNDEF to restore the previous #define. This allows for otherwise forbidden key words to be used in macros and elsewhere. Care must be taken using this feature!
Now this is interesting and has come cool possibilities; albeit its scoped nature hampers its usefulness a little (within namespaces, for example).
Juergen Kuehlwein wrote:...
#? :
new preprocessor operator for implementing quirk words like "any". This operator makes two comma separated parameters out of one parameter. If the original parameter contains a space (outside of a quoted string literal), the parameter is split into two parts. The first part is converted into a number, the ascii codes of the first four characters of the the first part form a LONG. The second part is passed as is.
If there is no space, "0, " is returned. In effect leading quirk words are separated from the actual parameter and passed as an extra parameter in form of the Ascii-code of the leading four characters of the quirk word. This allows for different quirk words. Passing it as string, would bias overload resolution. Converting and passing as number, doesn´t raise problems in this regard.
I had a different kind of 'de-stringize' in mind, but might prove useful. I'll have to test it more extensively to give you a better impression.
Juergen Kuehlwein wrote:...
"###" :
new preprocessor operator, which returns the number of arguments (count of commas + 1) in a variadic macro parameter.
Now this is a really useful one. Will work nicely along the #car and#cdr macros that were provided in the wiki (nice touch, whoever wrote them).
Juergen Kuehlwein wrote:...
"#&#" :
stringize uppercase, that is a macro parameter is returned as uppercase string - same as "#", but uppercase.
Useful for manglings, indeed.

May I suggest a 'destringize' also? That is, the reverse of the # operator: it converts a symbol to a string, but the reverse operation (a string to a valid symbol) is not readily available.

Also, a stringizing that takes spaces into account (akin to #? above) but in this case, it simply concatenates the given parameter, removing spaces: const integer const ptr would be stringized as constintegerconstptr.
Juergen Kuehlwein wrote:...
I will post examples and more explanations for all of this soon. I could supply Windows executables for beta testing these features. In general it would help, if others tested it in order to find problems or bugs i may have missed.
Yes, please do. I simply can't spare the time to build it from source ATM (let alone review the code), so the build would be very useful for me to perform quick tests and feed back. Thanks for all the work, Juergen.
Juergen Kuehlwein
Posts: 284
Joined: Mar 07, 2018 13:59
Location: Germany

Re: How to add new features? (syntax, user API)

Post by Juergen Kuehlwein »

@paul doe,

thanks for your interest! I will post a link to the executables and how to use them - maybe tomorrow.
stringizing that takes spaces into account
doable, but what for?
May I suggest a 'destringize' also?
could you expand on this too. I don´t understand, what exactly you want: "(a string to a valid symbol)" to my understanding this is what "#" already does. Return a literal string from a macro parameter (which can be a symbol).
Juergen Kuehlwein
Posts: 284
Joined: Mar 07, 2018 13:59
Location: Germany

Re: How to add new features? (syntax, user API)

Post by Juergen Kuehlwein »

Sample code for the first two new macro features

Code: Select all

'variadic parameter + sub-like syntax
sub printme(s1 as string = "def_1", s2 as string = "def_2", s3 as string = "def_3", s4 as string = "def_4", s5 as string = "def_5")
  print s1, s2, s3, s4, s5
end sub


#macro test2(a,b,c...)
'#macro test2(b,c...)
'#macro test2(c...)
  printme(a,b,c)
'  printme(b,c)
'  printme(c)
#endmacro


dim s1 as string = "s1"
dim s2 as string = "s2"

test2
test2 "1 param"
test2 "2 params", s2
test2("3 params", s1, s2)
test2("4 params", s1, s2, "one more")
test2 "5 params", s1, s2, "one more", "even more")
The new parameter count check doesn´t affect macros without a variadic parameter! Now it´s possible to have macros with a parameter range from zero to maximum allowed parameters and you can still use macro operators for parameters. That is, one macro can wrap a variety of functions with different parameter counts including optional parameters with a default value.

A macro required a function-like syntax, even if there was no return value. But BASIC often allows for a sub-like syntax too. E.g "LEN" has a return value, so n = LEN(<variable>) looks natural, but "FOR" or "DIM" don´t have a return value, so "For ..." is used - "For (..." looks strange in BASIC.

Statements without a return value usually aren´t followed by brackets. So in some cases it might be useful for a macro to mimic this behavior:

- ARRAY SORT, ...
- AUDIO PLAY, <some audio file>
- VIDEO PLAY, <some video file>

definitely looks better than

- ARRAY (SORT, ...)
- AUDIO (PLAY, <some audio file>)
- VIDEO (PLAY, <some video file>)
Juergen Kuehlwein
Posts: 284
Joined: Mar 07, 2018 13:59
Location: Germany

Re: How to add new features? (syntax, user API)

Post by Juergen Kuehlwein »

#REDEF:

Code: Select all

'#redef
sub printrange(x as long, y as long)
  print x, y
end sub  


#macro test1(a)
scope
#redef to ,

  printrange(a)

end scope
#endmacro


test1 1 to 5

In this case you can make use of the (otherwise reserved) key word "TO". In general all existing key word could be re-used in this manner for generating a BASIC-like syntax. But care must be taken, improper use (e.g. not restoring after #REDEF) will cause cryptic (because not foreseeable for the compiler and therefore not properly handled) errors.
Juergen Kuehlwein
Posts: 284
Joined: Mar 07, 2018 13:59
Location: Germany

Re: How to add new features? (syntax, user API)

Post by Juergen Kuehlwein »

#?:

Code: Select all

'#? operator
function testfunc(a as string, p as long, b as string) as long

  print a, hex(p, 8), b
  return 0

end function

#macro test(a,b)
    testfunc(a, #?b)
#endmacro


test("asdf", any "qwertz")
This makes "ANY" possible just like in INSTR or TRIM. In general any other quirk word could be inserted for special purposes. Omitting "ANY" in the code sample, will set p to zero, if there is a word before the seconf parameter ("qwertz" in this case) p will contain the first four character codes of this word (could expand this to a maximum of 8 and make it LONGINT, but i think 4 will do to distinguish between differrent words). In case of less than 4 characters ("ANY") zeros are prepended.
Juergen Kuehlwein
Posts: 284
Joined: Mar 07, 2018 13:59
Location: Germany

Re: How to add new features? (syntax, user API)

Post by Juergen Kuehlwein »

#&# and ###:

Code: Select all

'###, #?#
#macro test3(a,b,c...)
    print a

  #if #b = "YES"
    print "yes"
  #else
    print "no"  
  #endif

  #if #&#b = "YES"
    print "yes"
  #else
    print "no"  
  #endif
  
  print ###c
#endmacro


test3("asdf", yes, 1,2,3)
#&# (stringize uppercase) makes (among other things) testing strings in conditional compiling expressions easier, because it´s case insensitive now.

### returns the number of parameters contained in a variadic parameter. That is, it counts the number of separating commas + 1, may be used for conditional compiling expressions too. If there is only one parameter (no comma) or the parameter is empty, "1" is returned (even an empty parameter counts as a parameter).

Inside regular code (not in conditional compiling clauses) a procedure with optional parameters could do the same with one exception: you must specify a default value, so from the value alone you cannot tell, if it´s a valid value (which by chance is equal to the default) or the default value (indicating a missing parameter). In most cases making a clever choice for the default value helps avioding ambiguities, but sometimes it´s easier to just know the number of valid parameters.
Juergen Kuehlwein
Posts: 284
Joined: Mar 07, 2018 13:59
Location: Germany

Re: How to add new features? (syntax, user API)

Post by Juergen Kuehlwein »

This is a link to a shared folder on my google drive: https://drive.google.com/open?id=1e7oP1 ... RHp0G_HQQm. The folder "Macro" contains executables for win32/64, "File" contains executables for file related changes see: viewtopic.php?f=17&t=27856. Further explanations how to use it for testing will follow.

Added:
In order to use the test version (Windows executables) for the new macro and preprocessor freatures you should do the following:
- make by all means a backup of your exeisting installation, so you can always return to what you currently have.
- copy fbc.exe (there are two versions, one in win32 folder and another one in win64 folder) to the appropriate location. That is overwrite your existing fbc.exe. Both .exes were compiled with some debugging options, so they are bigger in size.
- it shouldn´t be necessary to copy the RTL files (fbrt0.o, libfb.a and libfbmt.a) to the "lib" folder, because the macro branch only changes the compiler and doesn´t change the RTL.


If you find problems or bugs, please report here. A mimimal version of the failing code is necessary for me to reproduce the problem and fix it. Thanks!
Post Reply