Study of symbol lookups in namespaces and types

[coderJeff]

Code: Select all

Dim Shared As Integer I
Print .I          '' OK

Dim As Integer J
Print .J          '' error 8: Undefined symbol, J in 'Print .J' with fbc 1.09
                  '' OK with fbc 1.08

Thanks. That makes sense. 'J' is local in the example. There is no global instance of 'J'.

Check this out:

Code: Select all

dim shared J as integer = 1
scope
	dim J as integer = 2
	print .J      '' incorrectly is '2' in fbc-1.08, and should be '1' as in fbc-1.09
end scope

I didn't disable the single '.' or repurpose it for some other scope lookup.

[fxm]

To sum up, the unqualified name look-ups for future fbc 1.09 version should verify the priority hierarchy as follows:
- (1) current namespace/type
- (2) base types (by 'Extends'), ranked from closest to furthest in inheritance hierarchy
- (3) parent namespaces, ranked from closest to furthest in ancestry hierarchy (including global namespace: always furthest in hierarchy)
- (4) imported namespaces (by 'Using'), without hierarchy between them

I pushed changes to fbc/master that fix #948 Overriding class members doesn't work inside methods of further-extended subclasses

I added quite a few tests to the test suite. It was kind of tricky at first to test namespaces directly in the test-suite because the framework is based on ... namespaces. :)

I have prepared my own fairly comprehensive suite of test programs (18) to verify the new priority hierarchy as defined above.
I will be back as soon as I test this with the new build of fbc, probably tomorrow.
(cross-checking of tests by several people is always safer)

[fxm]

Tested with the last build (2021-12-30) from St_W (https://users.freebasic-portal.de/stw/builds/)

Improvements, but still some faults about priority rules with unqualified names.

Unqualified symbol look-ups goal:
- (1) current namespace/type
- (2) base types (by 'Extends'), ranked from closest to furthest in inheritance hierarchy
- (3) parent namespaces, ranked from closest to furthest in ancestry hierarchy (including global namespace: always furthest in hierarchy)
- (4) imported namespaces (by 'Using'), without hierarchy between them

Faults observed about priority with unqualified names:
- From inside a type, (2) does not always take precedence over (3): (2) takes precedence over global namespace only.
- From inside a type, (3) does not always take precedence over (4): global namespace does not take precedence over (4). By cons, from inside a namespace, (3) always take precedence over (4).

In summary, using unqualified name:
- From inside a namespace, no more faults.
- From inside a type, 2 cases of fault as noted above:

• - parent namespaces (except global namespace) are in second priority (instead of third),
  - global namespace is in last priority (instead of the end of the third).

2 small parts taken from my tests, which fail when accessing unqualified variable name from inside type only:

• a)

  Code: Select all

  Dim Shared As Zstring * 32 duplicate = "   ..duplicate"
  
  Namespace M
      Dim As Zstring *32 duplicate = "   M.duplicate"
  End Namespace
  
  Namespace N
      Dim As Zstring * 32 duplicate = "   N.duplicate"
      Type Parent Extends Object
          Dim As Zstring * 32 duplicate = "   N.Parent.duplicate"
      End Type
      Type Child Extends Parent
          Declare Sub test()
      End Type
  End Namespace
  
  Sub N.Child.test()
      Using M
      Print duplicate
  End Sub
  
  Print "From Type:"
  Dim As N.Child c
  c.test()          '' expected: "N.Parent.duplicate"
  Print
  
  Sleep
  

  b)

  Code: Select all

  Dim Shared As Zstring * 32 duplicate = "   ..duplicate"
  
  Namespace M
      Dim As Zstring *32 duplicate = "   M.duplicate"
  End Namespace
  
  Namespace N
  '    Dim As Zstring * 32 duplicate = "   N.duplicate"
      Type Parent Extends Object
  '        Dim As Zstring * 32 duplicate = "   N.Parent.duplicate"
      End Type
      Type Child Extends Parent
          Declare Sub test()
      End Type
  End Namespace
  
  Sub N.Child.test()
      Using M
      Print duplicate
  End Sub
  
  Print "From Type:"
  Dim As N.Child c
  c.test()          '' expected: "..duplicate"
  Print
  
  Sleep
  

  or simpler:

  Code: Select all

  Dim Shared As Zstring * 32 duplicate = "   ..duplicate"
  
  Namespace M
      Dim As Zstring *32 duplicate = "   M.duplicate"
  End Namespace
  
  Type UDT Extends Object
      Declare Sub test()
  End Type
  
  Sub UDT.test()
      Using M
      Print duplicate
  End Sub
  
  Print "From Type:"
  Dim As UDT u
  u.test()          '' expected: "..duplicate"
  Print
  
  Sleep
  

  other fault maybe correlated with previous:

  Code: Select all

  Dim Shared As Zstring * 32 duplicate = "   ..duplicate"
  
  Namespace M
      Dim As Zstring *32 duplicate = "   M.duplicate"
  End Namespace
  
  Type UDT Extends Object
      Declare Sub test()
  End Type
  
  Sub UDT.test()
      Using M
      Print This.duplicate
  End Sub
  
  Print "From Type:"
  Dim As UDT u
  u.test()          '' expected: Element not defined
  Print
  
  Sleep
  

  and this other fault maybe always correlated:

  Code: Select all

  Sub duplicate()
      Print "   ..duplicate"
  End Sub
  
  Namespace M
      Sub duplicate()
          Print "   M.duplicate"
      End Sub
  End Namespace
  
  Type UDT Extends Object
      Declare Sub test()
  End Type
  
  Sub UDT.test()
      Using M
      This.duplicate()
  End Sub
  
  Print "From Type:"
  Dim As UDT u
  u.test()          '' expected: Element not defined
  Print
  
  Sleep
  

[coderJeff]

- From inside a type, 2 cases of fault as noted above:

• - parent namespaces (except global namespace) are in second priority (instead of third),
  - global namespace is in last priority (instead of the end of the third).

I believe I have this figured out. So that the 5 test cases in your last post behave as expected.

ENUMs will need more investigation which I'd like to defer until after more testing of the types. And I'm going to push on with a release.

[fxm]

Tested with the latest build (2021-12-31) from St_W (https://users.freebasic-portal.de/stw/builds/)

JEFF WELL DONE, AND HAPPY NEW YEAR EVERYONE :-)

All of my 18 test programs, to verify the new priority hierarchy, run successfully to check:

Unqualified name look-ups for future fbc 1.09 version should verify the priority hierarchy as follows:
- (1) current namespace/type
- (2) base types (by 'Extends'), ranked from closest to furthest in inheritance hierarchy
- (3) parent namespaces, ranked from closest to furthest in ancestry hierarchy (including global namespace: always furthest in hierarchy)
- (4) imported namespaces (by 'Using'), without hierarchy between them

Tests covering:
- from inside a type with inheritance hierarchy structure
- from inside a namespace with ancestry hierarchy structure
to access qualified/unqualified names like:
- variable names
- procedure names
- type names
(enum structure not yet tested)

[TJF]

ENUMs will need more investigation which I'd like to defer until after more testing of the types.

Just a propsal:

What about handling ENUMs internally as constants?

Code: Select all

ENUM
  G0
  G1
  G2 = 7
END ENUM

gets internally

Code: Select all

CONST AS INTEGER _
    G0 = 0 _
  , G1 = 1 _
  , G2 = 7 

and

Code: Select all

ENUM Named
  N0
  N1
  N2 = 7
END ENUM

gets internally

Code: Select all

TYPE Named_t
  AS CONST INTEGER _
    N0 = 0 _
  , N1 = 1 _
  , N2 = 7
END TYPE
DIM SHARED AS Named_t Named

Namespaces should already working for variables/types -> no further investigation.

We could get rid of the EXPLICIT keyword (named ENUMs are always explizit). Just ignore it in existing code.

And it should be easy to implement a new feature: typed ENUMs like

Code: Select all

ENUM AS BYTE
  T0
  T1
  T2 = 7
END ENUM

[fxm]

First observations on present ENUM structures and ambiguity resolution

Code: Select all

Enum [enumname [ Explicit ] ]

    symbolname [= expression] [, ...]
    ...

End Enum

Ambiguity resolution doesn't just work on the 'symbolname' alone.
Ambiguity resolution works on 'enumename' (or 'enumename.symbolname').

Unnamed ENUMs cannot therefore be used in the event of ambiguity. Only named ENUMs can be used.
Thus, the 'Explicit' qualifier has no effect on the ambiguity resolution behavior.

Otherwise, the enum name appears to check the same ambiguity precedence rules as for a type name.

Examples:

Code: Select all

' current namespace

Enum duplicate explicit
    enum1 = 1
End Enum

Namespace M
    Enum duplicate
        enum1 = 2
    End Enum
End namespace

Namespace N
    Enum duplicate explicit
        enum1 = 3
    End Enum
    Namespace P
        Enum duplicate
            enum1 = 4
        End Enum
        Sub test()
            Using M
'            Print enum1            '' Ambiguous symbol access, explicit scope resolution required
            #if typeof(duplicate) = "N.P.DUPLICATE"
                Print "N.P.duplicate"
            #endif
            #if typeof(duplicate) = "N.DUPLICATE"
                Print "N.duplicate"
            #endif
            #if typeof(duplicate) = "DUPLICATE"
                Print "..duplicate"
            #endif
            #if typeof(duplicate) = "M.DUPLICATE"
                Print "M.duplicate"
            #endif
            Print duplicate.enum1
        End Sub
    End Namespace
End Namespace

N.P.test()

Sleep

Code: Select all

' parent namespace

Enum duplicate explicit
    enum1 = 1
End Enum

Namespace M
    Enum duplicate
        enum1 = 2
    End Enum
End namespace

Namespace N
    Enum duplicate explicit
        enum1 = 3
    End Enum
    Namespace P
'        Enum duplicate
'            enum1 = 4
'        End Enum
        Sub test()
            Using M
'            Print enum1            '' Ambiguous symbol access, explicit scope resolution required
            #if typeof(duplicate) = "N.P.DUPLICATE"
                Print "N.P.duplicate"
            #endif
            #if typeof(duplicate) = "N.DUPLICATE"
                Print "N.duplicate"
            #endif
            #if typeof(duplicate) = "DUPLICATE"
                Print "..duplicate"
            #endif
            #if typeof(duplicate) = "M.DUPLICATE"
                Print "M.duplicate"
            #endif
            Print duplicate.enum1
        End Sub
    End Namespace
End Namespace

N.P.test()

Sleep

Code: Select all

' global namespace

Enum duplicate explicit
    enum1 = 1
End Enum

Namespace M
    Enum duplicate
        enum1 = 2
    End Enum
End namespace

Namespace N
'    Enum duplicate explicit
'        enum1 = 3
'    End Enum
    Namespace P
'        Enum duplicate
'            enum1 = 4
'        End Enum
        Sub test()
            Using M
'            Print enum1            '' Ambiguous symbol access, explicit scope resolution required
            #if typeof(duplicate) = "N.P.DUPLICATE"
                Print "N.P.duplicate"
            #endif
            #if typeof(duplicate) = "N.DUPLICATE"
                Print "N.duplicate"
            #endif
            #if typeof(duplicate) = "DUPLICATE"
                Print "..duplicate"
            #endif
            #if typeof(duplicate) = "M.DUPLICATE"
                Print "M.duplicate"
            #endif
            Print duplicate.enum1
        End Sub
    End Namespace
End Namespace

N.P.test()

Sleep

Code: Select all

' imported namespace

'Enum duplicate explicit
'    enum1 = 1
'End Enum

Namespace M
    Enum duplicate
        enum1 = 2
    End Enum
End namespace

Namespace N
'    Enum duplicate explicit
'        enum1 = 3
'    End Enum
    Namespace P
'        Enum duplicate
'            enum1 = 4
'        End Enum
        Sub test()
            Using M
'            Print enum1            '' Ambiguous symbol access, explicit scope resolution required
            #if typeof(duplicate) = "N.P.DUPLICATE"
                Print "N.P.duplicate"
            #endif
            #if typeof(duplicate) = "N.DUPLICATE"
                Print "N.duplicate"
            #endif
            #if typeof(duplicate) = "DUPLICATE"
                Print "..duplicate"
            #endif
            #if typeof(duplicate) = "M.DUPLICATE"
                Print "M.duplicate"
            #endif
            Print duplicate.enum1
        End Sub
    End Namespace
End Namespace

N.P.test()

Sleep

[coderJeff]

All of my 18 test programs, to verify the new priority hierarchy, run successfully to check:

Nice! I could not have done it without you. Thank-you, fxm.

[fxm]

Nice! I could not have done it without you. Thank-you, fxm.

It is always joy and gratification to be able to serve something to a developer that is far beyond my skills.

[fxm]

First observations on present ENUM structures and ambiguity resolution

Code: Select all

Enum [enumname [ Explicit ] ]

    symbolname [= expression] [, ...]
    ...

End Enum

Ambiguity resolution doesn't just work on the 'symbolname' alone.
Ambiguity resolution works on 'enumename' (or 'enumename.symbolname').

Unnamed ENUMs cannot therefore be used in the event of ambiguity. Only named ENUMs can be used.
Thus, the 'Explicit' qualifier has no effect on the ambiguity resolution behavior.

Otherwise, the enum name appears to check the same ambiguity precedence rules as for a type name.

In same way as for type structures, I added enum structures in my 18 test programs and everything looks OK compared to the behavior presumed above.

[fxm]

Name look-ups in namespaces and types

Unqualified name look-ups verify the priority hierarchy as follows:
- (1) current namespace/type
- (2) base types (by 'Extends'), ranked from closest to furthest in inheritance hierarchy
- (3) parent namespaces, ranked from closest to furthest in ancestry hierarchy (including global namespace: always furthest in hierarchy)
- (4) imported namespaces (by 'Using'), without hierarchy between them (regardless of their respective level of nesting)

Unqualified names for look-up can be:
- variable names
- procedure names
- type names
- enum names (not their field symbols alone)

a)
If a qualified name with a prefix referring to a namespace is used instead of an unqualified name, the previous rules for unqualified names are amended as follows:
- the look-up begins at the level of the namespace corresponding to the specified prefix (as in (1))
- failing that, then only namespaces imported into this namespace can be candidates for the look-up (as in (4))
- but never the parent namespaces of this namespace nor also the global namespace (not as in (3))

b)
If a qualified name with a prefix ('This' or 'Base') referring to a type is used instead of an unqualified name, the previous rules for unqualified names are amended as follows:
- the look-up begins at the level of the type or base type depending on the specified prefix 'This' or 'Base' (similarly to (1))
- failing that, then only higher base types from closest to furthest can be successively candidates for the look-up (as in (2))
- but never any namespaces, the current also excluded (not as in (3) or (4))

Examples for a):

Code: Select all

Dim Shared As Zstring * 32 duplicate = "..duplicate"

Namespace M
    Dim As Zstring * 32 duplicate = "M.duplicate"
    Namespace P
    End namespace
End Namespace

Namespace N
    Using M
End Namespace

Print M.duplicate     '' "M.duolicate"
Print N.duplicate     '' "M.duplicate"
'Print M.P.duplicate  '' error

Code: Select all

Dim Shared As Zstring * 32 duplicate = "..duplicate"

Namespace M
'    Dim As Zstring * 32 duplicate = "M.duplicate"
    Namespace P
    End namespace
End Namespace

Namespace N
    Using M
End Namespace

'Print M.duplicate    '' error
'Print N.duplicate    '' error
'Print M.P.duplicate  '' error

Examples for b):

Code: Select all

Dim Shared As Zstring * 32 duplicate = "..duplicate"

Namespace M
    Dim As Zstring *32 duplicate = "M.duplicate"
End Namespace

Namespace N
    Dim As Zstring * 32 duplicate = "N.duplicate"
    Type Parent Extends Object
        Dim As Zstring * 32 duplicate = "N.Parent.duplicate"
    End Type
    Type Child Extends Parent
        Dim As Zstring * 32 duplicate = "N.Child.duplicate"
    End Type
    Type GrandChild Extends Child
        Dim As Zstring * 32 duplicate = "N.GrandChild.duplicate"
        Declare Sub test()
    End Type
End Namespace

Sub N.GrandChild.test()
    Using M
    Print Base.duplicate
End Sub

Dim As N.GrandChild gc
gc.test()                '' "N.Child.duplicate"

Code: Select all

Dim Shared As Zstring * 32 duplicate = "..duplicate"

Namespace M
    Dim As Zstring *32 duplicate = "M.duplicate"
End Namespace

Namespace N
    Dim As Zstring * 32 duplicate = "N.duplicate"
    Type Parent Extends Object
        Dim As Zstring * 32 duplicate = "N.Parent.duplicate"
    End Type
    Type Child Extends Parent
'        Dim As Zstring * 32 duplicate = "N.Child.duplicate"
    End Type
    Type GrandChild Extends Child
        Dim As Zstring * 32 duplicate = "N.GrandChild.duplicate"
        Declare Sub test()
    End Type
End Namespace

Sub N.GrandChild.test()
    Using M
    Print Base.duplicate
End Sub

Dim As N.GrandChild gc
gc.test()                '' "N.Parent.duplicate"

Code: Select all

Dim Shared As Zstring * 32 duplicate = "..duplicate"

Namespace M
    Dim As Zstring *32 duplicate = "M.duplicate"
End Namespace

Namespace N
    Dim As Zstring * 32 duplicate = "N.duplicate"
    Type Parent Extends Object
'        Dim As Zstring * 32 duplicate = "N.Parent.duplicate"
    End Type
    Type Child Extends Parent
'        Dim As Zstring * 32 duplicate = "N.Child.duplicate"
    End Type
    Type GrandChild Extends Child
        Dim As Zstring * 32 duplicate = "N.GrandChild.duplicate"
        Declare Sub test()
    End Type
End Namespace

Sub N.GrandChild.test()
    Using M
    Print Base.duplicate
End Sub

Dim As N.GrandChild gc
gc.test()                '' error

[fxm]

Name look-ups in namespaces and types

Names can be unqualified or qualified.
An unqualified name is a name that does not have a prefix to specify what scope it comes from.
A qualified name includes a prefix to clarify the interpretive scope by overriding the default scope.

Scope priority hierarchy

Unqualified name look-ups verify the priority hierarchy as follows:

• [1] current namespace/type
• [2] base types (by 'Extends'), ranked from closest to furthest in inheritance hierarchy
• [3] parent namespaces, ranked from closest to furthest in ancestry hierarchy (including global namespace: always furthest in hierarchy)
• [4] imported namespaces (by 'Using'), without hierarchy between them (regardless of their respective level of nesting)
  
  Unqualified names for look-up can be:
  • variable names
  • procedure names
  • type/union names
  • enum names (not their field symbols alone)

If a qualified name with a prefix referring to a namespace is used instead of an unqualified name, the previous rules for unqualified names are amended as follows:

• the look-up begins at the level of the namespace corresponding to the specified prefix (as in [1])
• failing that, then only namespaces imported into this namespace can be candidates for the look-up (as in [4])
• but never the parent namespaces of this namespace nor also the global namespace (not as in [3])

If a qualified name with a prefix ('object-name'/'This' or 'Base') referring to a type is used instead of an unqualified name, the previous rules for unqualified names are amended as follows:

• the look-up begins at the level of the type or base type depending on the specified prefix 'object-name'/'This' or 'Base' (similarly to [1])
• failing that, then only higher base types from closest to furthest can be successively candidates for the look-up (as in [2])
• but never any namespaces, the current also excluded (not as in [3] or [4])

Full process

The full process simplifies the problem by dealing with it in two successive phases:

• using the scope priority hierarchy rules above, selection of a single scope among all those which are accessible by only taking into account the name for look-up, regardless of the signature supplied by the caller (if at least two scopes containing this name have equivalent accessibility priority, then the process is stopped with an ambiguity status)
• within the single scope retained if it exists, one final overload resolution taking into account the full signature is carried out (if there is no compatibility with the declared signature(s), the process is stopped without testing another accessible scope)

The problem is thus simplified to one single full overload resolution within a same scope (instead of a full overload resolution within each scope candidate, and then the difficult choice of the best result among the scope candidates).

The coding of a very optimized resolution process (the best of human intelligence) for a procedure name is cumbersome, because having to take into account from the start the complete signature of the called procedure (name of the procedure + type and number of parameters + calling convention + return type if it exists).
In addition, it must be taken into account that the signature deduced from the call may not necessarily be identical but just compatible with certain declared ones in the scope candidates.

Version:

• Since fbc 1.09.0.
• Before fbc 1.09.0, the priority hierarchy was very poorly managed with a lot of inconsistencies.

Do you think it would be interesting to have this kind of description (from this summarizing sheet above) with some enlightening examples in a new page of the 'Programmer's Guide':
'Name look-ups in namespaces and types', in the 'Other Topics' section for example ?

[coderJeff]

Do you think it would be interesting to have this kind of description (from this summarizing sheet above) with some enlightening examples in a new page of the 'Programmer's Guide':
'Name look-ups in namespaces and types', in the 'Other Topics' section for example ?

I think it is a very fine idea and a great starting point to expand upon. Your composition is an accurate reference.

Several wiki pages make use of the term "identifier" to have the same meaning as "name". Perhaps can use "identifier" throughout or simply have an explanation that "name" / "identifier" / "symbol" are used somewhat interchangeably. Potentially operators too? Operators have different rules from everything else (for now).

From Identifier Rules: An identifier is a symbolic name which uniquely identifies a variable, Type, Union, Enum, Function, Sub, or Property, within its scope or Namespace. (to be added? namespaces themselves also are identified by a name/identifier

Qualified versus unqualified is probably best explained by example.

[fxm]

Do you think it would be interesting to have this kind of description (from this summarizing sheet above) with some enlightening examples in a new page of the 'Programmer's Guide':
'Name look-ups in namespaces and types', in the 'Other Topics' section for example ?

Potentially operators too? Operators have different rules from everything else (for now).

- Unqualified function names are looked-up in the scopes considered by the 'unqualified name look-up' process described in my above post.
- Operator names are looked-up in the namespaces of their arguments in addition to the scopes considered by the 'unqualified name look-up' process: Argument-Dependent Look-up (ADL) => much more elaborate process.

Simple example 'absolute value', with only one argument for the function and the operator:

Code: Select all

Type A Extends Object
End Type
Operator Abs(Byref a0 As A) As Integer
    Return 2
End Operator

Function _abs(Byref a0 As A) As Integer
    Return 4
End Function

Namespace N
    Type B Extends Object
    End Type
    Operator Abs(Byref b0 As B) As Integer
        Return 1
    End Operator
    
    Function _abs(Byref b0 As B) As Integer
        Return 3
    End Function
    
    Sub test()
        Dim As B b0
        Dim As A a0
        Print Abs(b0)    '' ok: 1
        Print Abs(a0)    '' ok: 2
        Print _abs(b0)   '' ok: 3
        'Print _abs(a0)  '' error
    End Sub
End Namespace

N.test()

Sleep

[coderJeff]

I think you will find some odd behaviours when it comes to operators. They are defined in the global namespace even if the declaration is in a namespace. Which can be confusing. I believe there is improvements needed for operators.

Code: Select all

type T
	__ as integer
end type

namespace M
	declare operator ABS( byref as T ) as T
end namespace

namespace N
	'' uncomment for duplicate definition
	'' declare operator ABS( byref as T ) as T
end namespace

dim as T x, y
x = abs(y) '' this is found, because ABS(T) is actually global