GUI design

General FreeBASIC programming questions.
Psyche_Demon
Posts: 46
Joined: Jul 30, 2015 8:07
Location: England
Contact:

GUI design

Post by Psyche_Demon »

Hi there.
Interesting question for you.
I use screen reading software to access information on the computer, and the keyboard as my only means of input, as I have no sight whatsoever.
Consequently, as you can imagine, this sets me back a heck of a long way when making programs with GUI's.
Back in the days when I used to use Visual Basic 6 I was able to press a key to bring up a list of controls, use the cursor keys to navigate that list and place it on the form with enter. Then I could use the arrow keys to move between different controls, ctrl+arrow keys to reposition the selected control and shift+arrow keys to resize it. It would then give me all the coordinates. Using basic arithmetic I was then able to build the GUI so that controls at least didn't overlap. That helped the screen reader to read the label, state and content of the control without any difficulty. However I still had no idea whether it was visually readable or appealing.
I am now faced with an even more difficult issue. I am now using FreeBASIC, of course. The existing GUI builders seem to have trouble with screen readers (Perhaps because it tries to show your window over the builder window?), and so I would more or less have to guess what the coordinates should be - tricky as these are in pixels rather than inches or centimetres etc.
Knowing nothing about pixels, or any kind of manual GUI programming for that matter, I'm struggling slightly. And to be honest I don't want to be confined to using the console when a GUI is usually much nicer all round (depending on the software, that is).
I was thus thinking, would it be possible to create a GUI builder that had the possibility of using the keyboard in some way (preferably the method explained above as that seems the most logical to me), and even more importantly, use some form of audible signal to represent where the control is on the screen and how big it is?
Of course the audio doesn't have to specify detail (such as colour, control type, text, font etc), but position, size, and space between one control and the next. I'm thinking the X axis could be represented by panning, the Y axis could be represented by pitch, control size as a pan/pitch sweep etc. Control spacing, I'm not sure about how that would work.
These are just ideas. I would do it myself but having no knowledge of how things should look would make that rather a complicated task. Perhaps it can be incorporated into one of the existing GUI builders?
I'd be very interested to hear any thoughts or plans on this, as I think it could be a breakthrough for blind programmers looking to create GUI's. Of course some sighted assistance would still be required, but I would rather build a GUI myself and then ask a sighted person, "How does this look to you and how can I improve it?" rather than having to spend unnecessary time and money in hiring someone to do it for me for every project I wanted to create.
Cheers.
St_W
Posts: 1619
Joined: Feb 11, 2009 14:24
Location: Austria
Contact:

Re: GUI design

Post by St_W »

Hi,

I don't think FreeBasic is the best choice for you as of now as I don't expect such a GUI editor to come anytime soon. The thing is that FreeBasic is basically just a compiler and not a full-featured programming environment including IDE and tools, like VisualBasic or QuickBasic were. IDEs or GUI designers are developed as separate projects, and although there are some, there is not even a single one that I would consider nearly as comfortable and user-friendly as the one of VB6. Thus I doubt that there will be one that additionally provides help for visually impaired people.

So I would suggest to either continue using VB6, which is still supported on the latest Windows versions (the generated applications, not the IDE itself), or try to switch to VB6's successor Visual Basic .NET. Unfortunately I can't tell how well that works with screen readers, but Microsoft's software usually works quite well with assistive technology (at least compared to most third-party software).

Unfortunately FreeBasic simply doesn't have a decent IDE and GUI designer at this time and I don't see anything to come in the near future.
fxm
Moderator
Posts: 12082
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: GUI design

Post by fxm »

Psyche_Demon
Posts: 46
Joined: Jul 30, 2015 8:07
Location: England
Contact:

Re: GUI design

Post by Psyche_Demon »

Hi there,
Unfortunately Visual Studio .net is not accessible, at least to my screen reader.
Even if I were to use another language. The fact is that none of them have my ideas and concepts (I.E. optional audio representation of control sizes when building etc).
I am aware that FreeBASIC is a compiler and not an IDE but I am also aware that GUI builders and IDEs exist for it. I am also aware that the current builders aren't exactly screen reader friendly. That's why I'm suggesting a new tool designed to address this issue, not just for me but also for other blind programmers who might wish to program in FreeBASIC.
I am aware that it won't happen immediately. It will take time, as well as somebody being willing to put in the effort with both the design and implementation of the system. At this point in time I'm merely putting out an idea and asking how feasible it would be to build, and to establish whether anybody here would be willing to build it or whether I'll have to hire someone.
Cheers.
marcov
Posts: 3455
Joined: Jun 16, 2005 9:45
Location: Netherlands
Contact:

Re: GUI design

Post by marcov »

There are one or two screenreader users for Lazarus (http://lazarus.freepascal.org), and occasionally improvements are made in that area. Not all controls are winapi though, like the property grid, some are owner drawn, which afaik is one of the problems with screenreaders. As for audio representation, it is open source..

Alternately, GTK based applications and IDEs have a fairly good reputation for all kinds of assistive technologies. It might be worthwhile to see if these translate well to Windows, or are *nix only. Modifying for own purposes is more complex though. TJF used to advocate GTK in combination with FB, maybe he can tell more about the practicalities.
OSchmidt
Posts: 49
Joined: Jan 01, 2016 12:27
Contact:

Re: GUI design

Post by OSchmidt »

Psyche_Demon wrote:At this point in time I'm merely putting out an idea and asking how feasible it would be to build, and to establish whether anybody here would be willing to build it or whether I'll have to hire someone.
Not directly related to FreeBasic (although it would work also within FreeBasic with a few "COM-helpers").

But what about "building your own" in a "keep it simple"-manner?

A GUI-Builder for visually impaired people would IMO be best addressed when Widget-definitions start out "text-based"
(requiring support from the Text-Editor of course... not sure what "Notepad.exe" or the more powerful "NotePad++"
or "Programmers Notepad" have to offer in this regard).

What I mean is, that most GUI-Frameworks allow to create Controls/Widgets per Code-Line - as e.g. the
Set MyCtl = Controls.Add(...) call in VB6..., followed by a MyCtl.Move Left, Top, Width, Height

Here's the text for a *.vbs example which is using "vbRichClient5" as the GUI-Framework (a CairoWrapper-COM-lib
which can be used for GUI-development even per VBScript) - showing a Form, after creating a simple Button on it:
(the following requires a download and COM-registration per regsvr32 of vbRichClient5.dll and vbWidgets.dll)
Here's a zip which includes both COM-libs: http://vbRichClient.com/Downloads/vbRC5_vbWidgets.zip

Code: Select all

Set New_c   = WScript.CreateObject("vbRichClient5.cConstructor") 'create the vbRichClient5 Main-Constructor-Object
Set btnTest = WScript.CreateObject("vbWidgets.cwButton", "btnTest_") 'create a Button-Obj (which is not yet sited)
    btnTest.Caption = "Hello" 'give the Button a Caption
    
Set Form = New_c.Cairo.WidgetForms.Create(2, "Form-Caption", , 640, 480) 'create the Form-Obj (initial size 640x480)
	Form.Widgets.Add btnTest, "btnTest", 20, 20, 100, 30 'site the btnTest-Object we created earlier above
	Form.Show 1 'show the Form modally

Sub btnTest_Click() 'just to show that the VBS-EventBinding (second param in WScript.CreateObject) does work
    MsgBox "Hello World"
End Sub
That's it already with regards to minimum-code for a working GUI-example, which will give you already
"textual feedback" about the initial creation-size of the Form and the Button, the latter sited with:
Form.Widgets.Add btnTest, "btnTest", 20, 20, 100, 30

I'd imagine, that you would like to have feedback also, *after* the above Form was started (at least in "development-mode") -
e.g. when entering an area a Widget currently occupies on that Form with the Mouse... The Demo above could be
easily enhanced, to e.g. give the Left, Top, Width, Height of the "current Widget under cursor", along with its name
as output over the speech-API (quite easy to trigger from an available "centralized" MouseEnter-Event, which bubbles
up for all Widgets on a Form generically).
Another available "centralized Event" (instead of "MouseEnter") which could give that "speech-feedback" would be GotFocus.

Not sure whether something like that would help - but I could develop a little more enhanced Demo which shows that speech-output -
either for VB6 or for VBScript (with a bit more effort even for FreeBasic) if you like.

Olaf
OSchmidt
Posts: 49
Joined: Jan 01, 2016 12:27
Contact:

Re: GUI design

Post by OSchmidt »

Here's the little VBScript again, enhanced about what I meant with:
"Incorporating the MS-speech-API from centrally handled Events"
(reporting in the bubbling GotFocus-Event for both, the Button and the Text-Widget -
and in the bubbling Change-Event the current content of the currently focused Text-Widget on that Form)

Code: Select all

With CreateObject("WScript.Shell") 'this block auto-reshells a script to the 32Bit-WSH-version on Win64 (if needed)
	WS32on64 = .ExpandEnvironmentStrings("%SYSTEMROOT%\SysWOW64\wscript.exe")
	If StrComp(WScript.FullName, WS32on64, vbTextCompare) then 'So, when the Paths' differ, we...
	   If CreateObject("Scripting.FileSystemObject").FileExists(WS32on64) Then '...check if the target is there at all...
		 .Run """" & WS32on64 & """ """ & WScript.ScriptFullName & """", 1, False '...and re-launch the script to it!
		 WScript.Quit
	   End If
	End If
End With
 
'-------------- VBScript-Code starts here ----------------

Set Sp = Wscript.CreateObject("SAPI.SpVoice") 'create the Speech-API-HelperObject
Set New_c   = WScript.CreateObject("vbRichClient5.cConstructor") 'create the VbRichClient Main-Constructor-Object

Set btnTest = WScript.CreateObject("vbWidgets.cwButton", "btnTest_") 'create a Button-Widget (which is not yet sited)
    btnTest.Caption = "Hello" 'give the Button a Caption
Set txtTest = WScript.CreateObject("vbWidgets.cwTextBox", "txtTest_") 'create a Text-Widget (which is not yet sited)
    txtTest.Text = "The Text" 'give the TextBox some Input
    
Set Form = New_c.Cairo.WidgetForms.Create(2, "Form-Caption", , 640, 480) 'create the Form-Obj (initial size 640x480)
    Form.Widgets.Add btnTest, "Button1", 10, 10, 100, 30 'site the btnTest-WidgetObject we created earlier above
    Form.Widgets.Add txtTest, "Text1", 10, 45, 100, 25 'site the txtTest-WidgetObject we created earlier above
    
    WScript.ConnectObject Form, "Form_" '<- EventSink-Prefixing/Binding (in case an Object was instantiated not per WScript.CreateObject)
    Form.Show 1 'show the Form modally

Sub btnTest_Click() 'just to show that the VBS-EventBinding (second param in WScript.CreateObject) does work
    Speak "Hello World!"
    MsgBox "Hello World!"
End Sub

Sub Form_BubblingEvent(Sender, EventName, P1, P2, P3, P4, P5, P6, P7)
    Set W = Sender.Widget
    If EventName = "W_GotFocus" Then
        Select Case TypeName(Sender) 
          Case "cwButton":  Speak W.Key & ": position: x=" & W.Left & ", y=" & W.Top
          Case "cwTextBox": Speak W.Key & ": content: " & Sender.Text
        End Select
    ElseIf EventName = "Change" Then 'all TextBoxes report their changed content here
        Speak W.Key & "-content-change: " & Sender.Text
    End If
End Sub

Sub Speak(Text)
    Sp.Speak Text, 3 '3 = (1 OR 2) ... = SVSFlagsAsync OR SVSFPurgeBeforeSpeak 
End Sub
Olaf
Last edited by OSchmidt on Dec 24, 2016 9:45, edited 1 time in total.
Psyche_Demon
Posts: 46
Joined: Jul 30, 2015 8:07
Location: England
Contact:

Re: GUI design

Post by Psyche_Demon »

Hi there Olaf,
Again though. The problem I will have there is, A, knowing what the numbers represent. In the Windows API they represent pixels. In VB6 they can represent either pixels, inches, centimetres etc.
Once I have more of an idea about what all the numbers mean, I can then create a GUI that works with a screen reader. Great. But what I then don't know is how it would look from a sighted point of view. Do any controls overlap? Do they fit the screen? Are the controls big enough to see the text clearly?
Anything can represent numbers textually. It's knowing what those numbers mean to a sighted eye rather than trying to guess. Since I can't use my eyes, my other senses have to be sharper in order to perceive the world around me.
I used to make a lot of audio games for the blind. Where you might see a character walking across the screen, I would make a character walk across the stereo field using footsteps. Just as an object has its own image in a video game, so does it have its own sound in its audio equivalent. Usually, whatever can be represented visually can, in most cases, be represented in audio, even if the audio solution is a lot simpler and perhaps less realistic (such as a door constantly playing a handle rattling sound so that its position can be established), hence the reason I wondered whether it would be possible to make a GUI builder, representing a GUI in the same way.
BasicCoder2
Posts: 3906
Joined: Jan 01, 2009 7:03
Location: Australia

Re: GUI design

Post by BasicCoder2 »

Psyche_Demon wrote:Hi there.
Interesting question for you.
I use screen reading software to access information on the computer, and the keyboard as my only means of input, as I have no sight whatsoever.
Consequently, as you can imagine, this sets me back a heck of a long way when making programs with GUI's.
May I ask what screen reading software you use?
I would be interested in trying it if free to get an idea of what is involved.
Can you move the mouse over the screen and get audio feedback as to the object under it?
Buttons change color or their text when the mouse is over them. Can you get audio feedback such as mouse over a close button?
When the mouse is over the edge or corner of a window the cursor image changes so may audio feedback whenever the cursor image changes?
.
Psyche_Demon
Posts: 46
Joined: Jul 30, 2015 8:07
Location: England
Contact:

Re: GUI design

Post by Psyche_Demon »

Hi there,
I use a free, open source screen reader called NVDA.
NVDA does seem to have audible feedback when moving the mouse. However being blind since birth, and NVDA being nonexistent even in idea in my days of learning to use a computer, I have never used a mouse in my life. Apparently there are so many more things that a mouse can access that a keyboard can't, knowing exactly what they do, where they go and whether or not they are clickable, not to mention how much the mouse needs to be moved etc is something completely beyond me.
What I have done though is emailed the NVDA mailing list as well as this forum to find out if perhaps such a feature might already be present in NVDA that I am perhaps unaware of. To be honest, that would be even better than a GUI builder since then any blind person could technically program in any language and know exactly what the status of their GUI was.
Cheers.
greenink
Posts: 200
Joined: Jan 28, 2016 15:45

Re: GUI design

Post by greenink »

Tcl/Tk could be an option.
https://www.tcl.tk/software/tcltk/
BasicCoder2
Posts: 3906
Joined: Jan 01, 2009 7:03
Location: Australia

Re: GUI design

Post by BasicCoder2 »

When I closed my eyes the first thing of thought was how could I use the mouse to feel the screen. Even tactile feedback as a vibration would be possible via another usb port as the mouse moves over objects. My other interest happens to be electronics. Moving the mouse over the screen and feeling things as changing vibrations of the mouse might be useful? Maybe a speaker which could play vibrations to be felt with the other hand?

It would be easy to write a FreeBASIC program to test out some of these ideas. I find the problem an interesting one to work on. I will just have to find the FreeBASIC code for a command to output speech using the window API.

I see no reason a program couldn't be written using the mouse and audio feedback to design a simple GUI.

.
OSchmidt
Posts: 49
Joined: Jan 01, 2016 12:27
Contact:

Re: GUI design

Post by OSchmidt »

Psyche_Demon wrote: Hi there Olaf,
The problem I will have there is, A, knowing what the numbers represent. In the Windows API they represent pixels. In VB6 they can represent either pixels, inches, centimetres etc.
That's one reason, why I suggested to use the GUI-support of the vbRichClient5.dll.
It is "DPI-aware" and supports "zoomable GUIs", all based on "virtual Pixels".
These Virtual Pixels will match exact Pixels (with a resolution of 96 DPI) - as long as you
don't change the ZoomFactor on the Form.WidgetRoot from its default-value 1.

I've now did so in the next enhanced Demo below (have set Form.WidgetRoot.Zoom = 2)
This means, whilst visually the two Controls (Button and Text) are rendered twice as large,
their relative Coords and Dimensions remain - and are reported as they were (that's what
I mean with "virtual Pixels" - so - all you have to care about in your App is the Forms-Zoomfactor.
Psyche_Demon wrote: Usually, whatever can be represented visually can, in most cases, be represented in audio, even if the audio solution is a lot simpler and perhaps less realistic (such as a door constantly playing a handle rattling sound so that its position can be established), hence the reason I wondered whether it would be possible to make a GUI builder, representing a GUI in the same way.
I'm aware of that - hence my attempt to make the little Form-Demo "speak about their Controls".

Ok, below comes my take 3 for the whole thing - introducing:
- a Const-Definition (DevelopMode=True ... to be set to False, to make it "less noisy" when you ship the App)
- the two Widgets deliberately overlap now (in their initial Widgets.Add-Coordinates), to provoke a new speech-output which warns about that
- the Button-Click or -Press now deliberately sets the Form-Height to 100 (and when clicked again back to 480), to provoke the "Form-Visibility-check", which happens in the Form_Resize-Event.

I'd really suggest to play around with that stuff - because when you consider that I was able to come up with all that
in VBScript in about 30 invested minutes total (using only a sparse amount of code-lines), you shouldn't have larger
problems, adapting it to more specific needs, in case you want to give the whole idea a go...
I'd help with further ideas/enhancements as well, when you stumble over something.

Code: Select all

With CreateObject("WScript.Shell") 'this block auto-reshells a script to the 32Bit-WSH-version on Win64 (if needed)
	WS32on64 = .ExpandEnvironmentStrings("%SYSTEMROOT%\SysWOW64\wscript.exe")
	If StrComp(WScript.FullName, WS32on64, vbTextCompare) then 'So, when the Paths' differ, we...
	   If CreateObject("Scripting.FileSystemObject").FileExists(WS32on64) Then '...check if the target is there at all...
		 .Run """" & WS32on64 & """ """ & WScript.ScriptFullName & """", 1, False '...and re-launch the script to it!
		 WScript.Quit
	   End If
	End If
End With
 
'-------------- VBScript-Code starts here ----------------
Const DevelopMode = True

Set Sp = Wscript.CreateObject("SAPI.SpVoice") 'create the Speech-API-HelperObject
Set New_c   = WScript.CreateObject("vbRichClient5.cConstructor") 'create the VbRichClient Main-Constructor-Object

Set btnTest = WScript.CreateObject("vbWidgets.cwButton", "btnTest_") 'create a Button-Widget (which is not yet sited)
    btnTest.Caption = "Hello" 'give the Button a Caption
Set txtTest = WScript.CreateObject("vbWidgets.cwTextBox", "txtTest_") 'create a Text-Widget (which is not yet sited)
    txtTest.Text = "The Text" 'give the TextBox some Input
    
Set Form = New_c.Cairo.WidgetForms.Create(2, "Form-Caption", , 640, 480) 'create the Form-Obj (initial size 640x480)
    Form.WidgetRoot.Zoom = 2
    Form.Widgets.Add btnTest, "Button1", 10, 10, 100, 30 'site the btnTest-WidgetObject we created earlier above
    Form.Widgets.Add txtTest, "Text1", 10, 35, 200, 25 'site the txtTest-WidgetObject we created earlier above
    
    WScript.ConnectObject Form, "Form_" '<- EventSink-Prefix and -Binding the VBScript-way
    Form.Show 1 'show the Form modally (the Script will run until the Form gets closed)


Sub btnTest_Click() 'just to show that the VBS-EventBinding (second param in WScript.CreateObject) does work
    If Form.Height>100 Then Form.Height = 100 Else Form.Height = 480
End Sub

Sub Form_BubblingEvent(Sender, EventName, P1, P2, P3, P4, P5, P6, P7)
    Set W = Sender.Widget
    If EventName = "W_GotFocus" Then
        Dim S: S = W.Key
        Select Case TypeName(Sender) 
          Case "cwButton":  If DevelopMode Then S = S & GetControlInfo(W)
          Case "cwTextBox": S = S & ": content: " & Sender.Text: If DevelopMode Then S = S & GetControlInfo(W)
        End Select
        Speak S 
    ElseIf EventName = "Change" Then 'all TextBoxes report their changed content here
        Speak W.Key & "-content-change: " & Sender.Text
    End If
End Sub

Sub Form_ResizeWithDimensions(NewWidth, NewHeight)
    If DevelopMode Then CheckForOverlaps NewWidth / Form.WidgetRoot.Zoom, NewHeight / Form.WidgetRoot.Zoom
End Sub

Sub Speak(Text)
    Sp.Speak Text, 3 '3 = (1 OR 2) ... = SVSFlagsAsync OR SVSFPurgeBeforeSpeak 
End Sub

Sub CheckForOverlaps(FormWidth, FormHeight)
    Speak ""
    For i = 1 To Form.Widgets.Count  
        Set W = Form.Widgets(i).Widget
        If  W.Left < 0 Or W.Top < 0 Or W.Left + W.Width > FormWidth Or W.Top + W.Height > FormHeight Then 
            Speak W.Key & " is not entirely visible on the Form.": Exit Sub
        End If
        For j = i - 1 To 1 Step -1
          If IntersectionOf(W, Form.Widgets(j).Widget) Then
            Speak W.Key & GetControlInfo(W) & ", is overlapping the Widget " & Form.Widgets(j).Widget.Key & GetControlInfo(W): Exit Sub
          End If
        Next
    Next 
End Sub

Function GetControlInfo(W)
    GetControlInfo = ": x=" & W.Left & ", y=" & W.Top & ", dx=" & W.Width & ", dy=" & W.Height ' & ", Zoom=" & W.Root.Zoom
End Function

Function IntersectionOf(W1, W2)
  IntersectionOf = W1.Left < W2.Left + W2.Width And W2.Left < W1.Left + W1.Width And W1.Top < W2.Top + W2.Height And W2.Top < W1.Top + W1.Height  
End Function
Olaf
Psyche_Demon
Posts: 46
Joined: Jul 30, 2015 8:07
Location: England
Contact:

Re: GUI design

Post by Psyche_Demon »

Hi Olaf,
Again though. I have no way of comparing pixels, whether virtual or actual, with any physical size unit. At least with actual sizes (inches, centimetres etc) I can compare them to actual objects.
I understand how you are getting SAPI to read details about the control. But a screen reader can read out numbers by itself. What I need is an actual representation where I can literally try and imagine what my gui might look like.
Maybe I'm missing something crucial here. but also the following still needs to be answered.
1. Do all the controls fit on the screen? I.E. have any controls been lost due to overcalculating? What if someone has a smaller screen, what if someone has a bigger screen, etc. What happens to the controls in those scenarios.
2. Are any of my controls overlapping?
3. Are the controls and its text large enough to be read while not being so large that they exceed the screen's boundaries, or so large that they look like they've got beginner written all over them (metaphorically of course), and not so small that only a mouse could read it.
Also I think I recall you said you were using VBScript for these examples. If this is a COM object, how would this be used in FB?
Cheers.
BasicCoder2
Posts: 3906
Joined: Jan 01, 2009 7:03
Location: Australia

Re: GUI design

Post by BasicCoder2 »

You say that you are using FreeBASIC. So does someone set up FBIDE or FBEDIT and you type code into it? Does the software read out what you have typed? How do you find your way around the text? I just can't imagine the difficulty of using systems designed for sighted people.

When I make my way around the house at night with the lights off I rely entirely on touch and thus immediately think in terms of feeling my way around a computer screen using the mouse as a white cane but with extra features. There are touch screens as well so instead of the mouse you could move a finger over the screen with audio software to report what you are touching. I could imagine a 3D printer producing a printout of a screen where the control layout could actually be felt.

Maybe have cut out rectangles that can be moved around physically and then a webcam or scanner could generate the layout from that. I can think of all sorts of ideas but of course I don't know how practical they would be.

.
Post Reply