Reading a text file into a string array

New to FreeBASIC? Post your questions here.
Post Reply
jj2007
Posts: 2326
Joined: Oct 23, 2016 15:28
Location: Roma, Italia
Contact:

Reading a text file into a string array

Post by jj2007 »

The snippet below reads a text file (name supplied via commandline) into a string array. If it succeeds, the first and last 5 lines are printed.

My problem here is that I wanted to use ReDim to adjust the number of elements. Unfortunately, the commented out code below fails miserably, it crashes. I've studied the help file, and it's not clear to me. Can anybody shed light into the mysteries of dynamic string arrays?

Code: Select all

Dim shared recstr(1000) As string	' fixed size is no good!
function Recall(fname As string) As integer
  Dim As integer ct=0, maxct=1000
  If Open(fname For Binary Access Read As #1) = 0 Then
	While not eof(1)
		Line Input #1, recstr(ct)
		ct=ct+1
' 		if ct>=maxct then
' 			print "REDIM ";maxct;
' 			maxct=maxct*1.5
' 			Redim recstr(maxct)
' 			print " ok, maxct=";maxct
' 		endif
' 		print ct;chr(13);
	Wend
	Close #1
  Else
	Print "Error opening file"
  End If
  return ct
end function

Dim As double tickssec=Timer
Dim As Integer ticks, records
print "loading ";Command$(1);" took ";
records=Recall(Command$(1))
ticks=(Timer-tickssec)*1000
print ticks;" ms"
for i as integer=0 to records-1
  if i<5 or i>records-5 then
	print recstr(i)
  elseif i=5 then
	print "..."
  endif
next
sleep
MrSwiss
Posts: 3910
Joined: Jun 02, 2013 9:27
Location: Switzerland

Re: Reading a text file into a string array

Post by MrSwiss »

Dynamic string array definition methods are:

Code: Select all

Dim As String s_arr(Any)  '' initialize later (in procedure)
ReDim As String s_arr(1 to 1000)  '' with initial size
'' both above are valid, however:
Dim As String s_arr(1 to 1000)  '' fixed size (static)
fxm
Moderator
Posts: 12081
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: Reading a text file into a string array

Post by fxm »

And in the loop, the array must be resized with the 'Preserve' option:
Redim Preserve recstr(maxct)
srvaldez
Posts: 3373
Joined: Sep 25, 2005 21:54

Re: Reading a text file into a string array

Post by srvaldez »

hello jj2007
I can't get your algorithm to work, however, perhaps reading the entire file at once into a string buffer might be of interest to you viewtopic.php?f=2&t=26410
paul doe
Moderator
Posts: 1730
Joined: Jul 25, 2017 17:22
Location: Argentina

Re: Reading a text file into a string array

Post by paul doe »

jj2007 wrote:...Can anybody shed light into the mysteries of dynamic string arrays?
This code is more akin to what you're trying to do:

Code: Select all

function recall( _
  byref aFileName as const string, _
  text() as string, _
  byval size as uinteger = 1000 ) as boolean
  
  dim as integer aFileNumber = freeFile()
  dim as boolean result = false
  
  if( open( aFileName for input as aFileNumber ) = 0 ) then
    dim as uinteger current = 0
    
    redim text( 0 to size - 1 )
    
    do while( not eof( aFileNumber ) )
      if( current >= size ) then
        size *= 1.5
        
        redim preserve text( 0 to size - 1 )
      end if
      
      line input #aFileNumber, text( current )
      
      current += 1
    loop
    
    redim preserve text( 0 to current - 1 )
    
    result = true
  end if
  
  return( result )
end function

/'
  Test code
'/
dim as string text( any ) '' Pass the function an empty array, like this

if( recall( "test.txt", text() ) ) then
  for i as uinteger = 0 to ubound( text )
    ? text( i )
  next
else
  ? "Couldn't read requested file."
end if

sleep()
grindstone
Posts: 862
Joined: May 05, 2015 5:35
Location: Germany

Re: Reading a text file into a string array

Post by grindstone »

jj2007 wrote:Can anybody shed light into the mysteries of dynamic string arrays?
Your code only needs a few changes:

Code: Select all

ReDim recstr(0) As String   ' fixed size is no good!
Function Recall(fname As String, recstr() As String) As Integer
	Dim As Integer ct=0
	If Open(fname For Binary Access Read As #1) = 0 Then
		While Not Eof(1)
			ReDim Preserve recstr(ct)
			Line Input #1, recstr(ct)
			ct=ct+1
		Wend
		Close #1
	Else
		Print "Error opening file"
	End If
	Return ct
End Function

Dim As Double tickssec=Timer
Dim As Integer ticks, records
Print "loading ";Command$(1);" took ";
records=Recall(Command$(1), recstr())
ticks=(Timer-tickssec)*1000
Print ticks;" ms"
For i As Integer=0 To records-1
	If i<5 Or i>records-5 Then
		Print recstr(i)
	ElseIf i=5 Then
		Print "..."
	EndIf
Next
Sleep
dodicat
Posts: 7976
Joined: Jan 10, 2006 20:30
Location: Scotland

Re: Reading a text file into a string array

Post by dodicat »

Both line input and redim preserve are slow.
Test 3 methods
1)two line input loops with no redim 3 seconds
2) one line input with redim --jj2007 6 seconds
3) load to a string then split the string. < 1 second
Both my methods are 1 based arrays, jj2007 uses 0 based arrays.

Code: Select all




#include "file.bi"
'=========================   method 1 =================
Function Load(file As String,a() as string) As long
	If Fileexists(file)=0 Then Print file;" not found":Sleep:End
    Dim As Long  f=Freefile,counter
    Open file For Binary Access Read As #f
    dim as string tmp
    while not eof(f)
        counter+=1
        line input #f,tmp
    wend
   redim a(1 to counter):counter=0
   seek #f,1
   while not eof(f)
        counter+=1
       line input #f, a(counter) 
   wend
    Close #f
    Return counter
End Function

'=========================== method 2 =================
Function Recall(fname As String, recstr() As String) As Integer
   Dim As Integer ct=0
   If Open(fname For Binary Access Read As #1) = 0 Then
      While Not Eof(1)
         ReDim Preserve recstr(ct)
         Line Input #1, recstr(ct)
         ct=ct+1
      Wend
      Close #1
   Else
      Print "Error opening file"
   End If
   Return ct
End Function

'=============================method 3===================
Function loadfile(file As String) As String
	If Fileexists(file)=0 Then Print file;" not found":Sleep:End
    Dim As Long  f=Freefile
    Open file For Binary Access Read As #f
    Dim As String text
    If Lof(1) > 0 Then
        text = String(Lof(f), 0)
        Get #f, , text
    End If
    Close #f
    Return text
End Function

Function StringSplit(s_in As String,chars As String,result() As String) As Long
    Dim As Long ctr,ctr2,k,n,LC=Len(chars)
    Dim As boolean tally(Len(s_in))
    #macro check_instring()
    n=0
    While n<Lc
        If chars[n]=s_in[k] Then 
            tally(k)=true
            If (ctr2-1) Then ctr+=1
            ctr2=0
            Exit While
        End If
        n+=1
    Wend
    #endmacro
    
    #macro split()
    If tally(k) Then
        If (ctr2-1) Then ctr+=1:result(ctr)=Mid(s_in,k+2-ctr2,ctr2-1)
        ctr2=0
    End If
    #endmacro
    '==================  LOOP TWICE =======================
    For k  =0 To Len(s_in)-1
        ctr2+=1:check_instring()
    Next k
    If ctr Then Redim result(1 To ctr): ctr=0:ctr2=0 Else  Return 0
    For k  =0 To Len(s_in)-1
        ctr2+=1:split()
    Next k
    '===================== Last one ========================
    If ctr2>0 Then
        Redim Preserve result(1 To ctr+1)
        result(ctr+1)=Mid(s_in,k+1-ctr2,ctr2)
    End If
    Return Ubound(result)
End Function



'create a text file
sub savefile(filename as string,text as string)
    var h=freefile
    open filename for binary access write as #h
    put #h,1,text
    close #h
end sub

sub makeafile
dim as string s
for n as long=1 to 500000
   s+=str(n)+chr(13,10) 'a common line ending
next
savefile("test.txt",s)
end sub



makeafile



dim as double t

t=timer
redim as string a()
load("test.txt",a())
print "Time ";timer-t; "   Method 1 line input two loops"
for n as long=lbound(a) to 20
    print n,a(n)
next

t=timer
redim as string b()
recall("test.txt",b())
print "Time ";timer-t;"   Method 2 line input redim preserve"
for n as long=lbound(b) to 20
    print n,b(n)
next


t=timer
redim as string c()
var txt=loadfile("test.txt")
stringsplit(txt,chr(13,10),c())
print "Time ";timer-t;"   Method 3  split text"
for n as long=lbound(c) to 20
    print n,c(n)
next

print "done"
sleep
kill "test.txt"




   
jj2007
Posts: 2326
Joined: Oct 23, 2016 15:28
Location: Roma, Italia
Contact:

Re: Reading a text file into a string array

Post by jj2007 »

srvaldez wrote:I can't get your algorithm to work, however, perhaps reading the entire file at once into a string buffer might be of interest to you viewtopic.php?f=2&t=26410
This is what I use for MB Recall, and what dodicat uses in his fastest example. That could indeed be the way to go.

@dodicat: For which file size did you get 3/6 seconds? Your option #3 is pretty fast, but it misses empty strings.

@Paul: Your code works fine and is so far the fastest for the options using Line Input.
grindstone wrote:Your code only needs a few changes:
Thanxalot, works like a charm! What I don't understand, though - and the docs don't talk about it - is why ReDim on top works:

Code: Select all

ReDim recstr(0) As String
Function Recall(fname As String, recstr() As String) As Integer
   Dim As Integer ct=0
   If Open(fname For Binary Access Read As #1) = 0 Then
      While Not Eof(1)
         ReDim Preserve recstr(ct)
         Line Input #1, recstr(ct)
         ct=ct+1
      Wend
      Close #1
   Else
      Print "Error opening file"
   End If
   Return ct
End Function
... while a Dim recstr(0) As String on top makes it crash. Mysteries of FreeBasic...
dodicat
Posts: 7976
Joined: Jan 10, 2006 20:30
Location: Scotland

Re: Reading a text file into a string array

Post by dodicat »

I get 3,6 and about .2

Note, if you don't like 1 based arrays one dimensional you can convert to 0 based very easily.
example

Code: Select all


redim as double a(1 to 10)
randomize 1
for n as long=lbound(a) to ubound(a)  '1 to 10
    a(n)=rnd
next n

for n as long=lbound(a) to ubound(a)
    print n,a(n)
next n

print

redim preserve a(0 to 9)  

for n as long=lbound(a) to ubound(a)  '0 to 9
    print n,a(n)
next n
sleep
  
so in my string split use
redim preserve result(0 to ubound(result)-1)
before returning
(un noticable time penalty)
srvaldez
Posts: 3373
Joined: Sep 25, 2005 21:54

Re: Reading a text file into a string array

Post by srvaldez »

jj2007 wrote:What I don't understand, though - and the docs don't talk about it - is why ReDim on top works ... while a Dim recstr(0) As String on top makes it crash. Mysteries of FreeBasic...
I believe that Dim uses the stack while reDim uses the heap, hence it logical that a string that was allocated on the stack can't be reDimmed, though a compiler warning would be nice.
MrSwiss
Posts: 3910
Joined: Jun 02, 2013 9:27
Location: Switzerland

Re: Reading a text file into a string array

Post by MrSwiss »

The FB manual on: ReDim clearly states -- declaring a dynamic array.
Therefore, the 'mystery' claim, seems to be 'unfounded'!
fxm
Moderator
Posts: 12081
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: Reading a text file into a string array

Post by fxm »

Code: Select all

Dim Shared As String array1(0)
array1(0) = "1234"
Redim Shared As String array2(0)
array2(0) = "abcd"

Sub test ()
  Print "'" & array1(0) & "'"
  Redim Preserve array1(1)
  Print "'" & array1(0) & "'"
  array1(0) = "5678"
  Print "'" & array1(0) & "'"
  Print
  Print "'" & array2(0) & "'"
  Redim Preserve array2(1)
  Print "'" & array2(0) & "'"
  array2(0) = "efgh"
  Print "'" & array2(0) & "'"
  Print
End Sub

test()
Print "'" & array1(0) & "'"
Print "'" & array2(0) & "'"

Sleep
As the global 'array1()' is static (not dynamic), 'Redim Preserve array1(1)' in Sub creates a local dynamic array which hides the static global array.
jj2007
Posts: 2326
Joined: Oct 23, 2016 15:28
Location: Roma, Italia
Contact:

Re: Reading a text file into a string array

Post by jj2007 »

IMHO the manual needs some simple usage examples. The text might be all correct, technically, but its wording becomes understandable only once you have understood how it works, through trial and error and questions at the FB forum. Reminds me of the infamous Microsoft helicopter anecdote ;-)
Arrays can be declared as variable length in several ways: Using Dim with an empty set of indexes (Dim x()), using Dim with indexes that are variables or using the keyword ReDim, or using Any in place of the array bounds, or declaring it past the metacommand $Dynamic. Variable length arrays can't use initializers.
Anyway, it works fine now, here is the condensed version (needs a filename as commandline argument):

Code: Select all

Function Recall(fname As String, locArray() As String) As Integer
   Dim As Integer ct=0	' locArray is a local dynamic array
   If Open(fname For Binary Access Read As #1) = 0 Then
      While Not Eof(1)
         ReDim Preserve locArray(ct)
         Line Input #1, locArray(ct)
         ct=ct+1
      Wend
      Close #1
   Else
      Print "Error opening file"
   End If
   Return ct
End Function

Dim As Double tickssec=Timer	' we'll time the exercise
Dim As Integer ticks

Print "loading ";Command$(1);" took";

' ************* load a text file into a string array: **********
Dim MyStrings() As String
Dim as integer records=Recall(Command$(1), MyStrings())

ticks=(Timer-tickssec)*1000
Print ticks;" ms"

For i As Integer=0 To records-1		' print the first and last elements
   If i<10 Or i>=records-10 Then
      Print i, MyStrings(i)
   ElseIf i=10 Then
      Print " ..."
   EndIf
Next

Sleep
A big THANK YOU to all those who made helpful comments!
fxm
Moderator
Posts: 12081
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: Reading a text file into a string array

Post by fxm »

jj2007 wrote:IMHO the manual needs some simple usage examples.
There are simple examples of dynamic array usage at the REDIM documentation page.
grindstone
Posts: 862
Joined: May 05, 2015 5:35
Location: Germany

Re: Reading a text file into a string array

Post by grindstone »

@jj2007:
One remark: You can avoid connumerating the records by using "UBound(MyStrings)" instead of "records".
Post Reply