are memory streams impossible? (nope!)

[KlausPeterVDL]

that's pretty much the question.

the background...
i'd like to stream* things to a buffer, compress the buffer, then write it to disk.
on the way in, i'd like to read* the compressed file, uncompress it, then supply that block as 'file' input.
also compressed cdata sections

from searching here and in the wiki the c-rt seems to be bound so tightly to the language/compiler that unless there is something that completely escaped my notice, i don't see how i can get this done (without using actual files as buffers). help?!

* by stream and read i mean get/put/print #handle, <data>



(edited to change thread-title after angros came through with a solution. ty all who chimed in with alternative ideas!)

[gothon]

It is certainly possible to stream things in memory. The best way of going about it will depend on what you need to accomplish. If you only need to handle string data, you can use string handling methods.

If you need to stream data of arbitrary type, you can get the memory size of the data using the SizeOf operator. You can allocate memory space for it in your stream using either arrays, new/delete, or Allocate/DeAllocate. You can copy the data to the stream byte by byte using the crt/string.bi 'memcpy' function.

If speed is critical, you may wish to allocate extra memory for the stream using a doubling technique that will prevent frequent reallocation of the stream and the resultant excess recopying. If you have variable sized fields, size information for those fields must also be stored. If the data contains multiple types and you can't infer the correct types when reading it by reading it in the same order it was written, then you will need to write type information as-well. Storing type information is more complicated, but can still be done.

Ex:

Code: Select all

#Include "crt/string.bi"

Dim Stream() As Byte, StreamPosition As Integer = 0

#Macro StreamPut(X)
    If StreamPosition + SizeOf(X) > UBound(Stream) Then ReDim Preserve Stream(2*(StreamPosition + SizeOf(X)))
    memcpy @Stream(StreamPosition), @(X), SizeOf(X)
    StreamPosition += SizeOf(X)
#EndMacro

#Macro StreamGet(X)
    memcpy @(X), @Stream(StreamPosition), SizeOf(X)
    StreamPosition += SizeOf(X)
#EndMacro

Type Stuff
    UserName As String * 10
    ID As Integer
End Type

' Write Data to the memory stream
Dim Dat As Stuff, Y As Single = 3.141
Dat.UserName = "User 8"
Dat.ID = 8

StreamPut(Dat)
StreamPut(Y)

Dat.UserName = "User 4"
Dat.ID = 4

StreamPut(Dat)

'>> ToDo:  Compress, Save to Disk, Read From Disk, UnCompress

' Read data from memory stream
StreamPosition = 0

StreamGet(Dat)
Print Dat.UserName
Print Dat.ID

Y = 0
StreamGet(Y)
Print Y

StreamGet(Dat)
Print Dat.UserName
Print Dat.ID

Sleep

[KlausPeterVDL]

Hi Gothon, thank you,

i appreciate the effort, you really went all out; but what you're doing is a workaround, not a stream.
e.g. its not abusing open/put/get/print so that using them on-memory looks as it would when writing to disk.

given that i have a large codebase that does tons of custom writes I'll stick with the temp-files until i can see my way towards making things streamable by hand. for that the macro foo will work ;)

[fxm]

Why not use the Stream I/O functions from the C Standard Library?

[KlausPeterVDL]

Why not use the Stream I/O functions from the C Standard Library?

because it seems that i'm already using them! i read that page days ago when i was still thinking that maybe i was not seeing something, but no, there's no way that i can see to get 'behind' the write-, or 'before' the read-mechanics as one can't seem to inject anything into the rtl that would let one setup file handles for chunks of memory.. or is there a way?

hacking crt/stdio to get me cross-platform fake handles and read/write callbacks that operate on blocks is a bit much for me, but it would seem like that's what's missing. setbuf/setbufv look promising but are not _it.

[TJF]

I use

• GIO GResource (GLIB)

functions family for that kind of tasks.

[KlausPeterVDL]

.. for that kind of tasks.

i hear you loud and clear. with complete bindings for something deep and a new project that does not require access to inherited method i might look into gnome again, for now, its just the wrong kind of semantics. fb semantics might have had some clever way of using the filename to let me get into memory, at least that's what i was hoping for. ... i mean, who doesnt love basic for its side-effects ;)

anyway, good link! thanks.

[fxm]

There is a documentation page: File I/O with FreeBASIC

By the way, DOS386 (last updater), can you check the example?

[KlausPeterVDL]

There is a documentation page: File I/O with FreeBASIC

we should let this go.

i have a large codebase at my hands that would have benefit from a memory-stream semantics built into OPEN(). it does not exist and no matter how often i read the help the language isn't suddenly going to become well designed and i'm just going to roll with not having things that were available to me in delhi 1.0 alpha 16bit, just about 20 years ago.

dont take this as criticism, but the missing or incomplete semantics just mean that i wont ever be doing anything of scale beyond the project i'm updating. i appreciate your knowhow of how to get things done, but c'mon, 4 different syntax patterns for i/o... and an inability to call base.inherited() which would make designing any oop wrapper for them a nightmare on a good day ... and a plain old impossibility 'cause one can't tap into the guts of 'get/put/print' .. so that one could reuse existing code.

i'm going to take the option open to me and ship a helper program who's memory i can pipe things to/fro. done. ugly but i've got to move on before someone sends me more 'help' without saying anything authorative like .... "yes, you can not do here what you can do everywhere else."

basic is fun to me as long as i forget all about design. i'll keep it like that and wish you best-speed in getting method-inheritance and a decent (extendable) array/list structure going!

[marcov]

(
I assume Klaus means abstract stream support, I wondered about this already when the first answers came in. Afaik C++ STL also knows the concept.

SO the purpose is having a stream class that represents an abstract stream, taking an input stream and generating an output streams. Some streams (like reading a file/resource/memory block) only generate the output stream, some, like saving a file, only take the input. Some of these can be read/write though.

Others are more algorithms, transforming an incoming stream into something else (zlib (de/)compressing, base de/encoding, changing crlf to lf etc), escape non html characters etc etc.

So chaining a stream to an another stream to form a chain, like filestream(read file)->compress->encrypt->base64encode->http output stream, each step using a generic stream class that takes an input stream and is its own output stream. Which then can be passed to the next etc.

So the question probably becomes, do the proposed solutions allow chaining?

P.s. Klaus, this was already in Turbo Pascal TV and OWL, before Delphi, though the delphi mechanisms are slightly more elegant)
)

[Mentat]

It's really late at night, so what I'm saying may sound incredibly fantastic, but can't you map a desired memory buffer to a file descriptor with mmap, and then write to said descriptor like a stream? I know that's not how mmap is supposed to work, and I don't even know if mmap calls are available on windows.

[MichaelW]

Windows supports memory-mapped files, but AFAIK there is no built-in connection in any of the FreeBASIC-accessible stream, or stream-like, functions and no easy way to add a connection. Rolling your own set of specialized functions is doable, but that route seems to have been rejected here.

A simple demo/test:

Code: Select all

#include "file.bi"
#include "crt.bi"
#include "windows.bi"

dim as FILE ptr stream
dim as HANDLE hFile, hMap
dim as integer ptr p

kill "test.bin"

open "test.bin" for binary as 1

stream = cast( FILE ptr, fileattr( 1, fbFileAttrHandle ))

hFile = cast(HANDLE,_get_osfhandle(_fileno(stream)))

print hFile

hMap = CreateFileMapping( hFile, NULL, PAGE_READWRITE, 0, 1000000*4, NULL )

print hMap

p = MapViewOfFile( hMap, FILE_MAP_WRITE, 0, 0, 0 )

print hex(p);"h"

for i as integer = 0 to 999999
    p[i] = i
next

sleep

print FlushViewOfFile( p, 0 )

print UnmapViewOfFile( p )

print CloseHandle( hMap )

close

sleep

[angros47]

Maybe something like this could work?

First, you open a "dummy" file (to avoid really creating it, use the name "NUL" under windows); then, you get the file handle with FileAttr, and the file descriptor with fileno; after that, with _dup2 you redirect the file descriptor of the pipe on it, and you have a bidirectional pipe that you can access using PRINT #, GET, PUT...

Code: Select all

#include "windows.bi"
#include "file.bi"
#include "crt/stdio.bi"
#include "crt/fcntl.bi"
dim outAudio as integer

dim shared piped(2) as integer


_pipe(@piped(0),512, O_NOINHERIT)


sub _SoundThread(ByVal userdata As Any Ptr )

    dim as FILE ptr stream
    open "NUL" for input as #2
    stream = cast( FILE ptr, fileattr( 2, fbFileAttrHandle ))
    _dup2(piped(0),_fileno(stream))

    dim buffer as string
    do
        input #2, buffer
        print buffer;
    loop

end sub

dim thread_handle as any ptr
thread_handle = threadcreate( @_SoundThread)






open "NUL" for output as #1

dim as FILE ptr stream
stream = cast( FILE ptr, fileattr( 1, fbFileAttrHandle ))
_dup2(piped(1),_fileno(stream))


print #1,"Passed text"

dim a as integer
input a
end

[KlausPeterVDL]

Angros. I think you got it! that is definitely the right approach. its still blocking inside my single cpu vm and might be windows only, but its just what i was looking for! thank you!

[KlausPeterVDL]

below a working, single threaded example of passing 'print' through a buffer and back into your application.

- it relies on the windows runtime
-- needs a linux example

- it relies on a buffer provided by the operating system
-- so you need to buffer the input and data-processing seperately
-- more digging might reveal a way to pass a buffer to the os which one could use for processing

the method makes it possible to print to different streams that you collate later,
print and compress, or to pipe your console-code into something gui-like.

Code: Select all

    #include "windows.bi"
    #include "file.bi"
    #include "crt/stdio.bi"
    #include "crt/fcntl.bi"

	const fName="NUL"
	dim as integer fIn,fOut 'files
	dim as integer hIn,hOut 'stream-handles
	dim as integer pIn,pOut	'dupped to pipe-handles
	
	' _fileno http://msdn.microsoft.com/en-us/library/aa246862%28v=vs.60%29.aspx file handle of a stream
	' _pipe http://msdn.microsoft.com/en-us/library/aa298531%28v=vs.60%29.aspx creates a rw pipe
	' _dup2 http://msdn.microsoft.com/en-us/library/aa246796%28v=vs.60%29.aspx reassign a file handle 
	
	cls
	? "testing redir through ";fName
	?
	
	fIn=freefile:	open fName for input as #fIn:	?"open input#"&fIn, err
    fOut=freefile:	open fName for output as #fOut: ?"open output#"&fOut,err

    hIn= _fileno(cast(FILE ptr, fileattr( fIn, fbFileAttrHandle ))):	?"handle in#"& hIn, err
    hOut= _fileno(cast(FILE ptr, fileattr( fOut, fbFileAttrHandle ))):	?"handle out#"& hOut, err

	enum PIPES 
		READp
		WRITEp
	end enum
	
    dim shared piped(WRITEp) as integer

    ? "_pipe()", _
    	_pipe(@piped(0),512, O_NOINHERIT)
		pIn=piped(READp):	? "piped(READ)",pIn
		pOut=piped(WRITEp):	? "piped(WRITE)",pOut

		? "_dup2", _
		    _dup2(pIn,hIn)
		? "_dup2", _
		    _dup2(pOut,hOut)

	?
	'
	dim buffer as string="freebasic does polymorphic output!"
	
    ? #fOut, buffer:	?"print#"&fOut, err
 	lof(fOut) 'flush output buffer
	'
	input #fIn, buffer:	?"input#"&fIn, err, ">";buffer;"<"

sleep:end  

	'for a moment i was concerned that NUL might be a thing, so i cloned the code from above
	'to set up a 'second channel' through which to pipe things in order to see if it worked 
	'for real. it does: 

	'channel#2
	?
	? "channel#2"
	dim as integer fIn2,fOut2 'files
	dim as integer hIn2,hOut2 'stream-handles
	dim as integer pIn2,pOut2 'dupped to pipe-handles
	fIn2=freefile:	open fName for input as #fIn2:	?"open input#"&fIn2, err
    fOut2=freefile:	open fName for output as #fOut2: ?"open output#"&fOut2,err
    hIn2= _fileno(cast(FILE ptr, fileattr( fIn2, fbFileAttrHandle ))):	?"handle in#"& hIn2, err
    hOut2= _fileno(cast(FILE ptr, fileattr( fOut2, fbFileAttrHandle ))):?"handle out#"& hOut2, err
   	_pipe(@piped(0),512, O_NOINHERIT)
	pIn2=piped(READp):	? "piped(READ)",pIn2
	pOut2=piped(WRITEp):? "piped(WRITE)",pOut2
	? "_dup2", _dup2(pIn2,hIn2)
	? "_dup2", _dup2(pOut2,hOut2)

    ? #fOut2, buffer:	?"print#"&fOut2, err
 	lof(fOut2) 'flush output buffer
	'
	? "available to channel#1:",lof(fIn)
	? "available to channel#2:",lof(fIn2)
	?
	input #fIn2, buffer:	?"input#"&fIn2, err, ">";buffer;"<"

sleep

if it wasn't for the missing linux support this would be hot, even something that the fb gods might want to make part of the language. thanks again for digging angros!