Dodicat Zlib

General FreeBASIC programming questions.
Locked
albert
Posts: 6000
Joined: Sep 28, 2006 2:41
Location: California, USA

Dodicat Zlib

Post by albert »

@Dodicat

Here's your FB code to call Zlib from inside FB

Question: Can you make it so i can set the chunk size , and set the flag for Best Compression???

I think your code just uses the default 16,384 bits chunk size , and uses "default compression".
I'd like to play around with different input chunks , and different compression levels.

Code: Select all


'Dodicat from Scotland ( Zlib calling code )

Namespace Zlibrary

#inclib "zlib"
Extern "C"
    Declare Function compressBound(Byval sourceLen As Ulong) As Ulong
    Declare Function uncompress(Byval dest As Ubyte Ptr, Byval destLen As Uinteger Ptr, Byval source As  Ubyte Ptr, Byval sourceLen As Ulong) As Long
    Declare Function compress(Byval dest As Ubyte Ptr, Byval destLen As Uinteger Ptr, Byval source As  Ubyte Ptr, Byval sourceLen As Ulong) As Long
End Extern

Function getpassedinfo(text As String,Byref passed_length As Integer) As String
    Dim As String var1,var2
    Dim As Integer pst
    #macro splice(stri,char,var1,var2)
    pst=Instr(stri,char)
    var1="":var2=""
    If pst<>0 Then
        var1=Mid(stri,1,pst-1)
        var2=Mid(stri,pst+1)
    Else
        var1=stri
    End If
    #endmacro
    splice(text,"|",var1,var2)
    text=var2
    passed_length=Valint(var1)
    Return text
End Function


'=================   UNPACK ===============
Function unpack(file As String) As String
    Dim As Integer passed_length
    Dim As String text=getpassedinfo(file,passed_length)
    Dim As Integer stringlength,destinationlength
    stringlength=Len(text)
    destinationlength =passed_length
    Dim As Ubyte Ptr source
    Dim As Ubyte Ptr  destination =Callocate(destinationlength,1)
    source=@text[0]
    Var mistake=uncompress(destination,@destinationlength, source, stringlength)
    If mistake<>0 Then Print "There was an error":Sleep:End
    Dim As String uncompressed
    uncompressed=String(destinationlength,0)
    For i As Integer = 0 To destinationlength- 1
        uncompressed[i]=(destination[i])
    Next
    Deallocate destination
    Return uncompressed
End Function

'===================  PACK ============
Function pack(file As String) As String
    Dim As String text=file
    Dim As Integer stringlength,destinationlength
    stringlength=Len(text)
    destinationlength = compressBound(stringlength)
    Dim As Ubyte Ptr source
    Dim As Ubyte Ptr destination =Callocate(destinationlength,1)
    source=@text[0]
    Var mistake=compress(destination, @destinationlength, source, stringlength)
    If mistake <>0 Then Print "There was an error"
    Dim As String compressed
    compressed=String(destinationlength,0)
    For n As Integer=0 To destinationlength-1
        compressed[n]=destination[n]
    Next n
    compressed=stringlength &"|"+compressed
    Deallocate destination
    Return compressed
End Function

End Namespace

dodicat
Posts: 7983
Joined: Jan 10, 2006 20:30
Location: Scotland

Re: Dodicat Zlib

Post by dodicat »

Hi Albert.
There is a compress2 option which lets you set a level.

Code: Select all

 



Namespace Zlibrary

#define Z_NO_COMPRESSION         0
#define Z_BEST_SPEED             1
#define Z_BEST_COMPRESSION       9
#define Z_DEFAULT_COMPRESSION  (-1)

#inclib "zlib1"
Extern "C"
    Declare Function compressBound(Byval sourceLen As Ulong) As Ulong
    Declare Function uncompress(Byval dest As Ubyte Ptr, Byval destLen As Uinteger Ptr, Byval source As  Ubyte Ptr, Byval sourceLen As Ulong) As Long
    Declare Function compress(Byval dest As Ubyte Ptr, Byval destLen As Uinteger Ptr, Byval source As  Ubyte Ptr, Byval sourceLen As Ulong) As Long
    declare function compress2(byval dest as ubyte ptr, byval destLen as uinteger ptr, byval source as const ubyte ptr, byval sourceLen as uLong, byval level as long) as long
End Extern

Function getpassedinfo(text As String,Byref passed_length As Integer) As String
    Dim As String var1,var2
    Dim As Integer pst
    #macro splice(stri,char,var1,var2)
    pst=Instr(stri,char)
    var1="":var2=""
    If pst<>0 Then
        var1=Mid(stri,1,pst-1)
        var2=Mid(stri,pst+1)
    Else
        var1=stri
    End If
    #endmacro
    splice(text,"|",var1,var2)
    text=var2
    passed_length=Valint(var1)
    Return text
End Function


'=================   UNPACK ===============
Function unpack(file As String) As String
    Dim As Integer passed_length
    Dim As String text=getpassedinfo(file,passed_length)
    Dim As Integer stringlength,destinationlength
    stringlength=Len(text)
    destinationlength =passed_length
    Dim As Ubyte Ptr source
    Dim As Ubyte Ptr  destination =Callocate(destinationlength,1)
    source=@text[0]
    Var mistake=uncompress(destination,@destinationlength, source, stringlength)
    If mistake<>0 Then Print "There was an error":Sleep:End
    Dim As String uncompressed
    uncompressed=String(destinationlength,0)
    For i As Integer = 0 To destinationlength- 1
        uncompressed[i]=(destination[i])
    Next
    Deallocate destination
    Return uncompressed
End Function

'===================  PACK ============
Function pack(file As String) As String
    Dim As String text=file
    Dim As Integer stringlength,destinationlength
    stringlength=Len(text)
    destinationlength = compressBound(stringlength)
    Dim As Ubyte Ptr source
    Dim As Ubyte Ptr destination =Callocate(destinationlength,1)
    source=@text[0]
    Var mistake=compress2(destination, @destinationlength, source, stringlength,Z_BEST_COMPRESSION)''<----  use compress2
    If mistake <>0 Then Print "There was an error"
    Dim As String compressed
    compressed=String(destinationlength,0)
    For n As Integer=0 To destinationlength-1
        compressed[n]=destination[n]
    Next n
    compressed=stringlength &"|"+compressed
    Deallocate destination
    Return compressed
End Function

End Namespace

using Zlibrary

dim as string s="abcdefghijklmnopqrstuvwxyz"
s+=ucase(s)
for n as long=1 to 16
    s+=s
next n

print "original length   ";len(s)
var p=pack(s)
print "cpmpressed length ";len(p)
var up=unpack(p)
print "return length     ";len(up)
print iif(up=s,"OK","ERROR")

sleep

 
albert
Posts: 6000
Joined: Sep 28, 2006 2:41
Location: California, USA

Re: Dodicat Zlib

Post by albert »

@Dodicat

Thanks Dodicat!!

I'll play around with it some...
Maybe some of my formulas that didn't compress , will compress now??? ( I Hope )
angros47
Posts: 2323
Joined: Jun 21, 2005 19:04

Re: Dodicat Zlib

Post by angros47 »

Albert, you already got banned several times for that.
albert
Posts: 6000
Joined: Sep 28, 2006 2:41
Location: California, USA

Re: Dodicat Zlib

Post by albert »

@Dodicat

Setting the compress flag to Z_BEST_COMPRESSION ( 9 ) , Doesn't seem to help any..

In the below webpage , it says you can set a zip chunk size??
https://zlib.net/zlib_how.html

@angros47

What i posted above , is Dodicat's code...
I don't think they'll ban me for re-posting someone else's code..
dodicat
Posts: 7983
Joined: Jan 10, 2006 20:30
Location: Scotland

Re: Dodicat Zlib

Post by dodicat »

Albert.
I think the data matters more than choosing a compress flag.
I haven't studied the C code though.
I don't see a problem discussing zlib as far as banning and other punitive methods are concerned.
But you could always go C++ or another language for this if necessary.
If people keep popping up and complaining then this is 99% of the problem.
A bit like the realm of The Donald, once a few people stir the (you know what) then others follow like lemmings and the (you know what) starts flowing uphill.
I have done no work with zip compression, perhaps a review of that might be of help.
I think there are a few examples kicking around.
angros47
Posts: 2323
Joined: Jun 21, 2005 19:04

Re: Dodicat Zlib

Post by angros47 »

Dodicat, in my opinion one of the reasons why he got banned was that you kept encouraging him.
albert
Posts: 6000
Joined: Sep 28, 2006 2:41
Location: California, USA

Re: Dodicat Zlib

Post by albert »

@Dodicat

From the zlib manual....

I'd like to set the "strategy" and "data_type" vars???

==========================================================
Compression levels.

#define Z_NO_COMPRESSION 0
#define Z_BEST_SPEED 1
#define Z_BEST_COMPRESSION 9
#define Z_DEFAULT_COMPRESSION (-1)

Compression strategy — see deflateInit2() below for details.

#define Z_FILTERED 1
#define Z_HUFFMAN_ONLY 2
#define Z_RLE 3
#define Z_FIXED 4
#define Z_DEFAULT_STRATEGY 0

Possible values of the data_type field for deflate().

#define Z_BINARY 0
#define Z_TEXT 1
#define Z_ASCII Z_TEXT /* for compatibility with 1.2.2 and earlier */
#define Z_UNKNOWN 2
==========================================================
albert
Posts: 6000
Joined: Sep 28, 2006 2:41
Location: California, USA

Re: Dodicat Zlib

Post by albert »

@Dodicat

Here's your zlib1 code , i thought to post it here so i can search the forum for it , if need be...

Code: Select all


' copy zlib.bi to folder and rename it zlib1.bi
' download zlib1.dll and place in same folder

#include once "zlib1.bi"

Function getpassedinfo(text As String,Byref passed_length As Integer) As String
    Dim As String var1,var2
    Dim As Integer pst
    #macro splice(stri,char,var1,var2)
    pst=Instr(stri,char)
    var1="":var2=""
    If pst<>0 Then
        var1=Mid(stri,1,pst-1)
        var2=Mid(stri,pst+1)
    Else
        var1=stri
    End If
    #endmacro
    splice(text,"|",var1,var2)
    text=var2
    passed_length=Valint(var1)
    Return text
End Function


'=================   UNPACK ===============
Function unpack(file As String) As String
    Dim As Integer passed_length
    Dim As String text=getpassedinfo(file,passed_length)
    Dim As Integer stringlength,destinationlength
    stringlength=Len(text)
    destinationlength =passed_length
    Dim As Ubyte Ptr source
    Dim As Ubyte Ptr  destination =Callocate(destinationlength,1)
    source=@text[0]
    Var mistake=uncompress(destination,@destinationlength, source, stringlength)
    If mistake<>0 Then Print "There was an error":Sleep:End
    Dim As String uncompressed
    uncompressed=String(destinationlength,0)
    For i As Integer = 0 To destinationlength- 1
        uncompressed[i]=(destination[i])
    Next
    Deallocate destination
    Return uncompressed
End Function

'===================  PACK ============
Function pack(file As String) As String
    Dim As String text=file
    Dim As Integer stringlength,destinationlength
    stringlength=Len(text)
    destinationlength = compressBound(stringlength)
    Dim As Ubyte Ptr source
    Dim As Ubyte Ptr destination =Callocate(destinationlength,1)
    source=@text[0]
    Var mistake=compress2(destination, @destinationlength, source, stringlength,Z_BEST_COMPRESSION)''<----  use compress2
    If mistake <>0 Then Print "There was an error"
    Dim As String compressed
    compressed=String(destinationlength,0)
    For n As Integer=0 To destinationlength-1
        compressed[n]=destination[n]
    Next n
    compressed=stringlength &"|"+compressed
    Deallocate destination
    Return compressed
End Function

'declare function deflateInit2_(byval strm as z_streamp, byval level as long, byval method as long, byval windowBits as long, byval memLevel as long, byval strategy as long, byval version as const zstring ptr, byval stream_size as long) as long

dim as z_stream g'( BINARY , TEXT or ASCII

deflateInit2_(@g,Z_DEFAULT_COMPRESSION,Z_DEFLATED,15 or 16,8,Z_BINARY ,0,Z_TEXT or z_ASCII )

screen 19

dim as string s = ""
for n as longint = 1 to 1000000
    s+= chr( int( rnd * 256 ) )
next n

print "original length   ";len(s)
var p=pack(s)
print "cpmpressed length ";len(p)
var up=unpack(p)
print "return length     ";len(up)
print iif(up=s,"OK","ERROR")

sleep

albert
Posts: 6000
Joined: Sep 28, 2006 2:41
Location: California, USA

Re: Dodicat Zlib

Post by albert »

@Dodicat

I think i got a working formula... Get ready to be rich!!

I'm pondering the decompression...

If it's less or equal to a previous val , then you do nothing..
if it's greater than a previous val , then you add a "0" to the front..

Compresses 1,000,000 bytes , down to under 1,000 bytes , after 100 loops..
Compresses 7% on loop one , and 90% at loop thirty two.. and down to 900 or so , at loop 100.

The decompression should be straight forward... I'll let you know , when I've got the decompression done.
jj2007
Posts: 2326
Joined: Oct 23, 2016 15:28
Location: Roma, Italia
Contact:

Re: Dodicat Zlib

Post by jj2007 »

albert wrote:I'll let you know , when I've got the decompression done.
Sounds good! Not one minute earlier, please.
albert
Posts: 6000
Joined: Sep 28, 2006 2:41
Location: California, USA

Re: Dodicat Zlib

Post by albert »

@Dodicat

Decompression failure!!!
It seems ; that i forgot to address the case of , no bits being set..
it was returning a null string in those cases.. so it wasn't being added to the output...

So i set it to return a "0" if null , and it expands 3,500 bytes per loop of 1,000,000..
srvaldez
Posts: 3379
Joined: Sep 25, 2005 21:54

Re: Dodicat Zlib

Post by srvaldez »

albert, please stop with this compression insanity
albert
Posts: 6000
Joined: Sep 28, 2006 2:41
Location: California, USA

Re: Dodicat Zlib

Post by albert »

@srvaldez

Anyone who manages to compress random data,,
Would win worldwide acclaim , and could earn billions of dollars..

If you could zip a *.zip file and make it smaller??

Since I'm home bound , i can work 8 hours a day on it..
So far I've tried 10's of thousands of formulas.. so far no luck.. But i keep plugging along..

I've pondered and finally realize , that if it compresses , there might be an error or fault in the code..

So I'm no longer going to post compression code... I don't want to get banned again..

But what would ever posses you , to respond to a post directed @Dodicat
When a post starts with an @ , it's directed to a particular person , and no one else should respond..
albert
Posts: 6000
Joined: Sep 28, 2006 2:41
Location: California, USA

Re: Dodicat Zlib

Post by albert »

@Dodicat

I fumbled upon a quirk with Zlib...

If you make the random string s+= chr( int( rnd *128 ) )
It compresses on it's own , with no formula code..
Locked