TSNE problem.

New to FreeBASIC? Post your questions here.
Post Reply
PhiltheBear
Posts: 30
Joined: Jul 08, 2014 17:45

TSNE problem.

Post by PhiltheBear »

Or possibly not.

I used the supplied test_client.bas program with TSNE_V3.bi using one of my own websites as the website to get and all worked perfectly.

I am trying to produce a program that will get a .csv file from a website but if I program that site instead of mine I get a "No IPV6 supported" error. I have tested and my connection to the web is IPV4. What I'm having a problem with is that if I go to the target website via a browser (I've tried Edge, Chrome and Firefox) I can download the .csv file with no problem. Therefore, I can't see why there should be an IPV6 problem. This is my first attempt at trying to do something like this over the internet so it's almost certainly something simple but I have no idea what. Can anyone shed any light?

One other thing - the website I'm trying to connect to is https rather than http but I can't see why that would generate an IPV6 error.
jj2007
Posts: 2326
Joined: Oct 23, 2016 15:28
Location: Roma, Italia
Contact:

Re: TSNE problem.

Post by jj2007 »

Can you give the URL for our testing, or is it a problem?
PhiltheBear
Posts: 30
Joined: Jul 08, 2014 17:45

Re: TSNE problem.

Post by PhiltheBear »

https://promo.betfair.com/betfairsp/pri ... 072020.csv is the file I'm trying to get.

Normally I go to https://promo.betfair.com/betfairsp/prices and click on a file to download - but I'd like to automate the procedure to save some time.
jj2007
Posts: 2326
Joined: Oct 23, 2016 15:28
Location: Roma, Italia
Contact:

Re: TSNE problem.

Post by jj2007 »

Thanks, Phil. I've tested the URL with a WinInet-based program, and it downloads just fine. Same for URLDownloadToFileA...
grindstone
Posts: 862
Joined: May 05, 2015 5:35
Location: Germany

Re: TSNE problem.

Post by grindstone »

AFAIK TSNE can't handle https yet. I would recommend using libcurl for that kind of download.

Code: Select all

#Include Once "curl.bi"
#Include "vbcompat.bi"

Type tCurlInfo
	curl As CURL Ptr
	ff As Integer
	totlen As Double
	dllen As Double
	timeout As Double = 3.0
	process As boolean = FALSE
	content As String
End Type

Declare Function download_curl(url As String, tout As Double = 10.0) As String
Declare Function write_callback_curl Cdecl (buffer As Byte Ptr, size As Long, nitems As Long, info As tCurlInfo Ptr) As Integer

Function download_curl(url As String, tout As Double = 10.0) As String
	Dim As tCurlInfo curlinfo 
	Dim As String dlurl, tmp
	Dim As Double timeout
		
	curlinfo.curl = curl_easy_init()
	If curlinfo.curl = 0 Then
		Print "ERROR"
		Return ""
	End If

	'' set url and callback
	curl_easy_setopt(curlinfo.curl, CURLOPT_URL, url)
	curl_easy_setopt(curlinfo.curl, CURLOPT_WRITEDATA, @curlinfo)
	curl_easy_setopt(curlinfo.curl, CURLOPT_WRITEFUNCTION, @write_callback_curl)
	
	curl_easy_setopt(curlinfo.curl, CURLOPT_SSL_VERIFYPEER, 0)
	
	curl_easy_setopt(curlinfo.curl, CURLOPT_FOLLOWLOCATION, 1)
	
	curlinfo.timeout = tout
			
	curl_easy_perform(curlinfo.curl) 'execute..
	curl_easy_cleanup(curlinfo.curl) 'shutdown
	
	Close curlinfo.ff
	
	Return curlinfo.content
End Function

Function write_callback_curl Cdecl (buffer As Byte Ptr, size As Long, nitems As Long, info As tCurlInfo Ptr) As Integer
	Dim As Integer x, bytes = size * nitems
	Dim As Double tot_len, dl_len, timeout = Timer + info->timeout
			
	curl_easy_getinfo(info->curl, CURLINFO_CONTENT_LENGTH_DOWNLOAD, @info->totlen )
	curl_easy_getinfo(info->curl, CURLINFO_SIZE_DOWNLOAD, @info->dllen )
	
	If info->ff Then
		If info->totlen = -1 Then
			Print Format(info->dllen/1024/1024,"#.##");" MB (ESC to terminate)"
		Else
			Print Int(100 * info->dllen / info->totlen);"% of ";Format(info->totlen/1024/1024,"#.##");" MB (ESC to terminate)"
		EndIf
		Locate CsrLin - 1,1,0
		If info->totlen = info->dllen Then
			Print
		EndIf
		Put #info->ff, ,*buffer,bytes 'write downloaded bytes to file
	Else		
		For x = 0 To bytes - 1
			info->content += Chr(buffer[x])
		Next	
	EndIf
	
	If (InKey = Chr(27)) Or (Timer > timeout) Then
		Print
		Return 0
	EndIf
	
	Return bytes
End Function

Dim As String content

content = download_curl("https://promo.betfair.com/betfairsp/prices/dwbfpricesukwin13072020.csv")

Print "done"

Open "C:\downloadtest.csv" For Output As #1
Print #1, content;
Close

Sleep
jj2007
Posts: 2326
Joined: Oct 23, 2016 15:28
Location: Roma, Italia
Contact:

Re: TSNE problem.

Post by jj2007 »

grindstone wrote:AFAIK TSNE can't handle https yet. I would recommend using libcurl for that kind of download.

Code: Select all

#Include Once "curl.bi"[/quote]URLDownloadToFile is a one-liner that does the job - if you have no problem with AV warnings, and you are running Windows, I'd go for that.
PhiltheBear
Posts: 30
Joined: Jul 08, 2014 17:45

Re: TSNE problem.

Post by PhiltheBear »

grindstone wrote:AFAIK TSNE can't handle https yet. I would recommend using libcurl for that kind of download.
This all being new to me I copied your source and tried to run it. I got the following error:

C:\PROGRA~2\FREEBA~1\bin\win32\ld.exe: cannot find -lcurl (I'm using FBIde rather than command line)

I have no idea where -lcurl is supposed to be

jj2007 wrote: URLDownloadToFile is a one-liner that does the job - if you have no problem with AV warnings, and you are running Windows, I'd go for that.
I am running Windows but incorporating URLDownloadToFile is something I have no clue how to do. Including other people's libraries is a whole new experience for me :)
srvaldez
Posts: 3373
Joined: Sep 25, 2005 21:54

Re: TSNE problem.

Post by srvaldez »

you can download pre-compiled libcurl from https://curl.haxx.se/windows/
just copy libcurl.a to your FB lib/win32 folder
jj2007
Posts: 2326
Joined: Oct 23, 2016 15:28
Location: Roma, Italia
Contact:

Re: TSNE problem.

Post by jj2007 »

Sorry, I thought it was a one-liner, but FB is slightly more complicated...

Code: Select all

#include "Windows.bi"

Dim URLDownloadToFileA as function (byval pCaller as any ptr, _
      byval szURL as zstring ptr, _
      byval szFileName as zstring ptr, byval dwResv as uinteger, _
      byval lpfnCB as any ptr) as integer

URLDownloadToFileA = GetProcAddress(dylibload("urlmon"), "URLDownloadToFileA")

Dim DownloadOK As long=URLDownloadToFileA(0, @"https://promo.betfair.com/betfairsp/prices/dwbfpricesukwin13072020.csv", @"tmp.csv", 0, 0)
print "OK if zero: "; DownloadOK
sleep
dodicat
Posts: 7976
Joined: Jan 10, 2006 20:30
Location: Scotland

Re: TSNE problem.

Post by dodicat »

Win 10
One of these methods should work.

Code: Select all



#include "file.bi"
Sub savefile(filename As String,p As String)
  Dim As Integer n
  n=Freefile
  If Open (filename For Binary Access Write As #n)=0 Then
    Put #n,,p
    Close
  Else
    Print "Unable to save " + filename
  End If
End Sub

Function loadfile(file As String) As String
	If Fileexists(file)=0 Then Print file;" not found":Sleep:End
  Var  f=Freefile
  Open file For Binary Access Read As #f
  Dim As String text
  If Lof(f) > 0 Then
    text = String(Lof(f), 0)
    Get #f, , text
  End If
  Close #f
  Return text
End Function

Dim As String site="https://promo.betfair.com/betfairsp/prices/dwbfpricesukwin13072020.csv"


Sub download(site As String,newfile As String)
  If Instr(site," ") Then site=Chr(34)+site+Chr(34)
  savefile( Curdir +"\"+newfile,"") 'create an empty file
  shell "bitsadmin /create mydownload"
  Shell  "bitsadmin  /transfer mydownload  /download  /priority normal " + _
  site +" "+ Chr(34) +  Curdir +"\"+ newfile +Chr(34)
  shell "bitsadmin /complete mydownload"
  If Len(loadfile(newfile)) Then
    Shell Chr(34) +Curdir +  "\"+newfile + Chr(34)
  End If
End Sub

sub download2(site As String,newfile As String)
   If Instr(site," ") Then site=Chr(34)+site+Chr(34)
  shell "powershell Invoke-WebRequest "+site+ "/ -OutFile "+ curdir +"/"+ newfile
  If Len(loadfile(newfile)) Then
    Shell Chr(34) +Curdir +  "\"+newfile + Chr(34)
  End If
  end sub

sub cleanup destructor '(only for download using bitsadmin)
   shell "bitsadmin /complete mydownload"
   sleep 1000
   end sub

download2(site,"New.csv")

  
PhiltheBear
Posts: 30
Joined: Jul 08, 2014 17:45

Re: TSNE problem.

Post by PhiltheBear »

jj2007 wrote:Sorry, I thought it was a one-liner, but FB is slightly more complicated..
I got a strange result running this code (probably my fault). Firstly I copied and ran it as is without issue. Trying to modify it so that I could download a number of files I altered it bit by bit, firstly by putting the filenames "https://promo.betfair.com/betfairsp/pri ... 072020.csv" and "tmp.csv" into strings and using those strings. That didn't work until I removed the @ character, but then it did. However, it also produced an error when compiling for the line: URLDownloadToFileA = GetProcAddress(dylibload("urlmon"), "URLDownloadToFileA") which came up as "Suspicious pointer assignment"

Despite producing that error the program still appeared to work but only for one file. Trying to use multiple file names, one at a time, doesn't work*. I also downloaded the latest FreeBasic software - having discovered that windows.bi didn't exist on my PC (I had windows9.bi !). It was sort of interesting that it did work without, apparently, windows.bi. However, I still get the "Suspicious pointer assignment" error.

Can you offer any further ideas?

* I have a list of file names which I would use weekly e.g. dwbfpricesukwin17072020.csv, dwbfpricesukwin16072020.csv, dwbfpricesukwin15072020.csv,
dwbfpricesukwin14072020.csv, etc. I simply constructed a loop to read those in from a file and created for each the appropriate URL address and a file to store the results.
jj2007
Posts: 2326
Joined: Oct 23, 2016 15:28
Location: Roma, Italia
Contact:

Re: TSNE problem.

Post by jj2007 »

Phil,
That should work fine in a loop. When you print the URLs before your DownloadOK=URLDownloadToFileA(0, szUrl, destFile, 0, 0), do they all look ok? Can you post your code?

Btw the "Suspicious pointer assignment" is just a warning, you can ignore it. FB and WinAPI don't understand each other well ;-)
dodicat
Posts: 7976
Joined: Jan 10, 2006 20:30
Location: Scotland

Re: TSNE problem.

Post by dodicat »

This works here (win 10)
Takes a few seconds for powershell to fire up, after that only a few seconds more.

Code: Select all

 


#include "file.bi"

sub download(site As String,newfile As String)
   If Instr(site," ") Then site=Chr(34)+site+Chr(34)
  shell "powershell Invoke-WebRequest "+site+ "/ -OutFile "+ curdir +"/"+ newfile
end sub


dim as string site="https://promo.betfair.com/betfairsp/prices/"
Dim As String file(1 to 5)={"dwbfpricesukwin13072020.csv", _
                             "dwbfpricesukwin17072020.csv", _
                             "dwbfpricesukwin16072020.csv", _
                             "dwbfpricesukwin15072020.csv", _
                             "dwbfpricesukwin14072020.csv"}
                             
  for n as long=1 to ubound(file)
    var s=site+file(n)
    var g="new"+str(n)+".csv"
download(s,g)
print g;"   ";iif(fileexists(g),"OK","Error"); "  = ";file(n);"   size = ";filelen(g)
next n
print "press a key to delete these files"
sleep

for n as long=1 to ubound(file)
  kill "new"+str(n)+".csv"
  print iif(fileexists("new"+str(n)+".csv"),"Error deleting","OK")
next n
sleep

 
PhiltheBear
Posts: 30
Joined: Jul 08, 2014 17:45

Re: TSNE problem.

Post by PhiltheBear »

dodicat wrote:Win 10
One of these methods should work.
I was able to rework some of this and used it to do what I needed to do. Thank you.
PhiltheBear
Posts: 30
Joined: Jul 08, 2014 17:45

Re: TSNE problem.

Post by PhiltheBear »

I would like to thank you all for your assistance. I have managed to create a program that does exactly what I needed derived from the code you supplied. If I ever meet you in a hostelry the drinks will be on me.

Thank you all for stepping in and trying to help. It was much appreciated.
Post Reply