TSNE problem.

New to FreeBASIC? Post your questions here.
PhiltheBear
Posts: 26
Joined: Jul 08, 2014 17:45

TSNE problem.

Postby PhiltheBear » Jul 13, 2020 18:39

Or possibly not.

I used the supplied test_client.bas program with TSNE_V3.bi using one of my own websites as the website to get and all worked perfectly.

I am trying to produce a program that will get a .csv file from a website but if I program that site instead of mine I get a "No IPV6 supported" error. I have tested and my connection to the web is IPV4. What I'm having a problem with is that if I go to the target website via a browser (I've tried Edge, Chrome and Firefox) I can download the .csv file with no problem. Therefore, I can't see why there should be an IPV6 problem. This is my first attempt at trying to do something like this over the internet so it's almost certainly something simple but I have no idea what. Can anyone shed any light?

One other thing - the website I'm trying to connect to is https rather than http but I can't see why that would generate an IPV6 error.
jj2007
Posts: 1645
Joined: Oct 23, 2016 15:28
Location: Roma, Italia
Contact:

Re: TSNE problem.

Postby jj2007 » Jul 13, 2020 18:43

Can you give the URL for our testing, or is it a problem?
PhiltheBear
Posts: 26
Joined: Jul 08, 2014 17:45

Re: TSNE problem.

Postby PhiltheBear » Jul 13, 2020 21:24

https://promo.betfair.com/betfairsp/pri ... 072020.csv is the file I'm trying to get.

Normally I go to https://promo.betfair.com/betfairsp/prices and click on a file to download - but I'd like to automate the procedure to save some time.
jj2007
Posts: 1645
Joined: Oct 23, 2016 15:28
Location: Roma, Italia
Contact:

Re: TSNE problem.

Postby jj2007 » Jul 14, 2020 0:38

Thanks, Phil. I've tested the URL with a WinInet-based program, and it downloads just fine. Same for URLDownloadToFileA...
grindstone
Posts: 744
Joined: May 05, 2015 5:35
Location: Germany

Re: TSNE problem.

Postby grindstone » Jul 14, 2020 13:08

AFAIK TSNE can't handle https yet. I would recommend using libcurl for that kind of download.

Code: Select all

#Include Once "curl.bi"
#Include "vbcompat.bi"

Type tCurlInfo
   curl As CURL Ptr
   ff As Integer
   totlen As Double
   dllen As Double
   timeout As Double = 3.0
   process As boolean = FALSE
   content As String
End Type

Declare Function download_curl(url As String, tout As Double = 10.0) As String
Declare Function write_callback_curl Cdecl (buffer As Byte Ptr, size As Long, nitems As Long, info As tCurlInfo Ptr) As Integer

Function download_curl(url As String, tout As Double = 10.0) As String
   Dim As tCurlInfo curlinfo
   Dim As String dlurl, tmp
   Dim As Double timeout
      
   curlinfo.curl = curl_easy_init()
   If curlinfo.curl = 0 Then
      Print "ERROR"
      Return ""
   End If

   '' set url and callback
   curl_easy_setopt(curlinfo.curl, CURLOPT_URL, url)
   curl_easy_setopt(curlinfo.curl, CURLOPT_WRITEDATA, @curlinfo)
   curl_easy_setopt(curlinfo.curl, CURLOPT_WRITEFUNCTION, @write_callback_curl)
   
   curl_easy_setopt(curlinfo.curl, CURLOPT_SSL_VERIFYPEER, 0)
   
   curl_easy_setopt(curlinfo.curl, CURLOPT_FOLLOWLOCATION, 1)
   
   curlinfo.timeout = tout
         
   curl_easy_perform(curlinfo.curl) 'execute..
   curl_easy_cleanup(curlinfo.curl) 'shutdown
   
   Close curlinfo.ff
   
   Return curlinfo.content
End Function

Function write_callback_curl Cdecl (buffer As Byte Ptr, size As Long, nitems As Long, info As tCurlInfo Ptr) As Integer
   Dim As Integer x, bytes = size * nitems
   Dim As Double tot_len, dl_len, timeout = Timer + info->timeout
         
   curl_easy_getinfo(info->curl, CURLINFO_CONTENT_LENGTH_DOWNLOAD, @info->totlen )
   curl_easy_getinfo(info->curl, CURLINFO_SIZE_DOWNLOAD, @info->dllen )
   
   If info->ff Then
      If info->totlen = -1 Then
         Print Format(info->dllen/1024/1024,"#.##");" MB (ESC to terminate)"
      Else
         Print Int(100 * info->dllen / info->totlen);"% of ";Format(info->totlen/1024/1024,"#.##");" MB (ESC to terminate)"
      EndIf
      Locate CsrLin - 1,1,0
      If info->totlen = info->dllen Then
         Print
      EndIf
      Put #info->ff, ,*buffer,bytes 'write downloaded bytes to file
   Else      
      For x = 0 To bytes - 1
         info->content += Chr(buffer[x])
      Next   
   EndIf
   
   If (InKey = Chr(27)) Or (Timer > timeout) Then
      Print
      Return 0
   EndIf
   
   Return bytes
End Function

Dim As String content

content = download_curl("https://promo.betfair.com/betfairsp/prices/dwbfpricesukwin13072020.csv")

Print "done"

Open "C:\downloadtest.csv" For Output As #1
Print #1, content;
Close

Sleep
jj2007
Posts: 1645
Joined: Oct 23, 2016 15:28
Location: Roma, Italia
Contact:

Re: TSNE problem.

Postby jj2007 » Jul 14, 2020 14:07

grindstone wrote:AFAIK TSNE can't handle https yet. I would recommend using libcurl for that kind of download.[code]#Include Once "curl.bi"
URLDownloadToFile is a one-liner that does the job - if you have no problem with AV warnings, and you are running Windows, I'd go for that.
PhiltheBear
Posts: 26
Joined: Jul 08, 2014 17:45

Re: TSNE problem.

Postby PhiltheBear » Jul 14, 2020 15:05

grindstone wrote:AFAIK TSNE can't handle https yet. I would recommend using libcurl for that kind of download.


This all being new to me I copied your source and tried to run it. I got the following error:

C:\PROGRA~2\FREEBA~1\bin\win32\ld.exe: cannot find -lcurl (I'm using FBIde rather than command line)

I have no idea where -lcurl is supposed to be


jj2007 wrote: URLDownloadToFile is a one-liner that does the job - if you have no problem with AV warnings, and you are running Windows, I'd go for that.


I am running Windows but incorporating URLDownloadToFile is something I have no clue how to do. Including other people's libraries is a whole new experience for me :)
srvaldez
Posts: 2482
Joined: Sep 25, 2005 21:54

Re: TSNE problem.

Postby srvaldez » Jul 14, 2020 15:27

you can download pre-compiled libcurl from https://curl.haxx.se/windows/
just copy libcurl.a to your FB lib/win32 folder
jj2007
Posts: 1645
Joined: Oct 23, 2016 15:28
Location: Roma, Italia
Contact:

Re: TSNE problem.

Postby jj2007 » Jul 14, 2020 18:45

Sorry, I thought it was a one-liner, but FB is slightly more complicated...

Code: Select all

#include "Windows.bi"

Dim URLDownloadToFileA as function (byval pCaller as any ptr, _
      byval szURL as zstring ptr, _
      byval szFileName as zstring ptr, byval dwResv as uinteger, _
      byval lpfnCB as any ptr) as integer

URLDownloadToFileA = GetProcAddress(dylibload("urlmon"), "URLDownloadToFileA")

Dim DownloadOK As long=URLDownloadToFileA(0, @"https://promo.betfair.com/betfairsp/prices/dwbfpricesukwin13072020.csv", @"tmp.csv", 0, 0)
print "OK if zero: "; DownloadOK
sleep
dodicat
Posts: 6648
Joined: Jan 10, 2006 20:30
Location: Scotland

Re: TSNE problem.

Postby dodicat » Jul 14, 2020 18:50

Win 10
One of these methods should work.

Code: Select all



#include "file.bi"
Sub savefile(filename As String,p As String)
  Dim As Integer n
  n=Freefile
  If Open (filename For Binary Access Write As #n)=0 Then
    Put #n,,p
    Close
  Else
    Print "Unable to save " + filename
  End If
End Sub

Function loadfile(file As String) As String
   If Fileexists(file)=0 Then Print file;" not found":Sleep:End
  Var  f=Freefile
  Open file For Binary Access Read As #f
  Dim As String text
  If Lof(f) > 0 Then
    text = String(Lof(f), 0)
    Get #f, , text
  End If
  Close #f
  Return text
End Function

Dim As String site="https://promo.betfair.com/betfairsp/prices/dwbfpricesukwin13072020.csv"


Sub download(site As String,newfile As String)
  If Instr(site," ") Then site=Chr(34)+site+Chr(34)
  savefile( Curdir +"\"+newfile,"") 'create an empty file
  shell "bitsadmin /create mydownload"
  Shell  "bitsadmin  /transfer mydownload  /download  /priority normal " + _
  site +" "+ Chr(34) +  Curdir +"\"+ newfile +Chr(34)
  shell "bitsadmin /complete mydownload"
  If Len(loadfile(newfile)) Then
    Shell Chr(34) +Curdir +  "\"+newfile + Chr(34)
  End If
End Sub

sub download2(site As String,newfile As String)
   If Instr(site," ") Then site=Chr(34)+site+Chr(34)
  shell "powershell Invoke-WebRequest "+site+ "/ -OutFile "+ curdir +"/"+ newfile
  If Len(loadfile(newfile)) Then
    Shell Chr(34) +Curdir +  "\"+newfile + Chr(34)
  End If
  end sub

sub cleanup destructor '(only for download using bitsadmin)
   shell "bitsadmin /complete mydownload"
   sleep 1000
   end sub

download2(site,"New.csv")

 
PhiltheBear
Posts: 26
Joined: Jul 08, 2014 17:45

Re: TSNE problem.

Postby PhiltheBear » Jul 17, 2020 18:31

jj2007 wrote:Sorry, I thought it was a one-liner, but FB is slightly more complicated..


I got a strange result running this code (probably my fault). Firstly I copied and ran it as is without issue. Trying to modify it so that I could download a number of files I altered it bit by bit, firstly by putting the filenames "https://promo.betfair.com/betfairsp/prices/dwbfpricesukwin13072020.csv" and "tmp.csv" into strings and using those strings. That didn't work until I removed the @ character, but then it did. However, it also produced an error when compiling for the line: URLDownloadToFileA = GetProcAddress(dylibload("urlmon"), "URLDownloadToFileA") which came up as "Suspicious pointer assignment"

Despite producing that error the program still appeared to work but only for one file. Trying to use multiple file names, one at a time, doesn't work*. I also downloaded the latest FreeBasic software - having discovered that windows.bi didn't exist on my PC (I had windows9.bi !). It was sort of interesting that it did work without, apparently, windows.bi. However, I still get the "Suspicious pointer assignment" error.

Can you offer any further ideas?

* I have a list of file names which I would use weekly e.g. dwbfpricesukwin17072020.csv, dwbfpricesukwin16072020.csv, dwbfpricesukwin15072020.csv,
dwbfpricesukwin14072020.csv, etc. I simply constructed a loop to read those in from a file and created for each the appropriate URL address and a file to store the results.
jj2007
Posts: 1645
Joined: Oct 23, 2016 15:28
Location: Roma, Italia
Contact:

Re: TSNE problem.

Postby jj2007 » Jul 17, 2020 19:28

Phil,
That should work fine in a loop. When you print the URLs before your DownloadOK=URLDownloadToFileA(0, szUrl, destFile, 0, 0), do they all look ok? Can you post your code?

Btw the "Suspicious pointer assignment" is just a warning, you can ignore it. FB and WinAPI don't understand each other well ;-)
dodicat
Posts: 6648
Joined: Jan 10, 2006 20:30
Location: Scotland

Re: TSNE problem.

Postby dodicat » Jul 17, 2020 23:50

This works here (win 10)
Takes a few seconds for powershell to fire up, after that only a few seconds more.

Code: Select all

 


#include "file.bi"

sub download(site As String,newfile As String)
   If Instr(site," ") Then site=Chr(34)+site+Chr(34)
  shell "powershell Invoke-WebRequest "+site+ "/ -OutFile "+ curdir +"/"+ newfile
end sub


dim as string site="https://promo.betfair.com/betfairsp/prices/"
Dim As String file(1 to 5)={"dwbfpricesukwin13072020.csv", _
                             "dwbfpricesukwin17072020.csv", _
                             "dwbfpricesukwin16072020.csv", _
                             "dwbfpricesukwin15072020.csv", _
                             "dwbfpricesukwin14072020.csv"}
                             
  for n as long=1 to ubound(file)
    var s=site+file(n)
    var g="new"+str(n)+".csv"
download(s,g)
print g;"   ";iif(fileexists(g),"OK","Error"); "  = ";file(n);"   size = ";filelen(g)
next n
print "press a key to delete these files"
sleep

for n as long=1 to ubound(file)
  kill "new"+str(n)+".csv"
  print iif(fileexists("new"+str(n)+".csv"),"Error deleting","OK")
next n
sleep

 
PhiltheBear
Posts: 26
Joined: Jul 08, 2014 17:45

Re: TSNE problem.

Postby PhiltheBear » Jul 21, 2020 11:02

dodicat wrote:Win 10
One of these methods should work.


I was able to rework some of this and used it to do what I needed to do. Thank you.
PhiltheBear
Posts: 26
Joined: Jul 08, 2014 17:45

Re: TSNE problem.

Postby PhiltheBear » Jul 21, 2020 11:05

I would like to thank you all for your assistance. I have managed to create a program that does exactly what I needed derived from the code you supplied. If I ever meet you in a hostelry the drinks will be on me.

Thank you all for stepping in and trying to help. It was much appreciated.

Return to “Beginners”

Who is online

Users browsing this forum: No registered users and 5 guests