Problem in Append unicode text

New to FreeBASIC? Post your questions here.
Post Reply
Slinza
Posts: 5
Joined: Jan 30, 2019 17:21

Problem in Append unicode text

Post by Slinza »

Hello!

Tell me what could be the problem. Using the code below, I try to add text to the file, but an error is returned with code 2 (file not found).

Code: Select all

Open "test.txt" For Append Encoding "utf8" As #1
if err > 0 then
   print err
end if
Print #1, "extending test.txt"
Close #1
Using utf16 and utf32 also causes an error.

It works only if you write ascii. Also, if you write Output to the Append location, the text is written.
I tried the compiler version 1.06 and 1.05 (32 and 64 bit).
Tourist Trap
Posts: 2958
Joined: Jun 02, 2015 16:24

Re: Problem in Append unicode text

Post by Tourist Trap »

Hello. I tried your test, and it returns an error 2 and fails on appending anything unless it is specified "ASCII". It's surprising because the documentation says this:
The Input, Output and Append file modes also allow selection of a character encoding to be used when reading from or writing text to the disk file. ASCII or a Unicode encoding may be specified (see the description of the encoding_type parameter above).
jj2007
Posts: 2326
Joined: Oct 23, 2016 15:28
Location: Roma, Italia
Contact:

Re: Problem in Append unicode text

Post by jj2007 »

Under Windows, just use this:

Code: Select all

Open "test.txt" For Append Encoding "65001" As #1
if err > 0 then
   print err
end if
Print #1, "extending test.txt:"
Close #1
print "ok"
Sleep
Note that even this works:

Code: Select all

Open "Привет_мир.txt" For Append Encoding "65001" As #1
- provided your IDE does not provide a Utf8 BOM. And unfortunately the file name is NOT Utf8, despite the 65001=CP_UTF8.

The good news is that this works:

Code: Select all

Open "HelloWorld.txt" For Append Encoding "65001" As #1
if err > 0 then
   print err
end if
Print #1, chr(&hEF);chr(&hBB);chr(&hBF);"extending test.txt: Привет, мир"
Close #1
print "ok"
Sleep
Slinza
Posts: 5
Joined: Jan 30, 2019 17:21

Re: Problem in Append unicode text

Post by Slinza »

This code page designation passes without errors, but in the first version the text is saved in the Windows-1251 format.

In the second version, after manual insertion of BOM characters, the format became UTF-8-BOM, but Russian characters are displayed incorrectly.
Image
Last edited by Slinza on Feb 01, 2019 18:33, edited 4 times in total.
jj2007
Posts: 2326
Joined: Oct 23, 2016 15:28
Location: Roma, Italia
Contact:

Re: Problem in Append unicode text

Post by jj2007 »

Slinza wrote:but Russian characters are displayed incorrectly.
Printing Utf8 text to a file requires a Utf8-aware IDE.
Slinza
Posts: 5
Joined: Jan 30, 2019 17:21

Re: Problem in Append unicode text

Post by Slinza »

I use WinFBE, before compiling the source file I will encode into UTF-8 (BOM).
Which IDE do you use?
Slinza
Posts: 5
Joined: Jan 30, 2019 17:21

Re: Problem in Append unicode text

Post by Slinza »

At the moment, the only solution for me was the following example:

Code: Select all

#include once "crt/stdio.bi"

dim as FILE ptr flU
dim as wString ptr fileLog = @"unicode_test.txt"
dim as WString * 50 wTxt = !"Hello World!, Привет Мир!\n"

flU = _wfopen(fileLog, @"at,ccs=UTF-8")
if flU = 0 then
   printf(!"Error opening file %s.\n", fileLog)
   end
end if

fwrite(@wTxt, sizeof(wString), len(wTxt), flU)

if fclose(flU) then
   printf(!"Error closing file %s.\n", fileLog)
end if

printf(!"Ok.\n")

sleep
But still I would like to understand why the claimed function does not work correctly ...
dodicat
Posts: 7979
Joined: Jan 10, 2006 20:30
Location: Scotland

Re: Problem in Append unicode text

Post by dodicat »

You are correct Slinza, it doesn't work (append and utf-8)
Print or put fails to do anything, but the file is created.
So it looks like a bug as it stands.
jj2007 has chosen the utf8 codepage number for the string, and it seems to work that way.
But who would have thought of doing that? (apart from jj2007)
Imortis
Moderator
Posts: 1923
Joined: Jun 02, 2005 15:10
Location: USA
Contact:

Re: Problem in Append unicode text

Post by Imortis »

Bumping this thread because it seems this is still a problem. Is this an bug in the compiler or a misunderstanding of the requirements for UTF Append?

Does anyone have an more info on this?
Vinion
Posts: 19
Joined: Sep 08, 2022 6:27

Re: Problem in Append unicode text

Post by Vinion »

I confirm that there is a bug using append with Encoding

When I try this Open myFilePath For Append encoding "UTF8" As #1 then I cannot append to a file
But if I try Open myFilePath For Append As #1 then I can
coderJeff
Site Admin
Posts: 4317
Joined: Nov 04, 2005 14:23
Location: Ontario, Canada
Contact:

Re: Problem in Append unicode text

Post by coderJeff »

Problem with append should be fixed now, see :
https://github.com/freebasic/fbc/issues/393
https://github.com/freebasic/fbc/pull/394
https://github.com/freebasic/fbc/pull/395
Thanks to nsiatras on github for posting the report.

Just saw this, reading the entire thread:
jj2007 wrote: Feb 01, 2019 16:44 Under Windows, just use this:

Code: Select all

Open "test.txt" For Append Encoding "65001" As #1
...
"65001" is not a valid encoding and the file is opened in ascii by default. I guess that's another bug.

Code: Select all

#cmdline "-e"
#include once "file.bi"

Open "HelloWorld.txt" For Append Encoding "65001" As #1
if( err > 1 ) then
	print "unable to open file"
	end
end if

var encod = FileAttr( 1, fbFileAttrEncoding )

print "Encoding mode: ";
select case encod
case fbFileEncodASCII : print "fbFileEncodASCII"
case fbFileEncodUTF8  : print "fbFileEncodUTF8"
case fbFileEncodUTF16 : print "fbFileEncodUTF16"
case fbFileEncodUTF32 : print "fbFileEncodUTF32"
case else
	print "unknown"
end select 

close #1
sleep
Post Reply