luc14n0/glib
1
0
Fork 0
mirror of https://gitlab.gnome.org/GNOME/glib.git synced 2024-09-20 09:16:15 +02:00
Code Issues Packages Projects Releases Wiki Activity
cb8bfa758f
glib/tests/utf8.txt

292 lines
3.1 KiB
Plaintext
Raw Normal View History

move $enable_debug down below checks for GCC to avoid setting CFLAGS Fri Jan 5 11:25:42 2001 Owen Taylor <otaylor@redhat.com> * configure.in (PACKAGE): move $enable_debug down below checks for GCC to avoid setting CFLAGS prematurely, change checks to avoid adding -g twice. * gutf8.c (g_ucs4_to_utf8): Support len < 0 to mean 0 termination. * gutf8.c (g_utf8_to_ucs4): Terminate result with 0. * tests/mainloop-test.c (main): Fix uses of g_main_loop_destroy(). * tests/unicode-encoding.c tests/Makefile.am tests/utf8.txt: Tests for unicode-conversion code. * gconvert.c (g_convert, g_convert_with_fallback): work around a couple of GNU libc bugs. * gconvert.[ch] (g_{locale,filename}_{to,from}_utf8): Standardize arguments to match g_convert(). Document. * gunicode.[ch]: - Implement conversion functions to and from UTF-16 - Standardize unicode conversion functions on prototype like g_convert. - Add a lot of error checking to unicode conversion functions. * gunicode.[ch] (g_utf8_to_ucs4_fast): Add fast, non-checking variant of g_utf8_to_ucs4. * gutf8.c (g_utf8_validate): - add g_return_if_fail (str != NULL). - add checks for overlong strings, non-valid Unicode characters (>= 110000) and single surrogates.
2001-01-05 22:22:47 +01:00
# This file is derived from
#
# http://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-test.txt
#
# Which was created by Markus Kuhn <mkuhn@acm.org> - 2000-09-02
#
# lines begining with # and blank lines are ignored
#
# Beyond that, this file consists of a series of test cases. Each test case consists of
# 2 or 3 lines:
#
# 1. A UTF-8 string
# 2. A status
# VALID : The string is a valid UTF-8 representation of valid Unicode
# INCOMPLETE : The string has a partial character at the end
# NOTUNICODE : The string is valid UTF-8, but the characters represented
# are not valid unicode (
# OVERLONG : The string includes overlong sequences
# MALFORMED : The string is not valid UTF-8
# 3. If the status is VALID or NOTUNICODE, the UCS-4 representation of the string,
# as a series of hex numbers.
# 1 Some correct UTF-8 text
κόσμε
VALID
03ba 1f79 03c3 03bc 03b5
# 2.1 First possible sequence of a certain length
#
# FIXME - handle NULLS?
#
# [ NULL BYTE ]
#VALID
#0000
€
VALID
0080
ࠀ
VALID
0800
𐀀
VALID
00010000
<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>
NOTUNICODE
00200000
<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>
NOTUNICODE
04000000

VALID
0000007f
߿
VALID
000007ff
￿
More clean up for Unicode corrigendum #9 Fix some more testcases.
2013-03-19 04:28:27 +01:00
VALID
move $enable_debug down below checks for GCC to avoid setting CFLAGS Fri Jan 5 11:25:42 2001 Owen Taylor <otaylor@redhat.com> * configure.in (PACKAGE): move $enable_debug down below checks for GCC to avoid setting CFLAGS prematurely, change checks to avoid adding -g twice. * gutf8.c (g_ucs4_to_utf8): Support len < 0 to mean 0 termination. * gutf8.c (g_utf8_to_ucs4): Terminate result with 0. * tests/mainloop-test.c (main): Fix uses of g_main_loop_destroy(). * tests/unicode-encoding.c tests/Makefile.am tests/utf8.txt: Tests for unicode-conversion code. * gconvert.c (g_convert, g_convert_with_fallback): work around a couple of GNU libc bugs. * gconvert.[ch] (g_{locale,filename}_{to,from}_utf8): Standardize arguments to match g_convert(). Document. * gunicode.[ch]: - Implement conversion functions to and from UTF-16 - Standardize unicode conversion functions on prototype like g_convert. - Add a lot of error checking to unicode conversion functions. * gunicode.[ch] (g_utf8_to_ucs4_fast): Add fast, non-checking variant of g_utf8_to_ucs4. * gutf8.c (g_utf8_validate): - add g_return_if_fail (str != NULL). - add checks for overlong strings, non-valid Unicode characters (>= 110000) and single surrogates.
2001-01-05 22:22:47 +01:00
0000ffff
<EFBFBD><EFBFBD><EFBFBD><EFBFBD>
NOTUNICODE
001fffff
<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>
NOTUNICODE
03ffffff
<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>
NOTUNICODE
7fffffff
# 2.3 Other boundary conditions
퟿
VALID
d7ff
VALID
e000
<EFBFBD>
VALID
fffd
Change instances of U+10ffff to U+10fffd, since that is the last valid 2003-07-31 Noah Levitt <nlevitt@columbia.edu> * tests/utf8.txt: Change instances of U+10ffff to U+10fffd, since that is the last valid unicode character. Add check that U+10ffff is NOTUNICODE. (#118730)
2003-07-31 18:48:54 +02:00
􏿽
move $enable_debug down below checks for GCC to avoid setting CFLAGS Fri Jan 5 11:25:42 2001 Owen Taylor <otaylor@redhat.com> * configure.in (PACKAGE): move $enable_debug down below checks for GCC to avoid setting CFLAGS prematurely, change checks to avoid adding -g twice. * gutf8.c (g_ucs4_to_utf8): Support len < 0 to mean 0 termination. * gutf8.c (g_utf8_to_ucs4): Terminate result with 0. * tests/mainloop-test.c (main): Fix uses of g_main_loop_destroy(). * tests/unicode-encoding.c tests/Makefile.am tests/utf8.txt: Tests for unicode-conversion code. * gconvert.c (g_convert, g_convert_with_fallback): work around a couple of GNU libc bugs. * gconvert.[ch] (g_{locale,filename}_{to,from}_utf8): Standardize arguments to match g_convert(). Document. * gunicode.[ch]: - Implement conversion functions to and from UTF-16 - Standardize unicode conversion functions on prototype like g_convert. - Add a lot of error checking to unicode conversion functions. * gunicode.[ch] (g_utf8_to_ucs4_fast): Add fast, non-checking variant of g_utf8_to_ucs4. * gutf8.c (g_utf8_validate): - add g_return_if_fail (str != NULL). - add checks for overlong strings, non-valid Unicode characters (>= 110000) and single surrogates.
2001-01-05 22:22:47 +01:00
VALID
Change instances of U+10ffff to U+10fffd, since that is the last valid 2003-07-31 Noah Levitt <nlevitt@columbia.edu> * tests/utf8.txt: Change instances of U+10ffff to U+10fffd, since that is the last valid unicode character. Add check that U+10ffff is NOTUNICODE. (#118730)
2003-07-31 18:48:54 +02:00
0010fffd
􏿿
More clean up for Unicode corrigendum #9 Fix some more testcases.
2013-03-19 04:28:27 +01:00
VALID
move $enable_debug down below checks for GCC to avoid setting CFLAGS Fri Jan 5 11:25:42 2001 Owen Taylor <otaylor@redhat.com> * configure.in (PACKAGE): move $enable_debug down below checks for GCC to avoid setting CFLAGS prematurely, change checks to avoid adding -g twice. * gutf8.c (g_ucs4_to_utf8): Support len < 0 to mean 0 termination. * gutf8.c (g_utf8_to_ucs4): Terminate result with 0. * tests/mainloop-test.c (main): Fix uses of g_main_loop_destroy(). * tests/unicode-encoding.c tests/Makefile.am tests/utf8.txt: Tests for unicode-conversion code. * gconvert.c (g_convert, g_convert_with_fallback): work around a couple of GNU libc bugs. * gconvert.[ch] (g_{locale,filename}_{to,from}_utf8): Standardize arguments to match g_convert(). Document. * gunicode.[ch]: - Implement conversion functions to and from UTF-16 - Standardize unicode conversion functions on prototype like g_convert. - Add a lot of error checking to unicode conversion functions. * gunicode.[ch] (g_utf8_to_ucs4_fast): Add fast, non-checking variant of g_utf8_to_ucs4. * gutf8.c (g_utf8_validate): - add g_return_if_fail (str != NULL). - add checks for overlong strings, non-valid Unicode characters (>= 110000) and single surrogates.
2001-01-05 22:22:47 +01:00
0010ffff
<EFBFBD><EFBFBD><EFBFBD><EFBFBD>
NOTUNICODE
00110000
# 3.1 Unexpected continuation bytes
<EFBFBD>
MALFORMED
<EFBFBD>
MALFORMED
<EFBFBD><EFBFBD>
MALFORMED
<EFBFBD><EFBFBD><EFBFBD>
MALFORMED
<EFBFBD><EFBFBD><EFBFBD><EFBFBD>
MALFORMED
<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>
MALFORMED
<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>
MALFORMED
<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>
MALFORMED
<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>
MALFORMED
# 3.2 Lonely start characters
<EFBFBD> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20>
MALFORMED
<EFBFBD> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20>
MALFORMED
<EFBFBD> <20> <20> <20> <20> <20> <20> <20>
MALFORMED
<EFBFBD> <20> <20> <20>
MALFORMED
<EFBFBD> <20>
MALFORMED
# 3.3 Sequences with last continuation byte missing
<EFBFBD>
INCOMPLETE
<EFBFBD><EFBFBD>
INCOMPLETE
<EFBFBD><EFBFBD><EFBFBD>
INCOMPLETE
<EFBFBD><EFBFBD><EFBFBD><EFBFBD>
INCOMPLETE
<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>
INCOMPLETE
<EFBFBD>
INCOMPLETE
<EFBFBD><EFBFBD>
INCOMPLETE
<EFBFBD><EFBFBD><EFBFBD>
INCOMPLETE
<EFBFBD><EFBFBD><EFBFBD><EFBFBD>
INCOMPLETE
<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>
INCOMPLETE
# 3.4 Concatenation of incomplete sequences
<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>
MALFORMED
# 3.5 Impossible bytes
<EFBFBD>
MALFORMED
<EFBFBD>
MALFORMED
<EFBFBD><EFBFBD><EFBFBD><EFBFBD>
MALFORMED
# Examples of an overlong ASCII character
<EFBFBD><EFBFBD>
OVERLONG
<EFBFBD><EFBFBD><EFBFBD>
OVERLONG
<EFBFBD><EFBFBD><EFBFBD><EFBFBD>
OVERLONG
<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>
OVERLONG
<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>
OVERLONG
# Maximum overlong sequences
<EFBFBD><EFBFBD>
OVERLONG
<EFBFBD><EFBFBD><EFBFBD>
OVERLONG
<EFBFBD><EFBFBD><EFBFBD><EFBFBD>
OVERLONG
<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>
OVERLONG
<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>
OVERLONG
# Overlong representation of the NUL character
<EFBFBD><EFBFBD>
OVERLONG
<EFBFBD><EFBFBD><EFBFBD>
OVERLONG
<EFBFBD><EFBFBD><EFBFBD><EFBFBD>
OVERLONG
<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>
OVERLONG
<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>
OVERLONG
# Illegal code positions
# Single UTF-16 surrogates
<EFBFBD><EFBFBD><EFBFBD>
NOTUNICODE
d800
<EFBFBD><EFBFBD><EFBFBD>
NOTUNICODE
db7f
<EFBFBD><EFBFBD><EFBFBD>
NOTUNICODE
db80
<EFBFBD><EFBFBD><EFBFBD>
NOTUNICODE
dbff
<EFBFBD><EFBFBD><EFBFBD>
NOTUNICODE
dc00
<EFBFBD><EFBFBD><EFBFBD>
NOTUNICODE
df80
<EFBFBD><EFBFBD><EFBFBD>
NOTUNICODE
dfff
# Paired UTF-16 surrogates
<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>
NOTUNICODE
d800 dc00
<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>
NOTUNICODE
d800 dfff
<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>
NOTUNICODE
db7f dc00
<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>
NOTUNICODE
db7f dfff
<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>
NOTUNICODE
db80 dc00
<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>
NOTUNICODE
db80 dfff
<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>
NOTUNICODE
dbff dc00
<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>
NOTUNICODE
dbff dfff
################
#
# Some more tests, not from Markus Kuhn's file
#
# Mixed plane 0 and higher planes
Change instances of U+10ffff to U+10fffd, since that is the last valid 2003-07-31 Noah Levitt <nlevitt@columbia.edu> * tests/utf8.txt: Change instances of U+10ffff to U+10fffd, since that is the last valid unicode character. Add check that U+10ffff is NOTUNICODE. (#118730)
2003-07-31 18:48:54 +02:00
A𐀀B􏿽C
move $enable_debug down below checks for GCC to avoid setting CFLAGS Fri Jan 5 11:25:42 2001 Owen Taylor <otaylor@redhat.com> * configure.in (PACKAGE): move $enable_debug down below checks for GCC to avoid setting CFLAGS prematurely, change checks to avoid adding -g twice. * gutf8.c (g_ucs4_to_utf8): Support len < 0 to mean 0 termination. * gutf8.c (g_utf8_to_ucs4): Terminate result with 0. * tests/mainloop-test.c (main): Fix uses of g_main_loop_destroy(). * tests/unicode-encoding.c tests/Makefile.am tests/utf8.txt: Tests for unicode-conversion code. * gconvert.c (g_convert, g_convert_with_fallback): work around a couple of GNU libc bugs. * gconvert.[ch] (g_{locale,filename}_{to,from}_utf8): Standardize arguments to match g_convert(). Document. * gunicode.[ch]: - Implement conversion functions to and from UTF-16 - Standardize unicode conversion functions on prototype like g_convert. - Add a lot of error checking to unicode conversion functions. * gunicode.[ch] (g_utf8_to_ucs4_fast): Add fast, non-checking variant of g_utf8_to_ucs4. * gutf8.c (g_utf8_validate): - add g_return_if_fail (str != NULL). - add checks for overlong strings, non-valid Unicode characters (>= 110000) and single surrogates.
2001-01-05 22:22:47 +01:00
VALID
Change instances of U+10ffff to U+10fffd, since that is the last valid 2003-07-31 Noah Levitt <nlevitt@columbia.edu> * tests/utf8.txt: Change instances of U+10ffff to U+10fffd, since that is the last valid unicode character. Add check that U+10ffff is NOTUNICODE. (#118730)
2003-07-31 18:48:54 +02:00
41 00010000 42 10fffd 43
Reference in New Issue Copy Permalink