Support for Unicode 16 (two test failures)

[DonKult]

Hi,

running update_unicode.py against Unicode 16 results in files that work just fine (for me at least), but the changes since 15 seem to break two assumptions in the tests: The two tested characters are now in the indic extend set it seems.

With the following patch all tests pass again, but my knowledge of Unicode is limited and so the effect this change has.

--- a/cpp/ycm/tests/CodePoint_test.cpp
+++ b/cpp/ycm/tests/CodePoint_test.cpp
@@ -87,12 +87,16 @@
                 GraphemeBreakProperty::EXTEND,
                 IndicConjunctBreakProperty::EXTEND } },
   // Bengali vowel sign Aa
-  { "া", { "া", "া", "া", false, false, false, GraphemeBreakProperty::EXTEND } },
+  { "া", { "া", "া", "া", false, false, false,
+                GraphemeBreakProperty::EXTEND,
+                IndicConjunctBreakProperty::EXTEND } },
   // Zero-width non-joiner
   { "", { "", "", "", false, false, false,
                 GraphemeBreakProperty::EXTEND } },
   // Combining cyrillic millions sign
-  { "", { "", "", "", false, false, false, GraphemeBreakProperty::EXTEND } },
+  { "", { "", "", "", false, false, false,
+                GraphemeBreakProperty::EXTEND,
+                IndicConjunctBreakProperty::EXTEND } },

   // Zero-width joiner
   { "", { "", "", "", false, false, false,

[DonKult]

On top of the previous mentioned changes, Unicode 17 adapted the GraphemeBreakTest.txt to contain tests with <NULL> at the beginning/end. As it seems not overly useful to test for the behavior with an explicit \0 in a string-based API I have just opted to skip those instead of explicitly supporting them. Your mileage may vary.

--- a/update_unicode.py
+++ b/update_unicode.py
@@ -609,7 +609,9 @@
   res = []
   for line in test_contents:
     if line.startswith( '÷' ):
-      data = line.split( '#' )[ 0 ].rstrip().strip( '÷' ).strip()
+      data = line.split( '#' )[ 0 ].rstrip().strip( '÷' ).strip().removesuffix(' ÷ 0000')
+      if data.startswith('0000 ÷') or data == '0000':
+        continue
       all_data = data.replace( ' × ', ' ÷ ' ).replace( '÷ ', '' )
       all_data = JoinUnicodeToUtf8( all_data.split() )
       split_data = data.replace( '× ', '' ).split( ' ÷ ' )