Commit graph

102 commits

Author SHA1 Message Date
Markus Scherer
51e21af692 ICU-22707 Unicode 16 aug16 2024-08-19 17:15:00 -07:00
Elango Cheran
3663cc1e0f ICU-22707 Add support for property Modifier_Combining_Mark 2024-07-30 15:44:32 -07:00
Elango Cheran
06c077bd35 ICU-22503 add property Indic_Conjunct_Break 2024-07-26 14:47:39 -07:00
Markus Scherer
4312a4e5cb ICU-22707 Unicode 16 beta jun04 2024-07-18 23:56:34 +00:00
Markus Scherer
876816b0a1 ICU-22707 new short aliases ID_Status, ID_Type 2024-07-18 23:56:34 +00:00
Markus Scherer
c439dcdf27 ICU-22785 uprops.icu: coalesce scx+sc bits 2024-06-04 18:51:53 -07:00
Markus Scherer
0d8a3ccd11 ICU-22785 move Block bits from propsvec0 to new trie 2024-06-04 10:49:53 -07:00
Henri Sivonen
f5056cb46a ICU-22757 Remove allow list of known contractions with precomposed form from ICU4X mode of genuca
This assumes that future cases will work OK, since the addition that was seen in Unicode 16 alpha
was OK.
2024-05-13 11:25:01 -07:00
Markus Scherer
012c326755 ICU-22707 UCA 16 data first cut 2024-04-29 17:00:55 -07:00
Markus Scherer
ba1ecef7de ICU-22707 Unicode 16 security data first cut 2024-04-29 17:00:55 -07:00
Markus Scherer
94ef2757a3 ICU-22707 gennorm2 & C++ norm2impl support MaybeNo
- .nrm formatVersion 5
- updated data format doc & design doc
2024-04-29 17:00:55 -07:00
Markus Scherer
4d9612bfa2 ICU-22707 Age 6:2, run generate.sh 2024-04-29 17:00:55 -07:00
Markus Scherer
cce162bf4d ICU-11396 new properties Identifier_Status & Identifier_Type
See #2879
2024-03-20 13:20:14 -07:00
Markus Scherer
d1fa15bc1f ICU-22571 add Aran script code variant 2024-02-27 14:23:59 -08:00
Markus Scherer
d8659b476d ICU-22404 new properties IDS_Unary_Operator, ID_Compat_Math_*, NFKC_SCF 2023-09-16 14:41:51 -07:00
Markus Scherer
81a6edb287 ICU-22404 Unicode 15.1 data 20230811 plus UTS46 fix 2023-08-16 14:25:22 -07:00
Elango Cheran
2e45e6ec0e ICU-22404 Unicode 15.1 beta data files & API constants
See #2492

Co-authored-by: Andy Heninger <andy.heninger@gmail.com>
Co-authored-by: Robin Leroy <egg.robin.leroy@gmail.com>
2023-07-13 19:26:14 -07:00
Markus Scherer
f4687fc25a ICU-22221 update root collation again from CLDR 43 2023-04-06 08:20:03 -07:00
Markus Scherer
d86b1cebe1 ICU-22220 update root collation from CLDR 43 2023-02-22 17:13:13 -08:00
Markus Scherer
8050af5484 ICU-21980 Unicode 15 update 2022aug30 2022-08-31 16:15:42 -07:00
Henri Sivonen
3cefbd55c7 ICU-22028 Export collation and normalization data for ICU4X 2022-06-28 08:37:32 -07:00
Markus Scherer
3859735e3b ICU-21980 Unicode 15 collation data 2022-05-25 18:23:11 +00:00
Markus Scherer
1276b0a4f7 ICU-21980 generated data files 2022-05-25 18:23:11 +00:00
Markus Scherer
f9beb616a8 ICU-21652 add emoji properties of strings
- 7 new properties: API constants & property names
- u_stringHasBinaryProperty(s, property) & UCharacter.hasBinaryProperty(s, property)
- two additional source data files
- new genprops part for writing new binary data file uemoji.icu
- data for existing emoji properties moved from uprops.icu (hardcoded in C++) to uemoji.icu (always loaded)
- new EmojiProps implementation
2021-09-08 12:15:50 -07:00
Markus Scherer
41aa7159ea ICU-21635 Unicode 14 data files 20210820, line break LB30b.2
See #1807
2021-08-23 22:11:49 +00:00
Markus Scherer
d4c92ebcfc ICU-21635 Unicode 14 beta 2021-06-21 22:26:15 +00:00
Markus Scherer
f62693aa02 ICU-13416 change Armenian (hy) uppercase/titlecase of և ligature ech-yiwn 2020-08-30 18:19:10 -07:00
Markus Scherer
af9ef2650b ICU-20893 Unicode 13 data 2020feb19 2020-02-19 22:02:35 -08:00
Markus Scherer
a7e378d587 ICU-20893 Unicode 13 beta
See PR #915, see changes.txt
- Unicode 13 beta data as of 2019-nov-21
- uprops.icu format version 7.7 with more bits for Script/Script_Extensions
- more bits in spoof checker ScriptSet
- root line break rules adjusted for UAX 14 changes, from Andy
- line break tailorings not yet in sync with root
2019-11-21 17:35:53 -08:00
Markus Scherer
0565894534 ICU-20497 Unicode 12.1 2019-04-04 10:23:24 -07:00
Markus Scherer
ac4387a374 ICU-20203 Unicode 12 data 20190214 2019-02-15 11:37:34 -08:00
Markus Scherer
d2e3a8847d ICU-20111 move text layout properties data into a new ulayout.icu data file 2019-02-14 08:30:57 -08:00
Markus Scherer
ea7c030961 ICU-20203 update ICU to Unicode 12 beta
- data as of 2018-nov-26
- API constants for new blocks & scripts
- sync RBBIMonkeyTest.java test data with C++
2018-11-28 23:13:07 +01:00
Fredrik Roubert
8b7fa6a03f ICU-20228 Revert file from commit 10e79698d7.
This file was changed by mistake (and not caught by the presubmit tests
because of unrelated build failures).
2018-10-24 23:26:00 +02:00
Fredrik Roubert
10e79698d7 ICU-20228 Fix common typos for "language". 2018-10-24 20:54:10 +02:00
Markus Scherer
d2ec8987a7
ICU-8966 ICU-12850 add API/data/code for text layout properties InPC, InSC, vo (#92)
ICU-8966: Indic_Positional_Category & Indic_Syllabic_Category

ICU-12850: Vertical_Orientation
2018-09-27 14:27:39 -07:00
Markus Scherer
fe3eb3ed5c
ICU-13530 add UCPTrie/CodePointTrie, switch normalization to use it (#48)
* ICU-13530 copy C/C++ files UTrie2 -> UTrie3

X-SVN-Rev: 40754

* ICU-13530 UTrie3 new files copied from UTrie2: rename types/functions/macros

X-SVN-Rev: 40755

* ICU-13530 debug-print building each UTrie2

X-SVN-Rev: 40756

* ICU-13530 remove two-byte-UTF-8 errorValue block; move highValue from end of data array into header; add errorValue to header

X-SVN-Rev: 40762

* ICU-13530 UTrie3 U16_NEXT/PREV: errorValue for unpaired surrogates

X-SVN-Rev: 40763

* ICU-13530 no more separate values for lead surrogate code units

X-SVN-Rev: 40764

* ICU-13530 change from 11:5 trie bits to 10:6 for simpler UTF-8 code

X-SVN-Rev: 40766

* ICU-13530 UTrie2 build UTrie3 as well, print sizes

X-SVN-Rev: 40767

* ICU-13530 debug-print countSame, sumOverlaps, countInitial

X-SVN-Rev: 40768

* ICU-13530 debug-print whether trie is for CanonIterData

X-SVN-Rev: 40769

* ICU-13530 no index-shift for BMP data, no separate index-2 for 2-byte UTF-8; builder changes incomplete

X-SVN-Rev: 40777

* ICU-13530 remove errorValue and highStart from UNewTrie3

X-SVN-Rev: 40778

* ICU-13530 rewrite UTrie3 builder code

X-SVN-Rev: 40783

* ICU-13530 UTrie3 bug fixes

X-SVN-Rev: 40788

* ICU-13530 fully re-inline _UTRIE3_U8_NEXT()

X-SVN-Rev: 40790

* ICU-13530 find most common all-same data block for dataNullBlock and initialValue

X-SVN-Rev: 40792

* ICU-13530 UTrie3 iterator functions take start and return the end of a range, rather than callback call for each range

X-SVN-Rev: 40800

* ICU-13530 mask off unused data value bits before building a UTrie3 with values less than 32 bits wide

X-SVN-Rev: 40803

* ICU-13530 split utrie3builder.h out of utrie3.h

X-SVN-Rev: 40804

* ICU-13530 separate types UTrie3 vs. UTrie3Builder, implement builder as wrapper over C++ class Trie3Builder in .cpp

X-SVN-Rev: 40809

* ICU-13530 function to make a UTrie3Builder from a UTrie3

X-SVN-Rev: 40810

* ICU-13530 debug-print some data; some cleanup

X-SVN-Rev: 40865

* ICU-13530 BMP 10:6 but supplementary 10:6:4

X-SVN-Rev: 40984

* ICU-13530 move errorValue & highValue to the end of the data table, minimal padding to 4 bytes

X-SVN-Rev: 41011

* ICU-13530 index-1 table gap of index-2 null blocks

X-SVN-Rev: 41018

* ICU-13530 test with more than 128k compacted data

X-SVN-Rev: 41034

* ICU-13530 supplementary bits 11:5:4 saves a little space

X-SVN-Rev: 41039

* ICU-13530 supplementary bits 6:5:5:4 instead of gap: about same size but simpler

X-SVN-Rev: 41050

* ICU-13530 remove unnecessary utrie3_clone(built trie)

X-SVN-Rev: 41058

* ICU-13530 remove unnecessary UTrie3StringIterator

X-SVN-Rev: 41059

* ICU-13530 back to UTRIE3_GET...() macros *returning* data values

X-SVN-Rev: 41060

* ICU-13530 fast vs. small

X-SVN-Rev: 41066

* ICU-13530 always load NFC data, add simple normalization performance test

X-SVN-Rev: 41110

* ICU-13530 change normalization main trie to UTrie3 with special values for lead surrogates; forbid non-inert surrogate code *points* because unable to store values different from code *units*; runtime code work around that for code point lookup and iteration; adjust UTS 46 for normalization no longer mapping unpaired surrogates to U+FFFD

X-SVN-Rev: 41122

* ICU-13530 simplenormperf bug fix and NFC base line

X-SVN-Rev: 41126

* ICU-13530 move normalization getRange skipping lead surrogates to API getRangeSkipLead()

X-SVN-Rev: 41182

* ICU-13530 switch CanonIterData and gennorm2 Norms to UTrie3

X-SVN-Rev: 41183

* ICU-13530 remove unused overwrite parameter from setRange()

X-SVN-Rev: 41184

* ICU-13530 getRange skip lead -> fixed surrogates

X-SVN-Rev: 41219

* ICU-13530 minor cleanup

X-SVN-Rev: 41221

* ICU-13530 UTS 46 code map unpaired surrogates to U+FFFD before normalization

X-SVN-Rev: 41224

* ICU-13530 minor internal-docs cleanup

X-SVN-Rev: 41225

* ICU-13530 rename UTrie3 to UCPTrie, and other name changes

X-SVN-Rev: 41226

* ICU-13530 add 8-bit data option; add type-any & valueBits-any for fromBinary(); macros consistently source type then data width

X-SVN-Rev: 41234

* ICU-13530 scrub the API docs for the proposal

X-SVN-Rev: 41319

* ICU-13530 tag internal definitions as such, or move them to an internal header

X-SVN-Rev: 41320

* ICU-13530 Java API skeleton

X-SVN-Rev: 41326

* ICU-13530 API feedback: ValueWidth, MutableCodePointTrie, base CodePointMap, ...

X-SVN-Rev: 41382

* ICU-13530 add UCPTrie valueWidth field and padding, and combine data pointers into a union

X-SVN-Rev: 41408

* ICU-13530 switch some macros to using dataAccess parameter: separate index vs. data lookups, no macro variant for each value width

X-SVN-Rev: 41409

* ICU-13530 StringIterator is no longer a java.util.Iterator (bad fit)

X-SVN-Rev: 41455

* ICU-13530 CodePointTrie.java code complete

X-SVN-Rev: 41518

* ICU-13530 finish Java port incl test; keep C++ parallel

* ICU-13530 adjust API for feedback: rename HandleValue to FilterValue, change getRange+getRangeFixedSurr(bool allSurr) to enum RangeOption+getRange(enum option); change remaining C macros to use dataAccess for 16/32/8-bit value widths; fix/clarify some API docs

* ICU-13530 add javadoc

* ICU-13530 document UCPTrie binary data format

* ICU-13530 update .nrm formatVersion 3->4, document change in surrogate handling with new trie

* ICU-13530 re-hardcode NFC data

* move trie swapper code into new file; add new files to Windows project files; turn off trie debugging

* ICU-13530 minor cleanup

* ICU-13530 test more range starts; fix a C test leak

* ICU-13530 regenerate Java data from scratch

* ICU-13530 review feedback changes: API docs typos, more @internal, C++11 field initializers, fix potential leak in MutableCodePointTrie::fromUCPTrie()

* ICU-13530 rename interface FilterValue to ValueFilter
2018-09-27 14:27:38 -07:00
Markus Scherer
ebca759ea1 ICU-13630 Unicode 11 update from near-final data 20180521
X-SVN-Rev: 41426
2018-05-22 01:56:20 +00:00
Markus Scherer
03303a6cb6 ICU-13630 Unicode 11 beta data apr02 (security apr03), fix ICU4C tests except RBBI
X-SVN-Rev: 41191
2018-04-03 23:09:49 +00:00
Markus Scherer
af6a771267 ICU-13630 implement, test, use emoji property Extended_Pictographic
X-SVN-Rev: 41094
2018-03-12 05:53:02 +00:00
Markus Scherer
b3aec18a3c ICU-13630 ucase.icu formatVersion 4: more compressible exceptions, and more room for future exceptions growth
X-SVN-Rev: 41093
2018-03-12 00:15:40 +00:00
Markus Scherer
1752b5c8c9 ICU-13630 Unicode 11 beta data mar06, API constants for new property values
X-SVN-Rev: 41092
2018-03-09 23:53:02 +00:00
Markus Scherer
cf4cb10c3d ICU-13462 fix Script_Extensions for 5 characters: data generator needs to revert them from block scx to sc (merged from maint-60 r40667)
X-SVN-Rev: 40699
2017-12-05 20:53:14 +00:00
Markus Scherer
2f87cf4c46 ICU-10524 normalization one-way mapping with trailing ccc>1 has no compose-boundary-after
X-SVN-Rev: 40355
2017-08-25 22:46:12 +00:00
Markus Scherer
e6748afd82 ICU-13197 improved normalization data structure and code; .nrm formatVersion 3; merged from branches/markus/normv3 except for cherry-picks from trunk to there
X-SVN-Rev: 40265
2017-07-14 22:38:40 +00:00
Markus Scherer
6c1e41e0f2 ICU-13197 catch one more edge case where a mapping has no compose boundary after it
X-SVN-Rev: 40156
2017-06-07 07:13:34 +00:00
Markus Scherer
19d53e7641 ICU-13197 more gennorm2 cleanup, set smallFCD for some characters with algorithmic mappings
X-SVN-Rev: 40155
2017-06-07 05:23:53 +00:00
Markus Scherer
45da4f810d ICU-12985 final? Unicode 10 data from 20170605
X-SVN-Rev: 40153
2017-06-06 02:46:10 +00:00
Markus Scherer
cb722b8bcd ICU-12985 Unicode 10 minor data update
X-SVN-Rev: 40104
2017-05-03 23:26:47 +00:00
Markus Scherer
20bee936b1 ICU-12985 ppucd.txt more readable unassigned ranges; block compaction by size savings not value plurality reduces clutter
X-SVN-Rev: 40096
2017-05-02 22:53:28 +00:00