emacs/admin/notes/unicode

                                            -*-mode: text; coding: utf-8;-*-

Copyright (C) 2002-2024 Free Software Foundation, Inc.
See the end of the file for license conditions.

Importing a new Unicode Standard version into Emacs
-------------------------------------------------------------

Emacs uses the following files from the Unicode Character Database
(a.k.a. "UCD):

  . UnicodeData.txt
  . Blocks.txt
  . BidiBrackets.txt
  . BidiMirroring.txt
  . IVD_Sequences.txt
  . NormalizationTest.txt
  . PropertyValueAliases.txt
  . ScriptExtensions.txt
  . Scripts.txt
  . SpecialCasing.txt
  . confusables.txt
  . emoji-data.txt
  . emoji-zwj-sequences.txt
  . emoji-variation-sequences.txt
  . emoji-sequences.txt
  . BidiCharacterTest.txt

Emacs also uses the file emoji-test.txt which should be imported from
the Unicode's Public/emoji/ directory, and IdnaMappingTable.txt from
the Public/idna/ directory.

First, the first 15 files, emoji-test.txt and IdnaMappingTable.txt
need to be copied into admin/unidata/, and the file
https://www.unicode.org/copyright.html should be copied over
copyright.html in admin/unidata (some of them might need trailing
whitespace removed before they can be committed to the Emacs
repository).

Next, review the assignment of default values of the Bidi Class
property to blocks in the file extracted/DerivedBidiClass.txt from the
UCD (search for "unassigned" and "@missing" in that file).  Any
changes should be reflected in the unidata-gen.el file, where it sets
up the default values around line 210.

Then Emacs should be rebuilt for them to take effect.  Rebuilding
Emacs updates several derived files elsewhere in the Emacs source
tree, mainly in lisp/international/.

When Emacs is rebuilt for the first time after importing the new
files, pay attention to any warning or error messages.  In particular,
admin/unidata/unidata-gen.el will complain if UnicodeData.txt defines
new bidirectional attributes of characters, because unidata-gen.el,
bidi.c and dispextern.h need to be updated in that case; failure to do
so will cause aborts in redisplay.  unidata-gen.el will also complain
if the format of the Unicode Copyright notice in copyright.html
changed in significant ways; in that case, update the regular
expression in unidata-gen-file used to extract the copyright string.

Next, review the changes in UnicodeData.txt vs the previous version
used by Emacs.  Any changes, be it introduction of new scripts or
addition of codepoints to existing scripts, might need corresponding
changes in the data used for filling the category-table, case-table,
and char-width-table in characters.el.  The additional scripts should
cause automatic updates in charscript.el, but it is a good idea to
look at the results and see if any changes in admin/unidata/blocks.awk
are required.

The setting of char-width-table around line 1200 of characters.el
should be checked against the latest version of the Unicode file
EastAsianWidth.txt, and any discrepancies fixed: double-width
characters are those marked with W or F in that file.  Zero-width
characters are not taken from EastAsianWidth.txt, they are those whose
Unicode General Category property is one of Mn, Me, or Cf, and also
Hangul jungseong and jongseong characters (a.k.a. "Jamo medial vowels"
and "Jamo final consonants").  The list of "ambiguous-width
characters" recorded in the ambiguous-width-chars table (around line
1400 of characters.el) should also be checked against the latest
Unicode data in EastAsianWidth.txt.

Any new scripts added by UnicodeData.txt will also need updates to
script-representative-chars defined in fontset.el, and also the list
of OTF script tags in otf-script-alist, whose source is on this page:

  https://docs.microsoft.com/en-us/typography/opentype/spec/scripttags

Other databases in fontset.el might also need to be updated as needed.
One notable place to check is the function setup-default-fontset,
where new scripts will generally need some addition, most probably to
the list of "simple" scripts (search for "Simple").

The function 'ucs-names', defined in lisp/international/mule-cmds.el,
might need to be updated because it knows about used and unused ranges
of Unicode codepoints, which a new release of the Unicode Standard
could change.

The data used by ucs-normalize.el might need to be updated.
Specifically, the values of 'ucs-normalize-composition-exclusions' and
'check-range", defined at the beginning of ucs-normalize.el, should be
verified against the latest Unicode data files.

Next, test normalization functions against NormalizationTests.txt,
in the top-level directory run:

  make -C test lisp/international/ucs-normalize-tests

See commentary in test/lisp/international/ucs-normalize-tests.el
regarding failing lines.

Changes in UTS#46 and other relevant Unicode data might break our
features that detect confusable and suspicious text, so run
test/lisp-international/textsec-tests to check that:

  make -C test lisp/international/textsec-tests

The file BidiCharacterTest.txt should be copied to the test suite, and
if its format has changed, the file biditest.el there should be
modified to follow suit.  If there's trailing whitespace in
BidiCharacterTest.txt, it should be removed before committing the new
version.

src/macuvs.h is a generated file, but if it has changed as a result
of the updates, please commit it as well (see
admin/unidata/Makefile.in for an explanation).

Visit "emoji-data.txt" with the rebuilt Emacs, and check that an
appropriate font is being used for the emoji (by default Emacs uses
"Noto Color Emoji").  Running the following command in that buffer
will give you an idea of which codepoints are not supported by
whichever font Emacs is using.

(defun check-emoji-coverage (font-name-regexp)
"Display a buffer containing emoji codepoints for which FONT-NAME is not used.
This must be run from a buffer in the format of emoji-data.txt.
FONT-NAME-REGEXP is checked using `string-match'."
(interactive "MFont Name: ")
(save-excursion
(goto-char (point-min))
(let (res char name ifont)
  (while (re-search-forward "; Emoji_Presentation [^(]+(\\(.\\)[).]" nil t)
    (setq char (aref (match-string 1) 0))
    (setq ifont (car (internal-char-font nil char)))
    (when ifont
      (setq name (font-xlfd-name ifont)))
    (if (or (not ifont) (not (string-match font-name-regexp name)))
        (setq res (concat (string char) res))))
  (when res
    (with-output-to-temp-buffer "*Check-Emoji-Coverage*"
      (princ (format "Font not matching '%s' was used for the following characters:\n%s"
                     font-name-regexp (reverse res))))))))

Visit "emoji-zwj-sequences.txt" and "emoji-sequences.txt" with the
rebuilt Emacs, and check that the sample sequences are composed
properly.  Also check the Unicode style chart file available at
https://unicode.org/emoji/charts/emoji-style.txt for any issues
involving VS-15 and VS-16, if so you may need to update the value
generated for auto-composition-emoji-eligible-codepoints by
admin/unidata/emoji-zwj.awk.  Note that your emoji font might not have
glyphs for the newest codepoints yet.

Visit "emoji-variation-sequences.txt", and run the following lisp
fragment to actually insert the described codepoints, then check that
all the text in parentheses displays correctly (it can be helpful to
have `glyphless-char-display-control' customized to show hex codes for
variation selectors).

(save-excursion
 (goto-char (point-min))
 (while (re-search-forward "^\\([0-9A-Z]+\\) \\([0-9A-Z]+\\).*(\\([^)]+\\))" nil t)
   (let ((ch (string-to-number (match-string 1) 16))
         (sel (string-to-number (match-string 2) 16))
         (sp (match-string 3)))
     (replace-match (format "%s %c%c " sp ch sel) nil nil nil 3))))

Finally, etc/NEWS should be updated to announce the support for the
new Unicode version.

Problems, fixmes and other unicode-related issues
-------------------------------------------------------------

Notes by fx to record various things of variable importance.  Handa
needs to check them -- don't take too seriously, especially with
regard to completeness.

 * SINGLE_BYTE_CHAR_P returns true for Latin-1 characters, which has
   undesirable effects.  E.g.:
   (multibyte-string-p (let ((s "x")) (aset s 0 ?£) s)) => nil
   (multibyte-string-p (concat [?£])) => nil
   (text-char-description ?£) => "M-#"

	These examples are all fixed by the change of 2002-10-14, but
	there still exist questionable SINGLE_BYTE_CHAR_P in the
	code (keymap.c and print.c).

 * Rationalize character syntax and its relationship to the Unicode
   database.  (Applies mainly to symbol and punctuation syntax.)

 * Fontset handling and customization needs work.  We want to relate
   fonts to scripts, probably based on the Unicode blocks.  The
   presence of small-repertoire 10646-encoded fonts in XFree 4 is a
   pain, not currently worked round.

	With the change on 2002-07-26, multiple fonts can be
	specified in a fontset for a specific range of characters.
	Each range can also be specified by script.  Before using
	ISO10646 fonts, Emacs checks their repertories to avoid such
	fonts that don't have a glyph for a specific character.

	fx has worked on fontset customization, but was stymied by
	basic problems with the way the default face is dealt with
	(and something else, I think).  This needs revisiting.

 * Work is also needed on charset and coding system priorities.

 * The relevant bits of latin1-disp.el need porting (and probably
   re-naming/updating).  See also cyril-util.el.

 * Quail files need more work now the encoding is largely irrelevant.

 * What to do with the old coding categories stuff?

 * The preferred-coding-system property of charsets should probably be
   junked unless it can be made more useful now.

 * find-multibyte-characters needs looking at.

 * Implement Korean cp949/UHC, BIG5-HKSCS and any other important missing
   charsets.

 * Lazy-load tables for unify-charset somehow?

	Actually, Emacs clears out all charset maps and unify-map just
	before dumping, and they are loaded again on demand by the
	dumped emacs.  But, those maps (char tables) generated while
	temacs is running can't be removed from the dumped emacs.

 * iso-2022 charsets get unified on i/o.

	With the change on 2003-01-06, decoding routines put the 'charset'
	property onto decoded text, and iso-2022 encoder pay attention
	to it.  Thus, for instance, reading and writing by
	iso-2022-7bit preserve the original designation sequences.
	The property name 'preferred-charset' may be better?

	We may have to utilize this property to decide a font.

 * Revisit locale processing: look at treating the language and
   charset parts separately.  (Language should affect things like
   spelling and calendar, but that's not a Unicode issue.)

 * Handle Unicode combining characters usefully, e.g. diacritics, and
   handle more scripts specifically (à la Devanagari).  There are
   issues with canonicalization.

 * We need tabular input methods, e.g. for maths symbols.  (Not
   specific to Unicode.)

 * Need multibyte text in menus, e.g. for the above.  (Not specific to
   Unicode -- see Emacs etc/TODO, but now mostly works with gtk.)

 * Populate char-width-table correctly for Unicode characters and
   worry about what happens when double-width charsets covering
   non-CJK characters are unified.

 * There are type errors lurking, e.g. in
   Fcheck_coding_systems_region.  Define ENABLE_CHECKING to find them.

 * Old auto-save files, and similar files, such as Gnus drafts,
   containing non-ASCII characters probably won't be re-read correctly.


Source file encoding
--------------------

Most Emacs source files are encoded in UTF-8 (or in ASCII, which is a
subset), but there are a few exceptions, listed below.  Perhaps
someday many of these files will be converted to UTF-8, for
convenience when using tools like 'grep -r', but this might need
nontrivial changes to the build process.

 * chinese-big5

     These are verbatim copies of files taken from external sources.
     They haven't been converted to UTF-8.

	leim/CXTERM-DIC/4Corner.tit
	leim/CXTERM-DIC/ARRAY30.tit
	leim/CXTERM-DIC/ECDICT.tit
	leim/CXTERM-DIC/ETZY.tit
	leim/CXTERM-DIC/PY-b5.tit
	leim/CXTERM-DIC/Punct-b5.tit
	leim/CXTERM-DIC/QJ-b5.tit
	leim/CXTERM-DIC/ZOZY.tit
	leim/MISC-DIC/CTLau-b5.html
	leim/MISC-DIC/cangjie-table.b5

 * chinese-iso-8bit

     These are verbatim copies of files taken from external sources.
     They haven't been converted to UTF-8.

	leim/CXTERM-DIC/CCDOSPY.tit
	leim/CXTERM-DIC/Punct.tit
	leim/CXTERM-DIC/QJ.tit
	leim/CXTERM-DIC/SW.tit
	leim/CXTERM-DIC/TONEPY.tit
	leim/MISC-DIC/CTLau.html
	leim/MISC-DIC/pinyin.map
	leim/MISC-DIC/ziranma.cin

 * cp850

     This file contains non-ASCII characters in unibyte strings.  When
     editing a keyboard layout it's more convenient to see 'é' than
     '\202', and the MS-DOS compiler requires the single byte if a
     backslash escape is not being used.

	src/msdos.c

 * iso-latin-1

     This file is used to test Emacs encoding.

        test/lisp/gnus/mm-decode-resources/win1252-multipart.bin

 * iso-2022-cn-ext

     This file is externally generated from leim/MISC-DIC/cangjie-table.b5
     by Big5->CNS converter.  It hasn't been converted to UTF-8.

	leim/MISC-DIC/cangjie-table.cns

 * japanese-iso-8bit

     SKK-JISYO.L is a verbatim copy of a file taken from an external source.
     It hasn't been converted to UTF-8.

	leim/SKK-DIC/SKK-JISYO.L

 * japanese-shift-jis

     This is a verbatim copy of a file taken from an external source.
     It hasn't been converted to UTF-8.

	admin/charsets/mapfiles/cns2ucsdkw.txt

 * iso-2022-jp

     This contains just one CJK charset, but Emacs currently has no
     easy way to specify set-charset-priority on a per-file basis, so
     converting this file to UTF-8 might change the file's appearance
     when viewed by an Emacs that is operating in some other language
     environment.

	etc/tutorials/TUTORIAL.ja

 * utf-8-emacs

     These files contain characters that cannot be encoded in UTF-8.

	lisp/language/ethio-util.el
	lisp/language/ethiopic.el
	lisp/language/ind-util.el
	lisp/language/tibet-util.el
	lisp/language/tibetan.el
	lisp/leim/quail/ethiopic.el
	lisp/leim/quail/tibetan.el
	lisp/international/titdic-cnv.el

 * binary files

     These files contain binary data, and are not text files.
     Some of the entries in this list are patterns, and stand for any
     files with the listed extension.

	*.bmp
	*.cur
	*.gif
	*.gpg
	*.gz
	*.icns
	*.ico
	*.jpg
	*.kbx
	*.key
	*.pbm
	*.pdf
	*.pif
	*.png
	*.sig
	*.tiff
	*.webp
	*.zip
	etc/e/eterm-color
	etc/e/eterm-direct
	java/emacs.keystore

This file is part of GNU Emacs.

GNU Emacs is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

GNU Emacs is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with GNU Emacs.  If not, see <https://www.gnu.org/licenses/>.