Network Working Group                                       F. Yergeau
Request for Comments: 2044                           Alis Technologies
Category: Informational                                   October 1996


        UTF-8, a transformation format of Unicode and ISO 10646

Status of this Memo

   This memo provides information for the Internet community.  This memo
   does not specify an Internet standard of any kind.  Distribution of
   this memo is unlimited.

Abstract

   The Unicode Standard, version 1.1, and ISO/IEC 10646-1:1993 jointly
   define a 16 bit character set which encompasses most of the world's
   writing systems. 16-bit characters, however, are not compatible with
   many current applications and protocols, and this has led to the
   development of a few so-called UCS transformation formats (UTF), each
   with different characteristics.  UTF-8, the object of this memo, has
   the characteristic of preserving the full US-ASCII range: US-ASCII
   characters are encoded in one octet having the usual US-ASCII value,
   and any octet with such a value can only be an US-ASCII character.
   This provides compatibility with file systems, parsers and other
   software that rely on US-ASCII values but are transparent to other
   values.

1.  Introduction

   The Unicode Standard, version 1.1 [UNICODE], and ISO/IEC 10646-1:1993
   [ISO-10646] jointly define a 16 bit character set, UCS-2, which
   encompasses most of the world's writing systems.  ISO 10646 further
   defines a 31-bit character set, UCS-4, with currently no assignments
   outside of the region corresponding to UCS-2 (the Basic Multilingual
   Plane, BMP).  The UCS-2 and UCS-4 encodings, however, are hard to use
   in many current applications and protocols that assume 8 or even 7
   bit characters.  Even newer systems able to deal with 16 bit
   characters cannot process UCS-4 data. This situation has led to the
   development of so-called UCS transformation formats (UTF), each with
   different characteristics.

   UTF-1 has only historical interest, having been removed from ISO
   10646.  UTF-7 has the quality of encoding the full Unicode repertoire
   using only octets with the high-order bit clear (7 bit US-ASCII
   values, [US-ASCII]), and is thus deemed a mail-safe encoding
   ([RFC1642]).  UTF-8, the object of this memo, uses all bits of an
   octet, but has the quality of preserving the full US-ASCII range:




 
RFC 2044                         UTF-8                      October 1996


   US-ASCII characters are encoded in one octet having the normal US-
   ASCII value, and any octet with such a value can only stand for an
   US-ASCII character, and nothing else.

   UTF-16 is a scheme for transforming a subset of the UCS-4 repertoire
   into a pair of UCS-2 values from a reserved range.  UTF-16 impacts
   UTF-8 in that UCS-2 values from the reserved range must be treated
   specially in the UTF-8 transformation.

   UTF-8 encodes UCS-2 or UCS-4 characters as a varying number of
   octets, where the number of octets, and the value of each, depend on
   the integer value assigned to the character in ISO 10646.  This
   transformation format has the following characteristics (all values
   are in hexadecimal):

   -  Character values from 0000 0000 to 0000 007F (US-ASCII repertoire)
      correspond to octets 00 to 7F (7 bit US-ASCII values).

   -  US-ASCII values do not appear otherwise in a UTF-8 encoded charac-
      ter stream.  This provides compatibility with file systems or
      other software (e.g. the printf() function in C libraries) that
      parse based on US-ASCII values but are transparent to other val-
      ues.

   -  Round-trip conversion is easy between UTF-8 and either of UCS-4,
      UCS-2 or Unicode.

   -  The first octet of a multi-octet sequence indicates the number of
      octets in the sequence.

   -  Character boundaries are easily found from anywhere in an octet
      stream.

   -  The lexicographic sorting order of UCS-4 strings is preserved.  Of
      course this is of limited interest since the sort order is not
      culturally valid in either case.

   -  The octet values FE and FF never appear.