MAN page from Other perl-Unicode-String-2.07-1.i386.rpm

String

Section: User Contributed Perl Documentation (3)
Updated: 2003-03-11
Index

NAME

Unicode::String - String of Unicode characters (UCS2/UTF16)

SYNOPSIS

 use Unicode::String qw(utf8 latin1 utf16); $u = utf8("The Unicode Standard is a fixed-width, uniform "); $u .= utf8("encoding scheme for written characters and text");

 # convert to various external formats print $u->ucs4;      # 4 byte characters print $u->utf16;     # 2 byte characters + surrogates print $u->utf8;      # 1-4 byte characters print $u->utf7;      # 7-bit clean format print $u->latin1;    # lossy print $u->hex;       # a hexadecimal string

 # all these can be used to set string value or as constructor $u->latin1("Ĺ vćre eller ĺ ikke vćre"); $u = utf16("\0Ĺ\0 \0v\0ć\0r\0e");

 # string operations $u2 = $u->copy; $u->append($u2); $u->repeat(2); $u->chop;

 $u->length; $u->index($other); $u->index($other, $pos);

 $u->substr($offset); $u->substr($offset, $length); $u->substr($offset, $length, $substitute);

 # overloading $u .= "more"; $u = $u x 100; print "$u\n";

 # string <--> array of numbers @array = $u->unpack; $u->pack(@array);

 # misc $u->ord; $u = uchr($num);

DESCRIPTION

A Unicode::String object represents a sequence of Unicodecharacters. The Unicode Standard is a fixed-width, uniform encodingscheme for written characters and text. This encoding treatsalphabetic characters, ideographic characters, and symbolsidentically, which means that they can be used in any mixture and withequal facility. Unicode is modeled on the ASCII character set, butuses a 16-bit encoding to support full multilingual text.

Internally a Unicode::String object is a string of 2 byte values innetwork byte order (big-endian). The class provide various methods toconvert from and to various external formats, and all stringmanipulations are made on strings in this the internal 16-bit format.

The functions utf16(), utf8(), utf7(), ucs2(), ucs4(), latin1(),uchr() can be imported from the Unicode::String module and willwork as constructors initializing strings of the correspondingencoding. The ucs2() and utf16() are really aliases for the samefunction.

The Unicode::String objects overload various operators, so theywill normally work like plain 8-bit strings in Perl. This includesconversions to strings, numbers and booleans as well as assignment,concatenation and repetition.

METHODS

The following methods are available:

Unicode::String->stringify_as( [$enc] )

This class method specify which encoding will be used whenUnicode::String objects are implicitly converted to and from plainstrings. It define which encoding to assume for the argument of theUnicode::String constructor new(). Without an encoding argument,stringify_as() returns the current encoding ctor function. Theencoding argument ($enc) is a string with one of the following values:``ucs4'', ``ucs2'', ``utf16'', ``utf8'', ``utf7'', ``latin1'', ``hex''. The defaultis ``utf8''.

$us = Unicode::String->new( [$initial_value] )

This is the customary object constructor. Without argument, itcreates an empty Unicode::String object. If an $initial_valueargument is given, it is decoded according to the specifiedstringify_as() encoding and used to initialize the newly createdobject.

Normally you create Unicode::String objects by importing some ofthe encoding methods below as functions into your namespace andcalling them with an appropriate encoded argument.

$us->ucs4( [$newval] )

The UCS-4 encoding use 32 bits per character. The main benefit of thisencoding is that you don't have to deal with surrogate pairs. Encodedas a Perl string we use 4-bytes in network byte order for eachcharacter.

The ucs4() method always return the old value of $us and if given anargument decodes the UCS-4 string and set this as the new value of $us.The characters in $newval must be in the range 0x0 .. 0x10FFFF.Characters outside this range is ignored.

$us->ucs2( [$newval] )

$us->utf16( [$newval] )

The ucs2() and utf16() are really just different names for the samemethod. The UCS-2 encoding use 16 bits per character. The UTF-16encoding is identical to UCS-2, but includes the use of surrogatepairs. Surrogates make it possible to encode characters in the range0x010000 .. 0x10FFFF with the use of two consecutive 16-bit chars.Encoded as a Perl string we use 2-bytes in network byte order for eachcharacter (or surrogate code).

The ucs2() method always return the old value of $us and if given anargument set this as the new value of $us.

$us->utf8( [$newval] )

The UTF-8 encoding use 8-bit for the encoding of characters in therange 0x0 .. 0x7F, 16-bit for the encoding of characters in the range0x80 .. 0x7FF, 24-bit for the encoding of characters in the range0x800 .. 0xFFFF and 32-bit for characters in the range 0x01000.. 0x10FFFF. Americans like this encoding, because plain US-ASCIIcharacters are still US-ASCII. Another benefit is that the character'\0' only occurs as the encoding of 0x0, thus the normalNUL-terminated strings (popular in the C programming language) canstill be used.

The utf8() method always return the old value of $us encoded usingUTF-8 and if given an argument decodes the UTF-8 string and set this asthe new value of $us.

$us->utf7( [$newval] )

The UTF-7 encoding only use plain US-ASCII characters for theencoding. This makes it safe for transport through 8-bit strippingprotocols. Characters outside the US-ASCII range are base64-encodedand '+' is used as an escape character. The UTF-7 encoding isdescribed in RFC1642.

The utf7() method always return the old value of $us encoded usingUTF-7 and if given an argument decodes the UTF-7 string and set this asthe new value of $us.

If the (global) variable $Unicode::String::UTF7_OPTIONAL_DIRECT_CHARSis TRUE, then a wider range of characters are encoded as themselves.It is even TRUE by default. The characters affected by this are:

   ! " # $ % & * ; < = > @ [ ] ^ _ ` { | }

$us->latin1( [$newval] )

The first 256 codes of Unicode is identical to the ISO-8859-1 8-bitencoding, also known as Latin-1. The latin1() method always returnthe old value of $us and if given an argument set this as the newvalue of $us. Characters outside the 0x0 .. 0xFF range are ignoredwhen returning a Latin-1 string. If you want more control over themapping from Unicode to Latin-1, use the Unicode::Map8 class. Thisis also the way to deal with other 8-bit character sets.

$us->hex( [$newval] )

This method() return a plain ASCII string where each Unicode characteris represented by the ``U+XXXX'' string and separated by a single spacecharacter. This format can also be used to set the value of $us (inwhich case the ``U+'' is optional).

$us->as_string;

Converts a Unicode::String to a plain string according to thesetting of stringify_as(). The default stringify_as() method is``utf8''.

$us->as_num;

Converts a Unicode::String to a number. Currently only the digitsin the range 0x30 .. 0x39 are recognized. The plan is to eventuallysupport all Unicode digit characters.

$us->as_bool;

Converts a Unicode::String to a boolean value. Only the emptystring is FALSE. A string consisting of only the character U+0030 isconsidered TRUE, even if Perl consider ``0'' to be FALSE.

$us->repeat( $count );

Returns a new Unicode::String where the content of $us is repeated$count times. This operation is also overloaded as:

  $us x $count

$us->concat( $other_string );

Concatenates the string $us and the string $other_string. If$other_string is not an Unicode::String object, then it is firstpassed to the Unicode::String->new constructor function. Thisoperation is also overloaded as:

  $us . $other_string

$us->append( $other_string );

Appends the string $other_string to the value of $us. If$other_string is not an Unicode::String object, then it is firstpassed to the Unicode::String->new constructor function. Thisoperation is also overloaded as:

  $us .= $other_string

$us->copy;

Returns a copy of the current Unicode::String object. Thisoperation is overloaded as the assignment operator.

$us->length;

Returns the length of the Unicode::String. Surrogate pairs arestill counted as 2.

$us->byteswap;

This method will swap the bytes in the internal representation of theUnicode::String object.

Unicode reserve the character U+FEFF character as a byte order mark.This works because the swapped character, U+FFFE, is reserved to notbe valid. For strings that have the byte order mark as the firstcharacter, we can guaranty to get the byte order right with thefollowing code:

   $ustr->byteswap if $ustr->ord == 0xFFFE;

$us->unpack;

Returns a list of integers each representing an UTF-16 character code.

$us->pack( @uchr );

Sets the value of $us as a sequence of UTF-16 characters with thecharacters codes given as parameter.

$us->ord;

Returns the character code of the first character in $us. The ord()method deals with surrogate pairs, which gives us a result-range of0x0 .. 0x10FFFF. If the $us string is empty, undef is returned.

$us->chr( $code );

Sets the value of $us to be a string containing the character assignedcode $code. The argument $code must be an integer in the range 0x0.. 0x10FFFF. If the code is greater than 0xFFFF then a surrogate paircreated.

$us->name

In scalar context returns the official Unicode name of the firstcharacter in $us. In array context returns the name of all charactersin $us. Also see Unicode::CharName.

$us->substr( $offset, [$length, [$subst]] )

Returns a sub-string of $us. Works similar to the builtin substrfunction, but because we can't make LVALUE subs yet, you have to passthe string you want to assign to the sub-string as the 3rd parameter.

$us->index( $other, [$pos] );

Locates the position of $other within $us, possibly starting thesearch at position $pos.

$us->chop;

Chops off the last character of $us and returns it (as aUnicode::String object).

FUNCTIONS

The following utility functions are provided. They will be exportedon request.

byteswap2($str, ...): This function will swap 2 and 2 bytes in the strings passed asarguments. This can be used to fix up UTF-16 or UCS-2 strings fromlitle-endian systems. If this function is called in void context,then it will modify its arguments in-place. Otherwise, then swappedstrings are returned.
byteswap4($str, ...): The byteswap4 function works similar to byteswap2, but will reversethe order of 4 and 4 bytes. Can be used to fix litle-endian UCS-4strings.

COPYRIGHT

This library is free software; you can redistribute it and/ormodify it under the same terms as Perl itself.

This document was created byman2html,using the manual pages.