Category: Unicode

Match Unicode property values with a wildcard

Not all characters with a numerical value are “digits”, in the Perl sense. You saw in Match Unicode characters by property value, in which you selected characters by a Unicode property value. That showed up again in Perl v5.18 adds character class set operations. Perl v5.30 adds a slightly easier way to get at multiple numerical values at the same time. Now you can match Unicode property values with wildcards (Unicode TR 18), which are sorta like Perl patterns. Don’t get too excited, though, because these can be expensive.

Continue reading “Match Unicode property values with a wildcard”

Match Unicode character names with a pattern

Perl has some of the best Unicode support out there, and it keeps getting better. Perl v5.32 supports Unicode 13, and you can now apply patterns to character names. You probably don’t want to do that though.

First, the Unicode Character Database catalogs each character, giving it a code number, a name, and many other properties.

Continue reading “Match Unicode character names with a pattern”

Match only the same Unicode script

Earlier this year, this website was the target of some sort of attack in which a bot sent seemingly random data in its requests. The attack wasn’t that big of a deal since I easily blocked it with Cloudflare, but it was interesting. The apparently random data was actually a mix of Latin, Hangul, and Cyrillic. Domain hacks with unusual Unicode characters shows some of these exploits. Curiously, v5.28 added some regex feature that deals with this sort of nonsense.


Continue reading “Match only the same Unicode script”

Use Unicode 10 in Perl v5.28

Perl v5.28 updates to Unicode 10. There are 8,518 new characters, 7,473 which are in the CJK extension. There are 56 new emojis. And, the Bitcoin symbol, ₿. It adds a T. rex, 🦖, but we’re still waiting for a raptor. To Perl they are just characters like any other so you don’t need anything new to deal with them.

Continue reading “Use Unicode 10 in Perl v5.28”

Find the new emojis in Perl’s Unicode support

Perl v5.26 updates itself to Unicode 9. That’s not normally exciting news but people have been pretty enthusiastic about the 72 new emojis that come. As far as Perl cares, they are just valid code points like all of the other ones.

Continue reading “Find the new emojis in Perl’s Unicode support”

Look up Unicode properties with an inversion map

Perl comes with extracts of the Unicode character data, but it hasn’t been easy to look up all of the information Perl knows about a character. Perl v5.15.7 adds a way to created an inverted map based on the property that you want to access.

Continue reading “Look up Unicode properties with an inversion map”

Loose match Unicode character names

The charnames module can now handle loose name matching, as outlined in Unicode Standard Annex #44. This accounts for the various ways people are abusing things.

Consider the character 😻, (U+1F63B SMILING CAT FACE WITH HEART-SHAPED EYES). If you want to interpolate that into a string, you have to use the exact name:

use v5.16;
use open qw(:std :utf8);

say "\N{SMILING CAT FACE WITH HEART-SHAPED EYES}";

Starting with v5.16, the \N{} in a double-quoted string automatically imports :long and :short. There’s another one that you can import yourself, but it’s a bit costly.

Some people don’t like all uppercase strings, so they might want to type it out as title or lowercase:

use v5.16;
use open qw(:std :utf8);

say "\N{Smiling Cat Face With Heart-Shaped Eyes}";

That doesn’t work and you get an error:

Unknown charname 'Smiling Cat Face With Heart-Shaped Eyes'

Import :loose from charnames and it will works:

use v5.16;
use open qw(:std :utf8);
use charnames qw(:loose);

say "\N{Smiling Cat Face With Heart-Shaped Eyes}";

The loose naming rules involve three things, which makes the loose matching slow:

  • Ignore case folding
  • Ignore whitespace
  • Ignore “medial” hyphens (letters on either side)

So all of these work, even the one with consecutive hyphens:

use v5.16;
use open qw(:std :utf8);
use charnames qw(:loose);

say "\N{Smiling Cat Face With Heart Shaped Eyes}";
say "\N{SmilingCatFaceWithHeartShapedEyes}";
say "\N{Smiling-Cat-Face-With-Heart-Shaped-Eyes}";
say "\N{Smiling----Cat-Face-----With-Heart-----Shaped-Eyes}";

Some problematic names

This doesn’t work out well for some names, and Perl developer Karl Williamson made some comments about this to the Unicode Consortium in 2010. There are some names that have hyphens next to whitespace (so, not medial hyphens), but if you ignore whitespace first, then the hyphen isn’t next to whitespace.

Not only that, removing the hyphen can turn it into a character’s name into that for a completely different character:

  • U+0F68 TIBETAN LETTER A
  • U+0F60 TIBETAN LETTER -A
  • U+0FB8 TIBETAN SUBJOINED LETTER A
  • U+0FB0 TIBETAN SUBJOINED LETTER -A
  • U+116C HANGUL JUNGSEONG OE
  • U+1180 HANGUL JUNGSEONG O-E

Normalize your Perl source

Perl has had Unicode support since Perl 5.6, which means that most Perl tutorials have been bending the truth a bit when they tell you that a Perl identifier, the name that you give to variables, starts with [A-Za-z_] and continues with [0-9A-Za-z_]. With Unicode support, you have many more characters available to you, but moving outside the ASCII range has some problems. You can’t always tell what a variable name is just by looking at it (and this is a design bug in Perl: RT 96814). For instance, you don’t really don’t know what this variable is: Continue reading “Normalize your Perl source”