Word characters and separators

Word characters include all uppercase and lowercase letters, digits, and the following additional characters:

•_ (underscore)

•# (number/pound/hash sign)

•& (ampersand)

All other characters are separators (except in queries, wildcards ? and *, and special query characters ~, ", -, and !).

However, && by itself is not a word. It is a Boolean operator. When combined with at least one more word character, && can be part of a word. For example, a&&b is a word.

Query analysis and document indexing are not case-sensitive. Uppercase and lowercase letters are treated the same.

Regular expression definition of English word characters

The following regular expression provides, in succinct form, a complete specification of English word characters (except for treatment of && as a non-word):

[ A-Za-z0-9_#& ]+

Letters and digits in different character sets

Topics include:

•Letters and digits defined, page 38

•Letters and digits in files, page 38

Letters and digits defined

All letters and digits are word characters. What IAP considers a letter or digit depends on the character set encoding used. For US ASCII encoding, letters are uppercase and lowercase English letters (A-Z, a-z). For ISO 8859-1 (Latin-1) encoding, used for Western European languages, accented letters are included. Most ideographic characters, such as those used in Asian languages, are also considered letters.

Whatever the language and encoding used for a particular document (file or email message), IAP maps encoded characters to the Unicode 2.0 standard. The Unicode 2.0 standard is then used to determine if a given character is a letter or a digit (or neither):

•A letter is any Unicode character in one of the following Unicode categories: Ll (lowercase letter), Lu (uppercase letter), Lt (title case letter), Lm (modifier letter), or Lo (other letter).

•A digit is any Unicode character whose Unicode name contains the word DIGIT, provided it is not in the range \u2000 (en quad = en space) through \u2FFF (ideographic description - future).

Letters and digits in files

Although all letters and digits are word characters, their treatment in files (including email message attachments) depends on the character encoding used. You can search for any words in email message bodies and headers, regardless of the encoding.

You can search for words in files (including email body, header, attachments, and indexed documents) provided the character encoding is one the following:

38 Query expression syntax and matching