[Microsoft Word] Find and Replace for Unicode Ranges ≥ U+10000

Microsoft Word has “Find and Replace” function that can not only change text to other text but also change style, font or various properties of text.

It also has some regular-expression-like specification. This article describe how to specify a unicode range outside BMP (Basic Multilingual Plane), above U+10000, which is quite undocumented.

A Prerequisite: How to Apply Find and Replace to All Text by Word VBA

We typically want to replace all text including that in header/footer or textbox, and the Find and Replace dialog does this. However, doing this is a bit difficult in Word VBA. This is also quite undocumented.

This site gives comprehensive explanation to this problem, providing a perfect source code to Find and Replace all text.

Also, you may want to use Application.UndoRecord to treat all the Find and Replace as one action (which determines the behavior of Ctrl+Z).

You may have to combine this with what I present here.

Grapheme Cluster

Firstly, Find and Replace in MS Word seems to recognize characters only according to “grapheme cluster” in Unicode. In short, a grapheme cluster is a (set of) character(s) that can’t be separated by input cursor. For example, R͆ is a ligature of separate two characters, “R”(U+0052) and “͆”(U+0346), but the text cursor can’t usually go between the two characters and R͆ is apparently treated as a single character.

Therefore, when MS Word is ordered to replace “R” with “X”, it won’t change “R͆” to “X͆”. We must specify the whole “R͆” to replace “R͆”.

It is not that the character(s) “R͆” itself must be present at the specification. For example, a specification [A-Z]͆ can be used to change the font of all characters A͆, B͆, …, Z͆.

Surrogate pairs in UTF-16

MS Word seems to use UTF-16 encoding, so a Unicode character outside BMP (above U+10000) is internally a kind of “ligature” of two characters from 0xD800–0xD8FF (high surrogate) and 0xDC00–0xDCFF (low surrogate).

Therefore, we can’t specify a Unicode range with characters above U+10000 as usual.

For example, [𐀀-𐀏] (10000–1000F) appears to MS Word like [{D800}{DC00}-{D800}{DC0F}] , where {XXXX} (of course there is no such syntax in actual MS Word) indicates a “separate” character. Then word detects{DC00}-{D800} as invalid unicode range, because the beginning character of the range is larger than the ending character (just like [Z-A] is invalid).

So, to solve this problem, we need some expression like [{D800}{DC00}-{DC0F}] . You can also use range expression in both high and low surrogates: [{D800}-{D801}][{DC00}-{DFFF}] specifies all 2048 characters between U+10000 — U+107FF.

Note that you can only specify unicode characters in “a rectangle drawn on a 1024x1024 character table indexed by high surrogates and low surrogates” at once. Since Find and Replace has no “OR” expression, you may have to run Find and Replace three times at most in order to specify some unicode range (for example, U+10001—U+107FE) (this is a bit like a small mathematical problem).

Also, since they are not authentic characters, surrogate characters are difficult to input in the dialog. Using Alt+X in Word can produce surrogate characters (typically rendered as a rectangle) and they did work in my testing, but I recommend using Word VBA.

Code Example

This sample Word VBA code change the color of all characters between U+10000 — U+107FF to red, using Application.UndoRecord.

Sub Find_Replace_All_Text()
  Set objUndo = Application.UndoRecord
  objUndo.StartCustomRecord ("Replace All With VBA")
  'Fix the skipped blank Header/Footer problem.
  lngValidate = ActiveDocument.Sections(1).Headers(1).Range.StoryType
  'Iterate through all story types in the current document.
  Dim rngStory As Range
  For Each rngStory In ActiveDocument.StoryRanges
    'Iterate through all linked stories.
    Do
      Change_Red rngStory
      On Error Resume Next
      Select Case rngStory.StoryType
        Case 6, 7, 8, 9, 10, 11
          If rngStory.ShapeRange.Count > 0 Then
            For Each oShp In rngStory.ShapeRange
              If oShp.TextFrame.HasText Then
                Change_Red oShp.TextFrame.TextRange
              End If
            Next
          End If
        Case Else
          'Do Nothing
      End Select
      On Error GoTo 0
      'Get next linked story (if any)
      Set rngStory = rngStory.NextStoryRange
    Loop Until rngStory Is Nothing
  Next
  objUndo.EndCustomRecord
End Sub

Sub Change_Red(myRange As Range)
    Dim UniRange As String
    UniRange = "[" & ChrW(&HD800) & "-" & ChrW(&HD801) & "][" & ChrW(&HDC00) & "-" & ChrW(&HDFFF) & "]"

    myRange.Find.ClearFormatting
    myRange.Find.Replacement.ClearFormatting
    myRange.Find.Replacement.Font.Color = wdColorRed
    With myRange.Find
        .text = UniRange
        .Replacement.text = ""
        .Forward = True
        .Wrap = wdFindContinue
        .Format = True
        .MatchCase = False
        .MatchWholeWord = False
        .MatchByte = False
        .MatchAllWordForms = False
        .MatchSoundsLike = False
        .MatchFuzzy = False
        .MatchWildcards = True
        .Execute Replace:=wdReplaceAll
    End With
End Sub