Search function doesn't find results if diacritics/special characters are missing

[JoyfulGen]

I discovered this while searching for aliases of the willow flute. Two examples:

1. If you search for "seljefløyte" (with the slashed "o"), UMIL finds Willow Flute as a result. But if you search for "seljefloyte" (without the slashed "o"), UMIL finds nothing.
2. Similarly, if you search for "sälgflöjt" (with the umlauts), UMIL finds Willow Flute. But if you search for "salgflojt" (no umlauts), UMIL finds nothing.

I have a French Canadian keyboard that makes umlauts fairly easy, but I still find entering diacritics mildly annoying. I think that the search function should be able to find labels and aliases even if the diacritics are missing.

[ahankinson]

Solr does this through a process called "folding" where letters with diacritics are "folded" into their ASCII equivalents. You likely want this to happen both at index time and at query time.

The best folding filters are available in the Analysis Extras module, which you have to enable explicitly. I do this in the solr.in.sh file: SOLR_MODULES=analysis-extras

This gives you access to the ICU Folding Filter: https://solr.apache.org/guide/solr/latest/indexing-guide/filters.html#icu-folding-filter

We have this configured in RISM Online. Feel free to use this configuration: https://github.com/rism-digital/muscatplus_solr/blob/develop/configsets/muscatplus/conf/schema.xml#L231-L255

[ahankinson]

One thing to note: Not all non-ASCII characters fold into things that you expect. œ will fold to "oe", but ü will only fold to u -- not to "ue" (this is a common transformation of u with an umlaut.)

A very annoying case is when someone with a cyrillic keyboard starts typing Latin characters. РНОNЕ -- this will not come back with a search for "phone" because the PHO and E characters are from the Cyrillic set.