Enhancement: Arabic Search Normalization (Diacritics and Character Variants) #10483

New issue

Open

opened 2025-03-13 03:11:06 +00:00 by abdullahO2 · 0 comments

abdullahO2 commented

2025-03-13 03:11:06 +00:00

(Migrated from github.com)

Hello Organic Maps developers,

I'm writing to request an enhancement to the Arabic search functionality within Organic Maps. Currently, searching for Arabic place names is unreliable if the search term doesn't exactly match the diacritics (tashkeel), hamza variations (أ، إ، ئ، ؤ), and other orthographic details present in the underlying map data. This is a common problem, as many users omit these markings when searching.

OsmAnd recently addressed a similar issue by implementing normalization during both indexing and searching, as discussed here: [Enhanced Arabic Search Functionality: Addressing Diacritics, Character Variants, and Other Search-Related Improvements]. This significantly improves the accuracy and usability of Arabic search.

The solution involves normalizing both the indexed place names and the user's search query. A Java example of such a normalization function is shown below (this could be adapted to C++ or whichever language Organic Maps uses):

import java.text.Normalizer;
import java.util.regex.Pattern;

public class ArabicNormalizer {

    private static final Pattern DIACRITICS_PATTERN = Pattern.compile("\\p{Mn}");

    public static String normalizeArabic(String text) {
        if (text == null) {
            return null;
        }

        String normalized = Normalizer.normalize(text, Normalizer.Form.NFD);
        normalized = DIACRITICS_PATTERN.matcher(normalized).replaceAll(""); // Remove diacritics

        // Hamza variations
        normalized = normalized.replace("إ", "ا");
        normalized = normalized.replace("أ", "ا");
        normalized = normalized.replace("ئ", "ي");
        normalized = normalized.replace("ؤ", "و");

        // Other normalizations
        normalized = normalized.replace("آ", "ا");
        normalized = normalized.replace("ى", "ي");
        normalized = normalized.replace("ة", "ه");

        normalized = normalized.trim().replaceAll("\\s+", " "); // Normalize whitespace

        return normalized;
    }
}

Hello Organic Maps developers, I'm writing to request an enhancement to the Arabic search functionality within Organic Maps. Currently, searching for Arabic place names is unreliable if the search term doesn't exactly match the diacritics (tashkeel), hamza variations (أ، إ، ئ، ؤ), and other orthographic details present in the underlying map data. This is a common problem, as many users omit these markings when searching. OsmAnd recently addressed a similar issue by implementing normalization during both indexing and searching, as discussed here: [[Enhanced Arabic Search Functionality: Addressing Diacritics, Character Variants, and Other Search-Related Improvements](https://github.com/osmandapp/OsmAnd/issues/11709)]. This significantly improves the accuracy and usability of Arabic search. The solution involves normalizing both the indexed place names and the user's search query. A Java example of such a normalization function is shown below (this could be adapted to C++ or whichever language Organic Maps uses): ```java import java.text.Normalizer; import java.util.regex.Pattern; public class ArabicNormalizer { private static final Pattern DIACRITICS_PATTERN = Pattern.compile("\\p{Mn}"); public static String normalizeArabic(String text) { if (text == null) { return null; } String normalized = Normalizer.normalize(text, Normalizer.Form.NFD); normalized = DIACRITICS_PATTERN.matcher(normalized).replaceAll(""); // Remove diacritics // Hamza variations normalized = normalized.replace("إ", "ا"); normalized = normalized.replace("أ", "ا"); normalized = normalized.replace("ئ", "ي"); normalized = normalized.replace("ؤ", "و"); // Other normalizations normalized = normalized.replace("آ", "ا"); normalized = normalized.replace("ى", "ي"); normalized = normalized.replace("ة", "ه"); normalized = normalized.trim().replaceAll("\\s+", " "); // Normalize whitespace return normalized; } }

Rows
Columns