Many Natural Language Processing modules have been developed in English and other major languages, to extract useful insights from unstructured data, however not much work has been done for our local languages in Pakistan.
No work has been currently done in NLP with regards to Urdu, and one of the major reason why is that so is the lack of basic tools and a framework to process the Urdu language.
Therefore, in order to change that a Pakistani duo, Ikram Ali and Mujadad Rao, have developed an open source Python library for Urdu called UrduHack.
Ikram has over 7 years of experience in the software industry and is an avid Machine Learning practitioner with a bachelors in Computer Science from Virtual University, Pakistan. He is currently working as the Principal Software Engineer at Arbisoft, Lahore.
Mujadad has a Bachelors in Computer Science from the University of Central Punjab, and is currently employed as a Machine Learning Developer at Arbisoft.
Ikram Ali speaking to the local news talked about his idea on how one day he decided to research on the work being done on Urdu in the field of Natural Language Processing. He only found a few organizations working on making applications for Urdu and was disappointed to see only commercial work. Therefore, as a result, he set about to change that by making a full-fledged Urdu library.
Mujadad speaking about UrduHack said, “Our plan is to achieve the maximum possible heights with UrduHack. We want to make it a full-fledged Urdu NLP library which people can use to make thousands of interesting applications for desktop, mobile or web.”
Currently, the duo has managed to develop two core modules of the library, Normalization and Tokenization that are essential in cleaning and converting data from a cluttered form to a standard form. The library is still very much a work in progress according to the duo they are planning to use TensorFlow v2 in their upcoming modules later this month.
Their journey hasn’t been without its own set of challenges while developing the app and have faced a number of technical difficulties such as the use of Unicode for the Urdu script.
However, they were able to over the challenge by contacting Unicode Consortium and demanded a separate fixed Unicode block for Urdu.
The second challenge is finding reliable and authentic data in Urdu. The UrduHack team is actively looking for Urdu data available in digital form and if anyone has access to Urdu data, they can contact mujadad.ali@arbisoft.com.
Reference links: propakistani.pk
