Enhancing Sindhi word segmentation using subword representation learning and position-aware self-attention

• Verfasst von: Ali, Wazir [VerfasserIn]
• Verfasst von: Kumar, Jay [VerfasserIn]
• Verfasst von: Tumrani, Saifullah [VerfasserIn]
• Verfasst von: Nour, Redhwan [VerfasserIn]
• Verfasst von: Noor, Adeeb [VerfasserIn]
• Verfasst von: Xu, Zenglin [VerfasserIn]
• Titel: Enhancing Sindhi word segmentation using subword representation learning and position-aware self-attention
• Verf.angabe: Wazir Ali, Jay Kumar, Saifullah Tumrani, Redhwan Nour, Adeeb Noor, and Zenglin Xu (Senior Member, IEEE)
• Jahr: 2025
• Umfang: 10 S.
• Illustrationen: Illustrationen
• Fussnoten: Online veröffentlicht: 27. November 2024, Artikelversion: 13. Dezember 2024 ; Gesehen am 04.06.2025
• Titel Quelle: Enthalten in: Institute of Electrical and Electronics EngineersIEEE access
• Ort Quelle: New York, NY : IEEE, 2013
• Jahr Quelle: 2025
• Band/Heft Quelle: 13(2025), Seite 183133-183142
• ISSN Quelle: 2169-3536
• DOI: doi:10.1109/ACCESS.2024.3507382
• Datenträger: Online-Ressource
• Sprache: eng
• Sach-SW: Attention mechanism
• Sach-SW: Computer science
• Sach-SW: Context modeling
• Sach-SW: Labeling
• Sach-SW: Long short term memory
• Sach-SW: long short-term memory
• Sach-SW: neural network
• Sach-SW: Noise measurement
• Sach-SW: Recurrent neural networks
• Sach-SW: representation learning
• Sach-SW: Representation learning
• Sach-SW: Robustness
• Sach-SW: Tagging
• Sach-SW: White spaces
• Sach-SW: word segmentation
• K10plus-PPN: 1927457734

Abstract

Sindhi word segmentation is a challenging task due to space omission and insertion issues. The Sindhi language itself adds to this complexity. It’s cursive and consists of characters with inherent joining and non-joining properties, independent of word boundaries. Existing Sindhi word segmentation methods rely on designing and combining hand-crafted features. However, these methods have limitations, such as difficulty handling out-of-vocabulary words, limited robustness for other languages, and inefficiency with large amounts of noisy or raw text. Neural network-based models, in contrast, can automatically capture word boundary information without requiring prior knowledge. In this paper, we propose a Subword-Guided Neural Word Segmenter (SGNWS) that addresses word segmentation as a sequence labeling task. The SGNWS model incorporates subword representation learning through a bidirectional long short-term memory encoder, position-aware self-attention, and a conditional random field. Our empirical results demonstrate that the SGNWS model achieves state-of-the-art performance in Sindhi word segmentation on six datasets.