Exploring and identifying fine-grained accessibility issues in app store using fine-tuned deep learning

The Apple App Store (AAS) allows users to provide feedback on applications, offering developers insights into improving software performance. Researchers have utilized this feedback for software evolution activities, including features, issues, and nonfunctional requirements. However, end-user feedback has not been explored to identify accessibility-related challenges. This study proposes an automated approach to detect and classify accessibility issues by analyzing end-user reviews in the AAS. We crawled 178667 user reviews from 85 apps across 18 categories to represent a diverse sample. We developed a coding guideline to identify common accessibility issues, including Navigation and Interaction Problems (NAV), Input and Control Issues (INPUT), Compatibility with Assistive Technologies (CAT), Audio and visual accessibility issues (AUDIOVISUAL), and UI Accessibility Issues (UI). We manually annotated reviews using coding guidelines and content analysis to create a labeled dataset for training and evaluating deep learning(DL) algorithms to detect accessibility in user comments and classify them into categories. The experiments showed that fine-tuned DL classifiers achieved high accuracy in detecting accessibility and classifying them into specific types. For binary classification, the CNN classifier achieved 93% precision, while LSTM, BiLSTM, GRU, and BiGRU achieved accuracies from 76% to 87%. In fine-grained classification, CNN performed better with 97% accuracy, followed by BiGRU and BiLSTM at 96%. The BiLSTM and LSTM models demonstrated strong performance, with accuracies of 96% and 95%. These results show the potential of automated methods to improve identification of accessibility challenges, helping developers address these issues effectively and enhance user experience.