CROKAGE: effective solution recommendation for programming tasks by leveraging crowd knowledge

Abstract

Developers often search for relevant code examples on the web for their programming tasks. Unfortunately, they face three major problems. First, they frequently need to read and analyse multiple results from the search engines to obtain a satisfactory solution. Second, the search is impaired due to a lexical gap between the query (task description) and the information associated with the solution (e.g., code example). Third, the retrieved solution may not be comprehensive, i.e., the code segment might miss a succinct explanation. To address these three problems, we propose CROKAGE (Crowd Knowledge Answer Generator), a tool that takes the description of a programming task (the query) and delivers a comprehensive solution for the task. Our solutions contain not only relevant code examples but also their succinct explanations written by human developers. The search for code examples is modeled as an Information Retrieval (IR) problem. We first leverage the crowd knowledge stored in Stack Overflow to retrieve the candidate answers against a programming task. Then we use a multi-factor relevance mechanism to mitigate the lexical gap problem, and select the top quality answers related to the task. Finally, we perform natural language processing on the top quality answers and deliver the comprehensive solutions containing both code examples and code explanations unlike earlier studies. We evaluate and compare our approach against seven baselines, including the state-of-art. We show that CROKAGE outperforms the seven baselines in suggesting relevant solutions for 902 programming tasks (i.e., queries). Furthermore, a user study with 24 queries and 29 developers confirms the superiority of CROKAGE over the state-of-art tool in terms of relevance of the suggested code examples, benefit of the code explanations and the overall solution quality (code + explanation).