The technique is a bit more complex than simple text search. It uses actual click-through data to determine quality of the search result as it applies to the query about an API. The SWIM (synthesizing what I mean) paper can be found here:

https://arxiv.org/abs/1511.08497

Also of note from section 5.2: SWIM uses Bing clickthrough data to build the model.

Using a better (or simply more used) search engine like google search would likely improve the SWIM results.

EDIT: The method they use to compare the methods is BLEU which stands for Bi-Lingual Evaluation Understudy and was developed for automated machine translation evaluation. Apparently CS authors no longer bother with expanding acronyms the first time they are used. Paper is here:

http://www.aclweb.org/anthology/P02-1040.pdf

EDIT2: Also, for the BLEU comparison they compare the computer generated API sequence to a human-written API sequence. However, they give no details on who or how the human-written sequence is developed. Are the researchers coming up with their own API sequence? Are they using mechanical turk? Interns? There could be significant bias depending on how these human-written sequences are generated.

> Apparently CS authors no longer bother with expanding acronyms the first time they are used.

What does CS stand for? :p

But yeah, I might expect BLEU to be unexpanded in an NLP paper maybe, but not here

> What does CS stand for?

Haha. Well played. :) I just think it's good practice in general, no matter how common, to use the full name when the acronym is introduced. Especially if it is a method used in the paper.

They also test a Lucene based model, which is much worse on Top1, but similar to SWIM for Top 3 and Top 10.

Also, BLEU is a fairly widely used metric. I assumed they referenced the paper though?

Ok. I definitely see the value then.