I've build an interesting NN-based model, and I've been thinking about using some of the early layers as feature for a search engine.
The obvious thing to do is to dump the feature vectors into Elastic or Solr or something.
Ideally what I want is to:
1) Put 1024 dimensional vectors of floats into the index
2) Use plain cosine distance as the distance metric
2b) Customize the distance metric (or preferably use a preexisting and optimized earth-mover-distance implementation).
My initial Googling indicated that 1 and 2 are harder than I expected - it seems both Elastic and Solr don't have good representations for vectors, and assume you want BM25 or TF-IDF for your ranking.
Surely I'm missing something? Not super-keen on having to drop back to using Lucene.
I don't think you're missing anything. Elastic (I'm not very familiar with Solr) is built on assumptions from TFIDF etc., and a big one is that you reduce the size of your potential result set by looking up words in an index. Obviously that doesn't hold if your documents are represented as word vectors.
You can use something like annoy for searching vector space. It's not made specifically for text search, so you'll have to roll your own normalization and so on, but that's usually the less complicated part anyway.
