I have been using RethinkDB in production with clustering.
One thing I would really love to see more time put into is integration of a real consensus protocol like Paxos or Raft for properly recovering from failure.
Currently tables will become unwritable after a single machine fails and you need to update the blueprint/semilattice to use a new master manually. This is error prone (due to race conditions in replication of blueprint updates that can cause vector clock conflicts if you update it from more than one place) and generally not all that fun.
Error handling in these cases could also be better, the initscript it ships with doesn't make this much better (would be nicer if it used a symlink hack or something similar to have different initscripts for each db instance).
Currently it's hard to tell without manual inspection why a RethinkDB server crashed or didn't start without manual intervention.
That being said, I love using RethinkDB, ReQL is great - infact it's so good that it's generally justification enough for me to use it for new projects.
If clustering could be given more love it would be my #1 datastore for all projects probably. (See the Jepsen by aphyr series for an idea of what I am looking for here)
I'm happy you brought up clustering. Internally we've been quite frustrated with this part of the product, but until a few months ago we held off the development on it for two reasons: we wanted to collect more information from users on real use cases and behavior, and there were more immediate bottlenecks in the product.
We restarted heavy development on the clustering infrastructure two months ago, and just yesterday I played with the prototype of the first upcoming upgrade. It's a WIP but is absolutely delightful (you can see my tongue in cheek review of it at https://github.com/rethinkdb/rethinkdb/issues/2957).
Here are the parts that are already done and will be shipped soon:
- Vector clocks conflicts are now resolved automatically, no
more manual conflict resolution
- There is now a ReQL API for clustering that's dramatically
better than the current `rethinkdb admin` tool
- Much love has been put into presenting the abstractions to
users. Everything is cleaned up and simplified, it's easier
to understand and change, and even in advanced cases you
won't have to know anything about blueprints/semilattices.
- Really, this is about to get dramatically better. I can't
summarize it in a bullet point, we put an enormous amount of
effort into this in a thousand different places.
Here's what's coming immediately after that:
- Automatic failover
- Always-on resharding (no more resharding downtime)
(The reason why these latter updates are coming after the API overhaul is because they require a lot of simplification/refactoring/redesign internally as well as externally, and we wanted to do it piecemeal).
Thanks for writing up your feedback and sticking with RethinkDB despite the limitations of clustering 1.x. Multiple people are currently working very hard on this, and things are about to get a lot better.
This makes me very happy to hear. I've been using RethinkDB for a few months now and I love it, but the manual failover has made me a bit uneasy.
Any chance of a clue as to the kind of timeframe for this? Even a very rough idea would be fine as I can appreciate you might not want to (or be able to) commit to anything yet.
I think we'll be able to ship the new clustering API in ~two months (note, it's a huge and a massively delightful change). I'm hoping we'll be able to get failover out two months after that, but it's hard to give precise estimates looking that far out.
One thing I would really love to see more time put into is integration of a real consensus protocol like Paxos or Raft for properly recovering from failure.
Currently tables will become unwritable after a single machine fails and you need to update the blueprint/semilattice to use a new master manually. This is error prone (due to race conditions in replication of blueprint updates that can cause vector clock conflicts if you update it from more than one place) and generally not all that fun.
Error handling in these cases could also be better, the initscript it ships with doesn't make this much better (would be nicer if it used a symlink hack or something similar to have different initscripts for each db instance). Currently it's hard to tell without manual inspection why a RethinkDB server crashed or didn't start without manual intervention.
That being said, I love using RethinkDB, ReQL is great - infact it's so good that it's generally justification enough for me to use it for new projects.
If clustering could be given more love it would be my #1 datastore for all projects probably. (See the Jepsen by aphyr series for an idea of what I am looking for here)