I think part of it is that there's a (much dreaded) context switch from keyboard to mouse. Even if I can click a spot in text quicker with a mouse, I'll often just stick to the keyboard to retain my "flow state". I might trade off some time, but it keeps me slightly more focused.