> What's the biggest hurdle when trying to refactor the code?

I'd guess it's updating cross-references in prose and rewriting chapters of documentation which no longer make sense after your refactoring.