It may be a problem with the teaching though, rather than the subject matter itself. I understand why it takes a little bit of thinking to get used to, but not why it is fundamentally hard.
Now, algorithms can be difficult to understand and many of them use pointers... Do some people confuse the 2?
I don't think the difficulty is fundamental, and I don't think anyone here claimed that it was. But it is initially challenging nonetheless.
To tie this back in to the beginning of the thread, being able to understand indirection (such as in pointers and algorithms) is a far more important skill than understanding OOP.
Now, algorithms can be difficult to understand and many of them use pointers... Do some people confuse the 2?