The problem being that libertarianism places a heavy emphasis on absolute private property, which is a positive right. It's my right to evict you by force from occupying an empty apartment in my building unless you pay me the rent I want.

Now, you can certainly argue coherently in favor of absolute private property (though I would argue back, extensively and in detail, against the absolutism), but to do so, you need to start by admitting that it's a positive right.