This is also very true. The purpose of systems like Kubernetes/Mesos is to be set up once by a company, and for most developers to just interact with the CLI tool to run containers. Tools like Google Container Engine, make this into a software as a service product, where the cloud provider provides the API. In that world, as a developer, you never really think about the OS, just your app. But unlike in traditional PaaS, there is no framework that restricts what you can code (language, libraries, etc).
The main difference between an PaaS like Heroku and a general purpose cluster manager service like GKE is that the former can simplify my making certain assumptions about your workload.
For example if a service knows your container is serving a web application then it can sensibly provision load balancers, DNS, HTTPS, auto-scaling, static content caching, automatic QPS monitoring etc. to support your app with little explicit configuration. And with a commercial service you get an SLA for those things.
You can get much of this with a general purpose cluster as well, but of course you need to configure it yourself, and more importantly - debug it when it goes awry.
Sadly in the Cedar stack, Heroku did away with Varnish or static asset caching from their "Dynos". Instead all assets bundled with your app are served using Ruby and dyno time. While a simple and clean architecture, this either raises the cost or forces you to place static assets on other servers (or use CDNs). And you'll still need to debug someone else's stack just as much -- when you get a service-specific error for exceeding memory limits or having requests take too long because of instance-level queues, etc.
At this point, I never pick PaaS because it's less work, I pick it simply because I've used it before and it's easy to get started with. Production apps are never hands-off if you're the one developing them ;-)
> But unlike in traditional PaaS, there is no framework that restricts what you can code (language, libraries, etc).
I'm not sure how traditional PaaSes are limited in that sense. Heroku accepts a wide variety of software through the buildpacks mechanism. Cloud Foundry supports both buildpacks and docker images, with clean extension points to add further mechanisms -- for example, .NET-on-Windows deployments, which is under development.
Disclaimer: I've worked on Cloud Foundry and I'm employed by Pivotal.
