Any application running under the user id has full access to his private data.
Add the capability to do network communication and suddenly the all wolrd has access to $ HOME.
This is why in the container model of mobile OS and Windows/Mac OS X sandboxes, applications only get to see file handles to files choosen by the user.
