Yes, otherwise - when keys travel with secrets - there would be absolutely no security.

This is akin to disk encryption where you have to type password on boot, but then data is accessible later.