Page 85 discussed the Independent Progressive Party that he had registered to (later cancelled). According to the "Senate Fact-Finding Committee on Un-American Activities" the party was being run by the same people who ran the Communist Party. It also noted that a few people who joined it (and it seems this is where they categorized Feynman) were liberals who were dissatisfied with the two party system and had no Communist leanings.

Amusingly enough, there's a letter (page 110) to the president from an acquaintance accusing Feynman of being a Communist and a master of deception. Page 252+ elaborates a little more and identifies the acquaintance as female.