Well you our decimal numbering system is based on our digits, so that's why digit is another word for number. Computers only work on numbers, even more, integer numbers, the kind you can count with your digits. Most computers work with binary numbers, but that isn't a rule, in fact some older computers worked in other systems than binary. So calling them digital was more appropriate than binary.

I don't think many people call computers 'binary electronics', the usual term electrical engineers use would be discrete circuits. Note that that does not rule out other counting systems either.