It's easy - you have diminishing improvements in pedestrian survival rates. 40MPH+ is associated with a fatality rate of over 50%, whereas you get to 30MPH and you have 7%, and 20MPH is essentially zero: https://nacto.org/docs/usdg/relationship_between_speed_risk_...
Even if it is close to zero, how do we decide if the lives saved from going from a .1% fatality rate to a .09% fatality rate is worth the speed reduction or not?