>>But if they do make a point it's that we don't need to (and often can't) understand what will result from systems that have a built in feedback mechanism.
That may be true, but my point was more to the effect that there are things we fail to understand about human nature that drives decision making, actions, and goal setting within our own society, let alone other cultures. Consider how factors such as your own personal energy levels affect your decision making throughout the day (decision fatigue)...how are we going to model something like that with AI? Or consider that many different cultures around the world have very different societal goals than Americans do pursuing personal freedom and material well-being. How do we consider those different abstract goals, that most citizens don't even realize or understand why they pursue, because it's just how they have been raised. How do environmental factors drive decision making in different climates? Those who live in a very cold climate have to take different precautions and deal with different risks than those in warm climates do. What about locally available resources? Many business (and design) decisions are made based as much upon supply chain availability and risk management as they are customer demand. How does AI consider these factors when AI doesn't even know these things exist? These are the things I was talking about that we still have difficulty understanding about each other, let alone being able to program a robot to imitate.
Maybe the difference between a human intelligence and a super intelligence comes, in part, from not being affected by day to day mood/chemical processes that are probably related to keeping the body as a whole functioning. The super intelligence has an intelligence that's isolated from these needs.
I don't really have an answer to all your points, just a general feeling that AI needed be just like our "I" to be effectively superior.
That may be true, but my point was more to the effect that there are things we fail to understand about human nature that drives decision making, actions, and goal setting within our own society, let alone other cultures. Consider how factors such as your own personal energy levels affect your decision making throughout the day (decision fatigue)...how are we going to model something like that with AI? Or consider that many different cultures around the world have very different societal goals than Americans do pursuing personal freedom and material well-being. How do we consider those different abstract goals, that most citizens don't even realize or understand why they pursue, because it's just how they have been raised. How do environmental factors drive decision making in different climates? Those who live in a very cold climate have to take different precautions and deal with different risks than those in warm climates do. What about locally available resources? Many business (and design) decisions are made based as much upon supply chain availability and risk management as they are customer demand. How does AI consider these factors when AI doesn't even know these things exist? These are the things I was talking about that we still have difficulty understanding about each other, let alone being able to program a robot to imitate.