In recent years, autonomous driving technologies are being developed for vehicles and personal mobility devices including golf carts and autonomous wheelchairs for various use cases, not only outside areas but inside areas like shopping malls, hospitals and airpots. The main purpose of developing these autonomous vehicles is to avoid the traffic accidents caused by human errors, to assist people with walking, and to improve human comfort by relieving them from driving. Most relevant research focuses on the efficiency and safety of autonomous driving, however, in order to use by the widespread of people in the society, it is important to consider passenger comfort inside vehicles as well as safety and efficiency. Therefore, in this work, we emphasize the importance of considering passenger comfort in designing the control loop of autonomous navigation for the concept of comfortable intelligence in the future autonomous mobility. Moreover, passenger characteristics, in terms of ride comfort in an autonomous vehicle, have not been investigated with regard to safety and comfort, depending on each passenger's driving experience, habits, knowledge, personality, and preference. There are still few studies on the optimization of autonomous driving control reflecting passenger characteristics and different stress factors during the ride. In this study, passenger stress characteristics with different stress factors were objectively analyzed using physiological indices (heart rate and galvanic skin response sensors) during autonomous wheelchair usages. Two different experimental results from 12 participants suggest that there are always at least two types of passengers: one who experiences stress and the other who does not, depending on the stress factors considered. Moreover, with regard to the classification result for the stress reduction method, there are two types of passenger groups, for whom the solution method is, respectively, either effective or ineffective.