As a result of the growth of sensor-enabled mobile devices, in recent years, users can utilize diverse digital contents everywhere and anytime. However, the interfaces of mobile applications are often unnatural due to limited computational capability, miniaturized input/output controls, and so on. To complement the poor user interface (UI) and fully utilize mobility as feature of mobile devices, we explore possibilities for a new UI of mobile devices. This paper describes the method for recognizing and analyzing a user's continuous action including the user's various gestures and postures. The application example we created is mobile game called AM-Fishing game on mobile devices that employ the accelerometer as the main interaction modality. The demonstration shows the evaluation for the system usability.