Multimedia data and information management is an important task according to the development of media processing technology. Multimedia is a useful resource that people understand complex situations such as the elderly care domain. Appropriate annotation is beneficial in several tasks of information management, such as storing, retrieval, and summarization of data, from a semantic perspective. However, the metadata annotation for multimedia data remains problematic because metadata is obtained as a result of interpretation depending on domain-specific knowledge, and it needs well-controlled and comprehensive vocabulary for annotation. In this study, we proposed a collaborative methodology for developing ontologies and annotation with domain experts. The method includes (1) classification of knowledge types for collaborative construction of annotation data, (2) division of tasks among a team composed of domain experts, ontology engineers, and annotators, and (3) incremental approach to ontology development. We applied the proposed method to 11 videos on elderly care domain for the confirmation of its feasibility. We focused on annotation of actions occurring in these videos, thereby the annotated data is used as a support in evaluating staff skills. The application results show the content in the ontology during annotation increases monotonically. The number of “action concepts” is saturated and reused among the case studies. This demonstrates that the ontology is reusable and could represent various case studies by using a small number of “action concepts”. This study concludes by presenting lessons learnt from the case studies.