Bridging the Gap: Develop New CBVR Technologies for Online Career Training of Students with Hearing Impairments

Challenging Problems:

  • Recent advances on high-performance video compression, storage and communication technologies present an extraordinary opportunity to enable online IT-related training of the students with hearing impairments by recording, indexing and retrieval of large-scale lecture videos. In spite of some recent research progress on semnatic video classification and retrieval, incorporating CBVR technologies for online distance learning, especially for the students with hearing impairments, is still an open issue with many unsolved problems:
  • (a) Concept-Oriented Indexing and Retrieval of Large-Scale Lecture Videos: Most existing CBVR systems focus on achieving automatic low-level feature extraction, they cannot directly be extended for distance learning applications. In order to incorporate lecture videos and CBVR technologies for online distance learning application, there is urgent need to enable concept-oriented indexing and retrieval of large-scale lecture videos, so that the students with hearing impairments can easily access the lecture videos and have better understanding of lecture contents and obtain better training of relevant skills. To do this, we have to address several issues: how the students can specify their queries easily and intuitively? how the students can evaluate the query results quickly?
  • (b) Multi-Modal Video Caption: The students with hearing impairments largely depend on their vision, thus it is very important to provide text captions of video contents. Automatic speech recognition and lip reading are widely used for video captions, but they seriously suffer from the problem of lack of accuracy and robustness.
  • (c) Active Learning Environment Establishment: To make the students with hearing impairments to be active learners for our distance learning problem, it is very important to support two-way communication in an online distance learning system, so that we can increase national participation of the students with hearing impairments in science and engineering.

    • Research Focus:

    This project will tackle these challenging problems in a specific domain of lecture video:
  • develop a new framework to achieve concept-oriented indexng and retrieval of large-scale lecture videos, so that the students with hearing impairment can easily and effectively access our lecture materials and obtain good training of job skills;
  • integrate multi-modal technologies and resources to achieve multi-modal video caption integration and synchronization, so that the students with hearing impairments can understand the lecture video contents easily and accurately (what have been said in a lecture video);
  • establish an active learning environment for distance education, so that the students with hearing impairments can become active learners of advanced technologies and participate the society.

    • Relevant Publications:

  • J. Fan, H. Luo, Y. Gao, R. Jain, ``Incorporating Concept Ontology to Boost Hierarchical Classifier Training for Automatic Multi-Level Video Annotation", IEEE Trans. on Multimedia, special issue on Semantic Image and Video Indexing in Broad Domains, June, 2007.
  • J. Fan , H. Luo, A.K. Elmagarmid, ``Concept-Oriented Indexing of Video Database towards More Effective Retrieval and Browsing", IEEE Trans. on Image Processing, vol.13, no.7, pp.974-992, 2004.
  • J. Fan , X. Zhu, A.K. Elmagarmid, W.G. Aref, L. Wu, ``ClassView: Hierarchical Video Shot Classification, Indexing, and Accessing", IEEE Trans. on Multimedia, vol.6, no.1, pp.70-86, 2004.
  • H. Luo, J. Fan, ``Building Concept Ontology for Medical Video Annotation", ACM Multimedia, Santa Barbara, CA, 2006.
  • Y. Gao, J. Fan, ``Incorporate Concept Ontology to Enable Probabilistic Concept Reasoning for Multi-Level Image Annotation", ACM Multimedia Workshop on Multimedia Information Retrieval (MIR'06), Santa Barbara, CA, 2006.
  • Y. Gao, J. Fan, H. Luo, X. Xue, R. Jain, ``Automatic Image Annotation by Incorporating Feature Hierarchy and Boosting to Scale up SVM Classifiers", ACM Multimedia, Santa Barbara, CA, 2006.
  • H. Luo, J. Fan, J. Yang, B. Ribarsky, S. Satoh, ``Exploring large-scale video news via interactive visualization", IEEE VAST'06 (IEEE Symposium on Visual Analytics Science and Technology 2006), 2006.
  • H. Luo, J. Fan, ``Concept-oriented video skimming via semantic video classification" (demo paper), ACM Multimedia, New York, Oct. 10-15, 2004.
  • J. Fan, H. Luo, ``Semantic video classification by integrating unlabeled samples for classifier training" (poster paper), ACM SIGIR, Sheffield, UK, July 25-29, 2004.
  • J. Fan, H. Luo, J. Xiao, L. Wu, ``Semantic video classification and feature subset selection under context and concept uncertainty", ACM/IEEE Joint Conf. on Digital Libraries (JCDL'04), Tuson, AZ, June 7-11, 2004.
  • H. Luo, J. Fan, Y. Gao, G. Xu, ``Multi-Modal salient objects: General semantic building blocks for semantic video concept interpretation", Int. Conf. on Image and Video Retrieval (CIVR'04), Dublin, Ireland, 2004.

    • If we know what we were doing, it wouldn't be research, would it? ---Albert Einstein(1879-1955)---