Statistical Modeling, Semantic Interpretation, and Partial Matching of Images

Project Description:

• Problem Statement: Automatic semantic image annotation is becoming very important now because the naive users communicate with the image database via high-level semantic visual concepts, not low-level visual features as it has been treated by the traditional content-based image retrieval systems. Unfortunately, automatic semantic image annotation is still very challenging because of the following problems: (a) How the human being interprete the semantic meaning of an image? From the above figure, one can find that the human beings may interprete the semantic meaning of the images according to the individual objects, object-object relationships or global information. Some researchers mentioned that the human beings recognize and interprete an image based on the objects, but some researchers claimed that the human beings recognize an image without using the details of the image. How the human exactly interprete the semantic meaning of an image? How much psychological and psychophysical studies should be inputed? (b) How we can simulate the human vision system to interprete the semantic concepts of an image? It is very hard if not impossible because there is a semantic gap between the semantic visual concepts and the low-level visual features which the computer can calculate automatically for us. What is the rule for our techniques to do this? Learning from examples? (c) How the naive users specify their query concepts and how the database answer their queries? Most naive users may not have examples to specify their query concepts, and some of them may not know exactly what they want at all. How semantic image annotation system can support more effcient search engine?
• Proposed Research: In order to solve the above problems, we will focus on the following researches: (a) salient object detection. What's the salient object? How to detect the defined salient objects? How to characterize the salient objects? (b) automatic semantic image annotation via salient object detection. What's the relationship among the semantic visual concepts and the salient objects? How to evaluate these determined relationships? (c) image database management and search engine design. How to manage large-scale image database? What kind of image search engine can let the naive users to specify their queries?


Current Project Achievements



Salient Object Detection Flowchart:



Some Salient Object Detection Results from Natural Scenes:



Semantic Image Concept Interpretation with Dominant Compounds:



Statistical Image Modeling for Partial Matching:



Semantic Image Classification Flowchart:



Some Semantic Image Classification Results:



Concept Ontology Visualization for Interactive Image Access:

Concept Ontology Visualization with ICONS for Interactive Image Access:

System Demo for Semantic Image Classification

• System Demo1

  First, download this zip file; Second, generate a folder and unzip the file and save them in the same folder; Third, double click Player.exe

• System Demo2

  double click if you have media player!

a. Journal Publications:

• J. Fan , Y. Gao, H. Luo, G. Xu, ``Statistical modeling and conceptualization of natural images", Pattern Recognition, special issue on automatic understanding of photography, vol.38, no.6, pp.865-885, 2005.
• J. Fan , G. Zeng, M. Body, M.-S. Hacid, ``Seeded Region Growing: An Extensive and Comparative Study", Pattern Recognition Letters, vol.26, no.5, pp.1139-1156, 2005.

b. Conference Publications:

• Y. Gao, J. Fan, H. Luo, X. Xue, R. Jain, ``Automatic Image Annotation by Incorporating Feature Hierarchy and Boosting to Scale up SVM Classifiers", ACM Multimedia, Santa Barbara, CA, 2006.
• Y. Gao, J. Fan, ``Incorporate Concept Ontology to Enable Probabilistic Concept Reasoning for Multi-Level Image Annotation", ACM Multimedia Workshop on Multimedia Information Retrieval ( MIR'06), Santa Barbara, CA, 2006.
• J. Yang, J. Fan, D. Hubball, Y. Gao, H. Luo, B. Ribarsky, M. Ward, ``Semantic image browser: Bridging information visualization and automatic intelligent image analysis", IEEE VAST'06 (IEEE Symposium on Visual Analytics Science and Technology 2006), 2006.
• J. Fan, H. Luo, Y. Gao, M.-S. Hacid, ``Mining Image Databases on Semantics via Statistical Learning'', ACM SIGKDD Conference, Chicago, IL, August 20-25, 2005.
• J. Fan, H. Luo, Y. Gao, M.-S. Hacid, ``Mining Image Databases on Semantics via Statistical Learning'', ACM KDD Conference, Chicago, IL, August 20-25, 2005.
• J. Fan, H. Luo, Y. Gao, ``Learning the semantics of images by using unlabeled samples'', IEEE CVPR (Computer Vision and Pattern Recognition), San Diego, CA, June 20-26, 2005.
• J. Fan, Y. Gao, H. Luo, ``Multi-level annotation of natural scenes using dominant image components and semantic image concepts", ACM Multimedia, New York, Oct.10-15, 2004 ( best paper competition award).
• Y. Gao, J. Fan, G. Xu, ``Salient Objects: Semantic building blocks for image concept interpretation", Int. Conf. on Image and Video Retrieval (CIVR'04), July 21-23, 2004.
• J. Fan, H. Luo, L. Wu, ``Integrating unlabeled samples for semantic video/image classification", ACCV (Asian Conf. on Computer Vision), Jan. 27-30, 2004.
• J. Fan, Y. Gao, H. Luo, G. Xu, ``Automatic image annotation by using concept-sensitive salient objects for image content representation", ACM SIGIR, Sheffield, UK, July 25-29, 2004.
• J. Fan, H. Luo, J. Xiao, L. Wu, ``Semantic video classification and feature subset selection under context and concept uncertainty", ACM/IEEE Joint Conf. on Digital Libraries (JCDL'04), Tuson, AZ, June 7-11, 2004.
• J. Fan, Y. Gao, H. Luo, M.-S. Hacid, ``A novel framework for semantic image classification and benchmark", ACM SIGKDD Workshop on Multimedia Data Mining, 2003.
• X. Zhu, J. Fan, H. Luo, M.-S. Hacid, ``Using small samples for content-based image retrieval via relevance feedback'', ACM Multimedia Workshop on Multimedia Information Retrieval, 2002.


"You push the button, we do the rest" ---Slogan for Eastman Kodak---