Personal tools

Document Actions

LREC 2014 Tutorial - When Semantic Language Resources meet Cognitive Systems

Saturday 31 May 2014, Reykjavik, Iceland



Semantic language resources are increasingly being used beyond language technology applications to computer vision ones (e.g. large scale object recognition in Images-augmented WordNet, ImageNet) and cognitive robotics (for verbal interaction with humans and for verbalisation of visual scenes). This is the modern manifestation of a long-standing quest in Artificial Intelligence, regarding the integration of language with other modalities (images, gestures, body movements), or to put it more generally, the integration of symbolic and sensorimotor representations. Multimedia ontologies, collections of labelled images or video keyframes and knowledge-bases have appeared in different strands of Artificial Intelligence (AI) research. The automatic correlation of language and the denoted sensorimotor experiences has been a major challenge which is commonly known as the Semantic Gap problem.

On the other hand, there is growing experimental evidence that language is tightly related to perception and action. From Quillian’s view of semantic memory as a lexical network accessed through a spreading activation of knowledge, modern neuroscience provides new evidence on the structure of semantic memory and points to the fact that semantic information is multisensory, multimodal and distributed. Intelligent multimedia systems, become more and more informed by experimental research on how the human brain works, with the aspiration that a simulation or transfer of mechanisms from the human brain to artificial agents will be more promising in terms of scalability and generalisation. In such research landscape, semantic language resources need to inform and be informed systematically by Cognitive Systems Research.


This tutorial aims to provide a comprehensive overview of semantic language resources, from a new, interdisciplinary perspective: that of cognitive science. In doing so, the tutorial will relate semantic language resources with the evolving field of Cognitive Systems, pointing to needs, challenges and future directions of research. Furthermore, it will familiarise the audience with new types of semantic resources that integrate language with vision and action, i.e. resources that correlate language with images, and motoric representations of actions. The cognitive underpinnings of semantic language resources and their integration with non-verbal modalities will be ellaborated through reference to the latest theories and experimental findings on how the human semantic memory works. A case study of a multimodal semantic network for cognitive systems will be presented (the PRAXICON), whose structure is corroborated by experimental findings on how the human brain works and a practical, hands on experience with the resource will be provided to the participants.

In particular:

In the first part of the tutorial, we will position semantic language resources within intelligent multimedia systems and cognitive systems, elaborating on their current and potential contribution and presenting the challenges one faces in employing them in cognitive robotics, cognitive vision, and other intelligent multimedia system applications.

In the second part, we will give an overview of state-of-the-art semantic language resources, ranging from computational semantic lexicons to common-sense knowledge-bases. We will provide a comparative view of 25 semantic language resources -in total- that will comprise:

  • profiling of the resources (developers, dates, languages involved, size, interfaces, links to other resources, applications)
  • methodology used for their development, and
  • contents: semantic relations covered (ranging from lexical semantic relations to conceptual relations such as temporal inclusion, cause, effect, goal, entailment), inclusion of facts or common sense assertions, instance vs. class distinctions, terms, domain, affect, word sense distinctions, figurative language coverage, links to Ontologies.

Furthermore, a number of semantic resources will be presented, in which verbal and non-verbal information is coupled for addressing the different challenges in Cognitive Systems research. These resources are multifaceted, ranging from extensions of known semantic language resources (e.g. the ImageNet resource which couples an image database with WordNet), to labelled image collections (e.g. the Pascal Images Database), to small scale labelled motion capture databases, multimedia ontologies, multisensory and multimedia corpora (e.g. the POETICON corpus). The presentation will focus on the integration of language and other modalities in such resources. In the third part of the tutorial, we will present the cognitive underpinnings of semantic resources, starting from Quillian's lexical semantic networks and the  underlying model on how semantic memory works, to state of the art theories and experimental findings on the structure and contents of semantic memory.  The neuroscience perspective will point to directions in developing semantic resources for cognitive agents, which has been materialized through the PRAXICON, a  multisensory semantic network. A live demonstration of the PRAXICON and a hands-on training session will conclude the tutorial.

Outline of the tutorial
Part I. Introduction to Cognitive Systems from a Language Perspective
  • From Intelligent Systems to Multimedia Systems, to Cognitive Systems
  • Applications and Needs
  • The role of Semantic Language Resources in Cognitive Systems
  • The Semantic Gap Problem
Part IIa. Profiling Semantic Language Resources from a Cognitive Perspective
  • Types (Semantic Lexica, Common Sense Knowledge Bases, Ontologies)
  • Methodologies used for their development
  • Contents: focus on semantic relations
  • Extension trends
  • Cross-Resource Interfacing trends
Part IIb. Verbal and Non-verbal Symbiosis in Semantic Resources
  • Types (Labelled Image collections, Labelled Motion Capture Databases, Multimedia Ontologies, Multisensory and Multimedia Corpora)
  • Methodologies used for their development
  • Contents: focus on semantic relations
  • Challenges for going beyond symbiosis to integration
Part III. The Cognitive Underpinnings of Semantic Resources
  • From Semantic Networks to Semantic Memory
  • How can Neuroscience inform semantic language and/or multimodal resource development?
  • A case study & hands-on exploration of a computational semantic memory for cognitive systems (a multimodal semantic network)
The presenters
Katerina Pastra (Cognitive Systems Research Institute & ILSP)
Dimitris Mavroeidis (Cognitive Systems Research Institute & ILSP)
The research leading to these results has received funding from the European Union Seventh Framework Programme under grant agreements POETICON (n° 215843) and POETICON++ (n° 288382).
Terms and Conditions, Privacy Policy

European Commission • Seventh Framework Programme