My New Research Plan (2013-09)

I changed the plan.
There are two major changes.

• I would use simulator software rather than a real robot.
• I would focus on language acquisition rather than using conventional NLP.

---

Purpose

Research on language acquisition (symbol grounding) with a robot simulator.

Language Acquisition

Research will follow the human (infant) language acquisition process.  More concretely speaking, the simulated robot shall learn labels on the properties, movements and relations of objects, based on the cognition of bundles of coherent and continuous properties (Spelke's objects). It shall utter with learned labels. Moreover, it shall create adequate internal semantic representations and relate them with corresponding syntactic structures.
(See below for more details.)

Core cognitive architecture

The core cognitive architecture shall have the following functions.
Upon designing and implementing the cognitive architecture, generic mechanisms shall be (re)used whenever possible.

• Time-series pattern learner/recognizer
  having motor commands and sensory input as their input
• Attention and situation assessment
  on what will be learned and which action will be taken.
  It is part of the executive control function.
• Cognitive model based on association
  to memorize and recollect (temporal) generative patterns as associative sequences.
  Linguistic competence will be realized with this model.
  It contains backtracking mechanism based on the function of attention and situation assessment mentioned above.
• Episodic memory
  Patterns (the representation of situations -- combinations of abstract patterns created by (non-supervised) learning) positively assessed by the attention and situation assessment will be memorized.

 cf. A Figure of Human Cognitive Function

Platform

• Robot simulator
  Sigverse, etc.
• Visual data processing
  OpenCV, etc.
• Speech recognition/synthesis: option
  (Research can be carried out on the text basis.)
• Learning module
  SOINN, k-means, SOM, SVN, DeSTIN, HMM, etc.
  (To be used as plug-ins depending on the purpose)

A Tentative Research Steps

Phase I: Robot & World Setting

• Basic Ideas
• Locomotion: Rover［with wheels］ or Swimmer［like fish］
• Vision: chromatic, binocular & without saccade
• No manipulator
  Still the robot can interact with objects by physical contact.
  The robot may have a 'mouth-like' manipulator afterwards.
• The robot acts autonomously/spontaneously.Spontaneity may be motivated by the motion itself, attitude (posture) control, learning new motions, learning new objects, etc.  Learning by spontaneous acts can be said to be learning by 'playing' or curiosity.
• Designing and implementing on drives related to learning will not be done in Phase I.
• Simulator design & implementation
  Spontaneous actions should be designed elaboratively.

an example fish robot

Phase II: Recognizing Spelke's Objects

• Basic Ideas
• Spelke's Object: coherent, solid & inert bundle of features of a certain dimension that continues over time.
  Features: colors, shapes (jagginess), texture, visual depth, etc.
• While recognition of Spelke's objects may be preprogrammed, recognized objects become objects of categorization by means of non-supervised learning.  In this process, hierarchical (deep) learning would be done from the categorization of primitive features to the re-categorization of categorized patterns.
• Object recognition will be carried out within spontaneous actions of the robot.
• The robot shall gather information preferentially on 'novel' objects (curiosity-driven behavior) ('novelty' to be defined).
• Determining the recognition method & implementation
• Recognition experiments
• Determining (non-supervised) learning methods  & implementation
• Experiments on object categorization
• Designing novelty-driven recognition & behavior & implementation

Phase III: Labeling

• Basic Ideas
• The robot shall relate specific types of Spelke's objects it looks at with linguistic labels.
• Labels may be nominals representing shapes and adjectives representing features such as colors.
  Types of objects may be learned in a supervised manner with labels or have been categorized by non-supervised learning.
• The robot shall utter labels on recognizing types after learning association between the labels and types.
• The robot shall recognize/utter the syntactic pattern 'adjective + noun'.
• Determining the recognition method & implementation
• Designing and implementing mechanism for handling syntactic structure.
• Labeling experiment

Phase IV: Relation Learning

• Basic Ideas
• The robot shall learn labels for
• object locomotion such as moving up/down, right/left and closer/away
• orientational relations between objects such as above/below, right/left and short/thither
• Objects should be get the robot's attention by force (programming) or by certain preprogrammed mechanism of attention (such as attention to moving objects). 
• Designing & implementing labeling mechanism
• Experiments

Phase V: Linguistic Interaction

• Basic Ideas
• The robot shall answer to questions using labels learned in Phase III & Phase IV.
• The robot shall respond to requests on its behavior.
• The robot shall utter clarification questions.
• Designing & implementing mechanism for linguistic interaction
• Experiments

Phase VI: Episodic memory

• Basic Ideas
• Episodes (situations) to be memorized are the appearance of objects and changes in relations among them.
• The memory of novel objects and situations is prioritized.
• Designing & implementing episodic memory and attentional mechanism
• Designing & implementing episodic recollection & linguistic report.
• Experiments

Phase VII: More complicated syntax

• Basic Idea
  The robot shall understand/utter linguistic expressions having nested phrase structure.
• Designing & implementing nested phrase structure.
• Experiments