Intrinsically motivated reinforcement learning, life-long learning, autonomous development of intelligence in self-developping devices

Together with Frederic Kaplan, I am the co-inventor of 2 patents which describe a machine learning technology related to the capability of a learning machine to select autonomously its training examples and self-develop repertoires of skill through life-long learning. A part of this technology has been integrated in the Sony Qrio humanoid robot:

1. Automated Action-Selection System and Method , and Application Thereof to Training Prediction Machines and Driving the Development of Self-Developing Devices (27 july 2004)
   Abstract: In order to promote efficient learning of relationships inherent in a system or setup S described by system-state and context parameters, the next action to take, affecting the setup, is determined based on the knowledge gain expected to result from this action. Knowledge-gain is assessed “locally” by comparing the value of a knowledge-indicator parameter after the action with the value of this indicator on one or more previous occasions when the system-state/context parameter(s) and action variable(s)=had similar values to the current ones. Preferably the “level of knowledge” is assessed based on the accuracy of predictions made by a prediction module. This technique can be applied to train a prediction machine by causing it to participate in the selection of a sequence of actions. This technique can also be applied for managing development of a self-developing device or system, the self-developing device or system performing a sequence of actions selected according to the action-selection technique.
2. Architecture for self-developing devices (6th june 2003)
   Abstract: The present invention relates to an architecture for self-developing devices. More particularly, the invention relates to self-developing devices adapted so as to be capable of continuously developing new know-how (this is sometimes referred to as the capacity to engage in “lifelong learning”). A self-developing device (1) capable of open-ended development makes use of a special motivational system for selecting which action should be taken on the environment by an associated sensory-motor apparatus (2). For a given candidate action, a motivational module (11) calculates a reward associated with the corresponding values that would be taken by one or more motivational variables that are independent of the nature of the associated sensory-motor apparatus. Preferred motivational variables are dependent on the developmental history of the device (1), and include variables quantifying the predictability, familiarity and stability of sensory-motor variables serving as the inputs to the device (1). The sensory-motor variables represent the status of the external environment and/or the internal resources (3) of the sensory-motor apparatus (2) whose behaviour is controlled by the self-developing device (1). Open-ended development is enabled by attributing a reward which is proportional to the rate of change of the history-dependent motivational variables.

Emotional speech synthesis and recognition

I am the inventor of several worldwide patents describing various aspects of technologies for synthesis and recognition of emotional speech with/by computers. This technology has been integrated into the Qrio robot, and into Sony Playstation video games.

1. Method and apparatus for synthesizing an emotion conveyed on a sound, 13th july 2001
   Abstract: An emotion conveyed on a sound is synthesised by selectively modifying elementary sound portions (S) thereof prior to delivering the sound through an operator application step (S10, S16, S20) in which at least one operator (OP, OD; OI) is selectively applied to the elementary sound portions (S) to impose a specific modification in a characteristic, such as pitch and/or duration in accordance with an emotion to be synthesised.
   This can be achieved by applying at least one operator (OIrs, OIfs, OIsu, OIsd) to modify an intensity characteristic of an elementary sound portion, and/or an operator (OPrs, OPfs) for selectively causing the time evolution of the pitch of an elementary sound portion (S) to rise or fall according to an imposed slope characteristic (Prs, Pfs), and/or an operator (OPsu, OPsd) for selectively causing the time evolution of the pitch of an elementary sound portion (S) to rise or fall uniformly by a determined value (Psu, Psd), and/or an operator (ODd, ODc) for selectively causing the duration (t1) of an elementary sound portion (S) to increase or decrease by a determined value (D).
2. Method and apparatus for controlling the operation of an emotion synthesizing device, 14th august 2001
   Abstract: The invention controls the operation of an emotion synthesising device (12), notably of the type where the emotion is conveyed on a sound, having at least one input parameter (Pi) whose value (Ei) is used to set a type of emotion to be conveyed, by making at least one parameter a variable parameter (VPi) over a determined control range, thereby to confer a variability in an amount of the type of emotion to be conveyed. The variable parameter (VPi) can be made variable according to a variation model over the control range, the model relating a quantity of emotion control variable (δ) to the variable parameter, whereby said control variable is used to variably establish a value of said variable parameter.
   Preferably the variation obeys a linear model, the variable parameter being made to vary linearly with a variation in a quantity of emotion control variable (δ).
3. Emotion recognition method and device, 13th july 2001
   Abstract: Emotion recognition is performed by extracting a set comprising at least one feature derived from a signal, and processing the set of extracted feature(s) to detect an emotion therefrom. The voice signal is low pass filtered prior to extracting therefrom at least one feature of the set. The cut-off frequency for the low pass filtering is typically centered around 250 Hz. The features are e.g. statistical quantities extracted from sampling a signal of the intensity or pitch of the voice signal.
4. Method and apparatus for voice synthesis and robot apparatus, 11th may 2001
   Abstract: A robot apparatus (1) is capable of audibly expressing an emotion in a manner similar to that performed by a living animal. The robot apparatus (1) utters a sentence by means of voice synthesis by performing a process including the steps of: an emotional state discrimination step (S1) for discriminating an emotional state of an emotion model (73); a sentence output step (S2) for outputting a sentence representing a content to be uttered in the form of a voice; a parameter control step (S3) for controlling a parameter for use in voice synthesis, depending upon the emotional state discriminated in the emotional state discrimination step (S1); and a voice synthesis step (S4) for inputting, to a voice synthesis unit, the sentence output in the sentence output step (S2) and synthesizing a voice in accordance with the controlled parameter.
5. Sound processing method and device for modifying a sound characteristic, such as an impression of age associated to a voice, 14th august 2001
   Abstract: The invention allows to controllably modify a value of a characteristic, such as an age, associated to a sound contained in sound data (30) such as voice to which is associated a sampling frequency, the sound exhibiting a given value (A) of the characteristic for a first sampling frequency F1. This is achieved by :
   • producing from the sound data at least one sound data portion (UD) containing a sound portion to be reproduced with a modified value of the characteristic, the sound data portion having a first sampling duration R1 at the first sampling frequency F1,
   • establishing (36, 44-48) a second frequency F2 selected as a function of a desired modification in the value of the characteristic,
   • time adjusting (38) the sound data portion (UD) by elaborating a time-adjusted form (UDta) thereof on the basis of a sampling frequency equal to the second frequency F2, the time-adjusting step at least partially tending to cause the sampling duration of the time-adjusted form of sound data portion at the second frequency F2 to match the first sampling duration R1, and
   • playing the time adjusted sound data portion (UDta).
   Preferably, the time-adjusting step is performed by means of a Pitch-Synchronous Overlap and Add (PSOLA) algorithm (38) or the like.
6. Method and apparatus for speech synthesis program, recording medium, method and apparatus for generating constraint information and robot apparatus, 15 march 2002 (together with  Makoto Akabane, Erika Kobayashi, Kenichiro Kobayashi, Toshiyuki Kumakura, Tomoaki Nitta, Nobuhide Yamazaki)
   Abstract: The emotion is to be added to the synthesized speech as the prosodic feature of the language is maintained. In a speech synthesis device 200, a language processor 201 generates a string of pronunciation marks from the text, and a prosodic data generating unit 202 creates prosodic data, expressing the time duration, pitch, sound volume or the like parameters of phonemes, based on the string of pronunciation marks. A constraint information generating unit 203 is fed with the prosodic data and with the string of pronunciation marks to generate the constraint information which limits the changes in the parameters to add the so generated constraint information to the prosodic data. A emotion filter 204, fed with the prosodic data, to which has been added the constraint information, changes the parameters of the prosodic data, within the constraint, responsive to the feeling state information, imparted to it. A waveform generating unit 205 synthesizes the speech waveform based on the prosodic data the parameters of which have been changed.