From sensor fusion to self-organized control The software developed in the Replicator project, has to be seen in a large framework in which the ultimate goal is to reach levels of self-organized control, or what we also call meso-morphosis. This is the process of internal reconfiguration that has to occur alongside metamorphosis. A caterpillar does not just change from the outside into a butterfly, it is also changing from the inside. Certain central pattern generators can be reused in later stages, perhaps even olfactory or visual information is retained so that it's possible through this way to find the same kind of plant on which a butterfly was raised, to deposit her eggs.
The scientific challenge for sensor fusion and self-organized control: A reconfigurable robot should be able to follow a human as a snake and overcome door sills by transforming into a spider. The sensor data processing architecture should be such that this actually can be done in an agile way. This means, amongst others, turning off sensors that are unreliable or redundant. Neatly summarized: it means mesomorphic sensor fusion capable to run on reconfigurable robots.
The figure above shows how we might come to meso-morphosis (emergent self-organized control takes so long to write all the time) on the robots and how cognitive sensor fusion plays a role in this game:
At the total left you see maps that describe the sensory input in a 2D grid representation. Such representation can be formed for different modalities. A retinal map for vision, a cochleogram for sound. And, if, necessary, it might also be extended to even different modalities. Currently the focus is on those two modalities, because other partners within the Replicator project will profit most of preprocessing on this type of very complex sensory input. A distance sensor for example, can be used without a lot of preprocessing. The saliency model of Itti and Koch is implemented to fuse those modalities in a computational cheap and especially in a scalable way. (The resolution of the maps is easy to adjust.) The bottom-up saliency approach is especially feasible to implement later on top-down saliency mechanisms.
Below the time line it is shown that a developmental engine is built that creates via a gene regulatory network a topology of modules. In the context of sensor fusion those modules are filters that process sensor data. It hence enables the evolution of different topologies for a sensor fusion processing architecture. The developmental engine, called Linda, is very small and written in C and will be able to run on a Cortex M3 microcontroller. It might be possible to swap it later on in the project with one of the developmental engines delivered by other partners in the Replicator project. So, it is a developmental approach to a sensor fusion filter architecture. So, what's possible with such a developmental approach? It means that if a snake form is created, a set of sensory filters will be selected that best corresponds to this body form. For example, cameras might be placed on the head and the tail of the snake and the filter architecture might be streamlined throughout the snake body.
Back to above the time line. Liquid state machines [LSMs] can be used for sparse coding. It maps the sensory input data into a higher dimensional space, so that it subsequently can be linearly separated. LSMs or the simpler echo state networks, will be used to process sensory data coming from the line-edge detection, color detection, intensity detection filters to fed it into an associative memory structure. This type of memory stores associations, even across modalities. This type of machine learning techniques might make the difference between a slow robot and an agile, fast one.
For current implementations of this type of associative memory that also implement cognitive functionalities are adaptive resonance theory [ART] and global workspace theory [GWT]. Cognitive shouldn't be seen as a magic term over here. It refers to having attention mechanisms, and competition between processes as a form of "controller selection" or "action selection". Without a cognitive context, a robot will start looking for a bird in a tree, rather than focus on things that are actually important.
This type of cognitive architectures like ART or GWT have never been evolved before. If the developmental engine develops ART or GWT components, for different robot morphologies different layouts can be tried out. Concrete, this means that a spider form might rely more on its short-range sensors because it needs to overcome obstacles. A snake form might use longer-range sensors so it needs longer term memory and a longer attention span.
Related to the severe demands of a reconfigurable robots a static layout will most probably restricting the robot unnecessarily. A basic channel implementation has already been implemented in the form of SymbricatorRTOS hooks. This enables tasks to be connected to other tasks. A dynamic version of this channel architecture across the entire organisms. Reo is a formal description of such reconfigurable channels and an implementation of this type of formalism will be demanded from the middleware partners or has to be implemented by us. The reconfigurable middleware will enable the robot to actually morph internally as well as externally. Without it, we will only have spider robots with snake brains.
The marriage of a reconfigurable architecture and an evolvable cognitive architecture is subsequently visualized as emergent self-organized control [SOC] at the total right. At the bottom it is shown that this type of robots is able to reorganize their internal structure as soon as they dock. This can be seen very mundane as one robot turning off its laser scanner after docking and the other one turning off its camera. Although stated this simple, the cognitive architecture below it actually decides when this type of reconfigurations happen. Without this type of architecture a system will quickly loose track of what to turn off and what to turn on on which moments. Even less if a robot organism has to be able to perform different tasks, like searching for an energy socket versus crossing a door sill. Above the time line it shows that input from the environment and not just the robot morphology might guide this reconfiguration process. If it happens to be dark all the time, it might be convenient to just turn all cameras off... If the cameras on the spider legs move all the time, it will probably use the ones on the spider body.
Sensor fusion The following movie shows how a robot in the Symbricator3D simulator learns to drive its right and left wheels in the appropriate proportions, so that it correctly points at the salient object: a robot. Each time the robot is set in a neutral position. It then tries to center the most salient entity in the middle of its visual perceptive field. The weights that drive the right and left wheels are subsequently updated by reinforcement learning. So, this is sensor fusion in which multiple visual sub-modalities (orientation, intensity) are fused.
The robot that has learned to find salient objects by the method above can be used to find another robot in the simulator. For specific tasks, like this, it is important that not all (sub)modalities do get the same priority. The horizon for example is very visible for the filters that respond to horizontal lines. Hence, in this scenario this specific filter is turned off. This shows the need to be able to select an adaptive set of visual filters in different circumstances. This is one of the intended goals of self-organized control: a dynamic set of filters depending on the environment in which the robot acts.
For this self-organized approach towards sensory integration or in more general to a dynamic set of modules on a robot, there is an evolutionary and even developmental engine built, which is called the Linda engine. It communicates with the Symbricator3D simulator over TCP/IP. And in the meanwhile convenient helper tools are developed like: an m-bus that is able to multiplex TCP/IP connections to multiple processes across several machines; an abbey, which implements a thread pool with some additional agent-like properties e.g. defining dependencies in execution of tasks; and a GUI that is able to visualize which modules are turned on and off on the robot.
The software and movies are created by Ted Schmidt, Govert Buis and Anne van Rossum at Almende B.V. in the European FP7 project Replicator. Software is open-source and freely available at https://svn.almende.com/svn/replicator. The Symbricator3D software is built on Delta3D, and hence will be open-source too, somewhere in the future.
A run of the developmental engine. As can be seen some of the genomes are "neutral", they don't lead to cellular instructions and leave the initial topology just as it was with one I and one O module. Other ones are very active and create a diverse set of modules all over the grid.
The software and movie is created by Anne van Rossum at Almende B.V. in the European FP7 project Replicator. Software is open-source and freely available at https://svn.almende.com/svn/replicator.
Concentration flow of a gene product over a spatial grid
The concentration of one of the gene products (of in total around 30) in each cell of a spatial grid is written by gnuplot to a file, and later concatenated to a movie. It shows how the products flow from one cell to the other. In every cell where it crosses some threshold it causes a module (in this case a neuron) to undergo a morphogenetic instruction. Gene product "3" might code for a movement of the module to the north. Using this developmental paradigm it is possible to build a network of modules where the topology is not defined beforehand. The developmental engine used over here is an adapted version of the gene regulatory network by Josh Bongard.
The software and movie is created by Anne van Rossum at Almende B.V. in the European FP7 project Replicator. Software is open-source and freely available at https://svn.almende.com/svn/replicator.
Actually those movies are created later than the movie of the 9th of June. It shows a cochleogram representation in parallel with the retinal maps. The visual saliency maps are already used to drive the motors on both sides of the robot. So, vision is considered as the dominant modality. Sound intensity is used as a trigger to turn right versus left. As soon as the robot comes into view, the visual modality takes precedence.
Cochleogram representation and acoustic saliency maps in parallel with retinal and visual saliency maps. The acoustic saliency maps are not "easy to read" because frames from left and right sides are shown interleaved. It is just to show that acoustic saliency maps might be considered similar to visual saliency.
The software and movie is created by Ted Schmidt at Almende B.V. in the European FP7 project Replicator. Software is open-source and freely available in the near future...
Kennislink Kennislink is a Dutch forum for everybody, and especially students at secondary school, and a bit beyond, to become familiar with a diverse range of topics in science, You will find two articles written by me, together with Stev at:
[ Replicator ]
