Multifunctionality in a reservoir computer: Chaos: An Interdisciplinary Journal of Nonlinear Science: Vol 31, No 1

Advancements in machine learning often arise from a “two-way street” between neuroscientific observation and mathematical representation. In this paper, we stroll through such a street with inspiration from “multifunctional neural networks.” These are networks of neurons whose activity patterns can change on the demand of performing a given duty, but synapses remain fixed. We conceptualize multifunctionality in the context of dynamical systems and machine learning by using a reservoir computer as a means to realize this neurological feat in an artificial setting. More specifically, we train a Reservoir Computer to imitate the dynamics of numerous chaotic attractors from different sources based on a given initial condition. To do this, we design a training technique that “blends” and weights data from these chaotic attractors. We explore how different weightings and changes in the memory of this artificial neural network effect the desired learning outcomes. In doing so, we uncover some “behind-the-scenes” bifurcations of several other attractors found to be lurking within the prediction state space that interfere with the network capacity to express multifunctionality. Above all, this paper identifies some new application areas suitable to a reservoir computer and broadens the current understanding of the dynamical capabilities inherent to this learning system.

Multifunctionality is an essential element of biological neural networks.1–3 These multifunctional networks are distinct pools of neurons capable of performing a multitude of mutually exclusive tasks. To elaborate with an example, it was found that a subset of the same bundle of neurons in the brain of the medicinal leech (Hirudo medicinalis) can switch their activity pattern once it senses a change in its surroundings to drive either a swimming or crawling motion.4 It is reported that a cluster of neurons located in the pre-Bötzinger complex (a region of the mammalian brain stem) is responsible for regulating a switching between different respiratory patterns.5 Depending on a particular input, the neurons in this region of the brain can alter their activity pattern accordingly to elicit a switching between eupneic (regular) breathing, sighing, or gasping. Furthermore, it is argued that multifunctionality in neural networks may naturally emerge from an efficient use of limited resources (in this case, neurons) and thus an evolutionary advantage in enduring environmental changes, reflecting the developmental history of certain organisms.6

Nevertheless, what is ubiquitous among these multifunctional neural networks is that they in principle resemble a system with more than one modus operandi. Based on a particular input, there is a distinct activity pattern expressed by the neurons in the network in order to perform one of the many tasks required of it. When needed, these multifunctional neurons switch to another activity pattern to collectively execute a different task while the network connections remain fixed. Therefore, if an artificial neural network was trained to sustain more than one desired activity pattern, it would in this sense be multifunctional. From a dynamical system perspective, this type of behavior is akin to a multistable system or a system with a coexistence of attractors. For further reading on multistable systems, see Pisarchik and Feudel.7

There is much to be gained in the pursuit of artificial intelligence by articulating our current knowledge of biological neural networks and dynamical systems in machine learning environments. In this paper, we employ such a bilateral rationale to encapsulate multifunctionality in an artificial neural network by training a “Reservoir Computer”8–10 (RC) to facilitate the coexistence of more than one desired activity pattern.

The basis of our research involves using a recent formulation of a continuous-time RC, presented by Lu et al.17 Here, the RC was trained to perform short term predictions of a chaotic Lorenz system18 and reconstruct the “climate” (qualitatively similar dynamical behavior) of its famous butterfly shaped chaotic attractor. Taking this result into account, we train a RC to promote a coexistence of reconstructed chaotic attractors in its prediction state space, thus becoming multifunctional. In order to demonstrate the flexibility of our approach, we consider training scenarios in which the climate of these chaotic attractors is reconstructed from a variety of sources. For example, we consider the case where these chaotic attractors are generated from two different systems entirely. We devise a training technique to “blend” data with a certain weighting parameter from these chaotic attractors. The choice of this weighting along with a parameter involved in tuning the memory of the RC is critical to achieving multifunctionality. We investigate the optimal setting of these parameters, from which we infer the regions in this parameter space where the RC achieves multifunctionality.

However, while we train the RC to realize more than one chaotic attractor in its prediction state space, we find several “untrained attractors” also residing here. These attractors inhabit the prediction state space but were not part of the training and limit the regions in which multifunctionality is obtained. A bifurcation analysis of these untrained attractors reveals some interesting dynamics where, for example, one of these attractors undergoes a period-doubling route to chaos.

The structure of the rest of the paper is as follows. In Sec. II, we provide details of the RC approach to attractor reconstruction and present the training procedure we use to achieve multifunctionality in a RC. Next, in Sec. III, the problem of epitomizing multifunctionality in a RC is further conceptualized. The trajectories on the chaotic attractors we use as our training data are also given here. In Sec. IV, we present our main findings and then provide an extended discussion of our results in Sec. V.