HTML
-
The construction of intelligent machines that mimic BNNs has been pursued since soon after the invention of the modern computer. Most of the research into ANNs is based on software simulation using von Neumann computers. The concept of mimicking BNNs with electronic or photonic hardware, which is also called neuromorphic computing, was introduced in late 1980s12, 24. Compared with the ANNs directly based on biological neural cells, these ANNs rely on electronic or photonic systems containing artificial neurons to indirectly mimic the neurobiological architectures presented in BNNs.
-
The biological counterparts of artificial neurons are biological neurons, which represent the building blocks of BNNs. The BNNs consist of billions of neurons of different types and sizes. Figure 2a shows a schematic of a simplified biological neuron25 with the four basic fundamental units—dendrites, axons, somas and synapses.
ANNs follow a simplified model inspired by BNNs. The unit of this model, a simple formalised artificial neuron introduced by McCulloch and Pitts26, acts as a computational element in the ANN. The execution of a task involves the parallel activation of a large number of artificial neurons. As shown in Fig. 2b, one or more inputs (x) from the other neurons were sent into an artificial neuron and the inputs are summed up to produce outputs (y) to the other neurons on axons. Separated by weighted (w), a non-linear function known as an activation function or transfer function (f); the activation function can be a step function, Sigmoid function, etc. The mathematical form of the artificial neuron is as follows:
$$ y_j = f\left( {\mathop {\sum}\nolimits_{i = 0}^n {w_{ij}x_i} } \right) $$ (1) The functions of the key components (axon, dendrite, soma and synapse) in ANNs are summarised in Table 1. The analogy between artificial neurons and biological neurons holds that the interconnections of the signals stand for the axons and dendrites, the summation and threshold approximate the activation in the soma, and the connection weights and memory represent the synapses.
Components Functions Photonic devices Axon and dendrite Interconnection Free space12, waveguides13-15 Soma Summation and thresholding Photodetectors12-15, electric-optic modulators14, light sources (LED and laser)12, optical amplifiers52, saturable absorbers52, optical bistable devices53 Synapse Weighting Hologram12, MZI15, micro-ring resonators14 Memory Electronic memories12-15, reversible optical memories54 Table 1. Summary of the key components of artificial neural networks
Artificial neurons can be organised in any topological architecture to form ANNs. A common architecture, the feed forward ANN27, is shown in Fig. 2c. In this network, the information moves only in the forward direction. A series of artificial neurons are integrated to form the layer of input. Within the ANNs, the input signal is passed through one or more hidden layers. At the end of this structure, the output layer provides the results. Different from the current von Neumann computer that performs tasks by the pre-design of the programme, the ANN can learn to run a task through a sequence of training with examples. For example, one of the typical learning processes is the supervised learning method by updating weights with the backpropagation errors between known target values and the output values28.
The most common unsupervised learning method is the spike timing-dependent plasticity (STDP) algorithm29, inspired from the spiking nature of BNNs. Recent neurological research has shown that biological neurons encode information in the timing of single spikes29. Functionally, STDP constitutes a mechanism for implementing a Hebbian learning rule30, allowing for the learning process without supervision.
-
The ANNs based on electronics were first demonstrated with very large-scale integration (VLSI) systems24. The electronic wires on the chips can be used as electronic axons and dendrites. The combination of several transistors in VLSI are used to approximate electronic synapses and somas. ANNs with a large amount of artificial neurons based on VLSI have been developed11. However, the integrated level of such a kind of ANN is still low owing to the complexity of the architecture. Memristors, the fourth basic element in electronic circuits predicted in 197131 and demonstrated in 200832, provide a new path to demonstrate ANNs with a high integration of electronic devices.
Memristors are passive two-terminal circuit elements, usually with a metal/insulator/metal structure33 (Fig. 3a). Mimicking a biological synapse, the resistance of a memristor can be adjusted by tuning input amplitude of the charge or flux. It has been demonstrated that the synaptic functions can be implemented experimentally with nanoscale silicon-based memristors34. The learning function has been demonstrated in a circuit composed of memristor synapses and complementary metal-oxide semiconductor neurons35. An artificial soma function has been demonstrated with a neuristor built using two nanoscale Mott memristors36. An ANN unsupervised learning for pattern classification based on memristors has recently been demonstrated recently37.
Nanophotonics offers a new solution to fabricate memristors by the laser reduction of graphene oxide. Graphene has received significant attention owing to its superior electronic and optical properties38. Direct laser writing (DLW) with laser reduction can realise the simultaneous direct growth and patterning of rGO. Various rGO electronic devices such as capacitors39, gas sensors40 and multifunctional devices41 have been demonstrated.
A memristor based on rGO has been experimentally demonstrated. DLW is used to realise rGO fabrication for use as the bottom electrode of the memristor on flexible substrates (Fig. 3b). An insulating layer of 10 nm HfOx is blanket deposited by thermal evaporation. The top electrode is made by curable silver paste or using thermal evaporation. The fabricated Ag/HfOx/rGO structure exhibits stable switching up to 100 cycles42. Metal-free memristors have been fabricated through the DLW process with rGO/g-C3N4-NSs/rGO thin films43.
-
Photons are an ideal information carrier because of their specific properties, such as inherently massive parallelism, fast propagation speed and no side effects of mutual interference. Optical signals can be multiplexed in time44, space45, polarisation45, 46, angular momentum47 and wavelength45, 48 domains, and optical technologies may overcome the problems inherent to electronics. Thus, the research and development into optical interconnect technology has already led to the replacement of copper connections in computer chips and data centres by optical waveguides or fibres49, which can potentially improve the performance of ANNs based on software simulation and electronics. An ANN based on reservoir computing50 has been demonstrated with temporal multiplexing in a single optical fibre. ANNs based on nanophotonics offer a promising alternative approach with a faster response and lower power consumption.
ANNs based on nanophotonics can be achieved by providing key optoelectronic or optical components for the functions of ANNs (Table 1). Nanophotonic technology has several advantages in making interconnections with free space and waveguides, specifically with regard to broad bandwidth, low-loss and low-crosstalk. Photonic somas can be achieved either by optical–electrical and electrical–optical signal transferring with photodetectors12, 14, 15, electro-optic modulators14 and light sources (LED and laser)51 or by an all-optical process with lasers51, optical amplifiers52, saturable absorbers52 and optical bistable devices53. The weighting function can be achieved by optical switches such as holograms12, Mach–Zehnder interferometers (MZIs)15 and MRRs14. The memory function to record the weights in a synapse can be achieved by electronic memories or non-volatile optical memories12, 54.
-
In ANNs based on holography, the full parallelism of light in free space can be used and additional functionalities become available. These strengths have been recognised for many years, and early implementations utilised reconfigurable holograms for forming interconnections between optoelectronic artificial neurons12. The architect of the ANNs based on holography with optoelectronic artificial neurons is shown in Fig. 4a. The light sources and photodetectors combine as optoelectronic neurons. The input signal is generated by the modulation of the light source. The holographic interconnection is achieved by optical diffraction with holographic gratings. Holography gratings can be produced by using spatial light modulators (SLMs)55 and the photorefractive effect12. The detectors integrate the optical signal and transfer it into an electronic signal. The threshold function is achieved by processing the electronic signal with electronic circuits. Although free-space optics provide the media for the highly integrated large number of free-space interconnections and data communication at the speed of light, the hologram provides an approach for storing and implementing the weight matrix data between fully connected neural layers.
Fig. 4 Architectures of artificial neural networks based on holography.
a An ANN based on holography and composed of optoelectronic neurons12. b An ANN based on holography and composed of multiple passive layers56. c An ANN based on holography and composed with multiple passive layers enabled by 3D printing56. d Areal and volume densities of the artificial synapses and neurons of ANNs based on holography with different pixel sizesRecently, a all-optical deep learning framework in the terahertz region (0.4 THz) has been demonstrated as an all-optical deep learning network, in which the ANN consists of multiple layers of a hologram fabricated by 3D additive printing56. The ANN is created by using several layers of a diffractive surface, where each point on a given layer represents an artificial neuron (Fig. 4b). The learning process of such an ANN is based on a computer simulation to design the phase modulation of each layer (Fig. 4c), which can be fabricated with a 3D printer and based on plastic materials. The ANN cannot be tuned after the fabrication process. Compared with previous ANNs based on holography composed of optoelectronic neurons, this kind of ANN is based on all passive elements.
The ANNs based on holography can be easily scaled up using various fabrication techniques including electron beam lithography and photolithography. ANNs based on holography provide the potential for a large number of artificial synapses and neurons owing to the high parallelism in free space. If we assume that the artificial neurons in two adjacent layers of an ANN are all arbitrary interconnected. The relationship between the density of artificial neurons (Dn) and the density of artificial synapse (Ds) is:
$$ D_n = \root {2} \of {{D_s}} $$ (2) The maximum number of artificial synapse via holography is equal to the maximum number of gratings that can be supported by the hologram12. From the sampling theorem, the number of sinusoids or gratings that can be recorded in the medium is equal to the number of samples. Considering the pixel size of a hologram is δ in all dimension (δ > λ), the maximum number of independent artificial synapse is V/δ3 in a volumetric hologram or A/δ2 in a thin hologram, where V and A are the volume and area of the hologram, respectively. As shown in Fig. 4d, for a commercial SLM with a pixel size of 4 µm, the areal density of synapses (Dsa) is 6.25 × 104/mm2. A recent advancement in holography induced by laser has demonstrated a hologram with a pixel size of 0.55 µm46, which corresponds to a synapse density of 3.31 × 106/mm2. As this hologram is induced by the tight focusing of laser light, it has potential to achieve a volume synapse density (Dsv) of 6.01 × 109/mm3 if a volume hologram can be induced by laser in three dimensions. The corresponding maximum areal neuron density (Dna) and volume neuron density (Dna) can be calculated by Eq. (2), as shown in Fig. 4d. A recent advancement in the SPIN technique, with a feature size of 9 nm23 and optical signal multiplexing45, 47, can further improve the density of synapses and neurons.
Apart from plastic materials and SLMs, ANNs based on holography can be achieved with different materials (Table 1). The 3D two-photon polymerisation of polymers23 and chalcogenide glass57 can be used for ANNs based on holography. Holography imaging has been demonstrated by DLW with the photoreduction of graphene oxide46. A hologram with a thickness of 60 nm using a topological insulator material has been demonstrated by DLW58. ANNs based on the materials above are write-once only owing to the non-reversible photoinduced effect, requiring learning with pre-design structures in computers.
Reversible holograms have been demonstrated with photorefractive polymers59 and photorefractive polymers synthetised with nanoparticles60. A hologram based on an ultrathin layer of phase change material Ge2Sb2Te5 (GST)61 is also demonstrated. These holograms can be erased and written by the DLW technique owing to the thermally reconfigurability of GST. These reversible holography effects open a new avenue for ANNs with a closed-loop learning capability, which allows direct updates of the weighting in ANNs after receiving feedback from the outputs.
-
Integrated photonic circuits on a chip are an ideal platform for ANNs with a high compactness and high stability. As the photonic axons and dendrites, the low-loss waveguides transmit the optical signal. The development of integrated lasers, photodetectors and optical non-linear devices can serve as photonic somas. The weights can be achieved in the optical domain by using reconfigurable optical switches or splitters, which utilise the temperature dependence of the refractive index (the thermo-optics effect) to realise reconfigurable functionality by heating the devices in most of current devices.
Interest in integrated lasers with neuron-like-spiking behaviour has flourished over the past several years51. Biological neurons use spikes to send information. Recently, neurological research has discovered that information in biological neurons are encoded in the temporal domain of single spikes29. Lasers operating in the excitable regime are perfectly dynamic candidates but are ~ 8 orders-of magnitude faster. However, most experimental work has focused on isolated neurons or single chain of neurons (Fig. 5a).
Fig. 5 Artificial neural networks based on photonic circuits on a chip.
a Schematic representation of the basic operation of an excitatory and inhibition neuron with an integrated semiconductor laser on a photonic circuit51. b Schematic drawing of artificial synapses62. c The architecture of an artificial neural network based on photonic circuits15. d Integrated phase change memories on a photonic circuit as artificial synapses62Isolated artificial synapses on photonic circuits have been demonstrated by integrating a phase change memory (PCM) on an optical waveguide62 (Fig. 5b). As shown in Fig. 5b, the synapse is based on a tapered waveguide with PCM regions on top. The synaptic weight can be controlled by changing the number of optical pulses sent through the waveguide.
In parallel with the work on isolated artificial neurons and synapses, recent research has also investigated the capability of photonic circuits to mimic neural networks, focusing on the networks with connectivity to multiple neurons and layers. An ANN scheme based on silicon waveguide interconnections has been demonstrated by using a reconfigurable MZI to control the weights between the interconnections of neurons (Fig. 5c), which are laser and photodetectors15. Recently, a "broadcast-and-weight" scheme has been demonstrated to use wavelength-division multiplexing (WDM) to support a large number of reconfigurable interconnections enabled by micro-ring resonators (MRRs) on a silicon photonic chip14. A 294-fold acceleration against a conventional von Neumann computer is predicted.
The size of the reconfigurable MRRs or MZIs is from 625 to 20, 000 µm2, resulting in a synapse density of 50–1600/mm2. The neuron density is at the same level and is based on the size of the integrated lasers and photodetectors on the chip. As mentioned in the previous paragraphs, current ANNs utilise the thermo-optics effect in reconfigurable MZIs or MRRs to realise artificial synapse functionality by heating the devices, resulting in an additional power consumption of 10 mW per synapse to maintain the reconfigurable MRRs or MZIs. However, the weights can be set by integrated on-chip non-volatile memory, such as a PCM, which requires no power to maintain.
-
The ANNs based on photonics have a significantly higher rate and lower computation energy compared to those of their electronic counter parts. ANNs based on holography can achieve an ultrahigh density of synapses owing to the high parallel processing in free space. Nonetheless, current ANNs based on photonic circuits cannot compete with the density of electronics. However, novel implementations of ANNs based on nanoscale optical data storage63, photonic crystal nanocavity64 and plasmonic nanocavity65 technologies may become important to the future scaling down of the artificial neuron and synapses on a chip to the diffraction limit scale (~ 1 µm) or even beyond the diffraction limit. Three-dimensional photonic integration enabled by DLW could enable ANNs with larger number of artificial neurons and synapses by adding another spatial degree of freedom. Furthermore, the feeding of input signals through temporal multiplexing would be able to realise ANNs with larger number of artificial neurons and synapses effectively.
Apart from the demonstration of single lasers or laser chains, the coding of ANNs based on photonics has been demonstrated using analogue optical signals. ANNs based on photonics can be coded with the spiking method to further improve the signal robustness and enable unsupervised learning. Current ANNs based on photonics are derived from simplified artificial neuron models. The performance of ANNs can be further improved with a better understanding of the working mechanism of BNNs.
Working principles of indirect ANNs
Laser-written electronic memristors for ANNs based on electronics
Introduction of ANNs based on photonics
ANNs based on holography in free space
ANNs based on photonic circuits on a chip
Future perspectives
-
Unlike indirect ANNs based on electronics and photonics with input and output connections for signal detections, the spiking of biological neurons are the signals of direct ANNs. The detection of such weak electric signals at the nanoscale is of critical importance to studying neural activities. Novel devices that are ultrasensitive to weak electric fields, such as micro-electrode arrays and patch clamps, have been introduced to study neural activities. However, these devices have a low spatial resolution and are invasive, which might cause damage to the biological neurons. Nanophotonics, especially nanosensing enabled by nitrogen-vacancy (NV−) centres in bulk diamonds and NDs, has opened the possibility of the far-field detection of neural activities with nanoscale resolution.
-
The electrical potential associated with the passage of a spiking impulse along neurons is called action potential (AP). APs play a central role in the communication between individual neurons. The signals between neurons propagate along axons via APs. Different techniques can be implemented for the measurement of APs. Electrophysiology is a recording method with a patch-clamping configuration that remains the gold standard for the measurement of individual APs. The technique has an excellent temporal resolution and good signal-to-noise ratio, but the spatial resolution is limited to ~ 10 µm.
Nanophotonic techniques to measure APs offer many advantages, but they typically require high power, which can cause photodamage to neurons. In addition, voltage-sensitive fluorescent proteins must be genetically expressed, which may alter the neuronal functions. Owing to their optical and magnetic properties, NV− centres can be applied to study neural networks and the firing of neural cells to study brain activity97.
NV− centres have attracted significant attention in the recent years owing to their outstanding optical and magnetic properties98, 99. The ground state of an NV− centre is a triplet state with six unpaired electron spins, each one with an associated magnetic moment; the spin sublevels are ms = ±1, 0100. The energy sublevels ms = ±1 in the ground state can be perturbed with an external magnetic field. This paramagnetism of the ground state can be mathematically described by the spin Hamiltonian of the system100:
$$ H = D_{gs}S_z^2 + E_{gs}\left( {S_x^2 - S_y^2} \right) + g_{gs}\mu \overrightarrow B \cdot \overrightarrow S $$ (3) where Dgs is the ground state zero-field splitting, $ \vec S$ is the spin operator vector, Egs is the ground state strain-induced splitting coefficient, ggs the ground state g-factor and µ is the Bohr magneton101. In brief, the first term corresponds to the energy gap between the two sublevels in the ground state, the second term represents the energy shift owing to the strain from the lattice, and the last term is the Zeeman effect102, 103.
When NDs are stimulated by a microwave (mw) field, it is possible to redistribute the population of ground state electrons in NV− centres. In particular, by switching on an mw field at the zero-field splitting frequency, D = 2.87 GHz, there is a redistribution of the electrons from the most populated sublevel, ms = 0, to the less-populated sublevel, ms = ±1.
Under an excitation beam at wavelength 532 nm, an NV− centre is polarised in its ground state ms = 0. The application of an mw field at the zero-field splitting frequency causes an increase in the population of the ms = ±1 spin sublevels. This leads to an overall decrease in the fluorescence because of the non-radiative decay via the intermediate metastable state. The observed photoluminescence decrease is called the optically detected magnetic resonance (ODMR) signal. It is also possible to apply a magnetic field along the z axis that degenerates the energy sublevels ms = ±1.
In case of the weak-magnetic field regime, the magnetic field is calculated with the following104:
$$ B_{NV} = \frac{{\omega _2 - \omega _1}}{{2g_{gs}\mu }} $$ (4) Under this circumstance, the recorded ODMR signal shows two dips owing to the redistribution of the electrons in the sublevels ms = −1 and ms = +1. The separation between the recorded ODMR dips increases with the strength of the applied magnetic field.
In fact, APs are time-varying electrical fields, which lead to time-varying magnetic fields. The main principle behind sensing APs with NV− centres is the detection of time-varying magnetic fields generated by APs via ODMR signals. The measurement of the AP via magnetic field sensing confers important advantages: it is non-invasive, label-free and able to detect neuronal activity through intervening tissue and whole organisms.
In the first work of applying NV− centres to measure the axon transmembrane potential, the magnetic field generated by a single axon potential is modelled, magnetic field generated by a single axon potential is modelled and the magnetic field has been generated by a microwire on the surface of a diamond substrate, which simulates the AP of a morphologically reconstructed hippocampal CA1 pyramidal neuron. The detection system is composed of a commercial grade single crystal ultra-pure diamond substrate with a layer NV− defect centres with a standoff of 100 nm. The detection system is able to image planar neuron dynamics non-invasively with temporal resolution at millisecond and spatial resolution at micron spatial resolution (10 µmTHz−1/2]) within wide-field view105. NV− centres in bulk diamonds have been implemented to measure the APs of a giant axon in an invertebrate, Myxicola infundibulum97.
The bipolar azimuthal magnetic field associated with an AP is depicted in Fig. 7a, in the inset is the energy level of the NV− centres. A light beam at a wavelength of 532 nm is applied to the sensing NV− centre layer through the diamond at a sufficiently shallow angle so that the light reflects off the top diamond surface and therefore does not irradiate the living sample (Fig. 7b)97. The magnetic field from the AP measured via electrophysiology can be mathematically expressed as97:
$$ \phi _{in}(t):B(t) = {\int} {sd\phi _{in}dt} $$ (5) Fig. 7 Experimental overview of the measurement of action potential (AP) with NV centres97 and detection and location of nanodiamonds (NDs) in ECS cells107.
a A schematic of the bipolar azimuthal magnetic field associated with a giant axon. Inset: the energy-level diagram of an NV− centre. b A schematic of the microscope implemented for the magnetic measurements. c Measured AP voltage with the electrophysiology measurement. d Calculated magnetic field. e Measured magnetic field with NV− centres. f ECS cells treated with NDs. In red is the fluorescence intensity of NDs excited with light at a wavelength of 580 nm. The emission is collected in the wavelength range of 600-700 nm. In blue is the nucleus of the cell, and in yellow, the SSEA-1. g Differentiated neural cells. Fluorescent NDs are observed in the cytoplasm of undifferentiated ECS and differentiated cellsThe measured AP voltage and the calculated magnetic field are illustrated in Fig. 7c–e. The measurements with NV− centres are consistent with the prediction97.
With NV− centres, it is also possible to measure the AP propagation direction at a single point97. The results lay the groundwork for the real time, non-invasive 3D magnetic mapping of functional neuronal networks, ultimately with a circuit-scale (~ 1 cm) field-of-view.
-
An understanding of biological processes involves the tracking of several cells to study their differentiation and development. Owing to their photostability, their robustness against photobleaching and their biocompatibility, NDs are ideal candidates for long-term cell tracking106. A study has been reported on the tracking of neuron cells derived from a model of embryonal carcinoma stem cells with fluorescent NDs107. The NDs are detected with a confocal microscope (Fig. 7f). No effect on the morphological development of the cells is recorded nor do the NDs induce apoptosis during the neuronal differentiation (Fig. 7g). Therefore, the implementation of fluorescent NDs to track neural cell development could provide potential therapeutics for neural diseases107.
-
Studies on the detection of APs by measuring the ODMR signal with NV− centres in bulk diamonds have been reported97. There are several technical challenges with this technique. First, the magnetic field sensitivity needs to be improved to enable AP measurement from individual mammalian neurons97. Second, single-point detection needs to be developed for the 2D and 3D mapping of APs in neuronal networks. Third, the high-resolution imaging of AP magnetic fields is also required.
NDs can provide a new solution to the three challenges listed above. NDs has a wide range of applications in the life sciences in the life sciences compared with bulk diamonds108. Currently, NV− centres in NDs have been applied to study the formation and patterning of neural networks and to label and track the neuron cells75, 107. First, a higher density of NV− centres can yield a sensitivity improvement. NDs provide a substantially higher density of NV− centres than does bulk diamond. Some types of NDs contain ~ 1000 NV− centres in one ND with a diameter of 5 nm. Second, NDs enable 3D magnetic imaging. The magnetic imaging based on diamond layers can only achieve 2D imaging because the neuron networks can only be fixed on the surface of the diamond layers. However, NDs can be labelled everywhere inside the neuron networks, which makes it possible to obtain 3D magnetic images. Third, a higher spatial resolution can be achieved with NDs. Smaller NDs in biomedical imaging are highly required, and the blinking phenomenon is more prevalent when reducing the size of a ND is reduced109, 110, 111. Super-resolution imaging based on blinking fluorescence would allow the nanometric reconstruction of synapse connections by labelling the neuron cells with 5 nm oxide NDs109, 111.
The employment of the ODMR signal to interpret APs via magnetic field measurements could lay new groundwork for the development of nanoscale biomarkers for both the super-resolution optical imaging and magnetic sensing of ANNs.