# Storage Capacities of Twin-Multistate Quaternion Hopfield Neural Networks.

1. IntroductionA complex-valued Hopfield neural network (CHNN) is a multistate model of Hopfield neural network. CHNNs can deal with multilevel information and have been applied to the storage of image data [1-7]. They have also been extended using Clifford algebra, which includes a complex field, hyperbolic algebra, and quaternion field. Several models of hyperbolic Hopfield neural network have been proposed [8-11]. Isokawa et al. [12] proposed quaternion Hopfield neural networks (QHNNs) employing the split activation function. Minemoto et al. [13] studied QHNNs using the polar-represented activation function. Several other models of QHNNs have also been proposed [14-17]. In this work, we study the twin-multistate quaternion Hopfield neural networks (TMQHNNs) [18]. The neuron of a TMQHNN consists of a pair of complex-valued multistate neurons. The TMQHNN requires only half the connection weight parameters of the CHNN.

Storage capacity is an important issue in Hopfield neural networks. When a Hopfield neural network is given a training pattern, the weighted sum input is decomposed into main and crosstalk terms. The main term enables the Hopfield neural network to memorize the training patterns. The crosstalk term interferes with the storage of training patterns. The storage capacity of conventional Hopfield neural network has been investigated by evaluating the crosstalk term [19]. Jankowski et al. [1] and Kobayashi [20] applied this technique to CHNNs and rotor Hopfield neural networks (RHNNs), respectively. The RHNN is an extension of the CHNN using vectors and matrices [21]. In this work, we provide the Hebbian learning rule for TMQHNNs and evaluate the storage capacity based on Jankowski's concept. In the case of TMQHNNs, the cross talk term is decomposed into two complex parts. By evaluating both parts, we determine the storage capacity of TMQHNNs. The theory suggests that TMQHNNs have half the storage capacities of CHNNs. In addition, we compared the storage capacities of CHNNs and TMQHNNs by computer simulation.

The rest of this paper is organized as follows: Sections 2 and 3 introduce CHNNs and TMQHNNs, respectively. Section 4 provides the Hebbian learning rule for the TMQHNN and evaluates the storage capacity. It also contains descriptions of the computer simulations conducted to verify our analysis. Section 5 concludes this paper.

2. Complex-Valued Hopfield Neural Networks

The CHNNs are briefly described [1]. Let [z.sub.a] and [w.sub.ab] be the state of neuron a and the connection weight from neuron b to neuron a, respectively. The weighted sum input to neuron a is given by

[I.sub.a] = [[N.sub.C].summation over (b=1)][w.sub.ab][z.sub.b], (1)

where [N.sub.C] is the number of neurons. For the resolution factor K, we define [[theta].sub.K] = [pi]/K. For the weighted sum, input I = r exp (i[theta]), where r [greater than or equal to] 0 and 0 [less than or equal to] [theta] < 2[pi]; the complex-valued multistate activation function is defined by

[mathematical expression not reproducible]. (2)

We define the set of neuron states and denote it as follows: S = {1, exp (2i[[theta].sub.K]), ..., exp(2(K - 1)i[[theta].sub.K])}. The connection weights must satisfy the following conditions:

[bar.[w.sub.ab]] = [w.sub.ba], (3)

[w.sub.aa] [greater than or equal to] 0. (4)

Then, the CHNN converges to a fixed point.

Let [mathematical expression not reproducible] be the pth training pattern, where [P.sub.C] is the number of training patterns. The Hebbian learning rule is defined as

[w.sub.ab] = [1/[N.sub.C]][[P.sub.C].summation over (p=1)][z.sup.p.sub.a][z.sup.p.sub.b]. (5)

Then, the connection weights satisfy [bar.[w.sub.ab]] = [w.sub.ba]. Giving the qth training pattern to the CHNN, the weighted sum input to neuron a is

[I.sup.q.sub.a] = [[N.sub.C].summation over (b=1)][w.sub.ab][z.sup.q.sub.b] (6)

= [z.sup.q.sub.a] + [1/[N.sub.C]][summation over (p[not equal to]q)][[N.sub.C].summation over (b=1)][z.sup.p.sub.a][bar.[z.sup.p.sub.b]][z.sup.q.sub.b]. (7)

The second term of (7) is referred to as the crosstalk term. The crosstalk term interferes with the storage of training patterns. We define

[A.sup.q.sub.a] = [1/[N.sub.C]][summation over (p[not equal to]q)][[N.sub.C].summation over (b=1)][z.sup.p.sub.a][bar.[z.sup.p.sub.b][z.sup.q.sub.a]][z.sup.q.sub.b]. (8)

Then, we have

[I.sup.q.sub.a] = [z.sup.q.sub.a](1 + [A.sup.q.sub.a]). (9)

If [absolute value of arg(1 + [A.sup.q.sub.a])] < [[theta].sub.K], then we have [f.sub.C]([I.sup.q.sub.a]) = [z.sup.q.sub.a]. Therefore, if [absolute value of arg (1 + [A.sup.q.sub.a])] < [[theta].sub.K] for all a, the qth training pattern is a fixed point. We regard [N.sub.C][A.sup.q.sub.a] as the summation of [P.sub.C][N.sub.C] random variables of V for simplicity, although the summation consists of exactly (P - 1)[N.sub.C] terms. The real and imaginary parts of each random variable have the equal variance [sigma] and do not have correlations. Setting [P.sub.C] = [alpha][N.sub.C], [N.sub.C][A.sup.q.sub.a] is regarded as the summation of [alpha][N.sup.2.sub.C] random variables. Then, we have

[1/[square root of [alpha]]][A.sup.q.sub.a] = [1/[square root of [alpha]][N.sub.C]][summation over (p[not equal to]q)][[N.sub.C].summation over (b=1)][z.sup.p.sub.a][bar.[z.sup.p.sub.b]][bar.[z.sup.q.sub.a]][z.sup.q.sub.b]. (10)

Let [X.sub.A] and [Y.sub.A] be the real and imaginary parts of (1/[square root of [alpha]])[A.sup.q.sub.a], respectively. From the central limit theorem, we have

([X.sub.A], [Y.sub.A]) ~ [1/2[pi][[sigma].sup.2]] exp(-[[[X.sup.2.sub.A] + [Y.sup.2.sub.A]]/2[[sigma].sup.2]]). (11)

For [A.sup.q.sub.a] = [X'.sub.A] + [Y'.sub.A]i, we have

([X'.sub.A], [Y'.sub.A]) ~ [1/2[alpha][pi][[sigma].sup.2]] exp(- [[[X'.sup.2.sub.A] + [Y'.sup.2.sub.A]]/2[alpha][[sigma].sup.2]]). (12)

3. Twin-Multistate Quaternion Hopfield Neural Networks

A quaternion is expressed by q = [q.sub.0] + [q.sub.1]i + [q.sub.2]j + [q.sub.3]k using real numbers [q.sub.0], [q.sub.1], [q.sub.2], and [q.sub.3]. The imaginary units i, j, and k satisfy the following properties:

[i.sup.2] = [j.sup.2] = [k.sup.2] = -1, ij = -ji = k, jk = -kj = i, ki = -ik = j. (13)

The quaternions satisfy the associative and distributive laws. For a complex number c, we have the important equality:

jc = [bar.c]j. (14)

Putting x = [q.sub.0] + [q.sub.1]i and y = [q.sub.2] + [q.sub.3]i, the quaternion q is described as q = x + yj. For quaternions x + yj and x' + y' j, the addition and multiplication are described as

(x + yj) + (x' + y' j) = (x + x') + (y + y')j, (x + yj)(x' + y'j) = (xx' - y[bar.y']) + (xy' + [bar.x']y)j. (15)

The conjugate of q is defined as

[bar.q] = [q.sub.0] - [q.sub.1]i - [q.sub.2]j - [q.sub.3]k = [bar.x] - yj. (16)

Then, we have the equality

q[bar.q] = x[bar.x] + y[bar.y] = [[absolute value of x].sup.2] + [[absolute value of y].sup.2]. (17)

In the TMQHNNs, the neuron states and connection weights are represented by quaternions. These neuron states and connection weights are denoted in the same way as those of CHNNs. The number of neurons in a TMQHNN is denoted as [N.sub.Q]. The weighted sum input to neuron a is given by

[I.sub.a] = [[N.sub.Q].summation over (b=1)][w.sub.ab][z.sub.b]. (18)

For the weighted sum, input I = [I.sub.x] + [I.sub.y]j, the activation function is defined as

[f.sub.Q](I) = [f.sub.C]([I.sub.x]) + [f.sub.C]([I.sub.y])j. (19)

Therefore, the set of neuron states is S + Sj. The connection weights must satisfy conditions (3) and (4). Then, the TMQHNN converges to a fixed point.

4. Storage Capacity of Twin-Multistate Quaternion Hopfield Neural Networks

We provide the Hebbian learning rule for TMQHNNs. Let [mathematical expression not reproducible] be the pth training pattern, where [P.sub.Q] is the number of training patterns. The Hebbian learning rule is given by

[w.sub.ab] = [1/2[N.sub.Q]][[P.sub.Q].summation over (p=1)][z.sup.p.sub.a][bar.[z.sup.p.sub.b]]. (20)

Then, the connection weights satisfy [[bar.w].sub.ab] = [w.sub.ba]. Giving the qth training pattern to the TMQHNN, the weighted sum input to neuron a is

[I.sup.q.sub.a] = [summation over (b[not equal to]a)][w.sub.ab][z.sup.q.sub.b] (21)

= [z.sup.q.sub.a] + [1/[N.sub.Q]][summation over (p[not equal to]q)][[N.sub.Q].summation over (b=1)][z.sup.p.sub.a][bar.[z.sup.p.sub.b]][z.sup.q.sub.b]. (22)

The second term of (22) is also referred to as the crosstalk term and interferes the storage of training patterns. We decompose the quaternion [z.sup.p.sub.a] into a pair of complex numbers by [z.sup.p.sub.a] = [x.sup.p.sub.a] + [y.sup.p.sub.a]j to investigate the storage capacity. Then, we have

[mathematical expression not reproducible]. (23)

We define

[mathematical expression not reproducible]. (24)

Then, we have

[I.sup.q.sub.a] = [x.sup.q.sub.a](1 + [B.sup.q.sub.a]) + [y.sup.q.sub.a](1 + [C.sup.q.sub.a])j. (25)

If [absolute value of arg(1 + [B.sup.q.sub.a])] < [[theta].sub.K] and [absolute value of arg(1 + [C.sup.q.sub.a])] < [[theta].sub.K], then we have [f.sub.Q]([I.sup.q.sub.a]) = [z.sup.q.sub.a]. We regard 2[N.sub.Q][B.sup.q.sub.a] and 2[N.sub.Q][C.sup.q.sub.a] as the summations of 4[N.sub.Q][P.sub.Q] random variables of S. Then, [B.sup.q.sub.a] and [C.sup.q.sub.a] follow the same distributions. Thus, we can discuss only [B.sup.q.sub.a]. If the TMQHNN is used instead of the CHNN, [N.sub.C] = 2[N.sub.Q] is required, since a twin-multistate quaternion neuron consists of two complex-valued multistate neurons. Setting [P.sub.Q] = [beta][N.sub.C] = 2[beta][N.sub.Q], 2[N.sub.Q][B.sup.q.sub.a] is regarded as the summation of 8[beta][N.sup.2.sub.Q] random variables, and we have

[mathematical expression not reproducible]. (26)

Let [X.sub.B] and [Y.sub.B] be the real and imaginary parts of ([1/2][square root of [beta]])[B.sup.q.sub.a], respectively. From the central limit theorem, we have

([X.sub.B], [Y.sub.B]) ~ [1/2[pi][[sigma].sup.2]]exp(- [[[X.sup.2.sub.B]+[Y.sup.2.sub.B]]/2[[sigma].sup.2]]). (27)

Putting [B.sup.q.sub.a] = [X'.sub.B] + [Y'.sub.B]i, we have

([X'.sub.B], [Y'.sub.B]) ~ [1/4[beta][pi][[sigma].sup.2]] exp(-[[[X'.sup.2.sub.B]+[Y'.sup.2.sub.B]]/4[beta][[sigma].sup.2]]). (28)

We require the same distributions for (12) and (22) and obtained [alpha] = 2[beta]. Thus, the CHNN has double the storage capacity of the TMQHNN.

Computer simulations were conducted to verify our analysis. K was varied from 4-12 in steps of 2, and P was varied from 1-20. For each K and P, 100 sets of training patterns were generated randomly; the number of trials was 100. The CHNN and TMQHNN attempted to store the training patterns by the Hebbian learning rule. If all the training patterns were fixed, the trial was regarded as successful, otherwise, as failed. Figure 1 shows the simulation results. The horizontal and vertical axes indicate the number of training patterns and success rate, respectively. The simulation results showed that the storage capacity of the TMQHNN was a bit larger than half that of the CHNN.

5. Conclusions

A TMQHNN needs only half the connection weight parameters of CHNN and is expected to have a smaller storage capacity. However, the storage capacity had not yet been analyzed. In this work, we defined the Hebbian learning rule for TMQHNNs and analyzed their storage capacity. The analysis demonstrated that a TMQHNN had half of the storage capacity of a CHNN. In addition, a computer simulation was conducted to verify our analysis. The simulation results confirmed our analysis. In future, we intend to study the storage capacity using different methods [19, 22, 23].

https://doi.org/10.1155/2018/1275290

Data Availability

No data were used to support this study.

Conflicts of Interest

The author declares that there are no conflicts of interest regarding the publication of this article.

References

[1] S. Jankowski, A. Lozowski, and J. M. Zurada, "Complex-valued multistate neural associative memory," IEEE Transactions on Neural Networks, vol. 7, no. 6, pp. 1491-1496, 1996.

[2] H. Aoki and Y. Kosugi, "An image storage system using complex-valued associative memories," in Proceedings of International Conference on Pattern Recognition, vol. 2, pp. 626-629, Barcelona, Spain, September 2000.

[3] H. Aoki, "A complex-valued neuron to transform gray level images to phase information," in Proceedings of the International Conference on Neural Information Processing, vol. 3, pp. 1084-1088, Singapore, November 2002.

[4] G. Tanaka and K. Aihara, "Complex-valued multistate associative memory with nonlinear multilevel functions for gray-level image reconstruction," IEEE Transactions on Neural Networks, vol. 20, no. 9, pp. 1463-1473, 2009.

[5] M. K. Muezzinoglu, C. Guzelis, and J. M. Zurada, "A new design method for the complex-valued multistate Hopfield associative memory," IEEE Transactions on Neural Networks, vol. 14, no. 4, pp. 891-899, 2003.

[6] P. Zheng, "Threshold complex-valued neural associative memory," IEEE Transactions on Neural Networks and Learning Systems, vol. 25, no. 9, pp. 1714-1718, 2014.

[7] T. Isokawa, H. Yamamoto, H. Nishimura, T. Yumoto, N. Kamiura, and N. Matsui, "Complex-valued associative memories with projection and iterative learning rules," Journal of Artificial Intelligence and Soft Computing Research, vol. 8, no. 3, pp. 237-249, 2018.

[8] Y. Kuroe, S. Tanigawa, and H. Iima, "Models of Hopfield-type Clifford neural networks and their energy functions-hyperbolic and dual valued networks," in Proceedings of International Conference on Neural Information Processing, pp. 560-569, Doha, Qatar, November 2011.

[9] M. Kobayashi, "Hyperbolic Hopfield neural networks," IEEE Transactions on Neural Networks and Learning Systems, vol. 24, no. 2, pp. 335-341, 2013.

[10] M. Kobayashi, "Global hyperbolic Hopfield neural networks," IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences, vol. E99-A, no. 12, pp. 2511-2516, 2016.

[11] M. Kobayashi, "Hyperbolic Hopfield neural networks with directional multistate activation function," Neurocomputing, vol. 275, pp. 2217-2226, 2018.

[12] T. Isokawa, H. Nishimura, N. Kamiura, and N. Matsui, "Associative memory in quaternionic Hopfield neural network," International Journal of Neural Systems, vol. 18, no. 2, pp. 135-145, 2008.

[13] T. Minemoto, T. Isokawa, H. Nishimura, and N. Matsui, "Quaternionic multistate Hopfield neural network with extended projection rule," Artificial Life and Robotics, vol. 21, no. 1, pp. 106-111, 2016.

[14] T. Minemoto, T. Isokawa, H. Nishimura, and N. Matsui, "Pseudo-orthogonalization of memory patterns for complex-valued and quaternionic associative memories," Journal of Artificial Intelligence and Soft Computing Research, vol. 7, no. 4, pp. 257-264, 2017.

[15] M. Kobayashi, "Hybrid quaternionic Hopfield neural network," IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences, vol. E98-A, no. 7, pp. 1512-1518, 2015.

[16] M. Kobayashi, "Symmetric quaternionic Hopfield neural networks," Neurocomputing, vol. 240, pp. 110-114, 2017.

[17] M. Kobayashi, "Three-dimensional quaternionic Hopfield neural networks," IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences, vol. E100-A, no. 7, pp. 1575-1577, 2017.

[18] M. Kobayashi, "Quaternionic Hopfield neural networks with twin-multistate activation function," Neurocomputing, vol. 267, pp. 304-310, 2017.

[19] J. Hertz, A. Krogh, and R. G. Palmer, Introduction to the Theory of Neural Computation, Addison-Weasley, Redwood City, CA, USA, 1991.

[20] M. Kobayashi, "Storage capacity of rotor Hopfield neural networks," Neurocomputing, vol. 316, pp. 30-33, 2018.

[21] M. Kitahara and M. Kobayashi, "Projection rule for rotor Hopfield neural networks," IEEE Transactions on Neural Networks and Learning Systems, vol. 25, no. 7, pp. 1298-1307, 2014.

[22] J. Cook, "The mean-field theory of a Q-state neural network model," Journal of Physics A: Mathematical and General, vol. 22, no. 12, pp. 2057-2067, 1989.

[23] Y. Nakamura, K. Torii, and T. Munakata, "Neural-network model composed of multidimensional spin neurons," Physical Review E, vol. 51, no. 2, pp. 1538-1546, 1995.

Masaki Kobayashi [ID]

Mathematical Science Center, University of Yamanashi, Takeda 4-3-11, Kofu, Yamanashi 400-8511, Japan

Correspondence should be addressed to Masaki Kobayashi; k-masaki@yamanashi.ac.jp

Received 15 August 2018; Accepted 18 September 2018; Published 1 November 2018

Academic Editor: Reinoud Maex

Caption: Figure 1: The simulation result for storage capacities.

Printer friendly Cite/link Email Feedback | |

Title Annotation: | Research Article |
---|---|

Author: | Kobayashi, Masaki |

Publication: | Computational Intelligence and Neuroscience |

Article Type: | Report |

Date: | Jan 1, 2018 |

Words: | 2913 |

Previous Article: | Deep Learning Methods for Underwater Target Feature Extraction and Recognition. |

Next Article: | Classification of Asphalt Pavement Cracks Using Laplacian Pyramid-Based Image Processing and a Hybrid Computational Approach. |

Topics: |