Printer Friendly

Research on the design of intelligent instrument performance system based on the artificial intelligence graphics algorithm.

1. Introduction

Intelligent instrument performance system is the use of artificial intelligence and graphics algorithm to generate computer animation, the key frame animation is different from the traditional sense, the realization of the system includes the following work: firstly, the 3D hand model is built, then the music analysis and distribution of fingering, finally produce a continuous animation. Playing finger human hand is one of the most complex and sophisticated human structures. Hand animation is part of human animation, many research teams have been working on the research and development of computer simulation and joint animation, professor N.M.T halmann of human and Simulation Center at University of Pennsylvania in the United States, is particularly prominent in the research group of University of Geneva, led by Professor N.I.Badler halmann (Kokalj A., 2003). Both in terms of technology and visual effects, the simulation of human behavior in real life is the most difficult and the most challenging task.

Artificial intelligence is a wide range of cross and frontier science, with the development of electronic computers, artificial intelligence has been applied. Application of artificial intelligence to computer animation and games, on the one hand, it provides a new research method for computer animation (Abreu, A., Rocha, A., Cota, M. P., & Carvalho, J. V., 2015), on the other hand, computer animation and games have opened up a new area for artificial intelligence, artificial intelligence algorithm can better realize its intelligence, and do not need to consider the limitations of hardware devices on it, it provides a better platform for the study of the theory of artificial intelligence. Music is a kind of sound, and it is also a universal language in the world. It is the thought and emotion of human being. However, skillfully playing an instrument is not an easy thing, it needs to cooperate with flexible finger, people need to learn and practice to master (Cohen E, Lyche T, Riesenfeld R., 1980). We design and develop an intelligent algorithm based musical instrument performance animation system, it provides an effective tool for people to learn to play musical instruments. The system can be selected according to the analysis of the corresponding music playing fingering, at the same time, it analyzes the playing style and music emotion, and finally the performance of computer animation, this not only can help beginners learn to play techniques, but also can be used as entertainment to enjoy, at the same time it also provides a research platform for music research, artificial intelligence research, computer animation research, and the research and development of these fields (Kokalj A., 2003).

2. Materials and Methods

2.1. Basic Computer Graphics

Computer graphics is a discipline that studies how to generate, process and display a graph with a computer. Over the past 20 years, computer graphics has become one of the most important branches of computer science. At least two reasons for this phenomenon. First, the fig. 1 is the most easily acceptable form of information. This is not only because the eyes are the most important organs of human perception, but also because the vast majority of people's brains in the information is about the image of the information. Therefore, in order to form a graphical way of the most natural and most agile. Second, computer graphics itself is very attractive. Human exploration is to promote the scientific development of the largest power (Stollnitz E J, DeRose T D, Salesin D H., 1995). As a result, more and more scholars all over the world have joined the research work in this field, as a result, the scale of the computer graphics conference is increasing, and the results are becoming more and more wonderful, related industries are booming. The judgment and calculation of the intersection of polygon cutting is based on the line clipping of the polygon window, because in the polygon cutting of the intersection is the use of the entity of each edge of the polygon and polygon intersection, that is, line cutting. Line clipping algorithm for polygon window has several effective algorithms. For the convex polygon window of line clipping, a well-known algorithm is proposed by Cyrus and Beck. It can be divided into upper and lower two groups by judging whether the point product of the linear segment of the vector and the edge of the window is greater than zero. Then, the minimum and the maximum points in the lower group are respectively taken, the end of the visible part of the line segment. But it is not significant for the group of concave polygon window, so the Beck Cyrus algorithm is suitable for the line clipping of the convex polygon window (Ferrin T E, Huang C C, Jarvis L E, et al., 1988). Line clipping for general polygon window, some literatures propose an efficient algorithm (called the slope method). First, the algorithm is to select a fixed point on the line or the extension line which is clipped ([X.sub.f], [Y.sub.f]). This fixed point should be left (or under the lowest vertex) of the most left vertex of the polygon window. Then, calculate the slope of the line from the fixed point to the vertices of the polygon:

v[s.sub.i] = [[y.sub.i] - [Y.sub.f]]/[[x.sub.i]- [X.sub.f]] (1)

Where [v.sub.i] ([x.sub.i], [y.sub.i]) is the vertex of the i. This can then determine whether the slope of the straight line is the slope of the two adjacent vertices of the polygon. If, then, the edges of the two adjacent vertices are cut by a straight line or an extension line: Otherwise, do not intersect. If intersection, then calculate the intersection. In this paper, we use the method of fault tangent transformation, which is described below. A polygon window C has n sides, C's vertex is v. The coordinates are ([x.sub.i], [y.sub.i]), i = 1,2, ..., n; The cutting line is L, and the coordinates of A and B are ([x.sub.a], [y.sub.a]) and ([x.sub.b], [y.sub.b]), as shown in Fig. 1. Below using the case [x.sub.b] [greater than or equal to] [x.sub.a] to show that:

[increment of x] = [x.sub.b] - [x.sub.a] (2)

[increment of y] = [y.sub.b] - [y.sub.a] (3)

And assume that [increment of x] [not equal to] 0, [DELTA]y [not equal to] 0, or L will be parallel to a coordinate axis, and this situation does not need to cut the cut, make the implementation process more simple (Khalimsky E, Kopperman R, Meyer P R., 1990).


In the design of cutting algorithm, the most important consideration is to reduce unnecessary intersection. This method gives a very simple judging condition, and cannot calculate the non effective intersection of the extension line that falls on the window. The non effective intersection of the extension line that falls on the line to be clipped can only be determined after it is calculated to determine its validity (i.e., between the ends of the line). First, the Y and C are applied to the same L, and the L is transformed into a parallel (horizontal) direction of the X axis with the wrong tangent transformation. The matrix of the tangential transformation along the Y axis is [MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII], in the form of a transformation:


Where the x and y represent the coordinates of the wrong tangent transformation (the following are expressed in the form of the transformation of the point or coordinates). It is obvious that the X coordinates of these points have no effect on the coordinates, as shown in Fig 2. The intersection of the L and the Y axis is I(0, y), which is:

[y.sub.c] = [x.sub.a] x d + [y.sub.a] (5)

Easy to verify, after the wrong cut, straight AB into a straight line:

y = [y.sub.c] (6)

The error tangent transformation is an affine transformation, which cannot preserve the metric properties of a graph, it can cause a change in the shape of a graph, but the relative relationship between the lines is the same. Because of the C and L to make the wrong cut L into the level, so it is easy to judge and calculate the intersection (Webber B L, Phillips C B, Badler N I., 1993). And the X coordinates of the intersection and the intersection of the transformation are not changed, therefore, after (the wrong cut), it is necessary to carry out the anti-error transformation to the Y coordinates. From the above, the formula of the error is found, the calculation of the error is very small. After the wrong cut, easy to see, the condition of the intersection of an edge of C with the line AB is that the two vertexes after transform [v.sub.i-1] ([x.sub.i-1], [y.sub.i-1]) and [v.sub.i] ([x.sub.i], [y.sub.i]) are located in the two sides of linear y = [y.sub.c]. Or at least one of the vertices of the transformation is on the line. We first discuss the former case. At this time, the [v.sub.i-1] [v.sub.i] coordinates of the intersection of line y = [y.sub.c] and line x can be calculated as follows:



The above process can be described as follows:

Calculate [increment of x], [increment of y], d, [y.sub.c], [y'.sub.a] and [y'.sub.b]; For each [v.sub.i], calculate:

[y'.sub.i] = [x.sub.i] * d + [y.sub.i] ,(i = 1,2, ..., n) (8)

Make [y'.sub.0] = [y'.sub.n], 'that is j = 1. For each i(1 [less than or equal to] i [less than or equal to] n), followed by:

if ([y'.sub.i-1] > [y.sub.c]) AND ([y'.sub.i] < [y.sub.c]) OR ([y'.sub.i-1] < [y.sub.c]) AND ([y'.sub.i] > [y.sub.c])

then {Using Formula (7) to calculate the x coordinate I[x.sub.j] of the intersection point}

As mentioned in the previous section, the new algorithm determines the access of the intersection at the same time. The specific method is to determine the intersection of the intersection based on the parity of the intersection in the line(Kaufman A., 1990), the intersection of odd number is in the point of intersection, the intersection of the number is a point. As the cutting line is transformed into a horizontal line after the wrong cut, so the X coordinates of the points can be determined by the order of the points on the line. If the X coordinates of a point of intersection are obtained, the intersection is found to be located in the extension line of the cutting line and not between the two ends of the line A and B, the intersection is not inserted into the polygon list, but it is possible to remove the influence of the parity of the other nodes in the line. Our approach is that if the direction of the line is cut from A to B.

On the A side, the extension of the intersection of the line is not preserved but only the cumulative number of effective nodes to maintain the correct order of the right, on the B side of the extension of the intersection is not cumulative: If the direction of the cutting line is from [J.sub.1] to A, the number of points on the

B side of the extension line is only a total number of points(Tanaka A., 2000).

2.2. Music Performance Tips System Principle

The basic principle of the system of musical performance is to be placed in a row of the keyboard by a row, its role is to tell the player to play what keys at what time, so as to make the music without any knowledge of the background of the play can also play a complete piece of music, the principle block diagram is shown in Fig 3.
Figure 3--System Principle and Working Flow Chart

Video image capture

[right arrow]

Image preprocessing

[right arrow]

Image analysis, extract key information

[right arrow]

MIDI instruction

[right arrow]

Play MIDI sound

[right arrow]

Analysis of the next frame image

[right arrow]


Video image acquisition and image processing: the use of camera equipment to monitor the area of measurement, and the use of VFW for computer to recognize the digital signal. Image preprocessing: the image signal is sharpened, which can improve the visual effect of the image, the image's clarity is higher and it is more advantageous to the computer processing, analysis of various image features. Image analysis, extract key information: the use of background subtraction, the information from the player's hand is separated from the background to form a two value image, so as to extract the position information of the player's hand, and judge the corresponding key information (Morita H, Hashimoto S, Ohteru S., 1991). Prior to this process, the setting of the relevant parameters is required to determine the position of the player's hand. MIDI instruction: the position information of the player's hand is to determine the corresponding key information, MIDI format data generated in real time. Play MIDI sound: MIDI decoder based on the PC MIDI message real-time playback of a specific note. As the music is playing, the playing of the players is very fast, so the real-time performance of the system is very high, in addition, a variety of different color makes the scene there may be interference, so the video based virtual instrument system must have some robustness, to adapt to the changes in the environment. In the design must take into account the following aspects: Accuracy: due to the defects of the camera lens and other factors, the image will be distorted, in the target location, the image distortion should be considered, and the corresponding correction is carried out. In the case of the presence of the image and the interference of the environment, pixel information itself has a 2-3 pixel deviation, which corresponds to the actual size of the measured area, the position deviation (resolution 640 * 480) can be allowed for 1cm, so the process of image processing and identification must be kept in high accuracy to avoid error expansion.

Real time: in the process of performance training, the trainees' reaction speed is high, the trainee wants to play the music file in real time, LED real time real time to complete the corresponding action, the video based virtual instrument system without the keyboard must be trained in real-time detection and tracking, and maintain a high real-time performance. No keyboard virtual instrument system to be able to do the analog video signal each frame or a processing. The maximum operating frequency of the virtual instrument system is 60Hz, and the system must provide 60 times per second. The position coordinates of the training system is the whole performance training system to complete a closed loop operation, taking into account the time required by other systems, and the system must be trained to get the coordinates of the 10ms time. Robustness: although in the case of an ideal experiment or performance training, the ideal image can be obtained from the video based virtual instrument system without keyboard (Rovan J, Hayward V., 2000). But in practice, there are a lot of outside interference is inevitable, such as uneven lighting. Playing the scene is usually light illumination, according to their distribution, the difference of the illumination angle and the light material, the learning board is tested in different degrees of high brightness and dark areas. In addition, the influence of the outside world will make the local dark, or other objects reflect and use flash light will lead to the site immediately appear dark area and instantaneous bright spot. The light spot is reflected by the intensity of the light, and the color is light, if the intensity is large enough, it will be due to the saturation effect of the picture shows that the white light is weak, dark color imaging. These various reasons are caused by the change of illumination conditions and optical brightness is not uniform will reduce the accuracy of the system identification, and even lead to the failure of target identification. Therefore, it is necessary to study the color recognition algorithm which is insensitive to the change of light intensity, and the algorithm that can be adapted to these changes (Friberg A, Bresin R, Sundberg J., 2006).

2.3. The Overall Structure of the Intelligent Instrument Playing System

Intelligent instrument performance system based on intelligent algorithm is shown in Fig. 4, it consists of three parts: virtual space simulator: It contains both the hands and the model of the instrument (such as the piano), and when the instrument is played, the sound of the instrument is made.


The establishment of the hand model needs to consider the image, anatomy and robotics, the characteristics of the skeletal muscle model, the spring--damping model, the motion model and the dynamic model are considered, which not only considers the mechanical movement of the bone, but also considers the deformation of the surface, to establish a realistic model of the hands, consider establishing the solid model and the perfect fit for the sounds and movements of the instruments(Ryan J., 1991); Music analysis module: the classical music, the use of video analysis method, using image processing technology to extract playing fingering information and motion information, and access to fingering and playing style; it is also possible to use of intelligent algorithms, the music recognition and analysis.

Analysis of distribution of fingering, playing style and playing music emotion; intelligent module: According to the score distribution method, goal oriented model generation hand movement, the performance of the natural flow of the animation. We know that one of the main evaluation criteria of the animation system is the real sense of motion. The spatial coordinates and motion of the model can be generated by dynamic and kinematic constraints, but if the motion is too mechanical to lose the sense of reality will make the animation to reduce the appreciation (Sloboda J A., 1996).

Thus, the performance of the players is obtained by analyzing methods of key frame data through the video, an intermediate frame is generated by interpolation to generate a realistic performance of the performance, need to take into account the smart algorithm and the movement of the hand model, improve the speed of image analysis and data extraction. Through the intelligent algorithm generates proper distribution of fingering, virtual performer played according to the analysis to obtain the playing style and music emotion attractively. The musical lovers learning to provide very good auxiliary tools, for the creation of music creators provide auxiliary tools. The key to the whole system is based on computer image algorithm combined with intelligent algorithm extracting music playing fingering, capture the playing style and music emotion (Sloboda J A., 1996).

3. Results and Analysis


As shown in Figure 5, number 1-9 respectively for the musical instrument Hulusi, Guqin, Guzheng, Pipa, Bamboo flute, Erhu, Flute, Jing Hu and the Suona. The 114 sound high school, there are 4 in the pronunciation of the process did not change, and the remaining 110, are more or less change, there are a lot of high tone, some of the more than 10, and the 228 pitch points are only the highest and lowest part of the change. The positions of these pitches are different from those of the musical instruments, the highest point is usually in the beginning stage of the sound, the lowest in the end stage, the middle stage is high, low pitch, pitch alternation and irregular, these pitches are distributed throughout the entire course of the sound. That is to say, the instrument is played, relatively the most stable tone, the vast majority in the average of 31.83 tones, constantly changing, cannot be truly sustained and stable. The range of variation is 31.83, and it is far greater than that of the 6, why can't you find it? Preliminary analysis, there are two reasons for the change of the pitch is very fast, some shorter than the 0.1 seconds of the pitch of a person's ear: Two is the change is progressive or decreasing, not suddenly changed, a lot of instantaneous change is less than 6(Paradiso J, Sparacino F., 1997).

4. Conclusions

This paper introduced the design and implementation of intelligent instrument system on the basis of computer graphics, described the principle of music performance prompting system, and comprehensively used the knowledge of artificial intelligence, computer animation, AI graphics, robotics, music design, and so on. The musical score analysis of the musical instrument performance system mainly uses the intelligent robot method and computer graphics analysis method, so as to realize the natural and fluent intelligent performance. In a word, the system design in this paper can improve the overall design level of the musical instrument performance system.

Recebido/Submission: 11-09-2015

Aceitacao/Acceptance: 02-11-2015


This Paper was supported by the educational reform project in Liaoning province, department of education, the project name: "multiversity classification of performing arts professionals training research and practice of teaching mode", the project number: UPRP20140132, And the 2013 young teachers Special teaching reform project of Liaoning university of science and technology. The project number: QNJJ--2013-20.


Abreu, A., Rocha, A., Cota, M. P., & Carvalho, J. V. (2015). Caderneta Eletronica no Processo Ensino-Aprendizagem: Visao de Professores e Pais de alunos do ensino Basico e Secundario. RISTI--Revista Iberica de Sistemas e Tecnologias de Informacao, 2015(16), 108-128.

Cohen E, Lyche T, Riesenfeld R. (1980). Discrete B-splines and subdivision techniques in computer-aided geometric design and computer graphics. Computer graphics and image processing, 14(2), 87-111.

Ferrin T E, Huang C C, Jarvis L E, et al. (1988). The MIDAS display system. Journal of Molecular Graphics, 6(1), 13-27.

Friberg A, Bresin R, Sundberg J. (2006). Overview of the KTH rule system for musical performance. Advances in Cognitive Psychology, 2(2-3), 145-161.

Hunt A, Wanderley M M. (2002). Mapping performer parameters to synthesis engines. Organised sound , 7(02), 97-108.

Kaufman A. (1990). Volume visualization. The visual computer, 6(1), 1-10.

Khalimsky E, Kopperman R, Meyer P R. (1990). Computer graphics and connected topologies on finite ordered sets. Topology and its Applications , 36(1), 1-17.

Kokalj A. (2003). Computer graphics and graphical user interfaces as tools in simulations of matter at the atomic scale. Computational Materials Science, 28(2), 155-168.

Kokalj A. (2003). Computer graphics and graphical user interfaces as tools in simulations of matter at the atomic scale. Computational Materials Science, 28(2), 155-168.

Morita H, Hashimoto S, Ohteru S. (1991). A computer music system that follows a human conductor. Computer, 24(7), 44-53.

Paradiso J, Sparacino F. (1997). Optical tracking for music and dance performance. Optical 3-D Measurement Techniques IV, 11-18.

Rovan J, Hayward V. (2000). Typology of tactile sounds and their synthesis in gesture-driven computer music performance. Trends in gestural control of music, 297-320.

Ryan J. (1991). Some remarks on musical instrument design at STEIM. Contemporary music review, 6(1), 3-17.

Sloboda J A. (1996). The acquisition of musical performance expertise: Deconstructing the "talent" account of individual differences in musical expressivity. The road to excellence: The acquisition of expert performance in the arts and sciences, sports and games, 107-126.

Stollnitz E J, DeRose T D, Salesin D H. (1995). Wavelets for computer graphics: a primer. 1. Computer Graphics and Applications, IEEE, 15(3), 76-84.

Tanaka A. (2000). Musical performance practice on sensor-based instruments. Trends in Gestural Control of Music , 13(389-405), 284-292.

Webber B L, Phillips C B, Badler N I. (1993). Simulating humans: Computer graphics, animation, and control. Center for Human Modeling and Simulation, 68-72.

Zhong-shuang Liang (1)

(1) College of art, University of Science and Technology Liaoning 114051 PRC

DOI: 10.17013/risti.17A.41-51
COPYRIGHT 2016 AISTI (Iberian Association for Information Systems and Technologies)
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 2016 Gale, Cengage Learning. All rights reserved.

Article Details
Printer friendly Cite/link Email Feedback
Author:Liang, Zhong-shuang
Publication:RISTI (Revista Iberica de Sistemas e Tecnologias de Informacao)
Date:Mar 15, 2016
Previous Article:The modeling and simulation of flow shop scheduling problem based on adaptive genetic algorithm.
Next Article:Research on gesture driving virtual performance system based on graphics behavior.

Terms of use | Privacy policy | Copyright © 2019 Farlex, Inc. | Feedback | For webmasters