A grey relational analysis based approach to the evaluation of Brazilian postgraduate programs in master of business administration

Pauli A. A. Garcia

pauliadriano@id.uff.br

Fluminense Federal University – UFF, Niterói, Rio de Janeiro, Brazil

Fernanda A. de C. Duim

fernanda.duim@gmail.com

Fluminense Federal University – UFF, Niterói, Rio de Janeiro, Brazil

---

ABSTRACT

The education development in Brazil has been influenced by government policies, especially those directed at higher learning. To judge the effectiveness of these policies it is necessary to evaluate the quality of the teaching offered, particularly with respect to public education. In this context, there have been many initiatives for new postgraduate programs. To be accredited, these programs must be approved by the Coordination for the Improvement of Higher Education Personnel (CAPES), part of the Ministry of Education. Among the many criteria considered by CAPES, the academic production is the most important. Many works proposing approaches to classify college programs based on faculty bibliographical output have been published; among these, those based on data envelopment analysis stand out. The present work uses grey relational analysis based approach. Another difference is that the information is considered on a master’s program not yet accredited, that is, one that still needs to be evaluated by CAPES. A classification is established for this program in relation to the ones already accredited along with a way to identify the improvement points and an improvement factor for each attribute considered. The results indicate that the approach is efficient in relation to that based on traditional data envelopment analysis and suggest areas for future research in this area.

Keywords: Grey relational analysis; Educational evaluation; Multicriteria analysis.

---

INTRODUCTION

The role of education as a key element to alleviate many national social problems is increasingly acknowledged in Brazil. The Brazilian government is currently seeking to modify its policies at various school levels. This effort requires better ways to evaluate educational programs, to enable policymakers to reach informed decisions (Leite et al., 2006).

Particularly at the college level, the evaluation of educational performance is generally focused on results. This is even more so in the evaluation of postgraduate programs. Under the traditional “publish or perish” dictum, the main aim of researchers associated with master’s and doctoral programs is currently aimed at definitive publications, that is, at writing books, book chapters or papers in scientific journals. Other factors besides productivity are considered; however, scientific production is the leading criterion (Leite et al., 2006; Moreira, 2008).

Many new universities have sprung up significantly due to the current plan for university restructuring (REUNI – “Reestruturação Universitária”, in Portuguese) and the plan for spreading access to higher education on the outskirts of the country’s main cities, thus expanding the number of professors needed to make up their faculties. It is natural that, in this growth process, professors and administrators feel motivated to expand their programs beyond offering undergraduate degrees, by developing postgraduate programs as well. Logically these initiatives start with establishment of mastering programs, and then expand to doctoral programs. However, these new programs will unavoidably be evaluated in relation to the existing ones and their performance will thus be compared to that of traditional benchmark programs (Martins, 2002; Moreira, 2008).

Because of this need to identify benchmark programs, various works have proposed alternative approaches other than those in use by Coordination for the Improvement of Higher Education Personnel (CAPES). Among these approaches, the standouts are those based on data envelopment analysis (DEA) (Meza et al., 2003; Lins et al., 2004; Mello et al., 2008; Lobianco et Meza, 2008). These works identify benchmark programs in their respective academic fields, based particularly on aspects associated with productivity. Different approaches based on SWOT (Neves et Costa, 2006a), and ELECTRE (Miranda et Almeida, 2004; Neves et Costa, 2006b) are also proposed; however, the DEA ones are considered the most appropriated to deal with this type of problem.

Although the identification of benchmarks among postgraduate programs and the evaluation of their performance are not new subjects; however, the analysis of the insertion of a program not yet accredited in a determined field is. The contributions of the present paper to research on the evaluation of postgraduate programs are: (i) the identification of improvement points; and (ii) the positioning of a new program, that is, of a program pending evaluation for accreditation by CAPES, in relation to programs already approved. Another difference in relation to the traditional approaches used is the employment of grey relational analysis (GRA) to rank postgraduate programs. The GRA-based approach is, from a computational perspective, more efficient than the traditional DEA approach.

The paper is organized into five sections, including this introduction. The second section presents the concepts of grey relational analysis and the structure for evaluating postgraduate programs based on it. The third section features the data considered in the analysis and the results obtained. The fourth section discusses and compares the classification of the programs under analysis between grey relational analysis and data envelopment analysis. The fifth section presents the conclusion.

GREY THEORY

Grey systems theory was proposed by Professor Julong Deng (1982). The idea behind this theory is that data, such as the operational characteristics, mechanisms, structures and behavior of a determined system, are assumed to be deterministic and partially known. The behavior of the system is explored by relational analyses and construction of models (Lin et al., 2003; Wu et al., 1984 apud Chang et al., 2001).

Traditionally, a totally unknown system, without any data, is represented by a “black box”. In contrast, a system, about which full information is available, is considered a “white box”. Finally, systems featuring incomplete data are called grey systems. The grey elements are those with incomplete information, and a grey relation is one with incomplete information (Liu et Lin, 2006).

The aim of the present work is to establish grey relations by means of grey relational analysis.

Grey Relational Analysis

As a measurement technique, GRA aims to determine the relationship between a reference observation and comparison (or standard) observations. This relationship is established by means of grey relational coefficients (GRC) and grey relational grades (GRG) (Huang et Lee, 2004; Lin et al., 2009).

Consider a set of observations , where is a reference observation and are the observations to be compared with it. Each observation has n attributes and is denoted by . Generally, all the attributes must be processed in advance to normalize their values. This normalization is done as follows:

1 – If the attribute of the series is of the higher-the-better type, then the normalization is performed according to the following formula:

Here is the value of attribute j associated with series i and is the normalized value of attribute j of the series p. This formulation measures the effectiveness of the upper bound.

2 – If the attribute is of the lower-the-better type, then the normalization is performed according to the following formula:

Just as in the first formulation, is the value of attribute j associated with series i and is the normalized value of attribute j of the series p. This formulation measures the effectiveness of the lower bound.

3 – If the attribute is the nearer the reference the better, then the normalization is performed in the following form:

The normalization presented in formula (3) means that the nearer the value of the attribute of the comparative series is to the pre-specified value, ; the nearer to 1 the normalized the value will be.

After this preliminary normalization process of each attribute of each series, the GRC is calculated, as follows:

where is traditionally assumed to be equal to 0.5 (Deng, 1989; Zuo, 1995), i = j = 1, 2, ..., m e k = p = 1, 2, ..., n.

The GRC can be seen as the degree of similarity between attribute p of series i and attribute p of the standard series. The nearer this is to one, the greater the similarity is and the nearer to zero, the less the similarity is.

After the relational coefficients are calculated for each attribute of each series, the next step is to calculate the grey relational grade (GRG), as shown below.

where is the weight associated with the relational coefficient k, remembering that for each series there will be a relational coefficient for each attribute. Therefore, the weights reflect the importance of each attribute k and .

In this way, the series that has the highest grey relational grade is ranked higher than the others. Based on this concept, a ranking is established among the postgraduate programs evaluated by CAPES.

POSTGRADUATE PROGRAM CLASSIFICATION BASED ON GRA

As mentioned, the aim of this paper is to present a different approach to classify postgraduate programs accredited by CAPES, specifically a ranking based on the academic productivity of these programs.

In line with the discussion of GRA, the following setup is used to analyze the postgraduate programs:

a) Each program is considered a series;

b) The standard series is established based on the best attributes of all the series;

c) The attributes considered for each series are those shown in the comparative tables supplied by CAPES regarding bibliographical production.

The elements of the bibliographical production considered by CAPES (2007) are: (i) complete articles published in technical or scientific periodicals; (ii) books and book chapters. The periodical articles are further classified, based on the comparative table of 2004, 2005 and 2006, as: (i) International A, B and C; (ii) National A, B and C; and (iii) Local A, B and C. Books and chapters have the following sub-classification: (i) full text; (ii) chapters; (iii) collections; and (iv) short articles, such as in encyclopedias and other reference books, and others. All the above-mentioned data were provided by CAPES (2007), based on the triennial evaluation. From the programs recommended by CAPES, we only considered those issuing academic master’s degrees. The reason for this filter is that the program pending accreditation is a master’s program in business administration. Because of this, it would not be coherent to draw comparisons with established doctoral programs or professional master’s programs instead. The programs considered are shown in Table 1 below, together with the criteria adopted to classify the programs.

Table 1. Information on the master’s programs

Source: The authors adapted it from CAPES (2007)

In the table above, Faculty Members is related to the number of permanent professors in the program; Dissertations is related to the number of dissertations presented in the three years studied (2004, 2005 and 2006); International Periodicals refers to the number of articles published in international periodicals during this period; Brazilian Periodicals refers to the number of articles published in Brazilian periodicals in the period; Local Periodicals refers to the number of articles published in periodicals with local circulation; and Book Chapters and Others refers to the number of books, chapters, short reference articles, collections, etc. published in the period. The last line (in yellow) corresponds to the new program whose organizers intend to register it with CAPES, seeking to obtain accreditation.

According to the methodology presented, the data in Table 1 first had to be normalized for each program, considered to be an observational unit.

For the Faculty Members criterion, the normalization mode was the nearer the ratio between the Dissertations and Faculty Members was to 5, the better the program, meaning the application of equation (3). For the other criteria, the normalization was of the higher-the-better type, meaning application of equation (1). Table 2 presents the normalized figures for each series.

The normalization performed for the dissertation/professor ratio was based on our belief that five students per faculty advisor is an optimal ratio. However, this reference value can be changed as suitable, depending on the program and circumstances.

Table 2. Normalized data

Source: The authors

From the normalized data, the grey relational coefficients were then calculated according to equation (3). Table 3 shows these coefficients for each criterion of each series. The nearer to 1 the GRC is, the nearer the criterion associated with the series under analysis is to the target criterion associated with the standard series.

Table 3. Grey relational coefficients

Source: The authors

The next step was to use the grey relational coefficients to calculate the grey relational grades of each series with the standard series. At this initial moment, the GRG was calculated according to equation (4), considering the weights as being equal, that is, without making any distinction with respect to the importance of each attribute. Based on this assumption, the postgraduate programs were ranked as shown in Table 4.

This table shows the grey relational grade. The nearer this is to 1, the nearer the program is to the standard program, whose normalized attributes are all equal to 1. This allowed ranking the programs.

According to the ranking shown above, the program pending presentation for accreditation would be in eighth place. Obviously, this analysis is based on the criterion of bibliographical production, considering that the importance of each type of production is the same, meaning the same weight. This weighting scheme was also considered in the work of Lobianco et Meza (2008). However, this does not impair the proposal presented in this article, as it is clarified in the next section, which discusses the results obtained. Notice that, if one intends to consider the importance of each kind of bibliographical production, one can do that considering different weights in equation 5, i.e., one can consider different values for .

Table 4. Grey relational grade and program ranking

Source: The authors

RESULTS AND DISCUSSION

Table 5 compares the classification obtained by grey relational analysis with that established by CAPES, without considering the new program. In this table, the ranks were obtained based on the score given by CAPES to the programs. As can be seen, there are many programs with the same score. Therefore, the criterion of the average of tied ranks was used to establish the position of the programs with the same scores.

An analysis of the correlation between the classifications obtained by the two methods produced a Spearman correlation coefficient of , considered a weak correlation.

A possible explanation for this weak correlation between the rankings is that CAPES considers other factors, of a more subjective nature, to assign scores to the programs. Nevertheless, despite the differences in the rankings, the grey relational coefficients show how much the attributes of each program need to improve to attain the values assigned to the standard program, characterized by the standard series.

Table 5. Paired Ranking (CAPES x )

FSource: The authors

For example, with respect to the levels of the criteria for the new program, specifically the normalized indices, the improvement needed to attain the indices of the standard series are shown by the factors in Table 6.

Table 6. Improvement criteria factors for the new program.

*The factor concerned to the Faculty Members is 1.62 if one considers that the number of student will stay equal to 18

The interpretation of these factors is as follows: for each of the criteria presented in Table 6, for the new program to match the standard program, each of its production criteria has to be multiplied by the corresponding factor. For example, in the case of the number of publications in international journals, the new program’s production will have to rise 2.67 times. The same interpretation is held for the other criteria.

Comparison with the DEA-based Approach

The main purpose to proceed this comparison is because the DEA is the approach widely considered to achieve post-graduate program analysis (Lins et Meza, 2000; Meza et al. 2003; Lins et al., 2004; Mello et al. 2006, Mello et al., 2008; Lobianco et Meza, 2008).

The DEA technique, developed by Charnes et al. (1978), is an approach based on linear programming aimed to calculate the maximum efficiency of the decision making unit (DMU) under analysis. Such maximum efficiency is a relative measure, not an absolute one. In other words, it is calculated in relation to all the DMU under analysis. The traditional model is known as CCR due to the initials of its inventors, Charnes, Cooper and Rhodes (Lins et Meza, 2000).

where is the output level y of DMU k, is the input level x of DMU k, is the weight attributed to output y, is the weight attributed to input x, and is the efficiency measure of DMU 0, i.e., of the DMU under analysis.

The model represented by formula (5) is applied as many times as there are DMU under analysis. For more details on DEA models, see Charnes et Cooper (1990).

The above model was applied to the programs presented in Table 1, considering the number of professors as the input factor and the respective literature production values as the output factor (Lobianco et Meza, 2008). The result is shown in Table 7.

In this table, the more efficient programs are classified higher than the others. In other words, the programs with efficiency equal to 100% are called benchmark programs and are on the efficiency frontier. The other programs are within, or enveloped by, this efficiency frontier.

Table 7. DEA efficiency level

Source: The authors

Table 8 shows the results of comparing the rankings obtained by the DEA, without considering the presence of the new program, against the classifications established by CAPES.

Table 8. Paired position (CAPES vs. DEA)

Source: The authors

As before, the rankings were established considering the ties, to facilitate the statistical analyses based on rankings. From the ranks established for the programs, the Spearman correlation coefficient can be calculated between the classifications. For the data in Table 8, this coefficient is . This means there is a weak correlation between the rankings obtained by DEA and those established by CAPES (Sheskin, 2004).

Table 9 shows the comparison of the classifications, now considering the presence of the new program in both cases (data envelopment analysis and grey relational analysis).

Table 9. Paired position (DEA vs. GRA)

Source: The authors

Finally, the test of signs presented by Montgomery et Runger (2003) was performed. The result obtained was . A comparison of this value with the table value for α=0.05 and n=27 for a single-tailed test showed a critical value of 8. Since the value calculated is higher than the critical value, the null hypothesis cannot be rejected, that is, it cannot be said with 95% confidence that the two approaches used to classify the master’s in business administration programs produce different results. In other words, the two approaches are statistically equal. The correlation coefficient between the two classifications was , which is a good correlation.

CONCLUSIONS

The results obtained by grey relational analysis proved to be coherent with the data envelopment analysis approach. However, in terms of computational complexity, grey relational analysis is simpler and requires less computational cost. The reason is that, for DEA, a linear programming model has to be resolved for each DMU under analysis. On the other hand, grey relational analysis exclusively uses normalization procedures.

Another factor that deserves to be mentioned is that all the attributes in our analysis were weighted equally. In the data envelopment analysis technique, as presented by Lobianco et Meza (2008), different weights are considered in the model according to the needs of the DMU under analysis. That is, if one of the attributes contributes more for a particular DMU so that it is considered more efficient, this attribute will receive greater weighting than the other that can have a negative contribution. This latte can be unconsidered in the analysis, i.e., the model can consider a weight equal to zero.

In the DEA classification, the new program was considered 100% efficient. This can be related to the distribution of the weights when resolving the linear programming problems. Traditional models tend not to weight the attributes that impair the performance of the DMU under analysis. In these cases, the model tends to attribute a weight of zero to these attributes, as mentioned above. In contrast, in establishing the grey relational analysis for this case, all the attributes considered were equally weighted. However, this can be modified if there is information about the relative importance of each attribute to be considered in the analysis.

Other studies based on DEA with other models have been conducted, including: (i) cross evaluation (Meza et al., 2003), (ii) variable returns to scale (Faria et al., 2008), and (iii) weights restriction (Mello et al., 2006). The proposals of these models intend to provide results that incorporate important characteristics, such as: (i) distinction of the weights associated with different types of publication; (ii) variation in terms of the number of professors as being a possible scale variation; (iii) time in existence as being a possible scale variation; and (iv) greater distribution discriminatory power of the DMU by means of cross evaluation. These are some examples associated with the different possible uses of other models based on DEA. However, these models tend to become more complex as more characteristics are incorporated, and other shortcomings can be addressed to them. Obviously, DEA models have many issues that can lead to important analysis. In spite of that, for simple initial approach, GRA can be considered an alternative method for easily helping in this kind of decision making.

---

REFERENCES

Chang, C. L.; Liu, P. H.; Wel, C. C. (2001), “Failure mode and effects analysis using grey theory”, Integrated Manufacturing Systems, Vol. 12, No. 3, pp. 211-216.

Charnes, A.; Cooper, W. W. (1990), “Data Envelopment Analysis”, Operational Research, Vol. 90, pp. 641-645.

Coordenação de Aperfeiçoamento de Pessoa de Nível Superior – CAPES (2007). Planilhas Comparativas Trienal. Disponível em: www.capes.gov.br. Acesso em 12 dez. 2009.

Deng, J. L. (1982), “Control problems of grey systems”, Systems & Control Letters, Vol. 5, pp. 288-294.

Deng, J. L. (1989), Introduction to grey system theory, USA: The Journal of Grey System, vol.1, nº.1, pp.1-24. Disponível em: http://portal.acm.org. Acesso em: 13 ago. 2009.

Faria, F. P.; Jannuzzi, P. M.; S. S. J. (2008), “Eficiência dos gastos municipais em saúde e educação: uma investigação através da análise envoltória no estado do Rio de Janeiro”, Revista de Administração Pública, Vol. 42, pp. 155-177.

Huang, C. C.; Lee, H. M. (2004), “A grey-based nearest neighbor approach for missing attributes value prediction”, Applied Intelligence, Vol. 20, pp. 239-252.

Leite, M. F. B. et al. (2010), Métodos quantitativos na avaliação da Capes: uma pesquisa bibliográfica. FACEF Pesquisa, Vol. 9, No. 2, pp. 166-174, Disponível em: http://periodicos.unifacef.com.br/index.php/facefpesquisa/article/view/73. Acesso em: 20 mai. 2010.

Lin, Y. H; Lee, P. C.; Chang, T. P. (2009), “Practical expert diagnosis model based on the grey relational analysis technique”, Expert Systems with Application, Vol. 36, pp. 1523-1528.

Lin, Y., Chen, M.; Liu, S. (2003), “Theory of grey systems: Capturing uncertainties of grey information”, Kybernets, Vol. 33, pp. 196-218.

Lins, M. E.; Meza, L. A. (2000), “Análise Envoltória de Dados e perspectivas de integração no ambiente do apoio à decisão”, COPPE/UFRJ, Rio de Janeiro, RJ.

Lins, M. P. E.; Almeida, B.; Bartholo Junior, R. S. (2004), “Avaliação de desempenho na pós-graduação utilizando a Análise Envoltória de Dados: o caso da Engenharia de Produção”, Revista Brasileira de Pós-Graduação, Vol. 1, pp. 41-56.

Liu, S; Lin, Yi (2006). Grey information: theory and practical applications. Springer, London, UK.

Lobianco, A. T. M.; Meza, L. A. (2008), “Uma proposta para determinação de rankings no ensino superior utilizando a análise envoltória de dados”, Anais do XL SBPO, João Pessoa, PB, 2-5 set., 2008.

Martins, C. B. (2002), “A Formação do Sistema Nacional de Pós-Graduação”, In: Soares, M. S. A. (coord.), A Educação Superior no Brasil, Coordenação de Aperfeiçoamento de Pessoal de Nível Superior, Brasília, pp. 70-87.

Mello, J. C. C. B. S.; Gomes, E. G.; Meza, L. A. et al. (2006), “Engineering Post-Graduate Programmes: A quality and productivity analysis”, Studies in Education Evaluation, Vol. 32, pp. 136-152

Mello, M. H. C. S.; Mello, J. C. C. B. S.; Meza, L. A. (2008), “Modelos DEA para avaliar a atratividade dos cursos de Engenharia da UFF”, Anais do XXVIII ENEGEP, Rio de Janeiro, RJ, 13-16 out. 2008.

Meza, L. A.; Gomes, E. G.; Neto, L. B.; Coelho, P. H. G. (2003), “Avaliação do ensino nos cursos de pós-graduação em engenharia: um enfoque quantitativo da avaliação em conjunto”, Engenvista, Vol. 5, pp. 41-49.

Miranda, C. M. G.; Almeida, A. T. (2004), “Visão multicritério da avaliação de programas de pós-graduação pela CAPES: O caso da área Engenharia III baseado nos métodos Electre II e MAUT”, Gestão e Produção, Vol. 11, pp. 51-64.

Montgomery, D. C.; Runger, G. C. (2003). Applied Statistics and Probability for Engineers. John Wiley & Sons, New York.

Moreira, N. P. (2008), Análise da eficiência dos programas de pós-graduação acadêmicos em Administração, Contabilidade e Turismo. Dissertação de mestrado em Administração, Universidade Federal de Viçosa, Viçosa. Disponível em: <http://www.tede.ufv.br/tedesimplificado/tde_ arquivos>. Acesso em: 07 mai. 2010.

Neves, R. B.; Costa, H. G. (2006a), ”Swot, electre tri e sistemas de avaliação da CAPES”, Anais do XXVI ENEGEP, Fortaleza, CE, 9-11 out. 2006.

Neves, R. B.; Costa, H. G. (2006b), “Avaliação de programas de pós-graduação: proposta baseada na integração Electre Tri, SWOT e sistema CAPES”, Sistema & Gestão, Vol. 1, pp. 276-298.

Sheskin, D. J. (2004). Handbook of Parametric and Nonparametric Statistical Procedures. (3rd ed.). Chapman & Hall/CRC, New York.

Wu, H. H.; Deng, J. L.; Wen, K. L. (1984). Introduction to Grey Analysis, Kao-L., Taiwan.

Zuo, F (1995), “Determining method for grey relational distinguished coefficient”, ACM SIGICE Bulletin, Vol. 30, pp. 22-28.

---

Received: 20 Aug. 2014

Approved: 20 Sep. 2017

DOI: 10.20985/1980-5160.2017.v12n4.806

How to cite: Garcia, P.A.A.; Duim, F.A.C. (2017), “A grey relational analysis based approach to the evaluation of Brazilian postgraduate programs in master of business administration”, Sistemas & Gestão, Vol. 12, No. 4, pp. 391-400, available from: http://www.revistasg.uff.br/index.php/sg/article/view/806 (access day abbreviated month. year).