An analysis of the position and level of education of the users in their evaluation of integrated management systems
Warley Wanderson do Couto1, Antônio Artur de Souza2, Ewerton Alex Avelar2, Eloísa Helena Rodrigues Guimarães1, Luísa Raad Gervásio2
1 Pedro Leopoldo Foundation; 2 Federal University of Minas Gerais
ABSTRACT
This article presents the results of a research that aimed at analyzing the influence of the job position and the level of education of the users in their satisfaction with integrated management systems (the enterprise resource planning - ERP). We collected data through semi-structured questionnaires applied to 115 ERP users in 13 different companies, and performed data analysis using the following techniques: descriptive statistics, Spearman correlation coefficient and Chi-Square Automatic Interaction Detector (CHAID). The results demonstrated that the role and position of users have a direct influence on the evaluation of ERPs. In general, the evaluation of the technical and assistant users presented statistically higher results than the evaluation performed by users in supervisory and coordination positions. We could also observe a significant influence of the level of instruction of the users in their evaluation of the systems, especially because employees with lower level of education tended to evaluate more positively the characteristics of the ERPs.
Keywords: Integrated Management Systems; Evaluation of Information Systems; Users’ Position and Level of Education.
INTRODUCTION
According to Moscove et al. (2002), in the last decades, Information Technology (IT) had as much impact on society as the Industrial Revolution did in the centuries that preceded it. The advancement provided to organizations by the IT area has contributed significantly to the expansion of the economy of formerly restricted markets. Fonseca et Rodello (2016) emphasize that IT investments have become essential for companies to survive in competitive scenarios. In this sense, Johansson et al. (2016) state that as IT continually plays a central role in business, its budget tends to increase.
In this context, information systems (ISs) emerge as a usually crucial factor in reaching and sustaining the competitiveness of organizations in the market (Couto et al., 2015). According to Góes (2007), these systems have allowed organizations to maintain and manage their businesses on a world scale, following the market progresses. In particular, the accounting and finance areas became fundamental to the achievement of organizational goals and targets, and have been making intensive use of computerized IS. Based on the development of ISs, these areas have particularly benefited from Enterprise Resource Planning (ERP) to quickly and accurately provide information to support managers in decision-making. According to Fernandes et al. (2017, page 58), the ERPs constitute
a resource which is increasingly used by organizations, especially when the solution providers’ arguments are the survival of the company and the tangible and strategic benefits they can achieve through its use.
Davenport (1998) states that ERP systems can be considered one of the most innovative developments in the use of information technology in the 1990s. In turn, Souza et Iaronzinski (2007) point out that these systems offer possibilities to streamline the flow of information in the organization through the integration between departments, also allowing the reduction of IT costs and the updating of operations on-line. In this sense, Baykasoglu and Gölcük (2017) state that such systems increase the knowledge-processing capacity of companies when used effectively. However, these authors emphasize that not all companies are able to implement ERPs satisfactorily.
Given the important role of ISs in the business environment, the evaluation of these systems is very relevant (Passolongo, 2004). Several studies have been carried out on evaluation of information systems, especially ERPs. Among the most recent, studies from the following researchers stand out: Souza et Iaronzinski (2007), Souza et al. (2009), Kale et al. (2010), Souza et al. (2010), Américo et al. (2011), Couto (2011), Couto et Cunha (2012), Souza et al. (2012), Couto et al. (2015), Fernandes et al. (2017).
However, most of the aforementioned studies are limited to assessing user satisfaction and perception regarding ERP. Such studies assume, implicitly, that users, regardless of profile, are competent to carry out such analysis and evaluation of the ERP, which may not be true. For example, it is not known exactly whether the job position and the level of instruction of the user may exert influence on the result of their evaluation of the system. Therefore, a positive evaluation of any system may not be enough to assert whether the ERP is meeting the objectives of the organization or not. It is not uncommon to identify users who, due to lack of knowledge of the full potential of ERPs, evaluate the system in a positive way so as not to compromise their image and position in the hierarchy of the organization. Given this, our study aimed to answer the following research question: “Does the position and the education level of ERP users influence their satisfaction with such systems?”
In view of the above, the general goal of the research described here was to analyze if the position and the level of education of the users influence their level of satisfaction in relation to the ERPs employed in their organizations. For this purpose, we outlined the following specific goals: (a) to verify if the ERP user’s position has a significant influence on the level of satisfaction with the system; And (b) to verify if the level of education of the user of the ERP has a significant influence on their satisfaction with the system.
This article is structured in five sections (beginning on this introduction). In section 2, we provide a brief review of the literature on relevant topics for the adequate understanding of this work. Then, in section 3, we present the methodology used in the research. Subsequently, the research results are presented and discussed (section 4). Finally, in section 5, we report the final considerations regarding the study, followed by bibliographical references.
LITERATURE REVIEW
Evolution of Information Systems (ISs)
We estimate that the use of computerized IS by organizations began in the 1960s (Stair, 1998). According to Memória (2010), the vast majority of systems were developed by internal teams, since access to other software was still very restricted. Seeking to improve organizational management, companies started the development of more comprehensive systems, which could control the processes of purchase and storage of materials. This leads to the creation of the technique called Material Requirements Planning (MRP), which, according to Franco (2005), was the first predecessor to the ERP.
According to Corrêa et al. (2009), MRP is a module based on the manufacturing decision of finished products and which calculates the manufacturing needs (e.g.: materials, time, release and expiration of production orders). Using information from the register of product structures and inventories, the system also allows the issuance of reports that facilitate the management of production processes. Franco (2005) recalls that, initially, the MRP was linked to planning and production control, especially of organizations with industrial characteristics. This system was able to process the production orders in an integrated way with the sales department, since this, when receiving sales orders, passed them to the production department through the MRP, already estimating the production time, number of employees and raw material required to meet each request.
In the 1980s, MRP-II emerged, surpassing the restricted view of materials and extending control to the factory floor and other areas linked to production (Breternitz et Galhardi, 2011). The MRP II evolved from the MRP original design, with added modules such as: master production scheduling (MPS), rough cut capacity planning (RCCP), capacity requirements planning (CRP), statistical factory control (SFC), purchase order control (PUR) and sales and operations planning (S&OP), as described by Corrêa et al. (2009).
According to Marques (2008), MRP-II increased efficiency in planning and control of production plans, making the system integrated and interactive. MRP II met the evolution of the operational research models of the time, minimizing costs, losses and time of service or maximizing economic and financial production results (Franco, 2005). MRP-II then became a new management tool, which offered a vast amount of information capable of supporting, although not fully, the managerial decision-making process over resources other than production.
With input from raw materials, production, products in stock, orders from the sales sector and other information already absorbed in MRP modules, it was fundamental to offer a module that would act integrated to the fiscal and accounting area of the organizations. With the MRP-II as a robust and able choice to offer greater agility and reduction of costs, the expression ERP began to be used to denominate a new system. Thus, according to Acar et al. (2017), the ERP can be understood as an evolution of the MRP and MRP-II concepts.
The current scenario shows that the tendency of softwares is to enable more and more mobility and agility in the generation of information, making it easier to make decisions. Among the most recent technological innovations, the Internet and ERPs were the ones that had the greatest impact in the areas of accounting and finance, as Wernke and Bornia (2001) point out.
Integrated Management Systems (Enterprise Resources Planning – ERP)
According to Couto et al. (2015), the trend towards an increase in the number of companies that aim for greater agility in the flow of information through the use of IT is not recent. However, nowadays, due to the increasing competition motivated by the globalization of markets and access to information for decision making, this has become extremely important for the sustainability and survival of many companies. Given the importance of IT for organizations, a perfect understanding of the IS concept is vital for achieving the best result from the deployment of an ERP in any organization. Researchers observed that the goal of an organization, when implementing an ERP, lies in the very system definition presented by Laudon et. Laudon (2001) and O’Brien (2004) as follows: the integration of the various components of the IS. We can notice that there is no evolution in the systems concept, but only the emergence of tools that work in an integrated way.
Stair (1998) states that the integrated working system would be the basic principle to the interaction between elements and components in order to achieve the goals of its implementation. In the view of Acar et al. (2017), this integration is increasingly important in today’s companies. Such authors assert that current organizations are usually understood in a broader view, which includes suppliers, distributors, and customers, engaged in processes that deal with goods, services and information. Thus, an ERP, in general, becomes an essential system for companies to manage not only their internal activities, but also their supply chain, through the identification, capture, integration and storage of information created through the execution of business transactions from all internal and external processes of the company.
In a general way, we may define an ERP system as software that can be installed in all sectors of the organization, from production to the human resources area, even if they are geographically distant. Generally, the system receives the data referring to the countless transactions carried out, storing them in a single database. This way it is possible to perform a subsequent query and start several transactions automatically, from the initial data entry. Acar et al. (2017) state that there are many vendors of ERP software in the market today, some of which are very expensive and encompass a number of modules, while others are cheaper and focus only on certain business activities.
Baykasoglu et Gölcük (2017) highlight that the proper implementation of an ERP can provide competitive advantages to the company, from significant improvements in its efficiency, productivity and quality. Johansson et al. (2016) point out that when ERPs emerged, they were seen as a source of competitive advantage for companies. However, over time, this type of IS has become some kind of a requirement for manufacturing companies to remain in the market.
According to Baykasoglu et Gölcük (2017), despite its potential benefits, companies may have serious difficulties in implementing an ERP. O’Brien (2004) indicates that other authors deal with the components related to such a system, leading to a reflection about the relevance of monitoring people as part of the implementation process of integrated management systems. It’s worthy to stress that if the individuals are not sensitized and convinced of the importance of the ERP to the organization, it can significantly jeopardize the whole efforts for implementation of the system.
The implementation of an ERP can positively or negatively influence the operation of an organization, due to the complexity involved in the entire implementation process, as well as to the use of the system on a day-to-day basis. According to Acar et al. (2017), about one-third of ERP deployments fail due to a number of problems. According to Scott et Vessey (2000), the problems associated with software deployment are neither new nor specific to systems operating in an integrated manner; However, ERPs have been credited for the poor performance of various organizations. For this reason, the selection and implementation of an ERP should be judicious and include several factors that can contribute to the generation of effective benefits for organizations. In this sense, Fernandes et al. (2017: 59) state that:
The implementation and use of ERP systems require a high level of maturity of the company in terms of organization, processes and management, which could be a barrier for medium and especially small enterprises, especially when the system use and benefits extend from strategic to operational activities.
Another critical stage is the parameterization of the ERP, because it is the moment in which the system will be developed, aiming to accurately answer the operational processes of the organization. In this step, the users standardize a series of information that, during ERP operation, will result in the effectiveness of the reports and in all the good functioning of the system. Corrêa et al. (2009) point out that the ERP parameterization stage is one of the most important and also the most neglected activity by organizations that adopt an ERP. For the authors, although the parameterization of the system is considered essential for good performance, the subject is not well approached either by the academic or by the practical literature represented by the manuals of manufacturers and suppliers. According to these authors,
(...) the manuals, for example, explain what the parameters are and how they affect the calculations the system will perform. However, they neglect the treatment of how the decision maker must take his specific reality into account to then define the values of the parameters of the system (p. 107-108)
Methodologies and evaluation models for systems
In addition to the importance of the factors mentioned for a successful ERP implementation, we also must note the relevance of the methods and models used for the evaluation of this type of system. Given this, the evaluation of the proper implementation of systems has become a very active field of research (Baykasoglu et Gölcük, 2017). According to Johansson et al. (2016), the goal of the various tools and approaches to assessing an IS is to increase knowledge about planned IT investments and to create a basis for better decisions. Fernandes et al. (2017) discuss the importance of this evaluation to consider from operational to strategic aspects in companies. On the other hand, Fonseca et Rodello (2016) emphasize that the related costs and benefits generated by ERPs in companies is not yet clearly defined, and so they are difficult to evaluate, as it involves financial and non-financial factors, as well as tangible and intangible aspects. Despite this limitation, it is possible to use models already applied in previous researches to measure the results reached both in the deployment and in the post-implantation period of an ERP.
For Zwass (1992), the implementation of an IS must be preceded by a certain expectation regarding the quality of the information generated by the system. Therefore, it is essential to determine what information will be required from the implementation of an ERP, as well as the quality expected from this information.
According to Laudon and Laudon (1999), the most important criteria to be observed in the evaluation of systems are the following: (a) high level of use of IS by users; (b) user satisfaction regarding the compliance of their expectations with the information provided by the IS; (c) positive attitude of IS users and IT staff; (d) achievement of established goals for operation/implementation of the system; and (e) the organization’s financial return from deployment (i.e., cost reduction and/or increase in sales and profits).
To evaluate the information generated by the IS, Zwass (1992) points out that the search for quality information starts from the premise that it is possible to highlight some basic attributes that define satisfactory quality information. The author suggests the following attributes: convenience, accuracy, precision, completeness, conciseness, relevance and appropriate form, as shown in Table 1 below.
Table 1. Generated information characteristics
Source: Based on Zwass’ study (1992)
Alter (1996) has developed a methodology for systems evaluation called Work-Centered Analysis (WCA), aiming to understanding a work system developed by an IS in a way to enable an organization to decide whether to create a new system or promote improvements in those already used. The main use of the WCA model proposed by Alter (1996) is to perform the IS analysis keeping into account the following concepts: (i) “clients” (internal and external) are the personnel who receive and use information and data provided by the system, as end users of the information provided by the ERP; (ii) “products” are the system outputs, or the information provided to end users in the case of ERPs; (iii) “business processes” are the stages/participants of a process or activities that include people, information and other resources that create value for internal or external clients; (iv) “participants” are the people who develop the operational routines in the system, i.e. the users responsible for IS data input, manipulation of information and generation of output reports in the case of ERP; (v) “information” is the data received, created or altered by the system; and (vi) “technology” are the technological resources the IS uses.
The WCA methodology also includes five analytical perspectives for the systems, namely: (a) “architecture” presents how the system used by the organization or proposed for deployment develops its operational routines, highlighting its components, the way they are arranged and how they interact; (b) “performance” proposes verification of how the system operates and whether its operation is taking place correctly; (c) “infrastructure” includes the resources on which the system depends and which it shares with other systems used by the organization; (d) “context” stands for the technical and organizational environment in which the system develops its operational activities, including shareholders, competitive and regulatory affairs external to the company, policies, practices and organizational culture; and (e) “risks” consist of predictable events which could result in system degradation or failure. The risks can be related to the following three main aspects: accidents and malfunction (e.g., bugs and human failures); computer crime (e.g., hackers, viruses, forgery, and unauthorized bank transfers); and design flaws.
There are many studies in the literature on IS evaluation. Cardoso (2001), for example, researched the use of SAP R/3 ERP in the financial area of a steel mill using the performance and risk perspectives as proposed by WCA methodology. The application of this methodology also allows an analysis of the characteristics and attributes of the ERP information mentioned in Table 1, taking into account the six elements included, such as clients, products, business processes, participants, information and technology.
In addition, Passolongo (2004) evaluated whether the financial information generated by the financial information systems of three different companies met the informational needs of the administrators. The study was also based on the models and concepts presented by Zwass (1992) and Alter (1996), taking into account the characteristics and attributes of the information. They concluded that the ISs analyzed did not meet the information needs for decision making and failed to present enough flexibility to change according to the expectations of top management of the organizations surveyed.
A. Souza et al. (2012) evaluated the satisfaction of users of hospital information systems from eight organizations with the application of Zwass (1992) and Alter (1996) models. The applied methodology allowed the analysis not only of ISs as systems, but also of the reason for their existence in the organization. Using the WCA model, they assessed the IS, the information generated, the activities and resources to create value for internal and external users. The results of the study indicated that, although analyzed ISs meet most of the assessed attributes, such systems can still be improved. The research also concluded that the satisfaction of the users in relation to the information generated is greater than in relation to the IS itself, due to the need for improvement. B. Couto et Cunha (2012) also analyzed if SAP R/3 ERP can meet users’ information needs and provide support to decision-making processes, using part of the methodology developed by Zwass (1992) and Alter (1996). The study included information from the users of two medium-sized companies and, through the applied methodology, it was possible to observe that, although the system may present limitations and difficult access to information by users, SAP R/3 meets information needs to support decision making in the companies surveyed.
Other models and methodologies can also be applied depending on the objectives to be achieved, such as critical success factors (CSFs) (Baykasoglu et Gölcük, 2017; Parhizkar et Comuzzi, 2017); economic analysis tools, such as return on investment (ROI), cost benefit analysis (CBA), or total cost of ownership (TCO) (Johansson et al., 2016); And real options (Fonseca et Rodello, 2016).
However, the difficulties in evaluating ERPs can be understood due to the characteristics of the system itself, according to Fonseca et Rodello (2016). Nevertheless, these same authors emphasize the importance of evaluating to some extent “the value provided by ERP systems, which are increasingly frequent in business scenarios and demand increasing investments” (Fonseca et Rodello, 2016, p.159). Thus, for the development of the study described here, we sought to use the models mentioned and detailed previously because of their practical and positive results tested in previous studies.
METHODOLOGY
We adopted a quantitative approach for the development of this research. According to Collis et Hussey (2005, p.26), the quantitative research focuses on the measurement of phenomena, involving the collection and analysis of numerical data with the application of statistical tests. We decided, in this quantitative context, to carry out a survey. For Collis et Hussey (2005: 70), the survey is a positivist methodology, in which a sample of subjects is removed from a population and studied for inferences about this.
The research presented here is also characterized as exploratory and descriptive in its purpose. Still according to Collis et Hussey (2005), the exploratory research is carried out based on a problem or question of research with few previous studies. Also as Tripodi (1975) and Beall (2002) apud Souza et al. (2010) state, the exploratory research has the purpose of formulating a problem or questions that may provide more information about a topic still under-addressed. This research is descriptive because it aims to describe perceptions, expectations and observations of people who operate the ERP implanted in the organizations studied in this research. Finally, Collis et Hussey (2005) point out that this type of research is also used to identify and obtain information about the characteristics of a given problem or issue.
Firstly, we performed a bibliographic research based on the access to electronic magazine sites, annals of congresses, ProQuest database (www.search.proquest.com), CAPES electronic journals portal (www.periodicos.capes.gov.br) and theses and online university articles databases. This study focused on articles, dissertations and recent theses on the topic addressed here.
Subsequently, we provided questionnaires to a total of 115 users of ERPs, from the accounting and finance sectors of their companies. Respondents, necessarily, had to be users of the system used by the organization under study. We selected 13 organizations that use different systems for company management, such as RM and Datasul (by TOTVS) and SAP R/3 (by SAP). Our choice of organizations and users was based on the ease of access to information through employees who had good contact with the research team. Thus, the research sample can be classified as non-probabilistic for convenience, as according to Alencar (2007).
We applied semi-structured questionnaires (adapted from Souza et al., 2010) composed of closed questions with a 6-point Likert scale to supervisors, managers, directors and other ERP users, and kept confidential the names of respondents and organizations surveyed when the research was published. Table 2 below presents the scale used for the analysis of the data of each of the sections of the questionnaire that in this research are referred as “analysis category”. We adopted a numerical score for each one of the answers of the questionnaire, and users interviewed indicated their level of agreement or disagreement in relation to the presented assertions.
Table 2. Adopted Likert scale and scores for analysis category I
Source: Authors’ study
The analysis category I of the questionnaire included seven parts, as presented in Table 3. The goal was to collect data and evidence from more than one source, so that, in the end, the answers could converge to a rich data set about the research question and capture the complexity of the context that surrounds it, as recommended by Pozzebon et Freitas (1997).
Table 3. Aspects related to the evaluation of the information system
Source: Based on research by Souza et al. (2012, p. 11)
Table 4 as follows presents the scale used for the analysis categories II and III of the research and presents the score attributed by the users, representing their level of satisfaction with the information obtained through the ERP and with the results offered by the system. The Part II of the questionnaire focused on user satisfaction with the information provided by the ERP, including ten attributes for evaluation. This analysis category is described in Table 5. In part III (Table 6), we aimed at verifying the extent of satisfaction about users’ expectations regarding the information provided by the system. For this, the users answered five categories of evaluation, according to the scale presented in Table 2.
Table 4. Adopted Likert Scale and scores for Analysis Categories II and III
Source: Authors’ study
Table 5. Aspects related to user satisfaction with information
Source: Based on research by Souza et al. (2012, page 12).
Table 6. User Expectation regarding Information Provided
Source: Authors’ study.
We conducted data collection between the first half of 2011 and the first half of 2012. We contacted the person in charge of each of the organizations by phone, in addition to contacts by e-mail, in order to increase the representativeness of the sample with the Inclusion of the largest possible number of answered questionnaires. The total number of responses obtained was high, considering a total of 115 questionnaires, from 13 of the 15 organizations invited.
We proceeded with the analysis of the data collected after its processing with the use of the Statistical Package for the Social Sciences (SPSS) software, version 20. After the grouping of the answers obtained by the questionnaire, we sought to identify the trend of the answers in each analyzed item through descriptive statistical parameters and histograms of distribution.
We applied the Kolmogorov-Smirnov (KS) test in order to identify the possibility of using parametric techniques for data analysis. The KS test is the usual way of comparing a common sample to the theoretical distribution and is a set of uniform distributions between 0 and 1. The results indicated that, of all variables analyzed, none of the data had a normal distribution.
Once we confirmed that the data collected during the survey did not characterize a normal distribution, we defined a nonparametric data analysis. According to Triola (2008), the non-parametric methods have the following main characteristics: they (i) can be used in cases of categorical data; (ii) apply to a variety of situations; and (iii) do not require normally distributed populations.
We performed the correlation of the attributes and aspects of the categories of analysis applying Spearman’s correlation coefficient. A correlation coefficient is a numerical measure that corresponds to the strength of the relationship between two or more variables representing quantitative data (Triola, 2008). The correlation can be defined as in the positive or negative direction, which can vary between +1 and -1 (Landis et Koch, 1997).
In order to compare the indices of ERP characteristic constructs (analysis category I), information provided (analysis category II) and user satisfaction (analysis category III) with the profile variables, we adopted the Chi-Square Automatic Interaction Detector (CHAID). This technique, proposed by Kass (1980), allows us to analyze the relationship between a dependent variable and others at a categorical or continuous level. The result is presented in the form of a “tree” in which the predictor variables are better associated with the dependent variable. The resulting subsets show greater homogeneity internally in relation to the dependent variable, with the greatest possible heterogeneity among the subsets formed. We set the criteria of division or grouping used in this technique at 5%, that is, the subsets have a relevant difference with 95% accuracy.
RESULTS
Data descriptive analysis
As explained in the previous section, we carried out field research between the first half of 2011 and the first half of 2012, consulting 115 users who participated in the process of ERP implementation in the 13 companies that were the object of the study. With a distribution of 115 users, 30.43% of respondents to the data collection instrument work in coordination position and 69.57% work as Technical Assistant. Table 7 below shows the distribution of respondents by company. It should be noted that companies received fictitious names to keep data confidentiality.
Table 7. Users by educational level/organization
Fonte: Authors’ study.
We then performed the distribution of the users, presenting the quantity and percentage of users by level of education in relation to the total of the sample and in relation to the total per company.
It is noteworthy that all the interviewees participated in the implementation of the ERP system operated by the organizations under study (TOTVS RM or Datasul, or SAP R/3). The distribution of the percentage of companies by ERP can be seen in Figure 1 as follows.
Figure 1. Percentage of organizations by ERP
Source: Authors’ study.
The process of deploying an ERP may become unable to deliver the intended results if it is not properly operated by its users. In this sense, the planning stage should predict the investment in training of the users involved in each process to be integrated. Marques and Lazzarini Neto (2002) admit that successful ERP deployment can be considered innovative when there is conciliation between human resources training and IT investment.
Influence of the position and level of education in the evaluation of the ERP
As previously seen, several studies such as the ones by Cardoso (2001), Souza et Iaronzinski (2007), Jesus et Oliveira (2007), Souza et al. (2009), Kale et al. (2010), Couto (2011), Américo et al. (2011), Couto et Cunha (2012), Souza et al. (2012) emphasize the complexity of ERP implementation and, at the same time, the relevance of the user, both in the implementation process and in the maintenance of information for the operation of ERPs. However, most of research studies raise doubts about the users’ ability to evaluate the ERP used by the organization, since they may not have enough technical and/or professional knowledge to criticize the results and offer an adequate evaluation of the integrated system.
In order to elucidate this question and to meet the general goal of this study, we aimed at evaluating the influence of the position and the level of education of the users when comparing the result of their evaluation in relation to the ERP implanted in the organization. We started by applying the correlation method between attributes and aspects evaluated in the five categories of analysis with the position/role occupied by each user and their education level.
In general, the results point out that, the higher the position occupied by the users, the greater the tendency to evaluate the attributes more negatively. In other words, auxiliary/technical operational level personnel showed evaluation scores superior to coordination staff’s results, such as the Interface attribute. When the same attribute was analyzed based on education level, it was also noticed that the higher the level of instruction of the user, the lower the evaluation of the attribute.
It is possible that the divergent result obtained by the analysis between the positions, as well as between the users’ different levels of education, is connected to the tasks developed by each of them. The access to the operational routines is much more utilized by the users who register the information on a day to day basis, that is, the occupants of the assistant/technical positions. Such information, with rare exceptions, is recorded by coordinating positions. These latter users are often tied to more strategic levels within organizations and use routines that consolidate data, generate information for decision making, and in some organizations may not dominate operational routines, a fact that may justify the presented results.
The general evaluation of the system also presented an adverse result among the positions, as the second most relevant result and also considered strongly significant. For this attribute, Table 8 as follows details the results presented previously, specifying the evaluation among the different positions and noting that the overall evaluation of the ERP is different among users of the operational and coordination levels in the studied organizations. The attributes flexibility and ease of access also presented significant and relevant results for the verification of the goals of this study.
Table 8. User evaluation by position/role for the general evaluation attribute
Source: Authors’ study
Flexibility is related to the users’ assessment regarding the needs for generating new information from the system. Certainly, reporting is one of those needs and, as previously observed, in the organizations surveyed, this is a coordination assignment. As already seen, reports presented by ERP are limited and, although the systems offer possibilities of making different reports, users require programming know-how or contracting additional services from system provider.
Regarding the ease of access, the evaluation by the users includes assessing how easy is to access the ERP, as well as to gather information to execute the tasks. The evaluation for this attribute presented divergent results according to participants’ job positions, which is related to the attributions that each of them carries out, as highlighted above. This fact can affect evaluation of this attribute by organization coordination department, because in case of any inconsistency in the information during report stage, the supervisor/coordinator is dependent on solutions that are at the Assistant/Technical level, which prevents greater agility in obtaining the information. Analyzing the results of the Easy Access attribute in correlation with the level of education of the users, we also observed a significant variation. We confirmed that the higher the level of instruction of the user, the more critical the evaluations of this attribute.
We used the CHAID technique in order to evaluate the position/role relation with the attributes of analysis category I. The results were statistically significant regarding the difference of averages, presenting a tree below that presents significant difference between the participants’ position/role and the attributes evaluated for the ERP (Figure 2).
Figure 2. Difference of averages for ERP characteristics regarding participant position/roles
Source: Authors’ study
Legend: Características: characteristics; nó: node; média: average; desvio padrão: standard deviation; previsto: estimated; cargo função: position/role; supervisão/chefia: coordination/management; auxiliar/técnico: assistant/technician
We identified that users who occupy a lower hierarchical level (corresponding to 69.57%) tend to assign a more positive evaluation to the characteristics of the ERP (mean equal to 3,566) and to the general evaluation of the ERP (average equal to 3.883) than the users in higher hierarchical levels (corresponding to 30.43%). These provided a more negative evaluation for the characteristics (average equal to 3,090) and for the general evaluation of the ERP (average equal to 3.358).
For analysis category II, the correlation results followed the same trend of disparity between the evaluations, considering the position/role and level of instruction of the users. The coefficient that showed the most significant distortion among the evaluation of the attributes was the one referring to the completeness of the information.
We could notice that there is a lack of knowledge in technical/assistant level users regarding the real informational needs for decision making, since a percentage above 63% presented an evaluation between “Agree” and “Completely Agree” about the ERP ability to meet the company needs of information. The second attribute with the highest contrast among evaluations was associated with how clear the reports are. The evaluation of the assistant/technicians presented higher average scores than the supervision/coordination professionals’ results.
The quality of the information generated by the ERP must be constantly monitored by top management, since the information supports decision making and depends on the efficiency of the processes and how they are registered in the system. In this case, the evaluation of the positions also presented divergence due to the user profile. A percentage above 76% of the technical/assistant users “Agree” or “Completely Agree” that the ERP presents information quality, while just 51.4% of the coordination users present this same opinion.
Regarding the difference in average scores, we observed that the users on lower hierarchical levels (69.57% of the sample) also attributed a more positive evaluation to the information provided by the used ERP (average equal to 3.980) than the users who occupy the highest hierarchical levels (30.43% of the sample). The latter presented a more negative evaluation (mean equal to 3.313). The results by level of education showed a tendency similar to that identified for positions, indicating that the higher the level of education, the greater the divergence in the results of the evaluations. Analyzing the results by level of instruction for the three attributes that presented the highest coefficient of inverse correlation between levels of instruction, we found that the relevance and completeness of the information, followed by data retyping, were the most significant aspects regarding education level (significance in less than 1%).
The “generated information relevance” attribute is evaluated differently depending on the activities developed: for the supervisory/coordination positions, the provision of information by the system is very important for the results related to the decision making based on the information generated by the system. The same is observed for the completeness of information and data retyping, the causes of the differences being the same already evidenced in the analysis of these two attributes according to respondents’ position/role.
Table 9. Correlation between participant’s position/role and level of education and the attributes of analysis category III
Source: Authors’ study
Notes: *significance at less than 5.0%; **significance at less than 1.0%.
Category III is related to the user’s expectation regarding the information provided by the system and also presented a difference between the evaluations according to the profile of the users, as demonstrated in Table 9. For this last category of analysis, the aspects correlated with the position/role were more contrasting than the ones regarding the different levels of education.
The aspect regarding “information meeting the needs” is better evaluated among the assistant/technical users than among those at the coordination levels. We understand that users of operational positions need the system to finish daily tasks (i.e., inputs of invoices, payments, bank reconciliations and accounting), for which they say they are satisfied. On the other hand, the goal of coordinators is to use the information registered by those professionals, unlike them, they are not satisfied. We conclude, therefore, that, for the latter category, the difference of opinions is directly related to the role the users perform in the organizations, since it is directly linked to the goal of these users with the use of the ERP.
Figure 3 below shows the results that were significant regarding the difference in average scores. More specifically, there is the “tree” with the relevant difference between the job position and the attributes related to user satisfaction. We identified that users on technical hierarchical level (69.57% of the sample) are said to be more satisfied with the ERP (average equal to 4,011) while users from coordination (30.43% ) presented a more negative evaluation (average equal to 3,535).
Regarding the analysis for the educational level, we confirmed the hypothesis that the higher levels of education are more critical in relation to the objectivity, clarity and understanding of the information generated, as can be seen in Table 10. We observed that the management/coordination staff presents greater understanding regarding the information generated by the system, leading to a better analysis and interpretation of data generated by the ERP.
Figure 3. Difference of averages for position/role and user satisfaction
Source: Authors’ study
Legend: Satisfação do usuário: user satisfaction; nó: node; média: average; desvio padrão: standard deviation; previsto: estimated; cargo função: position/role; p-valor ajustado: Adjusted P-Value; supervisão/chefia: coordination/management; auxiliar/técnico: assistant/technician
Table 10. User evaluation by level of education regarding clarity, objectivity and easy understanding of the information
Source: Author’s study
In summary, for the organizations object of this study, it is confirmed that there is a very significant influence regarding the job position and the level of education in relation to the evaluation of the user of the ERPs. The results presented here confirm that the higher positions present different ratings than that of the operational level personnel. The same trend is also observed in most cases when the analysis is done observing users’ different levels of education.
FINAL CONSIDERATIONS
Considering specifically the question of research, we conclude that there is significant interference of users’ position/role and level of education in their evaluation of the system, the information generated and the expectations of results obtained through the ERP. It became clear that positions at more strategic levels require a greater level of understanding and interpretation of the information generated by the system.
Most assistant / technical positions perform operational functions of recording, reviewing, and adjusting information and documents in the system. Generally, the reports used by the operational levels are of low complexity and do not require in-depth knowledge about accounting and finance. Thus, the evaluation of the attributes of this study by the assistant/technician users was generally superior to the evaluation by the participants at management/coordination levels.
Based on the data obtained, we also confirmed that the users’ position/role directly interferes with their ERP assessment. We were able to identify through the statistical methods applied in the data analysis that, in almost all the attributes and aspects evaluated, there was a distortion between the evaluations, and so we see as a relevant idea to separate the results taking into account the different positions/roles in the organizations.
We verified that the level of instruction of the user also exerts influence in the evaluation of the system, mainly because the employees with lower level of education tend to give better evaluations for each item. Explanations of this result lie in the resistance and fear of losing the job or being poorly evaluated in processes of salary increase or promotion. It is important to remember that higher levels of education may provide to the user a broader view of various issues involving the operation of an ERP. These issues include knowledge of the type of business, its structure, operational processes, approval flows, and legal, tax, financial and accounting issues. In other words, the level of education of the user can help the user to offer better results for the organization in the operation of the ERP.
We conclude that the results presented in this study can be considered relevant for future research about satisfaction evaluation of ERP users, since we observed the consolidated results analyzed tend to be overestimated and lead to a misinterpretation regarding benefits provided by a used ERP. On the hand, we could confirm that stratifying evaluations by job position/role and/or education level may lead to interpretations closer to the real results offered by this kind of system.
Finally, the research contributes to the study of the problems that involve the process of deployment and maintenance of ERPs, adding new points of view to the existing theories and practices. However, the conclusions of this study are naturally conditioned to the constraints of a non-probabilistic sample and do not allow statistical generalizations to all types of situations that involve organizations using ERP. Therefore, we suggest new research to be conducted in order to extend this study, either by contemplating new issues, expanding the studied sample, focusing on specific ERPs or analyzing other factors that interfere in the satisfaction of ERP users. It is necessary to consider the feasibility of segmenting the categories of analysis presented (and others that may emerge in the future) or providing higher specification for the group of respondents (position, role, time in the company), among other possibilities.
REFERENCES
Acar, M. F. et al. (2017). Knowledge management and ERP: Complementary or contradictory? International Journal of Information Management, 37, 703–712. Recuperado em 08 julho, 2017, de https://www.researchgate.net/publication/227987750_Integrating_Knowledge_ Management_and_ERP_in_Enterprise_Information_Systems.
Alencar, E. (2007). Pesquisa em turismo. Lavras: FAEPE/UFLA.
Alter, S. (1996). Information systems: a management perspective. (3a ed.). San Indiana: Addison-Wesley Education Publisher Inc.
Américo, E. O. et al. (2011). Implantação do sistema de informação contábil ERP em indústrias nordestinas: uma análise da percepção dos usuários após a implantação de um ERP em duas indústrias nordestinas. Revista de Administração, Contabilidade e Sustentabilidade, Campina Grande, v. 1, n. 2, p. 1-17, set./dez.
Baykasoglu, A., Gölcük, I. (2017). Development of a two-phase structural model for evaluating ERP critical success factors along with a case study. Computers & Industrial Engineering, 106, 256-274. Recuperado em 08 julho, 2017, de http://www.sciencedirect.com/science/article/pii/S0360835217300761.
Breternitz, V. J., Galhardi, A. C. (2011). Contribuições ao processo de seleção de sistemas ERP (Enterprise Resource Planning) para pequenas e médias organizações. Revista Eletrônica de Tecnologia e Cultura, Jundiaí, v. 3, n. 2, p. 8-19, jun./ago.
Cardoso, D. (2001). Avaliação do SAP R/3 como instrumento para a gestão financeira: um estudo de caso no setor siderúrgico brasileiro. Dissertação (Mestrado em Administração) – Centro de Pós-graduação, Universidade Federal de Minas Gerais, Belo Horizonte, MG, Brasil.
Collis, J., Hussey, R. Pesquisa em Administração: um guia prático para alunos de graduação e pós-graduação (L. Simonini, Trad.). (2a ed.). Porto Alegre: Bookman, 2005.
Corrêa, H. L., Gianesi, I. G. N., Caon, M. (2009). Planejamento, programação e controle da produção: MRP II/ERP: conceitos, uso e implantação: base para SAP, Oracle Applications e outros softwares integrados de gestão. (5a ed.). São Paulo: Atlas, 2009.
Couto, W. W. (2011). Contabilidade como fonte de informações para acompanhamento da gestão orçamentária. Revista Contabilidade e Informação, 14(35), 39-48. Recuperado em 25 março, 2012, de http://www1.unijui.edu.br/revistas.
Couto, W. W. et al. (2015). Um estudo sobre a avaliação de sistemas integrados de gestão na região metropolitana de Belo Horizonte-MG. GEPROS. Gestão da Produção, Operações e Sistemas, 10(1) 69-86. Recuperado em 08 julho, 2017, de http://revista.feb.unesp.br/index.php/gepros/article/viewFile/1166/632.
Couto, W. W., Cunha, G. A. (2012, novembro). Avaliação da satisfação dos usuários do SAP R3: estudo de caso em empresas de médio porte. Anais do Congresso Virtual Brasileiro de Administração. São Paulo, SP, Brasil, 23-25. Recuperado em 23 novembro, 2012, de http://www.convibra.com.br
Davenport, T. H. (1998). Putting the Enterprise into the Enterprise system. Harvard Business Review, Nova York, v. 33, n. 6, p. 121-131. Fernandes, R. B. et al. (2017). Impacto da utilização de sistemas de ERP em dimensões estratégicas de pequenas e médias empresas. Exacta, 15(1), 57-74. Recuperado em 08 julho, 2017, de http://www.redalyc.org/articulo.oa?id=81050129005.
Fonseca, G. L., Rodello, I. A. (2016) Proposta de uma estrutura conceitual para avaliação de sistemas. Revista Eletrônica Sistemas & Gestão, 11(2), 150-160. Recuperado em 08 julho, 2017, de http://www.revistasg.uff.br/index.php/sg/article/view/747.
Franco, C. F. Jr. (2005). E-business: internet, tecnologia e sistemas de informação na administração de organizações. (3a ed.). São Paulo: Atlas.
Góes, W. M. (2007). Análise dos sistemas de informação do Hospital das Clínicas da Faculdade de Medicina de Ribeirão Preto: rumo ao sistema de informação gerencial. Dissertação (Mestrado em Saúde na Comunidade) – Faculdade de Medicina de Ribeirão Preto, Universidade de São Paulo, Ribeirão Preto, SP, Brasil. Recuperado em 08 outubro, 2011, de http://www.teses.usp.br
Jesus, R. G., Oliveira, M. O. F. (2007). Implementation of ERP Systems: technology and people in SAP R/3 implementation. Revista de Gestão da Tecnologia e Sistemas de Informação, São Paulo, 3(3), 315-330. Recuperado em 08 junho, 2012, de http://www.periodicos.capes.gov.br
Johansson, B., et al. (2016) After a Successful Business Case of ERP – What happens then? Procedia Computer Science, p. 100, 383-392. Recuperado em 08 julho, 2017, de http://www.sciencedirect.com/science/article/pii/S1877050916323419.
Kale, P. T., Banwait, S. S., Laroiya, S. C. (2010). Performance evaluation of ERP implementation in Indian SMEs. Journal of Manufacturing Technology Management, Chandigarh, 21(6), 758-780. Recuperado em 16 junho, 2012, de http://search.proquest.com
Kass, G. V. (1980). An exploratory technique for investigating large quantities of categorical data. Applied Statistics, Witwatersrand, 29(2), 119-127.
Landis, J. R., Koch, G. G. (1997). The measurement of observer agreement for categorical data. Biometrics, Washington, 33(1), 159-74.
Laudon, K. C., Laudon, J. P. (1999). Sistemas de Informação: com internet (D. C. Alencar, Trad.). (4a ed.). Rio de Janeiro: LTC.
Laudon, K. C., Laudon, J. P. (2001). Gerenciamento de sistemas de informação (A. Oliveira, Trad.). (3a ed.). Rio de Janeiro: LTC.
Marques, D. M. N. (2008). Implantação de um sistema MRP em ambiente de produção enxuta com alta diversidade de componentes e sazonalidade. Dissertação (Mestrado em Engenharia de Produção) – Escola de Engenharia de São Carlos, Universidade de São Paulo, São Carlos, SP, Brasil. Recuperado em 24 março, 2012, de http://www.teses.usp.br
Marques, M., Lazzarini, S. Neto. (2002). Capital humano e TI gerando vantagem competitiva. Revista de Administração de Organizações, São Paulo, 1(1), 1-16. Recuperado em 10 junho, 2012, de http://www.scielo.br Memória, C. A. S. (2010). A influência do ERP nos ativos intangíveis de organizações de TI de pequeno e médio porte: um estudo de caso no Distrito Federal. Dissertação (Mestrado em Administração de Organizações) – Escola Brasileira de Administração Pública e de Organizações (EBAPE), Fundação Getúlio Vargas, Rio de Janeiro, RJ, Brasil.
Moscove, S. A., Simkin, M. G., Bagranoff, N. A. (2002). Sistemas de informações contábeis (G. G. Goldschimidt, Trad.). São Paulo: Atlas.
O’Brien, J. A. (2004). Sistemas de Informação e as decisões gerenciais na era da Internet (C. K. Moreira, & D. K. Moreira, Trad.). 2a ed. São Paulo: Saraiva.
Parhizkar, M., Comuzzi, M. (2017). Impact analysis of ERP post-implementation modifications: Design, tool support and evaluation. Computers in Industry, 84, p. 25-38. Recuperado em 08 julho, 2017, de http://www.sciencedirect.com/science/article/pii/S0166361516302548
Passolongo, C. (2004). Avaliação de sistemas de informações financeiras: estudo de casos múltiplos. Dissertação (Mestrado em Administração) – Programa de Pós-Graduação em Administração, Universidade Estadual de Maringá e Universidade Estadual de Londrina, Maringá, PR, Brasil.
Pozzebon, M., Freitas, H. M. R. (1997, setembro). Pela aplicabilidade – com um maior rigor científico – dos estudos de caso em sistemas de informação. Anais do Encontro Nacional da Associação Nacional de Pós-graduação e Pesquisa em Administração, Angra dos Reis, RJ, Brasil, 21.
Scott, J. E., Vessey, I. (2000). Implementing enterprise resource planning systems: The role of learning from failure. Information Systems Frontiers, Holanda, 2(2), 213-232. Recuperado em 27 outubro, 2012, de http://search.proquest.com
Souza, A. A., et al. (2009, abril). Hospital integrated information system. Anais do Scientific Committee for the 6. CONTECSI, São Paulo. SP, Brasil.
Souza, A. A., et al. (2012, maio). Evaluation of Hospital Information Systems: A Study of Philanthropic and Public Hospitals Located in Belo Horizonte/MG. Anais do Scientific Committeefor the 9. CONTECSI, São Paulo, SP, Brasil.
Souza, A. A., et al. (2010, maio). An Evaluation of commercial-off-the-shelf hospital information systems available in the Brazilian market. Anais do Scientific Committee for the 7. CONTECSI, São Paulo, SP, Brasil.
Souza, V. P., Iaronzinski, A. Neto. (2007, outubro). Sistemas de Informação e Metodologias de Implantação. Anais do Encontro Nacional de Engenharia de Produção, Foz do Iguaçu, PR, Brasil, 27.
Stair, R. M. (1998). Princípios de sistemas de informação: uma abordagem gerencial (M. L. L. Vieira, & D. C. Alencar, Trad.). 2. ed. Rio de Janeiro: LTC.
Triola, M. F. (2008). Introdução à estatística (V. R. L. F. Flores, Trad.). (10a ed.). Rio de Janeiro: LTC. Wernke, R., Bornia, A. C. (2001). Considerações sobre o uso de sistemas informatizados na contabilidade. Fae Business School, Curitiba, 4(2), 53-66. Recuperado em 26 fevereiro, 2012, de http://www.fae.edu
Zwass, V. (1992). Management information systems. Michigan: WCB.