Tuesday, January 16, 2007

Measuring Intranet Effectiveness

Many businesses have deployed intranet technologies for business- to-user information sharing, but few have investigated the effectiveness of this investment as demonstrated by South Africa's The Porter Motor Group.

Introduction

In Cape Town, South Africa, The Porter Motor Group, an automotive dealer network, recently set out to evaluate user perceptions of the intranet, and the results have provided some valuable insights for future development.
As IT becomes increasingly recognized as a service rather than application provider, there is a greater need to develop service and customer orientation. User surveys are valuable tools for gathering qualitative information in a way that can be quantified.
Porter Group used a survey to provide a means of gaining insights with a minimum of data capture or complex analysis, with the goal of better understanding its effectiveness for directing actions towards service improvement. Possibly more important is that the research is a broadly applicable method of measurement. It is not specific to the Motor Industry and could find applicability across a wide range of different companies where assessment of intranet service quality needs to be evaluated.

IT as a service organization

Measuring user perceptions to determine intranet effectiveness is the same as measuring customer satisfaction in any other service delivery environment. While customer surveys are common in many industries, the IT organization needs to adopt a similar, albeit internal, approach to understanding user requirements and expectations.
Understanding the business value of technology has been a concern within the Porter Motor Group for some time and this survey is the beginning of a process that will drive IT projects for closer alignment to business requirements and improved service quality.
The primary aim of the survey was to measure intranet effectiveness and to establish an overall service quality index for service delivery. Analysis explored to what extent user expectations and perceptions differed and whether the instrument provided a valid measurement.
A service quality survey is used to determine a ‘service quality index’ based on the assumption that Service quality (Q) is the difference between user expectations (E) and user perceptions (P). The questionnaire asked for responses to the variables’ importance and performance across different service dimensions to arrive at the index. According to the literature, service quality should at least meet expectations, and therefore, Q should be equal to or greater than zero as a negative position, means service expectations are not being met.

The basis of the research

This research was conducted at a time when the Porter Motor Group (PMG) was attempting to introduce Knowledge Management concepts and the redevelopment of the Intranet offered an excellent opportunity for measuring effectiveness.
The original corporate intranet PMGNET was introduced in 1999 and users had become familiar with the concepts. Its introduction represented a milestone in the company’s history and facilitated structural changes along with greater emphasis on efficiency and communication. PMGNET content was provided by a community of information providers publishing corporate material and general information that contributed significantly to the collaborative culture.

Recognizing the dimensions of service quality

The questionnaire was based on the well-established SERVQUAL instrument developed by Parasuraman et al (1985, 1988) using a set of items (dimensions) that reflect a model of perceived Service Quality. Statements are used to measure expectation and perception (experience) for different dimensions as shown below:
Reliability: Provide accurate information and consistency. Intranet is dependable with minimum downtime, content and information is up-to-date and accurate, Intranet look and feel is user friendly.
Responsiveness: Fast Website, easy access and willingness to provide help. E-mail address for support and suggestions, Intranet response times are fast and reliable, Intranet content publishers always open to suggestions.
Assurance: Credible information, secure and allows users to inspire trust and confidence. Intranet content is appropriate and relative to the PMG, marketing and sales performance statistics promotes user confidence, the company Internet and E-mail use policy is accessible.
Empathy: Caring communication provides individual attention to users. Intranet content is easy to understand, knowledge centre is assisting users to collaborate, share and exchange ideas, community agent is encouraging team building.
Tangibles: Appearance of the Intranet site. Information is found with minimum key-clicks, the Intranet home page is visually appealing, navigation is consistent.
Decision makers and those with access to the internal network and Intranet were the main target of this study. Each question provided a 7-point scale on which the respondents could assess importance and performance of different aspects of the service as well as some demographic items to allow for analysis. The response rate by organizational role is shown in Table 1.

Role in the Organization Respondents Response Rate


Overall user responses are shown in Table 2, and while measurement of individual service quality dimensions varied, all rated above the mid-point of the scale, but negative GAP scores revealed a general lack of satisfaction in the service. Analysis on the five service dimensions shows clearly that in all instances expectation (importance) exceeds perception (satisfaction).

Expectations Perceptions



What this means to the Porter Motor Group
The negative service quality index results indicate that the Intranet is under-performing across all service quality dimensions, and suggests that the Intranet fails to meet expectations. While users generally indicated that the Intranet is important they are disappointed with its performance. Based on these results it is clear that organisational user expectations and perceptions are significant and consequently affect success and acceptability of the PMG Intranet.

The results highlight a number of deficiencies that are negatively affecting overall Intranet effectiveness. Furthermore, the low Intranet performance index suggests that the executive management team needs to become actively engaged in determining content strategy and in formulating policy guidelines. As well, equal attention should be given to the current and future role of IT as facilitator and content provider. What is evident however is that an overwhelming majority of users consider the intranet to be an important business tool and an effective medium for sharing and publishing information.

Conclusion
Although corporate intranets are often cited in the literature, there aren’t many studies of intranet performance and few recommendations on how to manage and measure Intranet service quality. This study represents real world insight into the implementation of a reliable and valid survey instrument that may be beneficial in assessing and measuring the service quality and performance of an intranet.

The questionnaire proved sufficiently usable since the number of questions is relatively small and more likely to elicit response. The long-term intention is to repeat this process to create comparisons as a means of assessing the effectiveness of proposed developments. Who better to ask than the user?


http://www.prescientdigital.com/articles/intranet-articles/measuring-intranet-effectiveness/