Sunday, January 21, 2007

Measuring Intranet Effectiveness


In Cape Town, South Africa, The Porter Motor Group, an automotive dealer network, recently set out to evaluate user perceptions of the intranet, and the results have provided some valuable insights for future development.
As IT becomes increasingly recognized as a service rather than application provider, there is a greater need to develop service and customer orientation. User surveys are valuable tools for gathering qualitative information in a way that can be quantified.
Porter Group used a survey to provide a means of gaining insights with a minimum of data capture or complex analysis, with the goal of better understanding its effectiveness for directing actions towards service improvement. Possibly more important is that the research is a broadly applicable method of measurement. It is not specific to the Motor Industry and could find applicability across a wide range of different companies where assessment of intranet service quality needs to be evaluated.

IT as a service organization

Measuring user perceptions to determine intranet effectiveness is the same as measuring customer satisfaction in any other service delivery environment. While customer surveys are common in many industries, the IT organization needs to adopt a similar, albeit internal, approach to understanding user requirements and expectations.
Understanding the business value of technology has been a concern within the Porter Motor Group for some time and this survey is the beginning of a process that will drive IT projects for closer alignment to business requirements and improved service quality.
The primary aim of the survey was to measure intranet effectiveness and to establish an overall service quality index for service delivery. Analysis explored to what extent user expectations and perceptions differed and whether the instrument provided a valid measurement.
A service quality survey is used to determine a ‘service quality index’ based on the assumption that Service quality (Q) is the difference between user expectations (E) and user perceptions (P). The questionnaire asked for responses to the variables’ importance and performance across different service dimensions to arrive at the index. According to the literature, service quality should at least meet expectations, and therefore, Q should be equal to or greater than zero as a negative position, means service expectations are not being met.

The basis of the research

This research was conducted at a time when the Porter Motor Group (PMG) was attempting to introduce Knowledge Management concepts and the redevelopment of the Intranet offered an excellent opportunity for measuring effectiveness.
The original corporate intranet PMGNET was introduced in 1999 and users had become familiar with the concepts. Its introduction represented a milestone in the company’s history and facilitated structural changes along with greater emphasis on efficiency and communication. PMGNET content was provided by a community of information providers publishing corporate material and general information that contributed significantly to the collaborative culture.

Recognizing the dimensions of service quality

The questionnaire was based on the well-established SERVQUAL instrument developed by Parasuraman et al (1985, 1988) using a set of items (dimensions) that reflect a model of perceived Service Quality. Statements are used to measure expectation and perception (experience) for different dimensions as shown below:
Reliability: Provide accurate information and consistency. Intranet is dependable with minimum downtime, content and information is up-to-date and accurate, Intranet look and feel is user friendly.
Responsiveness: Fast Website, easy access and willingness to provide help. E-mail address for support and suggestions, Intranet response times are fast and reliable, Intranet content publishers always open to suggestions.
Assurance: Credible information, secure and allows users to inspire trust and confidence. Intranet content is appropriate and relative to the PMG, marketing and sales performance statistics promotes user confidence, the company Internet and E-mail use policy is accessible.
Empathy: Caring communication provides individual attention to users. Intranet content is easy to understand, knowledge centre is assisting users to collaborate, share and exchange ideas, community agent is encouraging team building.
Tangibles: Appearance of the Intranet site. Information is found with minimum key-clicks, the Intranet home page is visually appealing, navigation is consistent.
Decision makers and those with access to the internal network and Intranet were the main target of this study. Each question provided a 7-point scale on which the respondents could assess importance and performance of different aspects of the service as well as some demographic items to allow for analysis.