CATEGORY 7: MEASURING EFFECTIVENESS

 

7P1. Selecting, managing, & distributing data and performance information to support programs & services

Data and performance information selection is driven largely by regulatory and reporting requirements, including: accreditations; internal and external audits; quality-related application and feedback processes; AQIP Action Project processes; HED and other state agencies; CUP Performance Effectiveness; Board of Regents reporting; budget preparation; and, most recently, development of the institutional scorecard, WNMU by the Numbers. All of these contribute to the ongoing review of organizational performance and capabilities. When the scorecard is fully developed in late 2009, key measures from each University division will link to strategies within the University’s strategic plan. Furthermore, these measures will be embedded in each vice president’s divisional operating plans. Selection of data and performance information will be based on priorities and data needs identified therein. Like the WNMU Strategic Plan, the Systems Portfolio, and QNM applications and feedback reports, the scorecard will be made available on the webpage.

 

Departments define data needs in conjunction with their assessment and program review processes, repetitive internal or external requests for information, and budget monitoring and performance evaluation responsibilities. Both institutional and departmental data may be stored in a hard copy file, Microsoft Office documents, on the shared drive, or – most commonly - as part of the student and administrative information system (Banner).

 

Uses of data vary by office. Academic units use summative and formative assessment data to better understand and support student learning, monitor budgets, determine class offerings, and advising assignments and loads. Student administrative support offices use data to monitor admissions inquiries, registrations, financial aid, and account balances. VP offices use data to support and monitor planning efforts, facilitate program development, and reallocate resources. The President and Vice Presidents use data, among other purposes, to measure institutional progress and performance.

 

7P2. Selecting, managing, & distributing data and performance information to support planning & improvement

As stated above, data and performance information to support planning and improvement are selected based on priorities and gaps identified in the SPP and by Action Project Teams, academic department assessments, committees, governance bodies and quality feedback reports.

 

For example, under Strategic Challenge 1, “Increase enrollment and student success,” Strategy 1.1 states: “Serve identified key markets and aggressively recruit and retain students from these markets, while remaining open to new opportunities as informed by University assessment methods and strategies.” Using this strategy as a framework, and using data collected from recruiters and various economic indicators in the Southwest, the Office of Admissions expanded its outreach program in four areas: southern California market; two-year college market in Arizona; two-year  colleges in New Mexico and El Paso; and the southern Colorado market. While enrollment is at an all-time high for Fall 2009, the number of students from these markets that will actually matriculate remains to be seen; however, data-informed recruitment efforts have paid off in the short-term.

 

Data distribution occurs in a number of ways, including divisional annual reports to the vice presidents, Assessment Convocation, Quality Council, Strategic Planning meetings, President’s Cabinet and VPs/SEAT meetings, and various other improvement-based, standing committee meetings such as Online Student Services and Online Policy Group. Meeting minutes are distributed via faculty and staff email lists and are filed in the Resource Room.

 

7P3. Determining data & performance information needs

Determining what data should be collected to support department or unit level needs derives largely from interactions unit leaders have with the various Councils and reports that units must file either for budget, program review, assessment, HED, IPEDS, or other internal or external reporting purposes. When repetitive data needs arise, the appropriate VP’s office usually sees to it that that data are included in existing standard data collection processes. The shared drive provides a storage mechanism that can easily be accessed by campus employees if data is not already a part of the Banner information system.

 

IT determines needs largely by asking users. The IT Director does this through discussions on campus, meetings such as President’s Cabinet and Academic Council, and through the SPP, outside conferences, and technology review publications.  Final determination must be based on alignment with Mission, Vision, and budget. Last year, faculty feedback to the IT Director did not indicate that wireless access was a priority; however, national trends in student preferences were very different, and – after attending a conference and networking with colleagues nationwide on this subject, the Director changed the wireless implementation plan on campus to expedite completion.

 

7P4. Performance data analysis & sharing

Organizational data collection, interpretation, integrity, and submission is overseen and largely conducted by the Director of Data Analysis and Research. The President’s Cabinet also provides a central coordination point for many activities related to analysis of data. Cabinet members’ reports update information and identify recommendations that affect a variety of performance measures ranging from enrollment data to financial and capital project status to information technology and athletic information.

 

Cabinet and the various council meetings are often the first place where an analysis is presented and discussed. Minutes of these meetings are sent to all faculty and staff via email. In addition, analyses are conveyed throughout the campus in a variety of ways including the shared derive, hard copy reports, committee and governance meetings, and other mechanisms identified in Figure 5P6.1.

 

7P5. Determining needs & priorities for comparative data

During the WNMU Decision Cycle, effort is made to identify appropriate comparative data needs. Often these information needs are augmented by the Office of Data Analysis and Research, or by committees, AQIP teams, and other groups working on a particular project or process and need comparative or benchmark data; departments preparing program reviews or accreditation reports; and units defending current or seeking increases in resource levels and newly mandated state or federal reports.

 

Criteria for the selection and use of external comparative data include similarity of organizations, cost of securing data, compatibility of definitional terms, best-in-class examples, relevance to institutional needs and priorities, and ease of access. Comparative data are used to provide input on setting performance standards, a context for data analysis, and identification of potential benchmarking institutions or practices. Where appropriate, comparisons to industry standards occur

 

We attempt to find peer institutions to compare with, although this can be problematic due to the University’s unique combination of environmental circumstances and roles within the community it serves. WNMU functions as both a regional university and a community college; furthermore, it is geographically remote and culturally diverse in both its student and employee populations.  Criteria for selecting peers generally must meet one or more (but rarely will they meet all) of the following: enrollment, budget, scope of academic programs offered, scope of technical/vocational/adult education programs and services offered, funding levels, and location.

 

7P6. Aligning department data analysis with organizational goals

Assessment reports for all academic programs are filed annually by each academic department with the Assessment Committee. These reports, which include department goals and their relationship to institutional goals, assessment methods utilized, data collected and analyzed, and the manner in which the results were used, are reviewed using a rubric developed and revised by the Assessment Committee. Reports are developed for each academic unit and a summary of all scores (0 to 4 indicating effectiveness of implementation of the unit’s assessment plan—non-existent, undeveloped, developed, established, exemplar) are provided to the department leaders and to the P/VPAA along with detailed comments based on the rubric from the reviewing team. Discussion of the reports is an agenda item for Academic Council and at subsequent department meetings. Results of 2009 academic department assessments are highlighted in Category 1.

 

7P7. Ensuring timeliness, accuracy, reliability, and security

The IT Department has primary responsibility for the information technology infrastructure at WNMU.  Faculty and staff provide inputs to ensure the organization remains current with information technologies in their disciplines, and students do the same through ASWNMU, the student government. Institution-wide technology concerns are also addressed in the SPP as a Strategic Challenge:  Sustain and improve a quality technology environment that supports students, faculty, staff, and communities in the use and value of instructional, administrative, and communication technologies.

 

Faculty make hardware and software requests on a regular basis.  Departments propose hardware purchases to the appropriate VP. Each semester the manager of the academic computing labs and classrooms asks faculty for a list of software to install on the lab and classroom computers. The IT Department and the academic departments negotiate the purchase of the software that is installed in classrooms and laboratories at the beginning of each semester.

 

Hardware reliability, security, and user-friendliness are ensured in the following ways:

§ IT staff review all purchase requisitions for computer hardware to ensure acquisition of compatible hardware that can be supported and maintained.

§ A risk-assessment document is maintained for the hardware components of the IT infrastructure.  Each component is coded according to its fragility and length of time it takes to resume function in the case of a breakdown (based on warranty, maintenance agreements, and redundancy). This document provides a means of prioritizing maintenance and upgrade of the network components to ensure overall reliability of the computing infrastructure.

§ Critical servers on the SC campus are housed in areas with card key locks, alarm systems and climate controls to ensure both reliability and security. 

§ The WNMU network is protected from attack from hackers by a firewall, internal partitioning, intrusion detection software and other software tools.

§ The Extended University employs a technical person at each center who coordinates local technical support with the ITD to ensure that WNMU standards, policies, and procedures are in place at these sites.

§ A computer hardware disposition policy in accordance with state law ensures that obsolete computing equipment is disposed of in an environmentally friendly manner consistent with state auditor rules/requirement.

§ Computer hardware is tagged and inventoried per WNMU policy and HED guidelines and are insured through State Risk Management against loss.

§ The Help Desk staff troubleshoots hardware and software problems over the telephone or in person.

 

WNMU ensures software reliability, security, and user friendliness in the following ways:

§ Centralized campus-wide licensing for software operating systems and applications.

§ IT Help Desk staff are trained in the use of and support of IT approved software packages.

§ Software for backing up servers is purchased and a schedule for backups maintained. 

§ Computer users are encouraged to perform data backups for personal workstations.

§ IT provides virus protection software and updates for all computers connected to the campus network.

§ A proxy server allows library users beyond the network to access electronic collections without having to use cumbersome usernames and passwords

 

WNMU protects the integrity of its data, information, and organizational knowledge through a data backup system that saves data to an alternative medium. The network firewall and virus protection software protect the network from external corruption.  Through network logins and user profiles, WNMU controls the privileges to view, revise, add, or delete shared information. WNMU ensures timeliness of data through the use of state-of-the art tools, such as SCT Banner, the Voyager integrated library system, Web page and portal software, and campus-wide licensing for a suite of office software. WNMU provides reports on schedule to its internal and external stakeholders, such as the reports to the HED. Hardware backups are also critical to supporting reliability of our information systems. WNMU replaces the Banner server every four years, keeping the institution on the “high reliability” side of the failure rate curve; in this case, we upgrade hardware faster than the state will allow us to depreciate it.

 

WNMU has established policies for the official repositories for different types of information in order to maintain the reliability of the data. The employee files maintained in the Human Resources Department, for example, are the official records for WNMU employees. The policy rules written into the student information database ensure reliability of student data. WNMU uses a variety of tools to ensure physical security, including limited physical access to repositories of key information. Only employees who need to enter those areas are given keys and/or codes. WNMU also uses environmental controls to protect data.  Buildings are maintained to minimize damage from fire and flood. Whenever possible, key information is made redundant using an alternative medium and saved in an off-site location.

 

WNMU achieves data accuracy by hiring qualified staff, training them to perform their jobs, incorporating data validation guidelines within key reporting fields, and supervising their work. WNMU follows FERPA guidelines in maintaining confidential information. For example, WNMU faculty is required to protect student grades by not posting them in a public place by the Banner-generated student ID numbers. User profiles determine the level of privilege a user has in viewing confidential information in the University databases.  Employees who only need to view finance records, for example, do not have privileges to view student records in the management information system.

 

 

7R1. Information system & knowledge management performance analysis

The IT Department uses a number of applications to monitor network, bandwidth, and application usage. The traffic profile for network use is looked at daily and various data bases are monitored to anticipate increased capacity needs. Banner query response time is also measured and addressed, as needed, particularly if IT receives a complaint. Figure 8R2.4 presents Information Technology Results including use of Banner/Oracle, telecommunications, servers, capacity, and IT support services.

 

7R2. Evidence that Measuring Effectiveness meets organization’s needs

With the creation of the Office of Data Analysis and Research several years ago, WNMU is in a much better position to gather, analyze, and learn from the information it can extract from Banner and other key data sources. The Data Integrity Team, headed by the Director of Data Analysis and Research, ensures that information submitted to HED, CUP, and other entities is accurate. Over the years this team has maintained a 99+ percent accuracy rating on data submitted to the New Mexico Higher Education Department.

 

WNMU’s information technology infrastructure allows collection of important data at the earliest point that it becomes available. Once collected, it is immediately available to analysts and end users. IT maintains highly reliable information storage systems that provide easy access for authorized data users, and does so with a budget that remains relatively flat while the organization grows.

 

7R3. Comparing results with other higher education organizations

IT does not conduct performance measurement for comparison purposes; rather, it looks at annual IT infrastructure dollars per student, an Educause benchmark. WNMU is currently maintaining a sound and secure IT system at slightly less than 50% of the national dollar-per-student average of Carnegie MA1 granting institutions, and we are approximately $180 below the New Mexico State University average of technology funding per student.

 

7I1. Recent improvements

Recent improvements include acquisition of Hotsos software, which will monitor Banner System 7 software and provide IT staff with information useful in optimizing that performance. Other network and usage monitoring tools have been added in the past two years, allowing for more focused analysis and troubleshooting. WNMU’s IT Department supports 99.9% up time with an aging infrastructure and very limited funding.

 

Other recent improvements include:

• Upgraded network access at learning centers: we now bypass core switches so that video signals aren’t as susceptible to jitter and delay, and connections between Silver City and the learning center sites were upgraded to ATM circuits, which are a synchronous, predictable technology that is a more robust transport system;
• Upgrade of lab & classroom computers from single- to multi-environment: IT now has the infrastructure to run PCs and Macs on the same hardware platform, improving performance and flexibility of labs and enhancing IT’s ability to respond to changes in instructional needs.
• IT recently completed the “WNMU IT Crisis Communications Plan.” This plan works in conjunction with the Public Relations/Marketing staff (“Director of Communications”) to ensure proper notification of stakeholders when an unplanned systems outage occurs.

 

 

 

7I2. Selecting specific processes & setting targets for improved performance results

The culture at WNMU determines selection of specific processes to improve through customer expectation and demands. For example, IT recently created a video overlay network to support ITV, which vastly improved image quality and overall performance of the system throughout all learning centers and receive sites. The ITV course delivery process is vital to our remote sites, and an improved process – with the ultimate target of no broadcast disruptions and good image quality – determined its selection.

 

Setting targets: Targets for improvement here as elsewhere may occur as a result of SPP actions, evaluation of the current measurement system, PMT and AQIP AP team recommendations, open campus discussions, or discussion in Cabinet, SEAT/VPs, or the various administrative councils.

 

Additional IT implementation priorities, including continued automation of online services, improved security, improved Help Desk performance, Extended University communications, portal and Web page improvement, general infrastructure upgrades and user training, have generally been met. Fine-tuning of automated online services needs to be done, and restoring an IT professional development training program, when resources allow, is a priority. Another IT target is to acquire new equipment in labs and classrooms every four years so that each student will see an equipment upgrade at least once during his or her tenure.

 

Current results and improvement priorities are made available to the campus through direct reports to Cabinet, various councils, governing bodies and their committees, as well as the common set of communication devices described in Section 5P7.1. The IT Director attends Student Senate meetings to report on IT priorities and accomplishments, especially with expenditures from student technology fees. In the case of measuring effectiveness, training will be another key way that communication of results and improvement will be reinforced.