ࡱ> ` X(bjbj m $ PPPP +$Rv(yyy+++++++$z-h/=+ yy=+ R+N ++fp  ɇ6PND3 h+0+30G00 ymyyy=+=+ yyy+ \lf4 lf  Annex 3: Theory of Benchmarking for e-Learning A Helicopter Review of the Literature This paper describes the process and outcomes of a rapid exercise to establish the state of knowledge of benchmarking e-learning activity, with particular emphasis on UK HE institutions and e-learning. It poses the problem, describes the methodology used and comments on the main papers found and main agencies involved. Finally it draws some conclusions sufficient to start a desk exercise on benchmarking elearning in The University of Manchester. The paper represents work in progress, but in our view it was sufficient to provide a credible basis to a list of benchmarks required to start the benchmarking exercise in The University of Manchester and in MBS. 0. Introduction In particular, respondents emphasised the importance of benchmarking the present state of e-learning in the HE sector [HEFCE] In their e-learning strategy document published on 8 March 2005 ( HYPERLINK "http://www.hefce.ac.uk/pubs/hefce/2005/05_12/" http://www.hefce.ac.uk/pubs/hefce/2005/05_12/), HEFCE stated that: 31. We agree with the respondents to our consultation that we should know more about the present state of all forms of e-learning in HE. This is essential to provide a baseline to judge the success of this strategy. However, understanding HE e-learning is not just a matter for HEFCE. Possibly more important is for us to help individual institutions understand their own positions on e-learning, to set their aspirations and goals for embedding e-learning and then to benchmark themselves and their progress against institutions with similar goals, and across the sector. We have therefore asked JISC and the Higher Education Academy to take forward a project with a view to identifying a benchmarking tool for HEIs. This tool may also then provide information, at a sector-wide anonymised level, to help us and our partners draw conclusions on the state of e-learning, progress towards embedding it, and the impact of our strategy However, since the HEFCE e-Learning Strategy has only just been published at the time of writing this annex (early March 2005) there is, not surprisingly, little progress towards such a benchmarking tool. Nor is there any existing tool or even general methodology oriented to benchmarking e-learning in UK HE although there are some for UK FE, the corporate sector and US HE. Thus it might seem that there are several near-misses however, the UK HE sector is notoriously unwilling to learn from even near neighbours (geographically or sectorally) and many of the earlier tools were created for special purposes some time ago and look dated. Thus for this part of the MBS study I had had to fall back on first principles to create the benchmarking tool. Having reflected on the problem, a simple benchmark taxonomy table was constructed. Then in order to refine and justify it, I looked at related work on benchmarking in HE and to some extent in FE and e-learning in corporate training. I also looked briefly at work in other countries (Australia, US, Canada, Netherlands) where there are competitors to The University of Manchester. Because my methodology is close to what JISC and the HE Academy have done in related projects, I suspect that JISC and the HE Academy will follow a similar approach when they come to do their sector-wide benchmarking project So although I cannot give any guarantees, I believe that the work here will be not be difficult to map into any sector-wide approach. In order to encourage that convergence, I have made the methodology and conclusions in this paper available to the HE Academy and submitted a work in progress paper to a forthcoming UK conference. There is at this time no other work in the UK that I can find which is directly applicable. 1. Literature Search Methodology I am not an expert in benchmarking but have good knowledge of e-learning internationally and have researched and taught change management, business process re-engineering and activity-based costing in universities, as well as participating in the evaluation of the e-learning initiative across all English FE Colleges. Consequently I am fairly confident that I have assessed (even if many would feel only at a superficial level) many of the main reports and activities in this area, despite the very limited time available. In a nutshell, there is very little in the HE literature which provides specific guidance on which benchmarks are appropriate for e-learning or on the topic of carrying out benchmark activities in e-learning. There is some relevant material in FE but its applicability to HE is likely to be debatable even among experts and likely to be contentious to the UK HE sector, unless extensively modified. Nevertheless, the review of a range of reports on commercial and university benchmarking did produce some indications of what benchmarks might be considered important and some guidance as to procedure. Both these aspects are described below. Many, if not most, of the proposed benchmarks are qualitative not quantitative. There is some consensus that a Likert 5-point scale is the best to use to capture the ranking aspect of these. While this approach is enshrined in the research literature I have extended this to a 6-point scale in order to allow an element of exceeding expectations to take place which seems particularly apt in a post-modern context for a university such as The University of Manchester. This 6-point scale also allows easier mapping of some relevant criteria. For a speedy literature search I followed standard real-world extreme research assistant operating procedure by starting with a Google search on benchmarking AND e-learning and spreading out from that, using skill and judgement, making sure that agencies and countries were covered which were likely to have information on this topic or at least the topic of benchmarking. 2. Review of the Benchmarking Literature Benchmarking is used in many industries and organisations. The Public Sector Benchmarking Service ( HYPERLINK "http://www.benchmarking.gov.uk/about_bench/types.asp" http://www.benchmarking.gov.uk/about_bench/types.asp) is relevant at a general level, and has, among other things, a useful set of definitions. 2.1 Benchmarking in HE Benchmarking in UK HE The standard public web reference relevant to the UK is Benchmarking in UK HE: An Overview (by Professor Norman Jackson, a Senior Advisor at the Higher Education Academy. This is available as a link from the Benchmarking for Self Improvement page ( HYPERLINK "http://www.heacademy.ac.uk/914.htm" http://www.heacademy.ac.uk/914.htm) on the HEA web site. Professor Jackson has also edited (with Helen Lund) a book [BHE] with a similar title. Professor Jackson observes that the term benchmarking has a wide range of interpretations. However, he suggests that going to the original definition (by Xerox) seems useful: a process of self-evaluation and self-improvement through the systematic and collaborative comparison of practice and performance with competitors in order to identify own strengths and weaknesses, and learn how to adapt and improve as conditions change Jacksons paper then goes on to describe many examples of benchmarking activity in the UK. However, none is directly relevant to e-learning although a couple are about aspects of libraries there are none about information technology even in general. Despite this, I feel that his conclusions are relevant to our study. The first paragraph of the conclusions is particularly instructive: The HE context differs from the world of business in using benchmarking for regulatory purposes as well as for improvement. This fact is sometimes not appreciated by benchmarking practitioners outside HE who are primarily focused on business processes. The rapid growth of benchmarking in UK HE partly reflects a search for a more effective way of regulating academic standards in a diverse, multipurpose mass HE system and partly is a consequence of the increasingly competitive environment in which HE institutions operate, and a political environment that ensures that public resources are used as effectively as possible. It also reflects the political realisation that benchmarking has the potential to promote change in-line with a range of social and economic agendas. Benchmarking in Commonwealth HE The former Commonwealth Higher Education Management Service (CHEMS) produced in 1998 a magisterial report Benchmarking in Higher Education: An International Review ( HYPERLINK "http://www.acu.ac.uk/chems/onlinepublications/961780238.pdf" http://www.acu.ac.uk/chems/onlinepublications/961780238.pdf). CHEMS flourished from 1993 to 2001 for more on the history of CHEMS and a list of its publications see  HYPERLINK "http://www.acu.ac.uk/chems/" http://www.acu.ac.uk/chems/. The report had two overview chapters and then covered North America, Australia, the UK and continental Europe (focusing mainly on the German-speaking areas). The report is 80 pages long. Again, it contains little of specific relevance to the challenge, but there are a number of very useful observations that helped me to refine the process. For example, chapter 2 has two pertinent observations: the range of approaches and definitions [for benchmarking] may perhaps be viewed most simply as a continuum, with a data driven and non-process focus at one end, and conceptualisations which integrate benchmarking with TQM as part of coordinated process-driven quality improvement programmes at the other. Fielden (1997) supports some of these conclusions by observing that a common misconception is that benchmarking is a relatively quick and inexpensive process. Rather, he notes that the converse is true, and it will take considerable time from both senior and middle level staff in universities if frustration and failure is to be avoided. However, such factors important as they are appear generic to almost all types of change management, and it is difficult to identify many key implementation factors which do not also apply to TQM, the implementation of ISO 9001, and to other quality systems. A key European agency The European Benchmarking Programme on University Management is now in its fifth year of operation. It describes itself as follows: This Benchmarking Programme offers a unique and cost effective opportunity for participating universities to compare their key management processes with those of other universities. This will help identify areas for change and assist in setting targets for improvement. [ESMU] Operated by the European Centre for Strategic Management of Universities (ESMU,  HYPERLINK "http://www.esmu.be/" http://www.esmu.be/), it was launched initially with the Association of Commonwealth Universities, so is likely to blend elements of a European and Commonwealth tradition of management; and thus seems particularly apt for UK universities. A group affiliated to ESMU is the HUMANE group (Heads of University Management &Administration Network in Europe), to which several UK universities belong, including The University of Manchester. In 2003, one of the four topics benchmarked was e-learning. No further information is available on the site. The general methodology for the benchmarking process is described in a document [ESMU] at  HYPERLINK "http://www.esmu.be/download/benchmarking/BENCH_YEAR5_INFO_NOTE.doc" http://www.esmu.be/download/benchmarking/BENCH_YEAR5_INFO_NOTE.doc. The conclusions I have drawn from EMSU are oriented to their methodology, rather than to specific criteria. Activity in UK HE agencies HEFCE The term benchmarking does not appear as a term in the site index to the HEFCE web site at  HYPERLINK "http://www.hefce.ac.uk/siteindex/" http://www.hefce.ac.uk/siteindex/ but there are 100 hits on the site for the term itself. However, many of the hits are to do with finance (especially the Transparency Review) and general university governance issues. None are to do with elearning and almost none to do with teaching and learning. Nevertheless, there are eight tenets of good practice in the HEFCE e-learning strategy: a. ICT is commonly accepted into all aspects of the student experience of higher education, with innovation for enhancement and flexible learning, connecting areas of HE with other aspects of life and work. b. Due to more coherence and collaboration, technical issues have been addressed to give better value for money. c. Students are able to access information, tutor support, expertise and guidance, and communicate with each other effectively wherever they are. They are able to check and record their achievement in a form designed for multiple uses to enable personal and professional development. d. Tutors have tools for course design to enable better communication between them and their students, giving feedback and targeted support. Individual teachers have access to information about the materials available, and support for continuous improvement of them. e. Subject communities are able to share materials in ways that enhance their ability to produce customised high quality courses. They are supported to work collaboratively in designing materials, which are effectively quality assured and widely disseminated. They have access to research information to inform curriculum development and research-based teaching. f. Institutions are able to build appropriate infrastructure and resources support for integrating registration and learning functions. They have links with regional networks of institutions to support progression and community involvement. g. Lifelong learning networks support connectivity between institutions to provide seamless access for students and staff. h. Staff are supported at all stages to develop appropriate skills in e-learning, and these skills are recognised in their roles and responsibilities and in reward structures. They have access to accreditation for their level of skills and professional practice in linking learning technology with teaching. It is obviously wise to build these into our benchmarking. JISC The JISC strategy 200406 ( HYPERLINK "http://www.jisc.ac.uk/strategy_jisc_04_06.html" http://www.jisc.ac.uk/strategy_jisc_04_06.html) makes just one reference to benchmark. This is under Aim Two, To provide advice to institutions to enable them to make economic, efficient and legally compliant use of ICT, respecting both the individuals and corporate rights and responsibilities. The JISC response to the DfES e-learning consultation emphasises that benchmarking is important. In its response ( HYPERLINK "http://www.jisc.ac.uk/dfes_elearning.html" http://www.jisc.ac.uk/dfes_elearning.html) to Question 5 on proposed action areas it states: JISC believes that overall, the action areas will help to realise the vision. There are other important areas which have not been given as much detail within the strategy as they should merit: Focus on international benchmarking in order to ensure that the UK remains highly competitive and at the forefront of developments in e-learning technologies Additional to this, a search of the JISC site yields a number of hits for benchmark. However, only one seems relevant. Funding Call 4/03 The risks associated with e-Learning investments in FE and HE ( HYPERLINK "http://www.jisc.ac.uk/index.cfm?name=funding_4_03" http://www.jisc.ac.uk/index.cfm?name=funding_4_03) calls for a study which, among other things, should identify any additional research, guidelines, self-help guides, training, best practices and benchmark or analysis tools that should be considered by the JISC, its services, the funding councils or other organisations to improve the effectiveness of strategic and investment planning in this area. Nothing further is available on the JISC site. The Higher Education Academy The HE Academy has a page specifically on benchmarking (Benchmarking for Self Improvement,  HYPERLINK "http://www.heacademy.ac.uk/914.htm" http://www.heacademy.ac.uk/914.htm). It then links to a paper on this topic, Benchmarking in UK HE: An Overview, by Norman Jackson which was described above. This also has a useful history of the topic, and a bibliography. The majority of the other hits on the term are to do with subject benchmarking and therefore not relevant. But there are some hits from the HEFCE planning material for the e-University. These are described (briefly) elsewhere. It is expected by HEFCE that the Higher Education Academy will be doing work, in collaborating with JISC, on benchmarking of e-learning, at some point in the not too distant future. The Leadership Foundation Though focused on leadership rather than management, the Leadership Foundation for Higher Education ( HYPERLINK "http://www.leadership-he.com/" http://www.leadership-he.com/) launched in March 2004, might be expected to make some reference to benchmarking. However, there is nothing relevant to this study on their web site. HE agencies in the other UK home nations Regarding Scotland, there are 50 hits on the SHEFC web site for the term. Most are to do with governance or finance, as with England but there are a few comments of more general relevance. In Wales there are no relevant hits on the HEFCW part of the Education and Learning Wales (ELWa) web site; but there are some passing references on the FE part. English Universities Benchmarking Club The English Universities Benchmarking Club (EUBC,  HYPERLINK "http://www.eubc.bham.ac.uk/" http://www.eubc.bham.ac.uk/) is a group of eight mainly research-intensive universities set up to undertake benchmarking, with support from the HEFCE fund for Developing Good Management Practice. The University of Manchester is a member. It aims to develop a benchmarking infrastructure to support ongoing benchmarking activities within each member organisation, oriented to student-facing processes, and to develop a methodology that will be recognised as Good Management Practice by other universities. The benchmarking topic areas of the Club do not have much to do with e-learning specifically, and their approach seems to have a focus on numerical performance indicators, but it would be useful to keep in touch with the Club and in particular to monitor the methodology and software tools used. Consortium for Excellence in Higher Education The Consortium for Excellence in Higher Education ( HYPERLINK "http://excellence.shu.ac.uk" http://excellence.shu.ac.uk) was established to evaluate the benefits of applying the European Foundation for Quality Management Excellence Model to the Higher Education Sector. The consortium was founded by Sheffield Hallam University, and original members included the Universities of Cranfield, Durham, Salford and Ulster. The Excellence Model is getting a lot of attention in a few universities, but not in most others. In addition, it is not clear as yet that it has much specific relevance to e-learning in particular, a perusal of the paper abstracts at the last conference of the Consortium (Mirror of Truth, held in June 2004 at Liverpool John Moores University) did not yield any references to either benchmarking or e-learning. Having said that, the general idea of excellence and particular approaches to fostering and measuring it is of great interest to universities and this particular approach should be kept under review. HE in Australia Uniserve Science, a development agency based at the University of Sydney, published in 2000 a 177-page manual Benchmarking in Australian Universities ( HYPERLINK "http://science.uniserve.edu.au/courses/benchmarking/benchmarking_manual.pdf" http://science.uniserve.edu.au/courses/benchmarking/benchmarking_manual.pdf). This was done under contract to the Australian Department of Education, Training and Youth Affairs (DETYA). While containing little of detailed relevance to e-learning, its tone is enabling rather than prescriptive and it seems (not surprisingly) to have a good understanding of the nature of a university and why it is unlike a business or government agency. In addition, it makes a number of detailed points which will assist people in devising an appropriate methodology for e-learning benchmarking activities, especially beyond the first desk research stage. Information Technology The following describes benchmark 5.14 and the 5-point scale to judge the level of an institution on it: Benchmark Rationale: Information Technology and Telecommunications are integral to the operation of a modern international university. For a university to be world-class its IT & T must at least sustain that status. Access to efficient, networked computing facilities, including access to university-wide information services (e.g., University web sites), and to the Internet, are aspects of the reasonable infrastructure expectations of staff members and students. The efficiency of those services is best measured in terms of availability and reliability. Complementary staff competencies are required for the services to be efficient. 12345IT & T agenda not fully worked out. Resource allocations ad hoc. 60% all staff and research students have access to the network from their work areas. Network arrangements provide only minimal research assistance. All students have teaching laboratory access to the network. Minimal provision of access to the network from off-campus. Network access is available 90% of the time. Re-engineering, and disaster management and recovery planning rudimentary. 60% of staff and students have the skills training/knowledge appropriate to their use of the network. Student acquisition of skills and training largely on own initiative. No planned programme for development of staff skills and knowledge.80% of staff and research students have dedicated access to the universitys network from their work areas. An IT & T agenda comparable to other universities. Substantial resources allocation. Network arrangements improve access to research information. All students have access to network from teaching and general-access laboratories. All staff and 50% of students have off-site access to the network. Network access is available 95% of the time. Effective planning, re-engineering, and disaster management and recovery practices. 80% staff and 70% of students possess the skills/knowledge appropriate to their use of the network. Staff training and development programme identifies skills required by staff members. Range of training and awareness opportunities provided to students. Annual evaluation of staff performance includes identifying training requirements.An IT & T agenda to give the university competitive advantage. Resources match the IT & T agenda. All staff and research students have dedicated access to the universitys network from their work areas. Network arrangements increasingly facilitate research outcomes. All students have access to the network from teaching and general access laboratories. All staff and students have off-site access to the network (whether or not they use it). Network access is available 99% of the time through effective planning, re-engineering, and disaster management and recovery practices. All staff and students possess the skills/knowledge appropriate to their use of the network. Staff training and development programme identifies skills required and ensures acquisition by appropriate staff members. Student skills training incorporated in the curriculum. Regular evaluation of staff performance and training requirements.This material is relevant to the benchmarks needed for e-learning. The other benchmarks do not add anything in the area of e-learning. Universitas21 Universitas21 is a global consortium of elite research-led universities and one of their areas of interest and activity is e-learning. There are some references to benchmarking; but not in e-learning. The omission of anything on learning and teaching is a little surprising, in view of the focus given to that in earlier announcements, and strong interest in e-learning from some members in particular, note the following from the University of British Columbia description of U21 at  HYPERLINK "http://www.ubcinternational.ubc.ca/universitas_21.htm" http://www.ubcinternational.ubc.ca/universitas_21.htm: establishment of rigorous international processes for benchmarking in key strategic areas of academic management, research, teaching, and learning. Benchmarking in HE e-Learning There is very little of direct relevance. However, there is much of more general relevance. Europe Coimbra Group The Coimbra Group ( HYPERLINK "http://www.coimbra-group.be" http://www.coimbra-group.be) is a group of around 30 high-ranking universities from across Europe. Members of the Coimbra Group ( HYPERLINK "http://www.coimbra-group.be/06_members.htm" http://www.coimbra-group.be/06_members.htm) include several institutions on the competitive radar of The University of Manchester. In the UK, Bristol, Cambridge, Edinburgh and Oxford are members. In early 2002 a survey of e-learning activity across the Coimbra Group was carried out under the leadership of Jeff Haywood (Edinburgh). A summary of the findings can be found at  HYPERLINK "http://www.coimbra-group.be/DOCUMENTS/summary.doc" http://www.coimbra-group.be/DOCUMENTS/summary.doc. This does mention benchmarking, in passing. Some evidence of further plans in this area is contained in other documents, in particular  HYPERLINK "http://www.coimbra-group.be/DOCUMENTS/portfolio.doc" http://www.coimbra-group.be/DOCUMENTS/portfolio.doc. Thus it is worth extracting relevant benchmarking criteria from the material. Looking at the survey, the following points come to mind: An important criterion on a 5-point scale is Q1. What is the position of e-learning in your universitys strategic planning, at central and faculty levels? The next question Q2. What developments do you expect in e-learning in your university in the next 5 years? cannot be a criterion as such, but we can turn it into a criterion on technology/pedagogy foresight. The next few questions do not turn readily into criteria, but the ones on collaboration suggest that this criterion is added. The Groups e-learning activities are still active two workshops Quality in eLearning and Open Source/Open Standards are being held in Edinburgh in March 2005. HE in the United States In her magisterial report Distance Learning: A Systems View, Rosemary Ruhig Du Mont described a range of benchmarking activities in the US relevant to e-learning: A number of research projects have focused on identifying the range of online student services needed to support students at a distance. In 1997 the Western Cooperative for Educational Telecommunications (WCET) received funding from the Fund for the Improvement of Post Secondary Education (FIPSE) to help western colleges and universities improve the availability and quality of support services provided to distance education students. One of the significant products to come out of the project was a report summarizing the student services being provided to distance education students by institutions of higher education (Dirr, 1999, Putting Principles into Practice). Also in 1997, the American Productivity and Quality Center (APQC) collaborated with the State Higher Education Executive Officers (SHEEO) to produce a comprehensive summary of best practices, Creating Electronic Student Services. In 1999, IBM and the Society for College and University Planning (SCUP) sponsored another benchmarking series of best practices case studies (EDUCAUSE, Institutional Readiness, 2001). WCET received a follow-up grant in February 2000 under the auspices of the U.S. Department of Education Learning Anytime Anywhere Partnership (LAAP) program. The grants purpose was to develop online student services modules and a set of guidelines for other institutions to use (Western Interstate Commission for Higher Education, 1999, p. 2; Krauth and Carbajal, 1999). The Guide to Developing Online Student Services, the final product of the LAAP grant, is intended to help higher education institutions develop effective online approaches to delivering student support services (Krauth and Carbajal, 2001, p. 1). The most recent compilation of best practices in online student services comes from the Instructional Telecommunications Council, which in 2001 published a volume on student services in distance education (Dalziel and Payne, 20001). The partnership between APQC and SHEEO actually started in 1996. The study referred to above (Creating Electronic Student Services) was the first such study. A second followed in April 1998. The web site takes up the story: The second study from the APQC/SHEEO partnership, entitled Faculty Instructional Development: Supporting Faculty Use of Technology in Teaching, began in April 1998. Dr. Tony Bates from the University of British Columbia provided content expertise throughout the course of the study. The purpose of this multi-organization benchmarking study was to identify and examine innovations, best practices, and key trends in the area of supporting the use of technology in teaching, as well as gain insights and learnings [sic] about the processes involved. The goal was to enable participants to direct their own faculty instructional development processes more effectively and identify any performance gaps. The study afforded participants the opportunity to gain a better understanding of issues and challenges involved in implementing technology-based education. A set of 14 key findings emerged from this study. The following quotes them in full: 1. Organizations that are responsive to their external environments are drawn to technology-based learning solutions. 2. Many best-practice organizations take a total immersion approach to technology involving the entire community of teachers and learners. 3. Best-practice organizations keep their focus on teaching and learning issues, not the technology itself. However, faculty members must reach a minimum comfort level with the technology before they can realize the deeper educational benefits. 4. There are no shortcuts; best-practice organizations provide sufficient time for planning and implementation of technology-based teaching initiatives. 5. Curriculum redesign is not taught to faculty members but rather emerges through project-oriented faculty development initiatives. 6. Faculty incentives come in many forms. Among the most powerful motivators is newfound pride in teaching. 7. A project team approach can produce a high-quality product and provide the faculty relief from technology overload. 8. A variety of departments coordinate instructional development services. Centralized structures and funds support overall organizational strategies, and decentralized structures and funds support just-in-time technical assistance. 9. Best-practice organizations have steadily moved toward strategic investments and firm criteria for funding projects. 10. Best-practice organizations do not wait for or depend on external funding for their faculty instructional development initiatives. 11. Faculty spokespeople and mentors are critical to effective dissemination strategies. 12. Effective partnerships for instructional development can leverage resources and improve quality. 13. Best-practice organizations use faculty and student evaluations to adjust instructional strategies. 14. Most best-practice organizations have not attempted to justify technology-based learning on the basis of cost savings. Improvements in learning effectiveness, relevance for the workplace, and widening access have been the key motivators. The work was done some years ago and some may find many of the results now rather obvious. However, I feel that that several of them help in formulating appropriate benchmarks. Canada The Commonwealth Benchmarking report of 1998 concluded that in the case of Canada, various institutional, political and union pressures had meant that there had been little progress on this topic. I carried out a Google search on benchmarking AND e-learning for material in the last 12 months it came up with nothing directly relevant. This seems to confirm the theory that benchmarking is still not seen as a Canadian priority. Netherlands The Netherlands is a country that the UK e-learning and networking community JISC, ALT and UKERNA especially look to as a source of ideas and cooperation. This is true even though the organisation of universities is still much under the control of the state and the ideas of competition much less developed. I carried out a Google search on benchmarking AND e-learning for material in the last 12 months it came up with nothing directly relevant. This seems to confirm the theory that benchmarking is still not seen as a Dutch priority either. 2.3 Benchmarking in Education outside HE Learning and Skills Council (England) The Learning and Skills Development Agency produced in 2002, under a grant from the Learning and Skills Council, a benchmarking guide Benchmarking for the Learning and Skills Sector. This is available at  HYPERLINK "http://www.nln.ac.uk/lsda/self_assessment/files/Benchmark.pdf" http://www.nln.ac.uk/lsda/self_assessment/files/Benchmark.pdf. It contains nothing specific to e-learning, but is full of useful information on benchmarking techniques and processes. Indeed, a combination of this manual and the Australian benchmarking guide would provide a very useful handbook for any future e-learning benchmarking exercise in UK HE provided that there is agreement on the benchmarks. National Learning Network As it says on their web site ( HYPERLINK "http://www.nln.ac.uk" http://www.nln.ac.uk): The national learning network (NLN) is a national partnership programme designed to increase the uptake of Information Learning Technology (ILT) across the learning and skills sector in England. Supported by the LSC and other sector bodies, the NLN achieves this by providing network infrastructure and a wide-ranging programme of support, information and training, as well as the development and provision of ILT materials for teaching and learning. The initiative began in 1999 with the aim of helping to transform post-16 education. To date, the Governments investment in the NLN totals 156 million over a five year period. Initially for the benefit of further education and sixth form colleges, the NLN programme of work is now being rolled out to workplace learning and Adult and Community Learning. Evaluation of the National Learning Network has been carried out in several phases by a team consisting of the Learning and Skills Development Agency and Sheffield Hallam University, with the assistance and support of Becta. (I declare an interest in having led the SHU team for the first three years of its life.) This work has generated a wealth of information including material highly relevant to the benchmarking of e-learning. In particular, there is a self-assessment tool to allow institutions to judge the extent to which they have embedded ILT into their operations. (Note that ILT in FE means essentially the same as ICT in HE.) The Guidelines for this tool ( HYPERLINK "http://www.nln.ac.uk/lsda/self_assessment/files/Self_assessment_tool_Guidelines.doc" http://www.nln.ac.uk/lsda/self_assessment/files/Self_assessment_tool_Guidelines.doc) describe it as follows: The ILT self-assessment tool has been developed from the FENTO ILT standards by the Learning and Skills Development Agency as part of the NLN initiative. It enables institutions to measure the extent to which they have embedded ILT into teaching and learning and to identify priorities for development Use of the tool will also support a valuable sector benchmarking exercise to obtain baseline data on the embedding of ILT into teaching and learning. The tool uses a 5-level classification of the level of ILT use. This is based on early work by MIT on the adoption of IT by companies, which was further refined by Becta. The model has levels as follows, from least to greatest use of ILT: Localised Coordinated Transformative Embedded Innovative Further information on this can be found in the Guidelines, but considerably more detail can be found in a chapter on the CITSCAPES Developmental Tool ( HYPERLINK "http://www.citscapes.ac.uk/products/phase1/ch10.pdf" http://www.citscapes.ac.uk/products/phase1/ch10.pdf), a version of the tool oriented to development of student skills in ICT. There is a translation of the levels into specific features. This is given on the next page: 5 LOCALISED4 CO-ORDINATED3 TRANSFORMATIVE2 EMBEDDED1 INNOVATIVEStrategic managementResponsibility for ILT delegated to identified staff.A co-ordinated approach to ILT development encouraged and supported.Staffing structure reviewed and appropriate new posts created and supported by senior management.Ensures that ILT is used across the curriculum and for management and administrative applications. Significant strategic commitment to use of ILT in learning.ILT managementTakes place mainly in isolation with little co-ordination of ILT across the institution.Central IT management function identified. Management involved in curriculum development to co-ordinate ILT practice across the institution. Contributes to planning of staff development.Acts as a catalyst for change. Management takes account of current applications of ILT in education. Supports the development of differentiated learning programmes through ILT.Monitors and supports ILT integration across the curriculum. Able to advise on models of good practice and innovation.Learning resources managementLearning resources are managed without reference to ILT resources.Senior member of staff has overall responsibility for all learning resources. Learning resource and ILT management are co-ordinated.Learning and ILT resource provision co-ordinated and integrated.Learning resources are available in a range of formats and locations to provide support for a range of needs.ILT strategyStrategy not developed but some staff, or departments, are integrating ILT in their schemes of work.Draft ILT strategy in place which bears reference to the overarching college mission. Extent of ILT use identified and recorded. Full inventory of resources available.Staff actively contribute to process of updating and expanding existing ILT strategy and to its implementation in the curriculum.ILT strategy takes account of changes in teaching and learning styles arising from the potential of ILTs exploitation.Staff developmentIndividual training for personal development is provided on an ad-hoc basisA co-ordinated approach to generic IT training e.g. spreadsheets, word processing, databases. Recognition of additional skills to support the integration of ILT in the curriculum.Curriculum- and MIS-based ILT training for most staff by internal and external trainers. Appropriate training for non-teaching staff. Recognition of new skills needed to facilitate changing teaching and learning styles.ILT is integrated intuitively into all areas of the work of the college. Staff take responsibility for identifying their own staff development needs. Staff trained in tutoring and timely intervention.Integration of curriculum and administration dataLimited ILT use in curriculum and in administration. MIS used for administration.Staff recognise the value of ILT in handling administration and curriculum data.Outputs used to support planning and decision making.Staff systematically use ILT systems to generate curriculum and management information. Flexible course delivery using ILT appropriately.Teaching and learning stylesIndividual tutors and learners explore the potential of ILT in learning in an ad-hoc way.ILT used to support and enhance existing teaching and learning practice across the institution.New, ILT-based approaches to teaching, supporting a range of learning styles, incorporated into curriculum planning, strategy and practice.Tutors recognise ILTs power to encourage higher order skills, e.g. problem solving. Suitable uses of ILT incorporated into learning strategies.Learner IT skillsSome staff exploit learners basic IT skills but with little attempt to integrate ILT into learning and assessment process.Curriculum areas provide contexts for the development of IT skills and their assessment. Generic skills may be developed through IT courses.Staff acknowledge high level of learner IT skills and devise appropriate learning situations which reflect and allow development of those skills.Learner use of IT is appropriate in the context of their learning experience and its application is regularity re-evaluated.Technical supportTechnical support sporadic and unreliable. No systematic procedures in place.Centrally managed and co-ordinated technical support. Support request, fault reporting, etc. procedures clearly defined.Non-academic support staff available to support student learning and staff development activities.Technical and learning support rotes have evolved to encompass developmental and advisory activities Efficient, client -driven resource deployment.FundingIT is funded on an ad-hoc basis.Centrally co-ordinated funding of IT through a single budget holder. ILT funding co-ordinated.Staff development represents a significant proportion of overall ILT funding programme.Innovative methods of funding ILT developments are explored and exploited. Physical resourcesIndividual departments control and explore potential of ILT resources.Provision of ILT facilities is centrally funded and co-ordinated. Provision recognises the importance of non curriculum-specific applications of ILT in the learning process.A mixed economy of provision leading to resource areas being developed throughout the institution, e.g. ILT in science or art and design areas.Open-access to ILT resources which are increasingly used for flexible and independent learning.External linksInformal links developed by individual departments that exploit ILT resources and/or expertise of commercial, industrial, academic and other institutions.The institutions links with external agencies centrally co-ordinated. Links regularly reviewed and considered for mutual benefit.Impact of external links on curriculum focus. The community and other external agendas provide support, e.g. local employers contribute to curriculum review and development.Contact with the external agencies influences the development of the institutions thinking on the educational use of ILT.Focus on community improvement through education.Record keepingIndividuals or departments use ILT for simple record-keeping e.g. word-processed student lists or simple databases.A co-ordinated and centralised approach to record keeping is implemented across the institution. Data entered mainly by administrative staff.Individual tutors actively engage with a centralised MIS. Some academic staff access the system on-line.Data entry and retrieval is an accepted part of every tutors practice. Diagnostic assessment and guidance on demand.Evaluation and assessmentReacts to external pressure, e.g. GNVQ.College looks outward (e.g. to other institutions) for examples of good practice.Systematic use of ILT for assessment, recording and reporting.ILT-based record systems used to inform curriculum development and planning In the institution.This classification has informed my proposed benchmarking approach. However, there are three main objections to its applicability in every detail: It is based on FE thinking and on the whole, UK HE likes to take its own view on such matters, especially when it has been using IT for many years, much longer than FE. The methodological base is 14-year old thinking from MIT about how IT transforms companies from a pre-IT situation but many companies and universities are now on at least their third wave of innovation with IT. Even the version developed by Becta is nearly 10 years old. 3. Review of the Literature from Virtual Universities and eUniversities The work on Critical Success Factors for e-universities was re-scrutinised from the standpoint of this analysis. However, I decided that it was too specific to be applied. Those interested in reading more should start with Chapter 1 of the eUniversity Compendium, Introduction to Virtual Universities and e-Universities ( HYPERLINK "http://www.heacademy.ac.uk/learningandteaching/eUniCompendium_chap01.doc" http://www.heacademy.ac.uk/learningandteaching/eUniCompendium_chap01.doc) The same conclusion applies in my view to the increasing volume of material on state-wide Virtual University consortia in the US. 4. Other Input to Classification Schemes 4.1 Early Adopter Theories In his classic book Diffusion of Innovations (1995), Everett Rogers described how innovations propagate throughout institutions or societies. He described five categories: innovators early adopters early majority late majority laggards He also described the typical bell-shaped curve giving the number of people in each category. One can turn this into the following scale for stage of adoption of e-learning: innovators only early adopters taking it up early adopters adopted it, early majority taking it up early majority adopted it, late majority taking it up all taken it up except laggards, who are now taking it up (or leaving or retiring) Given a desire for a 6th point of exceeding expectations I add that as: first wave embedded, second wave of innovation under way (e.g. m-learning after e-learning) There is a good review of Rogers theories in Orrs report at  HYPERLINK "http://www.stanford.edu/class/symbsys205/Diffusion%20of%20Innovations.htm" http://www.stanford.edu/class/symbsys205/Diffusion%20of%20Innovations.htm. 4.2 The e-Learning Maturity Model There is some interesting work in Australia/New Zealand by Marshall and Mitchell on what they call the e-Learning Maturity Model this seems likely to add another useful numeric measure to our portfolio. The work has been developed out of work on the Capability Maturity Model and from the SPICE approach to software process improvement. There is a project web site at  HYPERLINK "http://www.utdc.vuw.ac.nz/research/emm/" http://www.utdc.vuw.ac.nz/research/emm/. The model is described in [M&M] and backed up by a literature search in [Marshall]. The documents give a classification of processes relevant to e-learning, into: Learning with direct impact on pedagogical aspects of e-learning Development surrounding the creation and maintenance of e-learning resources Co-ordination surrounding the oversight and management of e-learning Evaluation surrounding the evaluation and quality control of e-learning throughout its entire lifecycle Organisation associated with institutional planning and management. The classification of processes and the orientation to the entire lifecycle has a substantial amount in common with that used for activity-based costing analysis of e-learning, in particular the CNL studies in the UK a key paper of which (speaking as one of the authors) was presented as [CNL] in Australia in 1999. The e-Learning Maturity Model has six levels of process capability: 5OptimisingContinual improvement in all aspects of the e-Learning process4ManagedEnsuring the quality of both the e-learning resources and student learning outcomes3DefinedDefined process for development and support of e-Learning2PlannedClear and measurable objectives for e-learning projects1InitialAd-hoc processes0Not performedNot done at allFor the present work I suggest re-normalising these with 0 becoming 1 in the Likert scale and 5 becoming 6, thus exceeding expectations. Few organisations could claim realistically to be at this top level yet. 4.3 Input from the US Quality in Distance Education Literature There is a large body of work in the US on Quality in Distance Education. While this is targeted to off-campus activity and much predates the widespread diffusion of e-learning into distance learning, I believe that it will be a valuable source of benchmark information. A readable yet scholarly introduction to the quality issue in e-learning is The Quality Dilemma in Online Education by Nancy Parker of Athabasca University. Although she makes only one reference to benchmarking (in other than its subject sense), the point is telling for the direction of future work: It has also been suggested that the thinking on quality assurance will have to shift dramatically, from external compliance-based approaches toward comparative benchmarking and mutual recognition arrangements for international quality standards. The Parker report also draws attention to the ground-breaking report by Phipps & Merisotis, Quality on the line: Benchmarks for success in Internet-based education, published in 2000. This study gives 24 benchmarks, in the sense of statements of good practice, that distance learning operations should adhere to. They are important enough to be listed in full below: Institutional Support Benchmarks A documented technology plan that includes electronic security measures (i.e., password protection, encryption, back-up systems) is in place and operational to ensure both quality standards and the integrity and validity of information. The reliability of the technology delivery system is as failsafe as possible. A centralized system provides support for building and maintaining the distance education infrastructure. Course Development Benchmarks Guidelines regarding minimum standards are used for course development, design, and delivery, while learning outcomesnot the availability of existing technologydetermine the technology being used to deliver course content. Instructional materials are reviewed periodically to ensure they meet program standards. Courses are designed to require students to engage themselves in analysis, synthesis, and evaluation as part of their course and program requirements. Teaching/Learning Benchmarks Student interaction with faculty and other students is an essential characteristic and is facilitated through a variety of ways, including voice-mail and/or e-mail. Feedback to student assignments and questions is constructive and provided in a timely manner. Students are instructed in the proper methods of effective research, including assessment of the validity of resources. Course Structure Benchmarks Before starting an online program, students are advised about the program to determine (1) if they possess the self-motivation and commitment to learn at a distance and (2) if they have access to the minimal technology required by the course design. Students are provided with supplemental course information that outlines course objectives, concepts, and ideas, and learning outcomes for each course are summarized in a clearly written, straightforward statement. Students have access to sufficient library resources that may include a virtual library accessible through the World Wide Web. Faculty and students agree upon expectations regarding times for student assignment completion and faculty response. Student Support Benchmarks Students receive information about programs, including admission requirements, tuition and fees, books and supplies, technical and proctoring requirements, and student support services. Students are provided with hands-on training and information to aid them in securing material through electronic databases, interlibrary loans, government archives, news services, and other sources. Throughout the duration of the course/program, students have access to technical assistance, including detailed instructions regarding the electronic media used, practice sessions prior to the beginning of the course, and convenient access to technical support staff. Questions directed to student service personnel are answered accurately and quickly, with a structured system in place to address student complaints. Faculty Support Benchmarks Technical assistance in course development is available to faculty, who are encouraged to use it. Faculty members are assisted in the transition from classroom teaching to online instruction and are assessed during the process. Instructor training and assistance, including peer mentoring, continues through the progression of the online course. Faculty members are provided with written resources to deal with issues arising from student use of electronically-accessed data. Evaluation and Assessment Benchmarks The programs educational effectiveness and teaching/learning process is assessed through an evaluation process that uses several methods and applies specific standards. Data on enrollment, costs, and successful/ innovative uses of technology are used to evaluate program effectiveness. Intended learning outcomes are reviewed regularly to ensure clarity, utility, and appropriateness. How one can use these benchmarks It is important to note that these benchmarks have already been distilled down from a longer list which was market researched with six institutions active in distance learning. I recommend two more adjustments: Removing some which with the benefit of five years more experience, can be seen to be irrelevant to success or best practice. Compositing some together. Finally it is important to note that these are not benchmarks but are aspirational statements of best practice, or at least of good practice. Thus each one has to be rewritten into a form which allows some element of quantitative measurement, in the 6-point scale with supporting narrative. For example: The programs educational effectiveness and teaching/learning process is assessed through an evaluation process that uses several methods and applies specific standards. becomes something like: Evaluation of educational effectiveness: frequency, depth and range of instruments used. 4.4 Costs of Networked Learning There are two main points of relevance from the CNL studies for JISC in the 1999-2001 period. Firstly, the 3-phase model of course development derived for CNL gives a reasonable classification of processes which (at the time) was checked against all other worldwide costing methodologies of the era, including in the US, Canada and Australia as well as UK and commercial practice. See [CNL] for some examples and a short bibliography. The model is as follows: Planning & Development Production & Delivery Maintenance & Evaluation. Many observers have pointed out that it breaks down as neatly into 6 phases. These correlate quite well with the process groupings discussed earlier by the way, it is part of the CNL approach that management is viewed as being outside the three phases, thus giving a seventh level of process this is the management as overhead viewpoint. Since the work was done some years and several jobs ago, even as the leader of the work I had to re-scrutinise in detail the CNL and related material for information about benchmarks. Most are about general management and financial processes, and a few about IT. None were about benchmarking in e-learning. 4.5 Work at Specific Universities Work at specific universities on e-strategies and e-learning strategies can give some useful insights into dimensions that we might build into our benchmarking. The universities were chosen originally because they are within the competitive radar of The University of Manchester, but they are likely to be sector exemplars for many UK universities. The University of Warwick Warwick has a well-regarded e-strategy ( HYPERLINK "http://www.estrategy.warwick.ac.uk/FinalDoc/" http://www.estrategy.warwick.ac.uk/FinalDoc/) including an e-learning strategy ( HYPERLINK "http://www.estrategy.warwick.ac.uk/FinalDoc/elearnDoc/elearndoc.html" http://www.estrategy.warwick.ac.uk/FinalDoc/elearnDoc/elearndoc.html) with an associated proposal to set up a new e-learning development unit ( HYPERLINK "http://www.estrategy.warwick.ac.uk/FinalDoc/elearnUnitDoc/elearnunitdoc.html" http://www.estrategy.warwick.ac.uk/FinalDoc/elearnUnitDoc/elearnunitdoc.html). A number of the topics raised in the e-learning strategy led on to refinements of my initial ideas. These include: Degree of development of Intellectual Property policies re e-learning Degree of development of staff recognition policies (including promotion, financial rewards, etc) for those with excellence in e-learning There is a further dimension Degree of progress in Computer-Assisted Assessment which may not be relevant to all HEIs but which should be at least part of a bundle of benchmarks in the area of Progress in use of e-tools. The University of Sydney The University of Sydney is one of the leading universities in Australia, usually ranked within the first three. It is a member of the Group of Eight leading research-led universities in Australia ( HYPERLINK "http://www.go8.edu.au/" http://www.go8.edu.au/) the Australian equivalent of the Russell Group in the UK. Sydney has a well-worked out and publicly available Learning and Teaching Strategy sets of documents ( HYPERLINK "http://www.usyd.edu.au/quality/teaching/docs/revised_tandl_plan_2004.pdf" http://www.usyd.edu.au/quality/teaching/docs/revised_tandl_plan_2004.pdf). It has a web page describing its benchmark activities, some of which cover aspects of e-learning ( HYPERLINK "http://www.usyd.edu.au/quality/teaching/mou.shtml" http://www.usyd.edu.au/quality/teaching/mou.shtml). UKeU It might have been thought that UK eUniversities Worldwide Limited (UKeU) would have carried out some benchmarking work in e-learning. However a systematic search of the UKeU Computer Archive revealed nothing relevant to this study. There were, however, many references to subject benchmarking, considerable use of the word in an informal (and unverifiable) sense (e.g. in marketing brochures and PR material), and use of the word in its proper sense in consultancy and market research reports from other agencies that have one way or another ended up in the Archives. This nil result confirms my own recollections as a former member of staff at UKeU. 5. References and Further Reading [ASTD] Training for the Next Economy: An ASTD State of the Industry Report on Trends in Employer-Provided Training in the United States see  HYPERLINK "http://www.astd.org/NR/rdonlyres/1CC4FE41-DE6A-435E-8440-B525C21D0972/0/State_of_the_Industry_Report.pdf" http://www.astd.org/NR/rdonlyres/1CC4FE41-DE6A-435E-8440-B525C21D0972/0/State_of_the_Industry_Report.pdf for details including how to order it. [BHE] Benchmarking for Higher Education, Edited by Norman Jackson and Helen Lund, Published by SRHE and Open University Press 2000 ISBN 0335 204538 (pb); 25.00 ISBN 0335 20454 6 (hb). [CHEMS] Benchmarking in Higher Education: An International Review, CHEMS, 1998,  HYPERLINK "http://www.acu.ac.uk/chems/onlinepublications/961780238.pdf" http://www.acu.ac.uk/chems/onlinepublications/961780238.pdf. [CNL] Paul Bacsich and Charlotte Ash, The hidden costs of networked learning the impact of a costing framework on educational practice, Proceedings of ASCILITE 99, Brisbane, 1999,  HYPERLINK "http://www.ascilite.org.au/conferences/brisbane99/papers/bacsichash.pdf" http://www.ascilite.org.au/conferences/brisbane99/papers/bacsichash.pdf. [HEFCE] HEFCE Strategy for e-Learning, Publication 2005/12, March 2005,  HYPERLINK "http://www.hefce.ac.uk/pubs/hefce/2005/05_12/" http://www.hefce.ac.uk/pubs/hefce/2005/05_12/. [IHEP] Phipps & Merisotis, Quality on the line: Benchmarks for success in Internet-based education, 2000,  HYPERLINK "http://www.ihep.org/Pubs/PDF/Quality.pdf" http://www.ihep.org/Pubs/PDF/Quality.pdf. [ILT] The Developing Impact of ILT, Final Report to the NLN Research and Evaluation Working Group by LSDA and SHU, December 2004, Summary Report at  HYPERLINK "http://www.nln.ac.uk/downloads/pdf/BEC11392_NLNComprep36pp.pdf" http://www.nln.ac.uk/downloads/pdf/BEC11392_NLNComprep36pp.pdf. [M&M] Stephen Marshall and Geoff Mitchell, Applying SPICE to e-Learning: An e-Learning Maturity Model, Sixth Australasian Computing Education Conference (ACE2004), Dunedin. Conferences in Research and Practice in Information Technology, Vol. 30, 2004,  HYPERLINK "http://crpit.com/confpapers/CRPITV30Marshall.pdf" http://crpit.com/confpapers/CRPITV30Marshall.pdf. [Marshall] Determination of New Zealand Tertiary Institution E-Learning Capability: An Application of an E-Learning Maturity Model Literature Review,  HYPERLINK "http://www.utdc.vuw.ac.nz/research/emm/documents/literature.pdf" http://www.utdc.vuw.ac.nz/research/emm/documents/literature.pdf. [Merwe] Antoinette van der Merwe, Implementing WebCT at Stellenbosch University: The integrated approach, University of Stellenbosch,  HYPERLINK "http://www.webct.com/service/viewcontentframe?contentID=2386007&pageName=index.html" http://www.webct.com/service/viewcontentframe?contentID=2386007&pageName=index.html. [Parker] Nancy Parker, The Quality Dilemma in Online Education, Chapter 16 of Theory and Practice of Online Learning, Athabasca University, 2004,  HYPERLINK "http://cde.athabascau.ca/online_book/ch16.html" http://cde.athabascau.ca/online_book/ch16.html [Rogers] Everett Rogers, Diffusion of Innovations, 1995. [Scanlan] Craig Scanlan, Reliability and Validity of a Student Scale for Assessing the Quality of Internet-Based Distance Learning, Online Journal of Distance Learning Administration, Volume VI, Number III, Fall 2003, State University of West Georgia, Distance Education Center,  HYPERLINK "http://www.westga.edu/~distance/ojdla/fall63/scanlan63.html" http://www.westga.edu/~distance/ojdla/fall63/scanlan63.html [SCONUL] SCONUL Benchmarking Manual, edited by J. Stephen Town, Looseleaf, ISBN 0 90021021 4. [Sydney] University of Sydney Teaching and Learning Plan 20042006, November 2003,  HYPERLINK "http://www.usyd.edu.au/quality/teaching/docs/revised_tandl_plan_2004.pdf" http://www.usyd.edu.au/quality/teaching/docs/revised_tandl_plan_2004.pdf. [TrPlace] Building a Strategic Plan for e-Learning, The Training Place, November 2004,  HYPERLINK "http://www.trainingplace.com/building.htm" http://www.trainingplace.com/building.htm. 6. The Benchmark Taxonomy In its first version the taxonomy was a rapidly developed tool to kick-start this specific desk research exercise in benchmarking. After reflecting for a day on appropriate benchmarks derived from my earlier work on evaluation, costing and critical success factors, a restless night and an early-morning writing session delivered an outline system. Then the top-level literature search described in this paper was done, which has allowed the original framework to be refined and back-filled, to some extent. The current version is presented on the next four pages (which are in landscape format). A considerably abbreviated version is in the main report. LevelFactor123456NotesInstrumentAdoption phase overall (Rogers)Innovators onlyEarly adopters taking it upEarly majority taking it upLate majority taking it upAll taken it up except some laggardsFirst wave embedded and universal, second wave startingHow many segments of the Rogers model are engagedInterviews, surveys, documentation in IT reports, etcILT-like phase (MIT)Individual lone rangersLocalised (Tonto has joined the team)Coordinated (e.g. by e-learning centre)Transformative (e.g. a PVC is driving it)EmbeddedInnovative (Second wave starting)MIT/Becta level as used in FEInterviews, surveys, documentation in IT reportseMM level overallMany e-learning processes not performedInitialPlannedDefinedManagedOptimisinge-Learning Maturity Model levelInterviews, surveys, documentation in IT reportsVLE stageNo VLEDifferent VLEs across departments VLEs reducing in number to around twoOne VLE chosen for future but not yet replaced former VLEsOne VLEOne VLE but with local variants given strong business case, and post-VLE activitiesDegree of coherence across institutionObservation, purchase ordersTools useNo use of tools beyond email, Web and the VLE minimum set.Some use of toolsWidespread use of at least one specific tool, e.g. assignment handling, CAAHEI-wide use of at least one toolHEI-wide use of several toolsUse of locally developed tools alsoScale, sophistication and depth of tools useInterviews, cross-checking with JISC and CETIS, etc.IT underpinning reliability90%95%99%99.5%99.9%99.95% 24x7x365Percentage uptime over service periodsIT underpinning performanceIT underpinning usabilityNo usability testing, no grasp of the conceptKey IT staff understand the concept, test some systemsExplicit usability testing of all key systemsMost services usable, with some internal evidence to back this upAll services usable, with internal evidence to back this upEvidence of usability involving external verificationLevel of provable usability of e-learning systemsAccessibilitye-learning material and services is not accessibleMuch e-learning material and most services conform to minimum standards of accessibilityAlmost all e-learning material and services conform to minimum standards of accessibilityAll e-learning material and services conform to at least minimum standards of accessibility, much to highere-learning material and services are accessible, and key components validated by external agenciesStrong evidence of conformance with letter and spirit of accessibility in all jurisdictions where students studyLevel of conformance to accessibility guidelinesSplit off separately for legal reasons. Seek advice from TechDIS over levelse-Learning StrategyNo e-Learning Strategy. No recent Learning and Teaching StrategySome mention of e-learning within the Learning and Teaching Strategye-Learning Strategy produced from time to time, e.g. under pressure from HEFCE or for particular grantsFrequently updated e-Learning Strategy, integrated with Learning and Teaching Strategy and perhaps some others Regularly updated e-Learning Strategy, integrated with Learning and Teaching Strategy and all related strategies (e.g. Distance Learning)Coherent regularly updated Strategy allowing adaptations to local needs, made public, etcDegree of strategic engagementReview of HEFCE, TQEF and other documents. Interview with PVC responsibleDecision-makingNo decision making regarding e-learning each project is differentDecision-making at meso level (school, department, faculty, etc)E-learning decisions (e.g. for VLEs) get taken but take a long time and are contested even after the decision is takenEffective decision-making for e-learning across the whole institution, including variations when justifiedDecisions taken in an organic way and efficient way, e.g. Round TableRobustness, sophistication and subtlety of decision-makingObservation and perusal of papersInstructional Design/PedagogyTerms not understood in the HEI.Terms well understood within the learning and teaching centre and among some academic staff in the field Pedagogic guidelines for the whole HEI, and acted onA culture where techno-pedagogic decisions are made naturallyLevel of practical evidence-based knowledge and application of instructional design and pedagogyInterviewsLearning materialLittle conformance of learning material to house style for editing or layoutRhetoric of quality, little conformance to any normsMost learning material conforms to explicit editorial and layout guidelinesAll learning material conforms to explicit editorial and layout guidelines but little embedding in the processHEI-wide standards for learning material, which are adhered to and embedded at any early stage, e.g. by style sheetsMuch learning material exceeds expectations.Level of fitness for purpose of learning materialPerusal of material, interviews.TrainingNo systematic training for e-learningSome systematic training for e-learning, e.g. in some facultiesHEI-wide training programme set up but little monitoring of attendance or encouragement to goHEI-wide training programme set up with monitoring of attendance and strong encouragement to goAll staff trained in VLE use, appropriate to job type and retrained when neededStaff increasingly keep themselves up to date, just in time, except when discontinuous system change occurs, when training is provided.Degree to which staff have competence in VLE and tools use, appropriate to job type%ages plus narrative. (Note: this may not involve training courses; but is likely to.)Academic workloadNo allowance given for the different workload pattern of e-learning coursesSome allowance given, but distortions in the system as shrewder staff flee the areas of overloadA work planning system which makes some attempt to cope, however crudely, with e-learning coursesWork planning system which recognises the main differences that e-learning courses have from traditionalSee the cell below.Sophistication of the work planning system for teachingDetailed and possibly anonymous interviews and questionnaires. Some union sensitivities likely in some HEIsCostsNo understanding of costsUnderstanding of costs in some departments e.g. business schoolGood understanding of costs Activity-Based Costing being used in partFull Activity-Based Costing used and adapted to e-learning situationLevel of understanding of costsInterviews and questionnaires PlanningIntegrated planning process for e-learning integrated with overall course planningIntegrated planning process allowing e.g. trade-offs of courses vs. buildings Interviews and questionnairesEvaluationNo evaluation of courses take place that is done by evaluation professionalsSome evaluation of courses takes place, either by professionals or internal staff advised by professionals or central agenciesEvaluation of key courses is done from time to time, by professionalsSome external evaluations are done of coursesRegular evaluation of all courses using a variety of measurement techniques and involving outside agencies if appropriateEvaluation built into an Excellence, TQM or other quality enhancement process including benchmarking aspectsLevel of thoroughness of evaluationInterviews with key evaluators. Perusal of conference and journal papersOrganisationNo appointments of e-learning staffAppointments of e-learning staff in at least some faculties but no specialist managers of these staffCentral unit or sub-unit set up to support e-learning developmentsCentral unit has some autonomy from IT or resources functionCentral unit has Director-level university manager in charge and links to support teams in facultiesBeginning of the withering away of explicit e-learning posts and structuresInterview with VC and relevant PVC(s)Technical support to academic staffNo specific technical support for the typical (unfunded) academic engaged in e-learningKey staff engaged in the main e-learning projects are well supported by technical staffAll staff engaged in e-learning process have nearby fast-response tech supportIncreasing technical sophistication of staff means that explicit tech support can reduceInterview with both top-level staff and selective interviews with grass-roots staff.Quality and ExcellenceConformance to QAA in a minimalist wayAn internal function which begins to focus on e-learning aspectsConformance to QAA precepts including those that impinge on e-learningAdoption of appropriate quality methodology (EFQM, etc) integrated with course quality mechanisms Active dialogue with QAA and wider quality agencies as to appropriate quality regimes for e-learningLevel of HEI overall commitment to quality and excellence agenda for e-learningInterviews, questionnaires, quality reviews, etc.Foresight on technology and pedagogyNo look-ahead functionSome individuals take it on themselves to do foresightSubscription to central agencies doing foresight (OBHE, JISC Observatory etc)Collaboration with central agencies doing foresightHEI Observatory functionForesight becomes embedded in course planning processLevel of institutional foresight functionInterviews, documentsCollaborationNo collaborationCollaboration at departmental levelCollaboration policy, patchily or superficially implemented Well-developed policy on collaboration and established partners (but not a closed list)HEI has explicit strategic approach to collaboration, and non-collaboration, as appropriateInterviews, documentsIPRNo IPR items in staff contractsIPR in staff contracts but not enforcedIPR embedded and enforced in staff, consultant and supplier contracts All of 5 plus use of open source, creative commons or other new IPR modelsLevel of IPR for staff, consultants and suppliersDocumentary evidence Staff recognition for e-learningNo recognition for staff, explicit pressure against (e.g. due to RAE)Formal structure for recognition (e.g. Teaching Fellows), no real progress Staff engaged only in the teaching process can reach a high level of salary and responsibilityLevel of staff recognition (not only and not necessarily financial) against the pressure for RAEDocumentary evidence     Benchmarking MBS for e-Learning Annex 3 (Benchmarking Theory) Paul Bacsich A3. PAGE 1 March 2005 Paul Bacsich A3. PAGE 26 March 2005  /U J K G J P V y z | ( ) * > 56LQPQHIflm{ȵޡ hShhshShp"0Jjhp"Ujhp"Uhp" hShp" h6 hSNHhSh hNHhhMCJaJhMht(?U | ? m]6_ "6#4$0%%(xgdgdgdgdgdgd'W([\.12DEegJKhqwx 13WXiTXouwxjk:;TWSThsh2ht( hNHhShhShNH hSh hShSR01   ; < D E l!m!!!!!!!!""J"K"""##y#z###$$$$$$$/%>%?%n%s%&&&&,'-'((%(&(((')()w)x))))))*ʽjhU hp"NH hShp"hp"hSjhU h0JjhUjhU hNHhF((d+S,-//y013#45555^7788: ;<}==-? 0^`0gdSgdgdgdxgdgd**|*}******+++++,,,!,",n-o--- / /00001122222Z2[222;3<33344}4~444455%5+556/60616R6S66666W7X7\7]7^7jhUhSj&hUjahU hNHhs h0Jj|hUhjhUF^7C8D8K8L899 : :::[:\:::::::;;;;<<==>>??h????????3@8@;@_@`@@@MANAhAiAAAAAAAAGBHB=C>CCCCCCCDD.D/DnDj hU hNH hSh h0JjhUjhUhh hSNHhSG-?h?m?@BBBbCDF1FNFGGH[IuIJJKHLoLNOgd & F gd & F x^gdgdgdgdgdSnDoDpDDDeEfEFFFFFFFFFGGHHIIJJJ%J&JJJKJJJJJKKKKKHLLLLLLLL&M,MFMHMYMhMMMMMMMNN⵬⚖hS hShjhUh6mH sH hmH sH hs h6j0 hUj/ hU hNHh h0JjhUj hU;NNNNNNNNO O+O?OOOOORPSP|P}P~PPPPP:Q;QQQ!R"RRRRRRRRRRRRRSS)T*TTTTTTT3U4U5UUUVVVVUWVWqWrW*X+XYYiYjYgZhZZZZjhUhs h0JjhUjhUhS hSh hNHhLOPQ0T@TWW7XZZZZZZ $$$Ifa$gd Lgdgdgdgd ZZZ[W[[[\<\\\3]w]x]tttttttttttt$If]^gd Lzkd$$Ifr H~C! 7 6 a ZZZQ[R[n[o[[[[[%\&\W\X\t\u\\\\\ ]]j]k]]]]]]]2^3^O^P^j^k^o^p^^^^^^^"_#___ ` ```@`A`a`b`d`e```````aaaaaaaa:b;bbbbbbbccZc[cxcycccddgdhdldmdhCJNHaJhCJaJ^x]]^9^v^^ _9___G````aAaaaAbb"ccc1dtd$If]^gd Ltduddd epgh"h~hhhTj}xsxnfxsaxgd & Fgdgdgdgdxgd{kd>$$If4r H~C! 7 6 af4 mdudeef)fAf`f}f~ffffff6g7g8gmgnghh"h+h^hbh|h}hhhhhhhhTiUiiiiiiiiAjBjjjkkGkHkIkzk{kkkkkkkkkjEhUj$hUj=hU hShShs h0JjhUjhU hShhS hNHhhCJaJ>kllFlGlHl{l|lllmmmmMnNnnnIoJohoiopp r!rirjrrrDsEsxsysssttuuvv;xWXxy<=}~hCJaJhhCJNHaJh]| lcccccW $$Ifa$gd L $Ifgd Lkd"$$If4ֈ8@ HPX $@ 8af4,zVlcccccWWWW $$Ifa$gd L $Ifgd Lkdf#$$If4ֈ8@ HPX $@ 8af4 wϪ`WWWWWW $Ifgd Lkd$$$If4ֈ8@ HPX $@`8af4 $$Ifa$gd L 0w%lcccccW $$Ifa$gd L $Ifgd Lkd$$$If4ֈ8@ HPX $@ 8af4&DmlcccccW $$Ifa$gd L $Ifgd Lkdh%$$If4ֈ8@ HPX $@ 8af4#bcdlcccccWWW $$Ifa$gd L $Ifgd Lkd&$$If4ֈ8@ HPX $@8af4 pqȲ+,PQ`xZ[ڵ۵ {|DEbmhirҺӺ,-={缴缫h2mH sH hH*mH sH hA@mH sH hmH sH  h0Jj(hUjhU hNHhshhCJNHaJhhCJaJh<ձ'fƲDzlcccccW $$Ifa$gd L $Ifgd Lkd&$$If4ֈ8@ HPX $@`8af4DzȲ[۴`Gɷle[[[VQQgdgd & F xgdxgdkde'$$If4ֈ8@ HPX $@ 8af4ɷȸ׸׹D=!Cm & F ^gd & F ^gd$$ & F ^gdgdgdgd{|ӻԻջPQ'(EFrsmn!":OPQg#YZ`x뭟hA@hmH sH hsmH sH ht(mH sH hCJNHaJmH sH hCJaJmH sH jC+hUmH sH hNHmH sH h0JmH sH j)hUmH sH hmH sH jhUmH sH 4NE $$Ifgd Lgd & F ^gd  tqfff $$Ifgd LkdX,$$IflF8 X hd t06    44 la<tuwqfff $$Ifgd Lkd-$$IflF8 X hd t06    44 la<qfff $$Ifgd Lkd-$$IflF8 X hd t06    44 la<qfff $$Ifgd Lkd_.$$IflF8 X hd t06    44 la<)9qhhh $Ifgd Lkd /$$IflF8 X hd t06    44 la<9:O`qje``[`R^gdrUgdgdgdxgdkd/$$IflF8 X hd t06    44 la<hilm[\&'BC:;+,)*VWBqjkZ[hrUmH sH hA@mH sH h2mH sH  hNHhhmH sH hNHmH sH P XsJf`7-Hkjb^gdrU & F 0^`0gdrUb1 *}rp & F 0^`0gdgd & Fgd & Fgdgdgd & F 0^`0gdrU^gdrUhi(,RSVvw{|deEFGǷ{j1hUmH sH h0JmH sH jf0hUmH sH jhUmH sH hmH sH hA@mH sH hrUmH sH hA@hmH sH hA@hA@mH sH hNHmH sH !hOJQJ]^JaJmH sH hmH sH 0f"<=z47a `]^`` `^``$gd & Fgdgdgd234kl rsEFKL@Bz{|뇃hrUh$ ht(mH sH j+7hUmH sH j5hUmH sH j4hUmH sH hNHmH sH hrUmH sH h0JmH sH j3hUmH sH hmH sH jhUmH sH 2|*+ =>WXpq}~gh#$?@vwx./78Z[j?hMUj>hMUj{=hMUj;hMUj:hMU hMNH hM0Jjh8hMUjhMUhM=45^_IJK34pqr)*<=>yz,-()`abjIhMUj;HhMUjFhMUjEhMUjChMUjkBhMU hMNHhM hM0JjhMUj2AhMU<{9:;<BCD $$Ifa$$If `^``babmoWX8;EL78EFJUV*+ <HIjkyzӿӪӪӪӿӿӪӪӿӿӪӪӪӿhM6CJNHaJhMCJNHaJhMCJaJhM56CJ\aJhM6CJaJht( hMNHhrUh$ hMjhMU hM0JCDELNPRTVXZTKKKKKK $$Ifa$$IfkdJ$$Ifl44\5', 3?`b!9 t0x3644 laf4X^ijJ|0Zc!Ff7SFfPFfL$If!)1<\#+ g-.LFf\FfYFf_V$IfEF[\uv<=[\-.89Lt '34STZ[iuv#$BCvwhM6CJNHaJhM6CJaJhMCJNHaJhMCJaJVLPTX^dtEs'YFfbFf_$IfYZ[iO      " c     d     $ e  FfwlFfOiFf'f$Ifw  ' ( 3 4 [ \ s t                "       2 3 ; < b c             2 3 < = d q r } ~          s t     hM6CJNHaJhM6CJaJhMCJNHaJhMCJaJV  H +, kvw W=jFfuFfrFfo$Ifqr ;<ef  01>?JKXYefvw;<opz{UVbcjcdKLhM6CJNHaJhM6CJaJhMCJNHaJhMCJaJV/?3tu4L7aFf?|Ffy$IfLZ[,-STtu./]^23[\~  #$IJjk  !"TUfg lmhM6CJNHaJhM6CJaJhMCJNHaJhMCJaJVF zXYfFfFfFfg$Ifm ;<DE\]rs?@XYfopjk!HIm*+BC  0 1 7 m n    ! !!!!!!!!""1"2"B"hM6CJNHaJhM6CJaJhMCJNHaJhMCJaJV3p!"HImp  7 ^    J!!!1"2"Ff/FfFf߈$IfB"C"Q"R"W"z"{"""""1#2#K#L#V#W#q#r#v########## $ $+$,$S$T$r$s$$$$$$$$%%% %!%7%8%H%I%|%}%%%%%%&&&&$&%&,&-&9&:&N&i&j& ' '4'5':';'B'|'}''''''hMCJNHaJhMCJaJhM6CJaJhM6CJNHaJV2"W"n"""'#@#v#######7$8$$$$%%%(%P%Q%R%%%FfFfW$If%&+&,&-&N&&&&&&B'''''''''''''(($ !(#a$gd.`FfϛFf$If'''''''''''''(((((((( (!("(,(<(=(>(?(E(F(H(I(J(T(V(W(X(ĶĶ hrU0Jhp 0JmHnHu h0Jjh0JUh2hhCJOJQJ^JaJh2CJOJQJ^JaJh?<jh?<UhMCJaJhMCJaJ%(-(.(U(V(W(X(  9r PT3  9r (#gd.`6&P 1h:p.`. A!"#$%`` 3&P 1h0A .!"#$% DyK yK \http://www.hefce.ac.uk/pubs/hefce/2005/05_12/IDyK 5http://www.benchmarking.gov.uk/about_bench/types.aspyK jhttp://www.benchmarking.gov.uk/about_bench/types.aspDyK #http://www.heacademy.ac.uk/914.htmyK Fhttp://www.heacademy.ac.uk/914.htmeDyK <http://www.acu.ac.uk/chems/onlinepublications/961780238.pdfyK xhttp://www.acu.ac.uk/chems/onlinepublications/961780238.pdfDyK http://www.acu.ac.uk/chems/yK 8http://www.acu.ac.uk/chems/DyK http://www.esmu.be/yK (http://www.esmu.be/DyK Chttp://www.esmu.be/download/benchmarking/BENCH_YEAR5_INFO_NOTE.docyK http://www.esmu.be/download/benchmarking/BENCH_YEAR5_INFO_NOTE.docDyK "http://www.hefce.ac.uk/siteindex/yK Dhttp://www.hefce.ac.uk/siteindex/1DyK /http://www.jisc.ac.uk/strategy_jisc_04_06.htmlyK ^http://www.jisc.ac.uk/strategy_jisc_04_06.htmlDyK *http://www.jisc.ac.uk/dfes_elearning.htmlyK Thttp://www.jisc.ac.uk/dfes_elearning.html=DyK 2http://www.jisc.ac.uk/index.cfm?name=funding_4_03yK dhttp://www.jisc.ac.uk/index.cfm?name=funding_4_03DyK #http://www.heacademy.ac.uk/914.htmyK Fhttp://www.heacademy.ac.uk/914.htmDyK http://www.leadership-he.com/yK <http://www.leadership-he.com/DyK http://www.eubc.bham.ac.uk/yK 8http://www.eubc.bham.ac.uk/DyK http://excellence.shu.ac.ukyK :http://excellence.shu.ac.uk/DyK Lhttp://science.uniserve.edu.au/courses/benchmarking/benchmarking_manual.pdfyK http://science.uniserve.edu.au/courses/benchmarking/benchmarking_manual.pdf$$If!vh5 575 565 #v #v7#v #v6#v :V 5 575 565 / 4a$$If!vh5 575 565 #v #v7#v #v6#v :V 45 575 565 / 4af4MDyK 6http://www.ubcinternational.ubc.ca/universitas_21.htmyK lhttp://www.ubcinternational.ubc.ca/universitas_21.htmDyK http://www.coimbra-group.beyK :http://www.coimbra-group.be/!DyK +http://www.coimbra-group.be/06_members.htmyK Vhttp://www.coimbra-group.be/06_members.htm=DyK 2http://www.coimbra-group.be/DOCUMENTS/summary.docyK dhttp://www.coimbra-group.be/DOCUMENTS/summary.docEDyK 4http://www.coimbra-group.be/DOCUMENTS/portfolio.docyK hhttp://www.coimbra-group.be/DOCUMENTS/portfolio.docmDyK >http://www.nln.ac.uk/lsda/self_assessment/files/Benchmark.pdfyK |http://www.nln.ac.uk/lsda/self_assessment/files/Benchmark.pdfDyK http://www.nln.ac.ukyK ,http://www.nln.ac.uk/DyK Thttp://www.nln.ac.uk/lsda/self_assessment/files/Self_assessment_tool_Guidelines.docyK http://www.nln.ac.uk/lsda/self_assessment/files/Self_assessment_tool_Guidelines.docEDyK 4http://www.citscapes.ac.uk/products/phase1/ch10.pdfyK hhttp://www.citscapes.ac.uk/products/phase1/ch10.pdf$$If!vh5@555558#v@#v#v8:V 445@558/ 4f4$$If!vh5@555558#v@#v#v8:V 4+5@558/ /  4f4$$If!vh5@555558#v@#v#v8:V 4+5@558/ /  4f4$$If!vh5@555558#v@#v#v8:V 4+5@558/ /  4f4$$If!vh5@555558#v@#v#v8:V 4+5@558/ / 4f4$$If!vh5@555558#v@#v#v8:V 45@558/ 4f4$$If!vh5@555558#v@#v#v8:V 4+5@558/ /  4f4$$If!vh5@555558#v@#v#v8:V 4+5@558/ /  4f4$$If!vh5@555558#v@#v#v8:V 4+5@558/ / 4f4$$If!vh5@555558#v@#v#v8:V 4+5@558/ 4f4$$If!vh5@555558#v@#v#v8:V 4+5@558/ / 4f4$$If!vh5@555558#v@#v#v8:V 4+5@558/ / 4f4$$If!vh5@555558#v@#v#v8:V 45@558/ 4f4$$If!vh5@555558#v@#v#v8:V 4+5@558/ /  4f4$$If!vh5@555558#v@#v#v8:V 4+5@558/ / 4f4DyK Ihttp://www.heacademy.ac.uk/learningandteaching/eUniCompendium_chap01.docyK http://www.heacademy.ac.uk/learningandteaching/eUniCompendium_chap01.docDyK Jhttp://www.stanford.edu/class/symbsys205/Diffusion%20of%20Innovations.htmyK http://www.stanford.edu/class/symbsys205/Diffusion of Innovations.htmDyK (http://www.utdc.vuw.ac.nz/research/emm/yK Phttp://www.utdc.vuw.ac.nz/research/emm/$$If<!vh5h55d#vh#v#vd:V l t065h55da<$$If<!vh5h55d#vh#v#vd:V l t065h55da<$$If<!vh5h55d#vh#v#vd:V l t065h55da<$$If<!vh5h55d#vh#v#vd:V l t065h55da<$$If<!vh5h55d#vh#v#vd:V l t065h55da<$$If<!vh5h55d#vh#v#vd:V l t065h55da<)DyK -http://www.estrategy.warwick.ac.uk/FinalDoc/yK Zhttp://www.estrategy.warwick.ac.uk/FinalDoc/DyK Ehttp://www.estrategy.warwick.ac.uk/FinalDoc/elearnDoc/elearndoc.htmlyK http://www.estrategy.warwick.ac.uk/FinalDoc/elearnDoc/elearndoc.htmlDyK Mhttp://www.estrategy.warwick.ac.uk/FinalDoc/elearnUnitDoc/elearnunitdoc.htmlyK http://www.estrategy.warwick.ac.uk/FinalDoc/elearnUnitDoc/elearnunitdoc.htmlDyK http://www.go8.edu.au/yK .http://www.go8.edu.au/DyK Ihttp://www.usyd.edu.au/quality/teaching/docs/revised_tandl_plan_2004.pdfyK http://www.usyd.edu.au/quality/teaching/docs/revised_tandl_plan_2004.pdf=DyK 2http://www.usyd.edu.au/quality/teaching/mou.shtmlyK dhttp://www.usyd.edu.au/quality/teaching/mou.shtmlDyK ihttp://www.astd.org/NR/rdonlyres/1CC4FE41-DE6A-435E-8440-B525C21D0972/0/State_of_the_Industry_Report.pdfyK http://www.astd.org/NR/rdonlyres/1CC4FE41-DE6A-435E-8440-B525C21D0972/0/State_of_the_Industry_Report.pdfeDyK <http://www.acu.ac.uk/chems/onlinepublications/961780238.pdfyK xhttp://www.acu.ac.uk/chems/onlinepublications/961780238.pdfDyK Hhttp://www.ascilite.org.au/conferences/brisbane99/papers/bacsichash.pdfyK http://www.ascilite.org.au/conferences/brisbane99/papers/bacsichash.pdf-DyK .http://www.hefce.ac.uk/pubs/hefce/2005/05_12/yK \http://www.hefce.ac.uk/pubs/hefce/2005/05_12/DyK )http://www.ihep.org/Pubs/PDF/Quality.pdfyK Rhttp://www.ihep.org/Pubs/PDF/Quality.pdfqDyK ?http://www.nln.ac.uk/downloads/pdf/BEC11392_NLNComprep36pp.pdfyK ~http://www.nln.ac.uk/downloads/pdf/BEC11392_NLNComprep36pp.pdf9DyK 1http://crpit.com/confpapers/CRPITV30Marshall.pdfyK bhttp://crpit.com/confpapers/CRPITV30Marshall.pdfuDyK @http://www.utdc.vuw.ac.nz/research/emm/documents/literature.pdfyK http://www.utdc.vuw.ac.nz/research/emm/documents/literature.pdfDyK Thttp://www.webct.com/service/viewcontentframe?contentID=2386007&pageName=index.htmlyK http://www.webct.com/service/viewcontentframe?contentID=2386007&pageName=index.html1DyK /http://cde.athabascau.ca/online_book/ch16.htmlyK ^http://cde.athabascau.ca/online_book/ch16.htmleDyK <http://www.westga.edu/~distance/ojdla/fall63/scanlan63.htmlyK xhttp://www.westga.edu/~distance/ojdla/fall63/scanlan63.htmlDyK Ihttp://www.usyd.edu.au/quality/teaching/docs/revised_tandl_plan_2004.pdfyK http://www.usyd.edu.au/quality/teaching/docs/revised_tandl_plan_2004.pdfDyK *http://www.trainingplace.com/building.htmyK Thttp://www.trainingplace.com/building.htm$$If!vh5?5b!559#v?#vb!#v#v9:V l44 t0x36+5?5b!559f4%$$If!v h5?5`55/5d5a5a55 9#v?#v`#v#v/#vd#va#v#v 9:V l44 t0x365?5`55/5d5a55 9f4kdK$$Ifl44 3 s!5', 3?`/daa9 t0x36$$$$44 laf4"$$If!v h5?5`55/5d5a5a55 9#v?#v`#v#v/#vd#va#v#v 9:V l4 t0x365?5`55/5d5a55 9f4kdN$$Ifl4 3 s!5', 3?`/daa9 t0x36$$$$44 laf4"$$If!v h5?5`55/5d5a5a55 9#v?#v`#v#v/#vd#va#v#v 9:V l4 t0x365?5`55/5d5a55 9f4kdR$$Ifl4 3 s!5', 3?`/daa9 t0x36$$$$44 laf4"$$If!v h5?5`55/5d5a5a55 9#v?#v`#v#v/#vd#va#v#v 9:V l4 t0x365?5`55/5d5a55 9f4kd;U$$Ifl4 3 s!5', 3?`/daa9 t0x36$$$$44 laf4"$$If!v h5?5`55/5d5a5a55 9#v?#v`#v#v/#vd#va#v#v 9:V l4 t0x365?5`55/5d5a55 9f4kdcX$$Ifl4 3 s!5', 3?`/daa9 t0x36$$$$44 laf4"$$If!v h5?5`55/5d5a5a55 9#v?#v`#v#v/#vd#va#v#v 9:V l4 t0x365?5`55/5d5a55 9f4kd[$$Ifl4 3 s!5', 3?`/daa9 t0x36$$$$44 laf4"$$If!v h5?5`55/5d5a5a55 9#v?#v`#v#v/#vd#va#v#v 9:V l4 t0x365?5`55/5d5a55 9f4kd^$$Ifl4 3 s!5', 3?`/daa9 t0x36$$$$44 laf4"$$If!v h5?5`55/5d5a5a55 9#v?#v`#v#v/#vd#va#v#v 9:V l4 t0x365?5`55/5d5a55 9f4kda$$Ifl4 3 s!5', 3?`/daa9 t0x36$$$$44 laf4"$$If!v h5?5`55/5d5a5a55 9#v?#v`#v#v/#vd#va#v#v 9:V l4 t0x365?5`55/5d5a55 9f4kde$$Ifl4 3 s!5', 3?`/daa9 t0x36$$$$44 laf4"$$If!v h5?5`55/5d5a5a55 9#v?#v`#v#v/#vd#va#v#v 9:V l4 t0x365?5`55/5d5a55 9f4kd+h$$Ifl4 3 s!5', 3?`/daa9 t0x36$$$$44 laf4"$$If!v h5?5`55/5d5a5a55 9#v?#v`#v#v/#vd#va#v#v 9:V l4 t0x365?5`55/5d5a55 9f4kdSk$$Ifl4 3 s!5', 3?`/daa9 t0x36$$$$44 laf4"$$If!v h5?5`55/5d5a5a55 9#v?#v`#v#v/#vd#va#v#v 9:V l4 t0x365?5`55/5d5a55 9f4kd{n$$Ifl4 3 s!5', 3?`/daa9 t0x36$$$$44 laf4"$$If!v h5?5`55/5d5a5a55 9#v?#v`#v#v/#vd#va#v#v 9:V l4 t0x365?5`55/5d5a55 9f4kdq$$Ifl4 3 s!5', 3?`/daa9 t0x36$$$$44 laf4"$$If!v h5?5`55/5d5a5a55 9#v?#v`#v#v/#vd#va#v#v 9:V l4 t0x365?5`55/5d5a55 9f4kdt$$Ifl4 3 s!5', 3?`/daa9 t0x36$$$$44 laf4"$$If!v h5?5`55/5d5a5a55 9#v?#v`#v#v/#vd#va#v#v 9:V l4 t0x365?5`55/5d5a55 9f4kdw$$Ifl4 3 s!5', 3?`/daa9 t0x36$$$$44 laf4"$$If!v h5?5`55/5d5a5a55 9#v?#v`#v#v/#vd#va#v#v 9:V l4 t0x365?5`55/5d5a55 9f4kd{$$Ifl4 3 s!5', 3?`/daa9 t0x36$$$$44 laf4"$$If!v h5?5`55/5d5a5a55 9#v?#v`#v#v/#vd#va#v#v 9:V l4 t0x365?5`55/5d5a55 9f4kdC~$$Ifl4 3 s!5', 3?`/daa9 t0x36$$$$44 laf4"$$If!v h5?5`55/5d5a5a55 9#v?#v`#v#v/#vd#va#v#v 9:V l4 t0x365?5`55/5d5a55 9f4kdk$$Ifl4 3 s!5', 3?`/daa9 t0x36$$$$44 laf4"$$If!v h5?5`55/5d5a5a55 9#v?#v`#v#v/#vd#va#v#v 9:V l4 t0x365?5`55/5d5a55 9f4kd$$Ifl4 3 s!5', 3?`/daa9 t0x36$$$$44 laf4"$$If!v h5?5`55/5d5a5a55 9#v?#v`#v#v/#vd#va#v#v 9:V l4 t0x365?5`55/5d5a55 9f4kd$$Ifl4 3 s!5', 3?`/daa9 t0x36$$$$44 laf4"$$If!v h5?5`55/5d5a5a55 9#v?#v`#v#v/#vd#va#v#v 9:V l4 t0x365?5`55/5d5a55 9f4kd$$Ifl4 3 s!5', 3?`/daa9 t0x36$$$$44 laf4"$$If!v h5?5`55/5d5a5a55 9#v?#v`#v#v/#vd#va#v#v 9:V l4 t0x365?5`55/5d5a55 9f4kd $$Ifl4 3 s!5', 3?`/daa9 t0x36$$$$44 laf4"$$If!v h5?5`55/5d5a5a55 9#v?#v`#v#v/#vd#va#v#v 9:V l4 t0x365?5`55/5d5a55 9f4kd3$$Ifl4 3 s!5', 3?`/daa9 t0x36$$$$44 laf4"$$If!v h5?5`55/5d5a5a55 9#v?#v`#v#v/#vd#va#v#v 9:V l4 t0x365?5`55/5d5a55 9f4kd[$$Ifl4 3 s!5', 3?`/daa9 t0x36$$$$44 laf4"$$If!v h5?5`55/5d5a5a55 9#v?#v`#v#v/#vd#va#v#v 9:V l4 t0x365?5`55/5d5a55 9f4kd$$Ifl4 3 s!5', 3?`/daa9 t0x36$$$$44 laf4"$$If!v h5?5`55/5d5a5a55 9#v?#v`#v#v/#vd#va#v#v 9:V l4 t0x365?5`55/5d5a55 9f4kd$$Ifl4 3 s!5', 3?`/daa9 t0x36$$$$44 laf4D@D NormalxCJ_HaJmH sH tH b@b  Heading 1$$$$@&a$5CJ KH OJQJ\^JaJ j@j  Heading 2'$$ 0x@&^`05OJQJ\]^J<@!<  Heading 3@& 5\aJ>@>  Heading 4 $$@&5\B@B  Heading 5 $$@& 6\]@@@ Heading 6$$@&a$6]:@:  Heading 7 <@&@@@  Heading 8 <@&6]DA@D Default Paragraph FontVi@V  Table Normal :V 44 la (k(No List >>  Footnote TextCJaJ@&@ Footnote ReferenceH*j^@j Normal (Web)ddd[$\$!B*CJOJQJaJmH phsH 6U@!6  Hyperlink >*B*ph,OB, TL $<<6OR6 Quote]^R @bR Footer 9r d CJOJQJaJtH \Oq\ Heading 3 Char'CJOJQJ]^J_HaJmH sH tH 4@4 Header  !.)@.  Page NumberFV@F FollowedHyperlink >*B* phPB@P Body Text $a$OJPJQJaJmH sH tO!t $Heading 2 + Left: 0",Hanging: 0.5"0^`0:X 8U|?m ] 6_640 d#S$%''y()+#,----^//002 34}55-7h7m78:::b;<>1>N>??@[AuABBCHDoDFGHI0L@LOO7PRRRRRRRRSWSSST01>01>01>H0-0[AH0-0B0B800HD0HD800G0G8000LH00L0O0O0O 0O 0O 0O 0O 0O0O0O0O0O0O0O0O0O0O0O0O 0O 0O0O0O0O0O0O0O0O0O0O0O0O 0O 0O0O0O0O0O0O0O0O0O0O0O 0O0O0O800\0\( 060`80`H0~`0`0`0` 0` 0` 0`0`80`0g0g0g0g0g0g0g0g0g0g0g0g0g0g0g0g0g0g0g0g0g0g0g80`0,}80`0~(06H00AH00000000 0 0 0 0 0000 0 0 0 0 0 00 0 0 0 0 000000000 00 0 0 0 0 0 00 0 0 0 0 0 00 0 0 0 0 0 00 0 0 0 0 00 00 0 0 0 0 00000 00 0 0 0 0 0 00 0 0 0 0 0 00 0 0 0 0 00000 00 0 0 0 00 0 00 0 0 0 0 0 00 0 0 0 0 0 00 0 0 0 0 000 00 0 0 0 0 0 00 0 0 00000(0ɯ0 0 0 0 0 00 0 0 0 0 00 00(0ɯ0!0! 0! 0! 0! 0! 0!0!0!0! 0! 0! 0! 0! 0! 0! 0! 0! 0! 0! 0!0! 0! 0! 0!0! 0! 0! 0!0! 0! 0! 0!0!(0ɯ0000H0 0 0 0H0 0 0 0H0 0 0 0H0 0 J 0 J 0 J 0 JH0 0 - 0- 0- 0-H0 0k 0k 0k 0kH0 0b 0b 0b800  0  0 0  0 0  0 (0ɯ0( 0ɯ( 0ɯ( 0ɯ00(0ɯ00H00"0" 0" 0"0"H000H0000z0z0z0z0z0z0z0z0z0z0z0z0z0z0z0z000000 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0 0 0 0@0y00@0y00@0y00@0y00@0@0@0@0@0@0y00|?m ] 6_640 d#S$%''y()+#,----h7m78:::b;<>1>N>??@[AuABBCHDoDFGHI0L@LOO7PRRRRRRRRSWSSSTJ@0{2@0?@0?:@0k@0AA@0AA:@0k@0D@0D:@0k@0HJ@0H@0jL@0jL@0jL @0jL @0jL @0jL @0jL @0jL @0jL@0jL@0jL@0jL@0jL@0jL@0jL@0jL@0jL@0jL@0jL @0jL @0jL@0jL@0jL@0jL@0jL@0jL@0jL@0jL@0jL@0jL@0jL@0jL @0jL @0jL@0jL@0jL@0jL@0jL@0jL@0jL@0jL@0jL@0jL@0jL @0jL @0jL@0jL:@0k@0Y@0Y*A 0 @0\:@0\J@0E]@0L]@0L]@0L]C 0L]C 0L]C 0L]@0L]:@0\@0~d@0~d@0~d@0~d@0~d@0~d@0~d@0~d@0~d@0~d@0~d@0~d@0~d@0~d@0~d@0~d@0~d@0~d@0~d@0~d@0~d@0~d@0~d:@0\@0'z:@0\@0{*@0 J@0/~@0X~J@0/~@01@01@01@01@01@01@01@ 01@ 01@ 01@ 01@ 01@01@01@01 @01 @01 @01 @01 @01 @01 @01 @01 @01 @01 @01 @01@01@01@01@01@01@01@01@01 @01 @01 @01 @01 @01 @01 `01 @01 @01 @01 @01 @01 @01 `01 @01 @01 @01 @01 @01 @01 `01 @01 @01 @01 @01 @01 @01 @01@01 @01 @01 @01 @01 @01 @01 @01@01@01@01@01 @01 @01 @01 @01 @01 @01 `01 @01 @01 @01 @01 @01 @01 `01 @01 @01 @01 @01 @01 @01 @01@01@01@01@01 @01 @01 @01 @01 @01 @01@01 `01 @01 @01 @01 @01 @01 @01 `01 @01 @01 @01 @01 @01 @01 @01 @01 @01 @01 @01 @01 @01 @01@01@01 @01 @01 @01 @01 @01 @01 `01 @01 @01@ 01@ 01@ 01@0@0@0@0*@0@0Ѭ@ 0Ѭ@ 0Ѭ@ 0Ѭ@ 0Ѭ@ 0Ѭ@0Ѭ@ 0Ѭ@ 0Ѭ@ 0Ѭ@ 0Ѭ@ 0Ѭ@0Ѭ@ 0Ѭ@0Ѭ*@0@0b@0b@ 0b@ 0b@ 0b@ 0b@ 0b@0b@0b@0b @0b @0b @0b @0b @0b @0b @0b @0b @0b @0b @0b @0b @0b @0b @0b @0b @0b @0b @0b @0b @0b @0b @0b @0b*@0@0T@0T@0T@0TJ@0T@ 0L@ 0L@ 0LJ@0T@ 0@ 0@ 0J@0T@ 0@ 0@ 0J@0T@ 0 @ 0 @ 0 @ 0 J@0T@ 0 }@ 0}@ 0}@ 0}J@0T@ 0@ 0@ 0@ 0J@0T@ 0@ 0@ 0:@0T@0Y@ 0Y@ 0Y@0Y@ 0Y@0Y@ 0Y*@0@0c*@ 0*@ 0*@ 0@0@0*@0@0@0J@0@0@0@ 0@ 0@0J@0@0@0J@00 MMtttttt*^7nDNZmdkфl{|bwLmB"'X((-?OZx]tdTj!o^ƣDzɷt9bDX!LY 2"%(X(W((;!!"|""")**},,--0.R.777h999.<o<<>>>AB%BDDDRH}HHL4MM^7_m_```TaaacHczcdGd{d5RvU #D{ԳdF3k r* =}g?w74^J3q=y,(aX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX]df!!8@0(  B S  ?-!.r!/h0\A"1z23`4dg5\6К7N8!9\:<ϙ;< ?=쉢>Dd?@|1ABCD? EۧF!G HԏIJ KԮL M<>F>@@@@BBBBdCdCCCEEEOIYI`IIIIKK KKK6L6LtLtLL^^^~`.abb'b2b@b@bbbgggg>h>hqqq,},}~}~}~~~^^IIPPZa88@yyůůWWbb!!tt@DOdd&&4~Y       !"#$%&'()+*,-./0126345789;:<>=?@ABCDEFGHIJKLMNOQPRSTUVWXYZ[\]^_b`acedfhgijklmnopqrstuwvxyz{|}~    ~~^hpp;##3#x+x+--;;E>M>M>@@@@BBBBkCkCCCEEEXI_IjIIII KKK!K!K?L?L~LLL^^^`4ab%b0b;bFbFbbbgggg@h@hqqq2}2}}}~~~eePPY`kk?LLǯǯ]]jj##vvBJXff0;;  Y     !"#$%&'()+*,-./0124563789;:<>=?@ABCDEFGHIJKLMNOQPRSTUVWXYZ[\]^_ab`cedfhgijklmnopqrstuwvxyz{|}~h*urn:schemas-microsoft-com:office:smarttagsCity0http://www.5iamas-microsoft-com:office:smarttagsB*urn:schemas-microsoft-com:office:smarttagscountry-regionV*urn:schemas-microsoft-com:office:smarttagsplacehttp://www.5iantlavalamp.com/=*urn:schemas-microsoft-com:office:smarttags PlaceName=*urn:schemas-microsoft-com:office:smarttags PlaceType Mh )zz{|* =}g?7_3z(    ! < = > I V Y )zz{|* =}g?7_3z(    ! < = > I V Y  GJPV|>LQ /#####$$$..22226888;;;;;;CC,EFEHEYEEEEEFFFF+G?GGGJJJJJJ^)^`^}^~^^``J`b`z`|`ccccqr|||}}}~~~~()6ʏ`xڭۭ bmrrҲӲOPQXcg##aq(,u{|@Bzz{|* =}g?7_3z(mo8   ! " ' ; > I J O U V Y  GJPV|>LQ /#####$$$..22226888;;;;;;CC,EFEHEYEEEEEFFFF+G?GGGJJJJJJ^)^`^}^~^^``J`b`z`|`ccccqr|||}}}~~~~()6ʏ`xڭۭ bmrrҲӲOPQXcg##aq(,u{|@Bzz{|* =}g?7_3z(mo8    ! " " < = > I J J Y :{ R  iqt _UN5jEBb~0H#X/& '$d)l .8Rd22X<+'6Bbqz6VL*g8nbZ,M;fpt@{9S(p4"Q GV2pF\WXD[$'`PbtB5c>B/k RF"Knކ*Wuni@n$xrZ<|=juWPgeyVL*zRŠ0^`0OJQJo(hH^`OJQJ^Jo(hHopp^p`OJQJo(hH@ @ ^@ `OJQJo(hH^`OJQJ^Jo(hHo^`OJQJo(hH^`OJQJo(hH^`OJQJ^Jo(hHoPP^P`OJQJo(hHh ^`hH.h ^`hH.h pLp^p`LhH.h @ @ ^@ `hH.h ^`hH.h L^`LhH.h ^`hH.h ^`hH.h PLP^P`LhH. hCJaJo(h ^`hH.h ^`hH.h pLp^p`LhH.h @ @ ^@ `hH.h ^`hH.h L^`LhH.h ^`hH.h ^`hH.h PLP^P`LhH.0^`0OJQJo(hH^`OJQJ^Jo(hHopp^p`OJQJo(hH@ @ ^@ `OJQJo(hH^`OJQJ^Jo(hHo^`OJQJo(hH^`OJQJo(hH^`OJQJ^Jo(hHoPP^P`OJQJo(hH0^`0OJQJo(hH^`OJQJ^Jo(hHopp^p`OJQJo(hH0^`0OJQJo(hH^`OJQJ^Jo(hHo^`OJQJo(hH^`OJQJo(hH^`OJQJ^Jo(hHoPP^P`OJQJo(hHh ^`hH.h ^`hH.h pLp^p`LhH.h @ @ ^@ `hH.h ^`hH.h L^`LhH.h ^`hH.h ^`hH.h PLP^P`LhH.h ^`hH.h ^`hH.h pLp^p`LhH.h @ @ ^@ `hH.h ^`hH.h L^`LhH.h ^`hH.h ^`hH.h PLP^P`LhH.h ^`hH.h ^`hH.h pLp^p`LhH.h @ @ ^@ `hH.h ^`hH.h L^`LhH.h ^`hH.h ^`hH.h PLP^P`LhH.0^`0OJQJo(hH^`OJQJ^Jo(hHopp^p`OJQJo(hH@ @ ^@ `OJQJo(hH^`OJQJ^Jo(hHo^`OJQJo(hH^`OJQJo(hH^`OJQJ^Jo(hHoPP^P`OJQJo(hH0^`0OJQJo(hH^`OJQJ^Jo(hHopp^p`OJQJo(hH@ @ ^@ `OJQJo(hH^`OJQJ^Jo(hHo^`OJQJo(hH^`OJQJo(hH^`OJQJ^Jo(hHoPP^P`OJQJo(hH0^`0o(.^`. L ^ `L.  ^ `.xx^x`.HLH^H`L.^`.^`.L^`L.h ^`hH.h ^`hH.h pLp^p`LhH.h @ @ ^@ `hH.h ^`hH.h L^`LhH.h ^`hH.h ^`hH.h PLP^P`LhH.h ^`hH.h ^`hH.h pLp^p`LhH.h @ @ ^@ `hH.h ^`hH.h L^`LhH.h ^`hH.h ^`hH.h PLP^P`LhH.h0^`0OJQJo(hHh^`OJQJ^Jo(hHohpp^p`OJQJo(hHh@ @ ^@ `OJQJo(hHh^`OJQJ^Jo(hHoh^`OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHohPP^P`OJQJo(hH 0^`0o(hH. ^`hH. pLp^p`LhH. @ @ ^@ `hH. ^`hH. L^`LhH. ^`hH. ^`hH. PLP^P`LhH.0^`0o(0^`0o(.0^`0o(..0^`0o(... 88^8`o( .... 88^8`o( ..... `^``o( ...... `^``o(....... ^`o(........h ^`o(hH.h ^`hH.h pLp^p`LhH.h @ @ ^@ `hH.h ^`hH.h L^`LhH.h ^`hH.h ^`hH.h PLP^P`LhH.0^`0OJQJo(hH^`OJQJ^Jo(hHopp^p`OJQJo(hH@ @ ^@ `OJQJo(hH^`OJQJ^Jo(hHo^`OJQJo(hH^`OJQJo(hH^`OJQJ^Jo(hHoPP^P`OJQJo(hHh ^`hH.h ^`hH.h pLp^p`LhH.h @ @ ^@ `hH.h ^`hH.h L^`LhH.h ^`hH.h ^`hH.h PLP^P`LhH.h ^`hH.h ^`hH.h pLp^p`LhH.h @ @ ^@ `hH.h ^`hH.h L^`LhH.h ^`hH.h ^`hH.h PLP^P`LhH.h0^`0OJQJo(hHh ^`hH.h pLp^p`LhH.h @ @ ^@ `hH.h ^`hH.h L^`LhH.h ^`hH.h ^`hH.h PLP^P`LhH.h ^`hH.h ^`hH.h pLp^p`LhH.h @ @ ^@ `hH.h ^`hH.h L^`LhH.h ^`hH.h ^`hH.h PLP^P`LhH.h ^`o(hH.h ^`hH.h pLp^p`LhH.h @ @ ^@ `hH.h ^`hH.h L^`LhH.h ^`hH.h ^`hH.h PLP^P`LhH.0^`0OJQJo(hH^`OJQJ^Jo(hHopp^p`OJQJo(hH@ @ ^@ `OJQJo(hH^`OJQJ^Jo(hHo^`OJQJo(hH^`OJQJo(hH^`OJQJ^Jo(hHoPP^P`OJQJo(hH0^`0OJQJo(hH^`OJQJ^Jo(hHopp^p`OJQJo(hH@ @ ^@ `OJQJo(hH^`OJQJ^Jo(hHo^`OJQJo(hH^`OJQJo(hH^`OJQJ^Jo(hHoPP^P`OJQJo(hH0^`0OJQJo(hHpp^p`OJQJ^Jo(hHo@ @ ^@ `OJQJo(hHx0x^x`0OJQJo(hH^`OJQJ^Jo(hHo^`OJQJo(hH^`OJQJo(hHPP^P`OJQJ^Jo(hHo  ^ `OJQJo(hH0^`0OJQJo(hH^`OJQJ^Jo(hHopp^p`OJQJo(hH@ @ ^@ `OJQJo(hH^`OJQJ^Jo(hHo^`OJQJo(hH^`OJQJo(hH^`OJQJ^Jo(hHoPP^P`OJQJo(hHh^`OJQJo(hHhpp^p`OJQJ^Jo(hHoh@ @ ^@ `OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHoh^`OJQJo(hHh^`OJQJo(hHhPP^P`OJQJ^Jo(hHoh  ^ `OJQJo(hHh ^`hH.h ^`hH.h pLp^p`LhH.h @ @ ^@ `hH.h ^`hH.h L^`LhH.h ^`hH.h ^`hH.h PLP^P`LhH. 0^`0o(hH. ^`hH. pLp^p`LhH. @ @ ^@ `hH. ^`hH. L^`LhH. ^`hH. ^`hH. PLP^P`LhH.g80d)pt@=ju_U0zZ,M; .k ~0H#Kn*Wunn$xr{9SjE<+'6biqqz6gey& 'D[\WB5c'` Q GV:{d2|                 |        2  |                                2        |        Z                 2        9                |        |                 |        |        |  |     |                 P*        p"t($ #=1?<FCSrUt L2,R[Su@2sp M.`(A@RRRRRRRwUxUXXt\u\̏͏ُ']gϒZߔ ,*^_4j›pМ\| ,zVwϢ0w%&Dm#bթ'fƪǪȪɹ˹ֹ tuwĺ)9::;<BCDELNPRTVX^ijJ|0Zc!)1<\#+ g-.LPTX^dtEs'YZ[iO "c d$eH +, kvw W  = j      /   ?   t u   4L7aF zXYf3p!"HImp 7^J12Wn'@v78(PQR,-NBY @-h X p@UnknownGz Times New Roman5Symbol3& z Arial7&  Verdana;" Helvetica; Batang?5 z Courier New;Wingdings"1he $f &)9)9"4d 2qHX?.`2%Theory of Benchmarking for e-LearningBenchmarking MBS for e-Learning Paul Bacsich Paul Bacsich                          Oh+'0HX p|    (Theory of Benchmarking for e-Learning Benchmarking MBS for e-LearningPaul Bacsich$Quality Review: Sara Frank BristowNormalPaul Bacsich9Microsoft Office Word@dN@؛@b6@6)՜.+,D՜.+,|8 px  Lynda BacsichMatic Media Ltd9 &Theory of Benchmarking for e-Learning Title, 8@ _PID_HLINKSA[*http://www.trainingplace.com/building.htm ]/Ihttp://www.usyd.edu.au/quality/teaching/docs/revised_tandl_plan_2004.pdf !&<http://www.westga.edu/~distance/ojdla/fall63/scanlan63.html vI/http://cde.athabascau.ca/online_book/ch16.html c$~Thttp://www.webct.com/service/viewcontentframe?contentID=2386007&pageName=index.html 4.{@http://www.utdc.vuw.ac.nz/research/emm/documents/literature.pdf Fx1http://crpit.com/confpapers/CRPITV30Marshall.pdf cu?http://www.nln.ac.uk/downloads/pdf/BEC11392_NLNComprep36pp.pdf Rr)http://www.ihep.org/Pubs/PDF/Quality.pdf (o.http://www.hefce.ac.uk/pubs/hefce/2005/05_12/ 6tlHhttp://www.ascilite.org.au/conferences/brisbane99/papers/bacsichash.pdf 4yi<http://www.acu.ac.uk/chems/onlinepublications/961780238.pdf  Xfihttp://www.astd.org/NR/rdonlyres/1CC4FE41-DE6A-435E-8440-B525C21D0972/0/State_of_the_Industry_Report.pdf LNc2http://www.usyd.edu.au/quality/teaching/mou.shtml ]/`Ihttp://www.usyd.edu.au/quality/teaching/docs/revised_tandl_plan_2004.pdf .*]http://www.go8.edu.au/ YZMhttp://www.estrategy.warwick.ac.uk/FinalDoc/elearnUnitDoc/elearnunitdoc.html YWEhttp://www.estrategy.warwick.ac.uk/FinalDoc/elearnDoc/elearndoc.html @T-http://www.estrategy.warwick.ac.uk/FinalDoc/ 2Q(http://www.utdc.vuw.ac.nz/research/emm/ NFhttp://www.stanford.edu/class/symbsys205/Diffusion of Innovations.htm {KIhttp://www.heacademy.ac.uk/learningandteaching/eUniCompendium_chap01.doc ,{H4http://www.citscapes.ac.uk/products/phase1/ch10.pdf ^EThttp://www.nln.ac.uk/lsda/self_assessment/files/Self_assessment_tool_Guidelines.doc Bhttp://www.nln.ac.uk/ j?>http://www.nln.ac.uk/lsda/self_assessment/files/Benchmark.pdf 5o<4http://www.coimbra-group.be/DOCUMENTS/portfolio.doc T92http://www.coimbra-group.be/DOCUMENTS/summary.doc 26+http://www.coimbra-group.be/06_members.htm A3http://www.coimbra-group.be/ iD06http://www.ubcinternational.ubc.ca/universitas_21.htm "-Lhttp://science.uniserve.edu.au/courses/benchmarking/benchmarking_manual.pdf Q*http://excellence.shu.ac.uk/ }#'http://www.eubc.bham.ac.uk/ $http://www.leadership-he.com/ #s!#http://www.heacademy.ac.uk/914.htm D_2http://www.jisc.ac.uk/index.cfm?name=funding_4_03 wI*http://www.jisc.ac.uk/dfes_elearning.html eX/http://www.jisc.ac.uk/strategy_jisc_04_06.html F"http://www.hefce.ac.uk/siteindex/ uZChttp://www.esmu.be/download/benchmarking/BENCH_YEAR5_INFO_NOTE.doc 4hhttp://www.esmu.be/ !n http://www.acu.ac.uk/chems/ 4y <http://www.acu.ac.uk/chems/onlinepublications/961780238.pdf #s#http://www.heacademy.ac.uk/914.htm ,W5http://www.benchmarking.gov.uk/about_bench/types.asp (.http://www.hefce.ac.uk/pubs/hefce/2005/05_12/   !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~      !"#$%&'()*+,-./012356789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~Root Entry F݇6Data ӝ1Table4L1WordDocumentmSummaryInformation(DocumentSummaryInformation8CompObjq  FMicrosoft Office Word Document MSWordDocWord.Document.89q