Definition of Competency or Understanding of Competency
To properly discuss the evaluation of library programs and services, I found it necessary to understand the difference between evaluation and assessment in the literature. According to Saunders, evaluation measures services in order to judge the overall quality or value of that service, whereas assessment tries to understand the “state and impact” of a service and provide feedback that can be used for improving the service (2016, p 213). This distinction is shared by Starr: evaluation makes a judgement against a set of standards, while assessment provides non-judgmental direction for “improving future performance” (Starr, 2014). Evaluation and assessment are bound together in an overall iterative evaluative cycle consisting of several steps:
The first step in the evaluative process is operationalization, that is to define the terms of the study, to consider how areas to be studied will be measured, and to establish standards and criteria against which researchers will measure (Saunders, 2016, p 227). The evaluative process forces researchers to consider assessment criteria— the methods, factors, and means of collecting data— to ensure that the data is measurable, and context is important (McClure, 2008, pp 183-185). What should be evaluated or assessed? The research done around reference services has historically focused on inputs (resources/infrastructure) and outputs (products/services), but since the 1990s has shifted to an emphasis on “evaluating quality and assessing outcomes” (Saunders, 2016, p 217; Tyckoson, 2008, p 140). A contemporary approach to evaluative research takes different perspectives into consideration, for instance not only the user, but stakeholders, librarians, or the library system as a whole (Saunders, 2016, p 219). Evaluation, by judging how successful library services or programs are, functions as a “validity check,” and thereby links itself to a non-judgmental assessment that simply asks: how might services might be improved? (McClure, 2008, p 181).
Evaluation standards or methods vary, but Saunders describes two basic types: quantitative and qualitative:
As the name implies, quantitative methods focus on information that can be represented numerically and might include frequency counts, percentages, and ratios. Common types of methods for gathering quantitative measures include surveys, questionnaires, or polls (with close-ended questions); transaction log analysis (including reviews of analytics of website use); and bibliometrics (including citation analysis). Qualitative research, on the other hand, relies on observation and description. Some common types of qualitative research include interviews, focus groups, document analysis, critical incident questionnaires, and unobtrusive observation. (2016, pp 227-228)
For instance, Tyckoson identifies four common methods of evaluating reference services: unobtrusive testing, obtrusive testing, willingness-to-return, and a more formal method, the Wisconsin-Ohio Reference Evaluation Program (WOREP)” (2008, p 141). A more codified set of standards for evaluating reference services can be found in the Reference and User Services Association (RUSA) document, Guidelines for Behavioral Performance of Reference and Information Service Providers, which uses five behavioral factors, measurable criteria by which reference success might be determined: approachability, interest, listening/inquiry, searching, and follow-up (Tyckoson (2008, p 140; RUSA, 1996; Guidelines, n.d.). Reference success with this methodology correlates librarian behavior to “positive patron satisfaction” (Tyckoson, 2012, p 597). In contrast, one might consider IFLA’s Guidelines for library services to prisoners, which contains no obvious evaluation criteria with regard to prison library services (Lehmann and Locke, 2005). Another set of criteria, similar to that created by RUSA, are McClure’s 6 quality standards (courtesy, accuracy, satisfaction, repeat users, awareness, cost) (Saunders, 2016, p 227). According to Saunders, evaluation and assessment both contain criteria or standards, and also answer questions (what, who, where, when, and how) regarding the research process (2016, p 226).
A good, modern example of a model for evaluating instruction is Kaplowitz’s Teaching Tripod, which consists of three interactive elements— expected learning outcomes (ELOs), the learning activity or activities to be analyzed, and assessment (Kaplowitz, 2014, p 112). In Kaplowitz’s model the evaluative standards are established in the writing of the ELOs, which involves constructing sentences with four components, audience, behavior, condition, and degree (Kaplowitz, 2014, pp 65-72). These ELOs, and their constituent parts, seemingly establish and define what is meant by “measurable criteria”. What are we to make of this tautology? For instance, Kaplowitz states that “all these assessments must have something to measure, and what they measure is the attainment of whatever ELO’s are associated with the class, course or program” (2014, p 114). I read this as saying that we measure criteria with measurable criteria, or that in this model, what is measurable depends on how one defines it ahead of time. Regardless of this chicken-and-egg question regarding measurable criteria, ELO’s are intended to answer two questions about the second part of the teaching tripod, the learning activity: what will the learners be able to do following instruction, and how will you know they can do it? (Kaplowitz, 2014, p 60).
The third element in the teaching tripod involves assessment of the learning activities. According to Kaplowitz, there are 4 types of behavioral assessments: reaction (did they like it?); learning (did they get it?); behavioral (can they do it?); and results (did it matter?), and can be made before, during, or after instruction (Kaplowitz, 2014, pp 114-115). Kaplowitz also lists 6 different “assessment categories”— forced choice; open-ended assessments; surveys & questionnaires; interviews & focus groups; classroom assessment techniques; and authentic assessments, (Kaplowitz, 2014, pp 125-127), which are essentially the same as the evaluation methods previously described (Saunders, 2016, pp 227-228). In fact, Kaplowitz appears to construct the assessment element of her Teaching Tripod as an amalgamation of both evaluation and assessment. For example, she declares that the assessment element can be seen as either “an example of constructive feedback or of accountability” depending how the assessment is implemented, scheduled, recorded, or to whom it is presented (Kaplowitz, 2014, p 127); “constructive feedback or accountability” recalls the purported differences we outlined earlier between evaluations and assessments. That the assessment element in Kaplowitz’s model has the flexibility to be used in this manner, makes it useful for discussing how evaluation (or assessment) can improve the design or provision of services and programs.
Evaluation is linked to planning, and that planning usually involves “an evaluation process to determine the degree to which the goals and objectives were accomplished. Planning is making choices of what to do and what not to do” (McClure, 2008, pp 180-183). This linkage can be seen in Saunder’s evaluation/assessment cycle previously mentioned (2016, p 215). Kaplowitz so thoroughly embodies planning in her instructional model, that she calls learning outcomes (ELOs), learning activities, and assessment, the three elements that define her model, as “vital to effective instructional design” (Kaplowitz, 2014, p 112), a remarkably tautological statement, considering that Kaplowitz’s model itself is an example of instructional design. Nevertheless, this is a great example of how evaluation might improve the design or provision of services and programs. Planning and evaluation make up stages of design plans, whether they be focused on creating or improving programs and services, or on judging the value of extant programs and services, and oftentimes the line between evaluation and assessment is blurry, even if the measurement criteria is clear and specific (McClure, 2008, pp 180-190). Both evaluation and assessment, for example, are essential elements in library marketing plans, where carefully linked objectives and outcomes are intentionally mapped to measurable factors, so that data can be aggregated, analyzed, and eventually applied back to those same marketing plans (Alman, 2016, p 341).
The evaluation of library programs and services using measurable criteria is critical for librarians and information organizations because ongoing assessment upholds the quality and usefulness of these programs and services, which is important within a very real political context (McClure, 2008, pp 189-190). For information professionals and organizations, it is less important what evaluative model is used, and more important that a culture of evaluation/assessment be cultivated (Saunders, 2016, p 237; McClure, 2008, p 180). Just as evaluation and planning are linked in an information organization, an evaluation/assessment culture can also be thought of as an “organizational culture” in that there are shared and “appropriate” ways of doing things that are baked into the system (McClure, 2008, p 180).
It is important to me that any kind of “organizational culture” or evaluation/assessment culture, be participatory and more importantly, democratic in structure, as all the moving parts of libraries, including services and programs, embody values (“intellectual freedom the common good, service to all, stewardship of the human record, free access to knowledge and information, nondiscrimination, etcetera”) (Gorman, 2008, p 21) that are democratic in nature. For a library to become a culture of evaluation, it must be intentionally designed by librarians capable of assessing both themselves and their environment. In a sense the chicken-and-egg metaphor is quite apt here, as it may be hard to discern which precedes the other— the library or the librarian. For myself, I find this competency important because at base it implies our responsibilities as information professionals, to be stewards of our institutions. Information organizations might want to dig into the ethical foundations of evaluation and understand how important it is to sow the seeds of critical thinking and honest self-assessment, so that an “organizational culture” can take root.
Preparation to Understand Competency N: Coursework and Work Experience
Working for a decade in a large, public library system, I have become somewhat cognizant of the standards and mechanisms used by the administration to evaluate programs and services, though their criteria has not always been apparent to me. An exception to this has been the recent introduction of standard core competencies as a way for staff and management to work together to assess performance and development. In this case, the competencies, and the attributes/criteria that make them up, are clearly laid out. The rollout of the competencies themselves seems to be part of a project, the design of which the administration hopes to improve through an iterative process.
I have also learned a lot through reading information science literature, particular full-text, peer-reviewed papers for my INFO 200, 204, and 210 classes. After reading a great number of these academic papers, one begins to understand some universal elements, such as defining terms, choosing a research methodology or methodologies, identifying objectives or expected outcomes, setting parameters, planning and chronicling the research, and so forth. Aside from the papers I cite below as evidence, my work constructing a collection development policy for INFO 266 showed me how evaluative methods, using measurable criteria, are designed for use in purchasing or weeding materials for a library collection. While that project focused on content, rather than services, the same essential process and critical analysis was involved, and therefore could be said to have prepared me for this competency.
I chose to present my evidentiary items in roughly the chronological order in which they were created, with the exception of the first item, which describes what I learned about evaluating programs during my summer internship. I chose to present these scanned items first because, taken as a whole they remain a standalone evidentiary item, which mostly illustrates my interest in, and pursuit of, information regarding how librarians evaluate their programs.
Evidence
Metrics for Evaluating Programs
Discussion of Evidentiary Items
https://documentcloud.adobe.com/link/review?uri=urn:aaid:scds:US:5ce6cf8c-ac9d-4e0c-b4c4-20d9d48e0e2b
My first evidentiary item consists of scanned documents from my internship (INFO 294) in XXXX’s Eastern region, where I planned and launched my own teen program, and assisted librarians on their programs. The first few pages here show how I reached out, early on in my internship, to April Gonzalez, XXXX’s Diversity Coordinator, to ask by what metrics XXXX might be evaluating its programs, and/or what metrics librarians might be using to gauge the effectiveness of their programs. My inquire was a practical one: I wanted to plan my own program with evaluation and/or assessment in mind. A follow-up document shows the learning outcomes I was using to track my project for SJSU, and the programs I worked on. After this are pages that show the planning for and eventual conversation with Ms. Gonzalez. A page from my logbook shows the results of my inquiry about evaluation methods and metrics, including discussion of Outcomes for Success, a book describing how to plan for the evaluation of library services. I saved a few pages for that book and scanned them as an example here. April also provided a page that list of operational definitions for diversity, equity, and inclusion that were in development; these definitions might eventually be used as measurable criteria in evaluating programs or services.
The following 11 pages consist of examples of programming planning sheets that I gathered from librarians over the course of my internship. Because the evaluation of XXXX programs is centrally managed, I knew I would not be finding an evaluative blueprint in these planning sheets. But what I did find was that XXXX librarians create their own methods for assessing their programs, and the vast majority use their self-created planning sheets as a way of tracking their work, and as a place to compile their data. I found that one or two (not shown here) used Excel spreadsheets to manage this compiled data. After these planning sheets I included 8-9 pages that show some of the work I did on my own project, and other’s programs. The relevant pages here are the log entries, which show the data I collected data from these programs, and how I organized these entries to map the activities to learning outcomes. The final scanned page shows the first page of a paper that April Gonzalez passed on to me, that involved research on immigrant and refugee entrepreneurs, and begins to describe how they coded and synthesized focus group (interview) data.
These pages are presented as an illustration of my interest in how the evaluation of programs takes place within a large public library, and my desire to understand how librarians might assess their own programs in lieu of a transparent and uniform evaluative model.
Organizational Analyses for The Historical Society of Pennsylvania, Part 1
Discussion of Evidentiary Items
https://documentcloud.adobe.com/link/review?uri=urn:aaid:scds:US:cd648d7c-283a-4baa-8401-02d6cdf32097
The major group project I worked on for Stenstrom’s Information Professions (INFO 204) class was a two-part organizational analysis of an information organization. My group chose to analyze the Historical Society of Pennsylvania (HSP), a research library. In the first part of our analysis we were to conduct a literature review, a SWOT analysis, and draw understanding from HSP’s mission, vision, and values statements. The primary work I did for this project was to conduct a thorough literature review on the historical evolution of mission statements for U.S. research libraries. One of my primary purposes for conducting the review, was to give our team a rich stratum of information with which we might reassess HSP’s mission, vision, and values statements, and reimagine another, more appropriate set of guiding statements.
My research shows how both public and academic visions for research libraries intertwine and overlap over time, how higher education and digitalization were forces that shaped both historical societies and their missions, and how important it is to not make presumptions about user needs. Finally, some of my research led to some truisms about how mission statements ought to be derived from outreach and understanding the library’s community base, how mission statements fail not from language or focus but from undefined goals, and how they generally need to be more visible to the public. Once I had sifted through and analyzed my research, I was able to construct new mission, vison, and values statements by tying the core HSP values we wanted to retain with insights from the research. The words were drawn from original statements, but I modified them to be more in-line for a modern, forward-looking institution. I also chose my words carefully, understanding that statements used in a strategic plan can be considered measurable criteria by which HSP will evaluate itself in the future. For instance, the goal of “being a culturally valuable entity,” was deliberately chosen as an outcome towards which the library could be expected to strive, and for which it could reasonably be assessed over time.
My contribution to the SWOT analysis was of a more general nature, focusing mostly on encouraging my co-researchers to fill out more detail in important areas such as sociocultural factors or naming HSP’s major competitors. As with the literature review, I approached this task with the understanding that the information that we fed into the SWOT analysis would provide hard data with myriad, practical uses in forging a new strategic plan. For example, data of this sort can lead to developing measurable criteria for future assessments or it might be used as a way of developing standards for evaluating services. Within the parameters of the strategic plan of course, I knew that the data would be used as context for the goals we would set out to achieve, and the means by which we would achieve them. Still, as with the mission, vision, and values statements I wrote, I consciously pushed for greater complexity in the SWOT analysis, understanding that assessment and planning go hand in hand, and once our strategic plan was in place, the components upon which it was constructed would be used as a guide for what, when, where, why, and how to assess our progress.
Working on the HSP Organizational Analysis project shows that I understand how assessment and planning go hand and hand, and the role that research plays in both evaluating plans and in constructing a plan that will have future evaluative uses. I show that I understand what goes into evaluating or assessing diverse entities such as strategic plans, research papers, or even entire information organizations. Finally, I demonstrate an understanding of planning and evaluation that is cognizant of inputs and outputs, of accurate and relevant data being used to forge measurable criteria, and of the need for using both qualitative and quantitative research to improve whatever we are designing.
Reference Observation Assignment
Discussion of Evidentiary Items
https://documentcloud.adobe.com/link/review?uri=urn:aaid:scds:US:d90f9742-4017-4276-8517-0e420e054f7c
In this assignment for my INFO 210 class I arranged to observe a reference librarian over the course of a single “shift” and observe their interactions with patrons, where I would keep a field journal of notes on the reference desk experience and reference interviews, followed by an analysis of what I observed, including how the service was designed and delivered, and any examples of patron feedback. Part of the assignment was to keep the librarian, the user, and the myself (the observer) in mind, an important factor in modern evaluation of information services. I chose to observe a virtual reference service assistant performing phone interviews at AskKCLS, a virtual chat service.
I decide to filter my observations for this assignment through the IFLA and RUSA guidelines for reference service professionals (Guidelines, n.d.; RUSA, 1996), two documents that list and codify criteria by which reference service activity can be measured. After observing a reference service assistant over the course of one work shift, I analyze and evaluate the reference interviews against the measurable criteria in the IFLA and RUSA guidelines. For instance, at the beginning of an inquiry I note the assistant using good virtual reference practices such as being prepared, acknowledging the patron, and using a neutral but friendly tone of voice, and over the course of the phone call I observe them using closed and open questions, explaining their search strategy to the patron, and citing authoritative sources (Shachaf & Horowitz, 2008, pp 25-30).
Taking the data points from my evaluation of the reference services assistant (using measurable criteria), I provide a reasoned evaluation of how services might be improved. I provide suggestions for redesigning the chat service including the idea that a return to mentoring might reinvigorate a simple reliance on IFLA/RUSA guidelines, allowing for behaviors like being “passionately curious” to be properly valued in actual service delivery. I argue that when we have two information behavior data sets to draw upon (the user and the library associate) we can plan and design better reference services.
This assignment shows that I am able to evaluate an actual working instance of reference services using measurable criteria and, based on that data, proffer design improvements of those services. I show that I understand how documents like the IFLA/RUSA guidelines, are produced through an iterative practice of evaluation and planning, and that these standards we create are improved through their use in evaluating actual service provided in the field. Finally, I demonstrate an understanding that our evaluations of services improve when we take into consideration all the perspectives in play, including the patron, the observer, and the information professional.
Instructional Design Plan Draft
Discussion of Evidentiary Items
https://documentcloud.adobe.com/link/review?uri=urn:aaid:scds:US:b2928b0f-4d6e-4545-ad1c-3fa147695dd6
There are several evaluative elements woven into the instructional design plan I co-created for INFO 250. My partner and I chose Kaplowitz’s Teaching Tripod ID Model because of its flexible modern design, containing only three elements— expected learning outcomes (ELOs); learning activities; and assessment. I did the bulk of the work in transforming this rough draft into a five-minute screencast introduction, as well as a six-page paper capable of being orally presented in a 15-minute timeframe. I accomplished this by removing the scaffolding from our rough draft. In order to remove what we did not need, I had to understand the components of Kaplowitz’s model well, as each leg of the tripod informs every other leg.
I performed evaluation and assessment of services throughout the design of this plan. I and my partner interviewed our librarian clients, as part of a needs assessment. Using this data as a starting point she and I wrote instructional goals and evaluated and defined learner characteristics and behavior. The constructing of the ELOs was a decidedly collaborative effort where I focused on editing and structure and deferred to my partner regarding final wording and the exact degree to which students would be measured. My work here was to be sure that our terms were defined, that ELOs were parallel in structure, and that the elements making up the ELOs were not only represented, but did in fact consist of measurable criteria. Another example of this was in the instructional strategies, where my partner conceived of how the instruction would function, and I double and triple-checked to make sure that a) all measurable criteria was properly applied, b) that all measurable criteria could be traced back to the appropriate ELO, and c) that the instruction (program) was described accurately. The section on formative and summative evaluation was a culmination of this approach, where my partner and I went through a cycle whereby she would conclude that a tutorial would produce a certain evaluative result, and I would demonstrate that this result could not be achieved. This iterative, collaborative approach finally resulted in my being able to describe how our instruction could and would deliver certain evaluative results.
My summation of our instructional design is, in a sense, an argument for how evaluation and assessment can improve information services. I write how I and my partner’s evaluation and assessment of our client’s instructional interface, their students’ needs and behaviors, the library environment, and the librarians’ information needs were boiled down into measurable criteria, and using Kaplowitz’s evaluative model, a new and improved instructional interface designed.
My work on this Instructional Design Plan illustrates my understanding of evaluative elements within an instructional design model like Kaplowitz’s Teaching Tripod, and their functions. I demonstrate that I can perform a needs assessment or design formative and summative assessments. I show that I know how to develop measurable criteria, and that I am capable of ensuring that assessments are properly tied to such criteria. Lastly, I can articulate and demonstrate how evaluation and assessment can transform and improve instruction.
Conclusion
My interest and curiosity in how information professionals and organizations evaluate their programs and services, comes across in how instinctively I dig for data that describes this process. Because I understand how planning and evaluation go hand-in-hand, I am cognizant of inputs and outputs, of how accurate and relevant data is used to forge measurable criteria, and of the need for using both qualitative and quantitative research to improve whatever we are designing. Showing myself capable of evaluating an information service, and alternately capable of using evaluative methods and models to re-evaluate, and thereby redesign existing information entities, means having a skill applicable in any information organization. Understanding how this iterative practice of evaluation and planning produces standards and guidelines of measurable criteria, means always knowing how I might contribute within my chosen profession. And finally, demonstrating that I can design formative and summative assessments tied to measurable criteria, means that I have a necessary tool should I want to evaluate my own library program, or any information service or program.
References
Alman, S. W. (2018). Communication, marketing, and outreach strategies. In Hirsh, S. (Ed.). Information services today: An introduction. Lanham, Maryland: Rowman & Littlefield
Gorman, M. (2008). Professional ethics and values in a changing world. In Haycock, K. & Sheldon, B.E. (Eds.). The portable MLIS: Insights from the experts.15-22. Westport: Libraries Unlimited.
Guidelines for behavioral performance of reference and information service providers. (n.d.). Retrieved June 26, 2016, from: http://www.ala.org/rusa/resources/guidelines/guidelinesbehavioral
International Federation of Library Associations (2005). IFLA digital reference guidelines. Retrieved from http://www.ifla.org/VII/s36/pubs/drg03.htm
Kaplowitz, J.R. (2014). Designing information literacy instruction: The teaching tripod approach. Lanham, MD: Rowman & Littlefield
Lehmann, V. & Locke, J. (2005). Guidelines for library services to prisoners (3rd ed.). IFLA
Professional Report, 92. The Hague: International Federation of Library Associations
and Institutions.
McClure, C.R. (2008). Learning and using evaluation: A practical introduction. In Haycock, K. &
Sheldon, B.E. (Eds.). The portable MLIS: Insights from the experts. Westport, Connecticut: Libraries Unlimited.
Reference and User Services Association. (1996). Guidelines for behavioral performance of reference and information services professionals. RQ, 36(2), 200-203. Retrieved from http://www.ala.org/rusa/resources/guidelines/guidelinesbehavioral
Saunders, L. (2016). Evaluation and assessment of reference services. In Smith, L.C. and Wong,
M.A. (Eds.). Reference and information services: An Introduction, 212-243. Santa Barbara, CA: Libraries Unlimited
Shachaf, P., & Horowitz, S. M. (2008). Virtual reference service evaluation: Adherence to RUSA behavioral guidelines and IFLA digital reference guidelines. Library & Information Science Research, 30(2). Retrieved from http://eprints.rclis.org/12414/1/virtualReferenceServiceEvaluation.pdf
Starr S. (2014). Moving from evaluation to assessment. Journal of the Medical Library Association: JMLA, 102(4), 227–229. Retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4188047/
Tyckoson, D. A. (2012). Issues and trends in the management of reference services: A historical perspective. Journal of Library Administration, 51(3), 259-278. Retrieved from https://content.ebscohost.com/ContentServer.aspT=P&P=AN&K=503012446&S=R&D=lls&EbscoContent=dGJyMNLr40Sep7U4y9fwOLCmsEiep7ZSsK64TbCWxWXS&ContentCustomer=dGJyMPGtsVCuprFLuePfgeyx4YXr2QAA
Tyckoson, D. A. (2008). Reference service: The personal side of librarianship. In Haycock, K. &
Sheldon, B.E. (Eds.). The portable MLIS: Insights from the experts, 127-46. Westport, Connecticut: Libraries Unlimited.
To properly discuss the evaluation of library programs and services, I found it necessary to understand the difference between evaluation and assessment in the literature. According to Saunders, evaluation measures services in order to judge the overall quality or value of that service, whereas assessment tries to understand the “state and impact” of a service and provide feedback that can be used for improving the service (2016, p 213). This distinction is shared by Starr: evaluation makes a judgement against a set of standards, while assessment provides non-judgmental direction for “improving future performance” (Starr, 2014). Evaluation and assessment are bound together in an overall iterative evaluative cycle consisting of several steps:
- Choosing a focus and setting goals
- Gathering and analyzing data
- Making decisions
- Sharing results
- Revisiting goals (Saunders, 2016, p 215)
The first step in the evaluative process is operationalization, that is to define the terms of the study, to consider how areas to be studied will be measured, and to establish standards and criteria against which researchers will measure (Saunders, 2016, p 227). The evaluative process forces researchers to consider assessment criteria— the methods, factors, and means of collecting data— to ensure that the data is measurable, and context is important (McClure, 2008, pp 183-185). What should be evaluated or assessed? The research done around reference services has historically focused on inputs (resources/infrastructure) and outputs (products/services), but since the 1990s has shifted to an emphasis on “evaluating quality and assessing outcomes” (Saunders, 2016, p 217; Tyckoson, 2008, p 140). A contemporary approach to evaluative research takes different perspectives into consideration, for instance not only the user, but stakeholders, librarians, or the library system as a whole (Saunders, 2016, p 219). Evaluation, by judging how successful library services or programs are, functions as a “validity check,” and thereby links itself to a non-judgmental assessment that simply asks: how might services might be improved? (McClure, 2008, p 181).
Evaluation standards or methods vary, but Saunders describes two basic types: quantitative and qualitative:
As the name implies, quantitative methods focus on information that can be represented numerically and might include frequency counts, percentages, and ratios. Common types of methods for gathering quantitative measures include surveys, questionnaires, or polls (with close-ended questions); transaction log analysis (including reviews of analytics of website use); and bibliometrics (including citation analysis). Qualitative research, on the other hand, relies on observation and description. Some common types of qualitative research include interviews, focus groups, document analysis, critical incident questionnaires, and unobtrusive observation. (2016, pp 227-228)
For instance, Tyckoson identifies four common methods of evaluating reference services: unobtrusive testing, obtrusive testing, willingness-to-return, and a more formal method, the Wisconsin-Ohio Reference Evaluation Program (WOREP)” (2008, p 141). A more codified set of standards for evaluating reference services can be found in the Reference and User Services Association (RUSA) document, Guidelines for Behavioral Performance of Reference and Information Service Providers, which uses five behavioral factors, measurable criteria by which reference success might be determined: approachability, interest, listening/inquiry, searching, and follow-up (Tyckoson (2008, p 140; RUSA, 1996; Guidelines, n.d.). Reference success with this methodology correlates librarian behavior to “positive patron satisfaction” (Tyckoson, 2012, p 597). In contrast, one might consider IFLA’s Guidelines for library services to prisoners, which contains no obvious evaluation criteria with regard to prison library services (Lehmann and Locke, 2005). Another set of criteria, similar to that created by RUSA, are McClure’s 6 quality standards (courtesy, accuracy, satisfaction, repeat users, awareness, cost) (Saunders, 2016, p 227). According to Saunders, evaluation and assessment both contain criteria or standards, and also answer questions (what, who, where, when, and how) regarding the research process (2016, p 226).
A good, modern example of a model for evaluating instruction is Kaplowitz’s Teaching Tripod, which consists of three interactive elements— expected learning outcomes (ELOs), the learning activity or activities to be analyzed, and assessment (Kaplowitz, 2014, p 112). In Kaplowitz’s model the evaluative standards are established in the writing of the ELOs, which involves constructing sentences with four components, audience, behavior, condition, and degree (Kaplowitz, 2014, pp 65-72). These ELOs, and their constituent parts, seemingly establish and define what is meant by “measurable criteria”. What are we to make of this tautology? For instance, Kaplowitz states that “all these assessments must have something to measure, and what they measure is the attainment of whatever ELO’s are associated with the class, course or program” (2014, p 114). I read this as saying that we measure criteria with measurable criteria, or that in this model, what is measurable depends on how one defines it ahead of time. Regardless of this chicken-and-egg question regarding measurable criteria, ELO’s are intended to answer two questions about the second part of the teaching tripod, the learning activity: what will the learners be able to do following instruction, and how will you know they can do it? (Kaplowitz, 2014, p 60).
The third element in the teaching tripod involves assessment of the learning activities. According to Kaplowitz, there are 4 types of behavioral assessments: reaction (did they like it?); learning (did they get it?); behavioral (can they do it?); and results (did it matter?), and can be made before, during, or after instruction (Kaplowitz, 2014, pp 114-115). Kaplowitz also lists 6 different “assessment categories”— forced choice; open-ended assessments; surveys & questionnaires; interviews & focus groups; classroom assessment techniques; and authentic assessments, (Kaplowitz, 2014, pp 125-127), which are essentially the same as the evaluation methods previously described (Saunders, 2016, pp 227-228). In fact, Kaplowitz appears to construct the assessment element of her Teaching Tripod as an amalgamation of both evaluation and assessment. For example, she declares that the assessment element can be seen as either “an example of constructive feedback or of accountability” depending how the assessment is implemented, scheduled, recorded, or to whom it is presented (Kaplowitz, 2014, p 127); “constructive feedback or accountability” recalls the purported differences we outlined earlier between evaluations and assessments. That the assessment element in Kaplowitz’s model has the flexibility to be used in this manner, makes it useful for discussing how evaluation (or assessment) can improve the design or provision of services and programs.
Evaluation is linked to planning, and that planning usually involves “an evaluation process to determine the degree to which the goals and objectives were accomplished. Planning is making choices of what to do and what not to do” (McClure, 2008, pp 180-183). This linkage can be seen in Saunder’s evaluation/assessment cycle previously mentioned (2016, p 215). Kaplowitz so thoroughly embodies planning in her instructional model, that she calls learning outcomes (ELOs), learning activities, and assessment, the three elements that define her model, as “vital to effective instructional design” (Kaplowitz, 2014, p 112), a remarkably tautological statement, considering that Kaplowitz’s model itself is an example of instructional design. Nevertheless, this is a great example of how evaluation might improve the design or provision of services and programs. Planning and evaluation make up stages of design plans, whether they be focused on creating or improving programs and services, or on judging the value of extant programs and services, and oftentimes the line between evaluation and assessment is blurry, even if the measurement criteria is clear and specific (McClure, 2008, pp 180-190). Both evaluation and assessment, for example, are essential elements in library marketing plans, where carefully linked objectives and outcomes are intentionally mapped to measurable factors, so that data can be aggregated, analyzed, and eventually applied back to those same marketing plans (Alman, 2016, p 341).
The evaluation of library programs and services using measurable criteria is critical for librarians and information organizations because ongoing assessment upholds the quality and usefulness of these programs and services, which is important within a very real political context (McClure, 2008, pp 189-190). For information professionals and organizations, it is less important what evaluative model is used, and more important that a culture of evaluation/assessment be cultivated (Saunders, 2016, p 237; McClure, 2008, p 180). Just as evaluation and planning are linked in an information organization, an evaluation/assessment culture can also be thought of as an “organizational culture” in that there are shared and “appropriate” ways of doing things that are baked into the system (McClure, 2008, p 180).
It is important to me that any kind of “organizational culture” or evaluation/assessment culture, be participatory and more importantly, democratic in structure, as all the moving parts of libraries, including services and programs, embody values (“intellectual freedom the common good, service to all, stewardship of the human record, free access to knowledge and information, nondiscrimination, etcetera”) (Gorman, 2008, p 21) that are democratic in nature. For a library to become a culture of evaluation, it must be intentionally designed by librarians capable of assessing both themselves and their environment. In a sense the chicken-and-egg metaphor is quite apt here, as it may be hard to discern which precedes the other— the library or the librarian. For myself, I find this competency important because at base it implies our responsibilities as information professionals, to be stewards of our institutions. Information organizations might want to dig into the ethical foundations of evaluation and understand how important it is to sow the seeds of critical thinking and honest self-assessment, so that an “organizational culture” can take root.
Preparation to Understand Competency N: Coursework and Work Experience
Working for a decade in a large, public library system, I have become somewhat cognizant of the standards and mechanisms used by the administration to evaluate programs and services, though their criteria has not always been apparent to me. An exception to this has been the recent introduction of standard core competencies as a way for staff and management to work together to assess performance and development. In this case, the competencies, and the attributes/criteria that make them up, are clearly laid out. The rollout of the competencies themselves seems to be part of a project, the design of which the administration hopes to improve through an iterative process.
I have also learned a lot through reading information science literature, particular full-text, peer-reviewed papers for my INFO 200, 204, and 210 classes. After reading a great number of these academic papers, one begins to understand some universal elements, such as defining terms, choosing a research methodology or methodologies, identifying objectives or expected outcomes, setting parameters, planning and chronicling the research, and so forth. Aside from the papers I cite below as evidence, my work constructing a collection development policy for INFO 266 showed me how evaluative methods, using measurable criteria, are designed for use in purchasing or weeding materials for a library collection. While that project focused on content, rather than services, the same essential process and critical analysis was involved, and therefore could be said to have prepared me for this competency.
I chose to present my evidentiary items in roughly the chronological order in which they were created, with the exception of the first item, which describes what I learned about evaluating programs during my summer internship. I chose to present these scanned items first because, taken as a whole they remain a standalone evidentiary item, which mostly illustrates my interest in, and pursuit of, information regarding how librarians evaluate their programs.
Evidence
Metrics for Evaluating Programs
Discussion of Evidentiary Items
https://documentcloud.adobe.com/link/review?uri=urn:aaid:scds:US:5ce6cf8c-ac9d-4e0c-b4c4-20d9d48e0e2b
My first evidentiary item consists of scanned documents from my internship (INFO 294) in XXXX’s Eastern region, where I planned and launched my own teen program, and assisted librarians on their programs. The first few pages here show how I reached out, early on in my internship, to April Gonzalez, XXXX’s Diversity Coordinator, to ask by what metrics XXXX might be evaluating its programs, and/or what metrics librarians might be using to gauge the effectiveness of their programs. My inquire was a practical one: I wanted to plan my own program with evaluation and/or assessment in mind. A follow-up document shows the learning outcomes I was using to track my project for SJSU, and the programs I worked on. After this are pages that show the planning for and eventual conversation with Ms. Gonzalez. A page from my logbook shows the results of my inquiry about evaluation methods and metrics, including discussion of Outcomes for Success, a book describing how to plan for the evaluation of library services. I saved a few pages for that book and scanned them as an example here. April also provided a page that list of operational definitions for diversity, equity, and inclusion that were in development; these definitions might eventually be used as measurable criteria in evaluating programs or services.
The following 11 pages consist of examples of programming planning sheets that I gathered from librarians over the course of my internship. Because the evaluation of XXXX programs is centrally managed, I knew I would not be finding an evaluative blueprint in these planning sheets. But what I did find was that XXXX librarians create their own methods for assessing their programs, and the vast majority use their self-created planning sheets as a way of tracking their work, and as a place to compile their data. I found that one or two (not shown here) used Excel spreadsheets to manage this compiled data. After these planning sheets I included 8-9 pages that show some of the work I did on my own project, and other’s programs. The relevant pages here are the log entries, which show the data I collected data from these programs, and how I organized these entries to map the activities to learning outcomes. The final scanned page shows the first page of a paper that April Gonzalez passed on to me, that involved research on immigrant and refugee entrepreneurs, and begins to describe how they coded and synthesized focus group (interview) data.
These pages are presented as an illustration of my interest in how the evaluation of programs takes place within a large public library, and my desire to understand how librarians might assess their own programs in lieu of a transparent and uniform evaluative model.
Organizational Analyses for The Historical Society of Pennsylvania, Part 1
Discussion of Evidentiary Items
https://documentcloud.adobe.com/link/review?uri=urn:aaid:scds:US:cd648d7c-283a-4baa-8401-02d6cdf32097
The major group project I worked on for Stenstrom’s Information Professions (INFO 204) class was a two-part organizational analysis of an information organization. My group chose to analyze the Historical Society of Pennsylvania (HSP), a research library. In the first part of our analysis we were to conduct a literature review, a SWOT analysis, and draw understanding from HSP’s mission, vision, and values statements. The primary work I did for this project was to conduct a thorough literature review on the historical evolution of mission statements for U.S. research libraries. One of my primary purposes for conducting the review, was to give our team a rich stratum of information with which we might reassess HSP’s mission, vision, and values statements, and reimagine another, more appropriate set of guiding statements.
My research shows how both public and academic visions for research libraries intertwine and overlap over time, how higher education and digitalization were forces that shaped both historical societies and their missions, and how important it is to not make presumptions about user needs. Finally, some of my research led to some truisms about how mission statements ought to be derived from outreach and understanding the library’s community base, how mission statements fail not from language or focus but from undefined goals, and how they generally need to be more visible to the public. Once I had sifted through and analyzed my research, I was able to construct new mission, vison, and values statements by tying the core HSP values we wanted to retain with insights from the research. The words were drawn from original statements, but I modified them to be more in-line for a modern, forward-looking institution. I also chose my words carefully, understanding that statements used in a strategic plan can be considered measurable criteria by which HSP will evaluate itself in the future. For instance, the goal of “being a culturally valuable entity,” was deliberately chosen as an outcome towards which the library could be expected to strive, and for which it could reasonably be assessed over time.
My contribution to the SWOT analysis was of a more general nature, focusing mostly on encouraging my co-researchers to fill out more detail in important areas such as sociocultural factors or naming HSP’s major competitors. As with the literature review, I approached this task with the understanding that the information that we fed into the SWOT analysis would provide hard data with myriad, practical uses in forging a new strategic plan. For example, data of this sort can lead to developing measurable criteria for future assessments or it might be used as a way of developing standards for evaluating services. Within the parameters of the strategic plan of course, I knew that the data would be used as context for the goals we would set out to achieve, and the means by which we would achieve them. Still, as with the mission, vision, and values statements I wrote, I consciously pushed for greater complexity in the SWOT analysis, understanding that assessment and planning go hand in hand, and once our strategic plan was in place, the components upon which it was constructed would be used as a guide for what, when, where, why, and how to assess our progress.
Working on the HSP Organizational Analysis project shows that I understand how assessment and planning go hand and hand, and the role that research plays in both evaluating plans and in constructing a plan that will have future evaluative uses. I show that I understand what goes into evaluating or assessing diverse entities such as strategic plans, research papers, or even entire information organizations. Finally, I demonstrate an understanding of planning and evaluation that is cognizant of inputs and outputs, of accurate and relevant data being used to forge measurable criteria, and of the need for using both qualitative and quantitative research to improve whatever we are designing.
Reference Observation Assignment
Discussion of Evidentiary Items
https://documentcloud.adobe.com/link/review?uri=urn:aaid:scds:US:d90f9742-4017-4276-8517-0e420e054f7c
In this assignment for my INFO 210 class I arranged to observe a reference librarian over the course of a single “shift” and observe their interactions with patrons, where I would keep a field journal of notes on the reference desk experience and reference interviews, followed by an analysis of what I observed, including how the service was designed and delivered, and any examples of patron feedback. Part of the assignment was to keep the librarian, the user, and the myself (the observer) in mind, an important factor in modern evaluation of information services. I chose to observe a virtual reference service assistant performing phone interviews at AskKCLS, a virtual chat service.
I decide to filter my observations for this assignment through the IFLA and RUSA guidelines for reference service professionals (Guidelines, n.d.; RUSA, 1996), two documents that list and codify criteria by which reference service activity can be measured. After observing a reference service assistant over the course of one work shift, I analyze and evaluate the reference interviews against the measurable criteria in the IFLA and RUSA guidelines. For instance, at the beginning of an inquiry I note the assistant using good virtual reference practices such as being prepared, acknowledging the patron, and using a neutral but friendly tone of voice, and over the course of the phone call I observe them using closed and open questions, explaining their search strategy to the patron, and citing authoritative sources (Shachaf & Horowitz, 2008, pp 25-30).
Taking the data points from my evaluation of the reference services assistant (using measurable criteria), I provide a reasoned evaluation of how services might be improved. I provide suggestions for redesigning the chat service including the idea that a return to mentoring might reinvigorate a simple reliance on IFLA/RUSA guidelines, allowing for behaviors like being “passionately curious” to be properly valued in actual service delivery. I argue that when we have two information behavior data sets to draw upon (the user and the library associate) we can plan and design better reference services.
This assignment shows that I am able to evaluate an actual working instance of reference services using measurable criteria and, based on that data, proffer design improvements of those services. I show that I understand how documents like the IFLA/RUSA guidelines, are produced through an iterative practice of evaluation and planning, and that these standards we create are improved through their use in evaluating actual service provided in the field. Finally, I demonstrate an understanding that our evaluations of services improve when we take into consideration all the perspectives in play, including the patron, the observer, and the information professional.
Instructional Design Plan Draft
Discussion of Evidentiary Items
https://documentcloud.adobe.com/link/review?uri=urn:aaid:scds:US:b2928b0f-4d6e-4545-ad1c-3fa147695dd6
There are several evaluative elements woven into the instructional design plan I co-created for INFO 250. My partner and I chose Kaplowitz’s Teaching Tripod ID Model because of its flexible modern design, containing only three elements— expected learning outcomes (ELOs); learning activities; and assessment. I did the bulk of the work in transforming this rough draft into a five-minute screencast introduction, as well as a six-page paper capable of being orally presented in a 15-minute timeframe. I accomplished this by removing the scaffolding from our rough draft. In order to remove what we did not need, I had to understand the components of Kaplowitz’s model well, as each leg of the tripod informs every other leg.
I performed evaluation and assessment of services throughout the design of this plan. I and my partner interviewed our librarian clients, as part of a needs assessment. Using this data as a starting point she and I wrote instructional goals and evaluated and defined learner characteristics and behavior. The constructing of the ELOs was a decidedly collaborative effort where I focused on editing and structure and deferred to my partner regarding final wording and the exact degree to which students would be measured. My work here was to be sure that our terms were defined, that ELOs were parallel in structure, and that the elements making up the ELOs were not only represented, but did in fact consist of measurable criteria. Another example of this was in the instructional strategies, where my partner conceived of how the instruction would function, and I double and triple-checked to make sure that a) all measurable criteria was properly applied, b) that all measurable criteria could be traced back to the appropriate ELO, and c) that the instruction (program) was described accurately. The section on formative and summative evaluation was a culmination of this approach, where my partner and I went through a cycle whereby she would conclude that a tutorial would produce a certain evaluative result, and I would demonstrate that this result could not be achieved. This iterative, collaborative approach finally resulted in my being able to describe how our instruction could and would deliver certain evaluative results.
My summation of our instructional design is, in a sense, an argument for how evaluation and assessment can improve information services. I write how I and my partner’s evaluation and assessment of our client’s instructional interface, their students’ needs and behaviors, the library environment, and the librarians’ information needs were boiled down into measurable criteria, and using Kaplowitz’s evaluative model, a new and improved instructional interface designed.
My work on this Instructional Design Plan illustrates my understanding of evaluative elements within an instructional design model like Kaplowitz’s Teaching Tripod, and their functions. I demonstrate that I can perform a needs assessment or design formative and summative assessments. I show that I know how to develop measurable criteria, and that I am capable of ensuring that assessments are properly tied to such criteria. Lastly, I can articulate and demonstrate how evaluation and assessment can transform and improve instruction.
Conclusion
My interest and curiosity in how information professionals and organizations evaluate their programs and services, comes across in how instinctively I dig for data that describes this process. Because I understand how planning and evaluation go hand-in-hand, I am cognizant of inputs and outputs, of how accurate and relevant data is used to forge measurable criteria, and of the need for using both qualitative and quantitative research to improve whatever we are designing. Showing myself capable of evaluating an information service, and alternately capable of using evaluative methods and models to re-evaluate, and thereby redesign existing information entities, means having a skill applicable in any information organization. Understanding how this iterative practice of evaluation and planning produces standards and guidelines of measurable criteria, means always knowing how I might contribute within my chosen profession. And finally, demonstrating that I can design formative and summative assessments tied to measurable criteria, means that I have a necessary tool should I want to evaluate my own library program, or any information service or program.
References
Alman, S. W. (2018). Communication, marketing, and outreach strategies. In Hirsh, S. (Ed.). Information services today: An introduction. Lanham, Maryland: Rowman & Littlefield
Gorman, M. (2008). Professional ethics and values in a changing world. In Haycock, K. & Sheldon, B.E. (Eds.). The portable MLIS: Insights from the experts.15-22. Westport: Libraries Unlimited.
Guidelines for behavioral performance of reference and information service providers. (n.d.). Retrieved June 26, 2016, from: http://www.ala.org/rusa/resources/guidelines/guidelinesbehavioral
International Federation of Library Associations (2005). IFLA digital reference guidelines. Retrieved from http://www.ifla.org/VII/s36/pubs/drg03.htm
Kaplowitz, J.R. (2014). Designing information literacy instruction: The teaching tripod approach. Lanham, MD: Rowman & Littlefield
Lehmann, V. & Locke, J. (2005). Guidelines for library services to prisoners (3rd ed.). IFLA
Professional Report, 92. The Hague: International Federation of Library Associations
and Institutions.
McClure, C.R. (2008). Learning and using evaluation: A practical introduction. In Haycock, K. &
Sheldon, B.E. (Eds.). The portable MLIS: Insights from the experts. Westport, Connecticut: Libraries Unlimited.
Reference and User Services Association. (1996). Guidelines for behavioral performance of reference and information services professionals. RQ, 36(2), 200-203. Retrieved from http://www.ala.org/rusa/resources/guidelines/guidelinesbehavioral
Saunders, L. (2016). Evaluation and assessment of reference services. In Smith, L.C. and Wong,
M.A. (Eds.). Reference and information services: An Introduction, 212-243. Santa Barbara, CA: Libraries Unlimited
Shachaf, P., & Horowitz, S. M. (2008). Virtual reference service evaluation: Adherence to RUSA behavioral guidelines and IFLA digital reference guidelines. Library & Information Science Research, 30(2). Retrieved from http://eprints.rclis.org/12414/1/virtualReferenceServiceEvaluation.pdf
Starr S. (2014). Moving from evaluation to assessment. Journal of the Medical Library Association: JMLA, 102(4), 227–229. Retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4188047/
Tyckoson, D. A. (2012). Issues and trends in the management of reference services: A historical perspective. Journal of Library Administration, 51(3), 259-278. Retrieved from https://content.ebscohost.com/ContentServer.aspT=P&P=AN&K=503012446&S=R&D=lls&EbscoContent=dGJyMNLr40Sep7U4y9fwOLCmsEiep7ZSsK64TbCWxWXS&ContentCustomer=dGJyMPGtsVCuprFLuePfgeyx4YXr2QAA
Tyckoson, D. A. (2008). Reference service: The personal side of librarianship. In Haycock, K. &
Sheldon, B.E. (Eds.). The portable MLIS: Insights from the experts, 127-46. Westport, Connecticut: Libraries Unlimited.