Measure and Evaluate is a project initiated and administered by the Stockholm County Library during 2008-2010 with a grant from the Swedish National Arts Council. For many years, the national library statistics were criticized and discussed and various initiatives have been undertaken to evolve statistical scales of measurement and despite these efforts, statistics have shown only a marginal improvement.
The criticism lies in how statistics of this kind fail to present an accurate picture of public library operations, how various areas of importance are not measured at all and how definitions remain unclear. To complement the national statistics, Stockholm County Library has over a long period collected complementary data from the county libraries and compiled key data for Stockholm County. In its work with county statistics, Stockholm County Library showed a lack of coherent analysis, which led to the initiation of the project Measure and Evaluate. The purpose of Measure and Evaluate is:
“… to find new scales of measurement that allow for a broad and reasonable presentation of library programs and account for the scope, quality and efficiency as its basis of operational developments. The result will be a paper on library statistics, applying a theoretical and discursive background, operable suggestions and applications” (from project application to the Arts Council).
About the book Measure and Evaluate
The project resulted in a book, Measure and Evaluate – about statistics and efficiency at public libraries, (2010). The book’s emphasis has been placed on its theoretical and discursive background. The reasoning in the book concerns public library service in general yet one of the chapters, written by Lena Lundgren, deals specifically with existing and prospective statistics regarding the activities of children.
Another chapter formulates some proposals for taxonomies. They are meant to be used as a basis for discussions when implementing partnerships with another sector. Taxonomies can be used to categorize standards of quality.
In a previous project, called Goals and Measures, four libraries in Region Västra Götaland were involved in testing performance indicators. Kerstin Wockatz informs about the experience in one of the chapters.
Do statistics describe operational activities?
To what extent do statistics describe today’s public library activities? How do measurements relate to stated objectives? Do we have the key data we need to pursue operational objectives? How should we interpret scenarios that emerge from our results? Who should we compare ourselves to in order to answer whether the results are good or bad?
An important question in statistics is what platform for comparisons should prevail. Should one only compare data on an annual basis and internally, one’s neighbouring municipality, the municipalities included in the same municipal group or among those of other countries? Or should it be compared to other activities?
To a certain extent maybe, it is about how we view the concept of performance or what we mean by efficiency. Is an efficient operation by definition of a higher standard? To use resources more efficiently one has to deal with the formulation of efficiency in terms of goals. To understand the purpose as to why scales of measure- ments are being used and linking those to objectives will determine whether statistics can be used in operational developments. It is crucial that by using the governing documents we actually formulate the intent or purpose of the activities we perform in libraries.
Internal and external efficiency
The project has partly been based on notions of external and internal efficiency:
External efficiency measures the performance of the right choices and that the operation is possible and utilized by the projected target groups.
Internal efficiency can be translated as meaning a description of daily operations. The operation is supplied withresources and in a number of activities these are converted into a result. This result should preferably provide an effect that is linked to the objectives. Internal efficiency measures whether things are being done the correct way.
We need parameters to describe both internal and external efficiency. Above all, it is necessary to consider what indicators are saying and how they can be combined. For example: If the circulation rate, circulation / inventory, increases so also should loans increase per capita in order to be able to say that overall efficiency has increased. That is, the joint ratios of the internal and external efficiency together define the balance between different types of efficiency that interact to hopefully form a quality-assured operational activity.
Efficiency analysis of reaction rate calculation models
There are numerous calculation models for measuring rates of efficiency. The Danish researcher Niels Ole Pors has constructed a model which is based on the Balanced Scorecard model. The project Goals and Measures let four municipal libraries in Västra Götaland apply some of the key data suggested by Pors. The point of the Balanced Scorecard is to measure and evaluate the key data from all perspectives that exist in the model so as to obtain the balance of operational activities. Västra Götaland’s experience clearly demonstrates that single measurements say very little about an operation in an overall context.
It is challenging to draw any conclusions as to whether a particular model is applicable or not merely by reviewing certain key data. Models are often based on an image of the organisation as a stable environment where the efficiency criteria is measured at position A are also expected to apply to position B. It seems to be that we need other ways of looking at the effectiveness of the concept. Efficiency might include the ability to manage conflicting interests and competing values. If one uses this approach to view efficiency, it means that efficiency is not served by optimizing a particular method, but instead there is a balance between the (perhaps conflicting) values which over time favours operational activities. The challenge facing an organisation is to find such a balance and ways of working whereby values relate favourably to each other over time.
Measuring levels of quality using taxonomies
The taxonomies presented in the book differ from one another. While we have sought the same number of levels there are discrepancies if you place taxonomies next to each other. For example, it is not obvious that level 2 means the same degree of interaction in all taxonomies. This is partly because we are at different stages in terms of cooperation between different activities. Libraries, for example, collaborated together for a long time with the school sector and in many places around the country there are ongoing formal agreements. Agreements concerning cooperation between libraries and elderly care are less common. As our activities evolve so hopefully the highest levels evolve even more and maybe some of the levels will merge.
Another difference between taxonomies is the degree of a formalized political assignment. Public libraries carry mandate to collaborate and formulate municipal library strategic plans. Elderly care, child care centres and preschools have also formulated assignments where cooperation with libraries can benefit parts of their work. But the voluntary sector of associations has itself no mandate to cooperate with a library, although some organisations would certainly benefit from collaborative ventures.
It is important to note that taxonomies have been developed within a library context, and from the library staff ’s diverse experiences in cooperative ventures. Although the taxonomies describe cooperation, their formulations nevertheless remain based on a library’s assignment. It remains uncertain whether they would look the same if we increasingly cooperated with each respective activity when we have developed taxonomies. Of course this is a troublesome fact and may emphasize the project’s ambition to initiate a work process and to provide a warm welcome to all initiatives to further discussions and to the further development of taxonomies. This also applies to the creation of new taxonomies.
Expectations are that seminars and workshops will be set up, applying our knowledge to embrace the use of keyfigures, how they can be combined and what they say about our activities.We also hope that libraries want to test different models in their entirety. We also hope that national players such as Sweden’s County Librarians, Local Authorities and County Councils, the National Library and the Swedish Arts Council keep contributing to ideas taking root, their dissemination and challenging them to further evolvement.
Stockholm County Library
malin.ogland AT kultur.stockholm.se
Translated by Jonathan Pearman