Measuring scholarly impact beyond the library and the lab

If you're a scholar, how do you know if the work you do is influential? Researchers, universities and publishers have traditionally measured the impact of scholarly work by counting the number of publications, and how often those publications have been cited.

But in a world where research comes in many forms, citations are only part of the story. What traditional impact metrics don't catch are scholarly contributions that don’t come in standard journal article form, such as datasets, software and code, all of which influence research and teaching in ways that aren’t reflected in the citation record.

Biologist Amy Zanne of George Washington University understands this first hand.

In the mid-2000s, Zanne and colleagues began compiling data for what would eventually become the largest wood density database to date -- representing wood densities for more than 8400 plant taxa worldwide. "It's a one-stop shop for wood density data," Zanne said.

In 2009, Zanne uploaded the data to a digital data repository for other researchers to use. And use it they did. The data set has since been downloaded nearly 3,000 times, making it the most downloaded data set in the Dryad data repository.

"The database is being used in lots of different countries, and not just in academic settings -- It’s also showing up in technical reports and white papers from agencies like Earthwatch and the Forest Service -- so the data are also helping to inform policy to some degree," Zanne said.

What’s less clear is how this kind of research productivity will be measured or evaluated.

"Scholars make all sorts of contributions that are almost invisible," said UNC Chapel Hill biologist Todd Vision. "The impact of those things isn’t currently being measured, because they don’t count towards annual review or tenure and promotion packages," Vision added.

A new movement called altmetrics -- short for alternative metrics -- aims to provide a more complete and timelier picture of all the different ways that scholars contribute to their field and the impact and reach of their work.

It takes months and even years for citations to accumulate. But altmetrics tools such as ImpactStory, for example, give up-to-date impact data in minutes, not months.

Thanks to a $125 grant from the Sloan Foundation, the team of researchers behind the ImpactStory project have developed a web application that collects and displays a wide range of usage information. Citations, to be sure, but also page views, clicks, downloads, blogs, media coverage, and other indicators that the work is being noticed.

Advocates argue that ImpactStory and other altmetrics tools have the added benefit of capturing usage patterns by users outside the ivory tower, such as industry practitioners and the lay public -- thus capturing a broader picture of who’s using your work and how.

Learn more at
[ more ]