At the fifth SIG/MET workshop, held during the 2015 ASIS&T Annual Meeting, the group shared papers, posters and discussions exploring developments in information measurement. The opening session on bibliometric case studies examined interdisciplinarity among consumers of academic research, increased funding for coauthors of previously funded authors and a classification of acknowledgement types. A session on information retrieval in relation to bibliometrics included studies on overcoming the limits of computational linguistics in very large corpora, an interactive context explorer of bibliographic data called Ariadne and comparative approaches to visualizing the structure of a very large dataset. The alternative metrics session covered application of altmetrics for analyzing public policy documents, a novel usage indicator promoting article discovery, the basis for connections among faculty members using Twitter and the heavy use of Twitter among academics. The daylong workshop included awards for best papers, best student papers and two featured presentations on the application and use of metrics.


special interest groups
information retrieval
electronic visualization

ASIS&T Annual Meeting Pre-conference Activities


METRICS 2015: Workshop on Informetric and Scientometric Research

by Stefanie Haustein

For the fifth time SIG/MET, ASIS&T’s special interest group for the measurement of information production and use, held its workshop in conjunction with the ASIS&T Annual Meeting. The full-day workshop was organized by SIG/MET chairs and officers Isabella Peters, ZBW Leibniz Information Centre for Economics, Germany; Stefanie Haustein, University of Montreal, Canada; Chaoqun Ni, Simmons College; Kun Lu, University of Oklahoma; and Timothy D. Bowman, University of Turku, Finland. Fifteen presentations and 3 posters at METRICS 2015 attracted 45 participants. In addition to submitting a peer-reviewed poster ahead of the workshop, participants were for the first time invited to bring posters with their latest findings to discuss during two open poster sessions. Workshop presentations, posters and discussions revolved around SIG/MET’s central topics related to the measuring of information, including metrics research and application in bibliometrics, informetrics, altmetrics and information retrieval, as well as social network analysis and visualization of scholarly communication.


Figure 1. Attendees of the METRICS 2015 workshop at the ASIS&T Annual Meeting in St. Louis


The workshop opened with a session devoted to bibliometric case studies. George Lan, Elsevier, presented on the impact of interdisciplinary publications. He differentiated between multiple audiences or consumers of academic research, comparing the effect of the level of interdisciplinarity of papers based on their citations from articles and patents as well as downloads. Philippe Mongeon, University of Montreal, demonstrated the positive effect of funding on researchers’ collaboration networks. Based on researchers in Quebec, he found a linear positive correlation between the funding amount and the number of new co-authors of funded authors. While Mongeon’s analysis of collaboration patterns was quantitative and based on co-authorship patterns, Katherine W. McCain, Drexel University, took a different and more qualitative approach to identify scientific collaboration: she analyzed personal mentions of researchers in the acknowledgement texts of research papers using the starlet sea anemone as a model organism. Acknowledgements were classified into 10 different types including “providing animals,” “providing comments on manuscript” or “inspiration, valediction.”

The second session of the METRICS workshop focused on information retrieval and its relationship with bibliometrics. Kun Lu, University of Oklahoma, and Dietmar Wolfram, University of Wisconsin-Milwaukee, dedicated their study to the problem of computational limitations that occur due to an ever-growing amount of data and increasingly large corpora. In their presentation entitled “Vocabulary Size and Its Effect on Topic Representation for Informetric and Information Retrieval Data Processing,” they showed, based on an analysis of three datasets, that cutting off the so-called long tail could increase processing without affecting retrieval results. Andrea Scharnhorst, Royal Netherlands Academy of Arts and Sciences KNAW, introduced Ariadne, an interactive context explorer for bibliographic data. Ariadne is trained on 65 million publications from OCLC’s ArticleFirst database but can be used as an exploratory tool for any dataset. Ariadne’s algorithms were tested and compared to other methods of topic extraction and visualization in the study by Theresa Velden, University of Michigan, and colleagues, who concluded the information retrieval session. The contribution entitled “Same Data, Different Results? On a Comparative Topic Extraction Exercise” presented a systematic comparison of different approaches and algorithms to visualize the structure of an astrophysics and astronomy dataset with more than 110,000 papers. Velden emphasized that several solutions are valid and that the aim of the exercise was to find the best solution for a given purpose. So-called “blind spots” of some of the methods were identified and visualized.

Similar to past workshops, SIG/MET recognized outstanding contributions of students with a Best Student Paper Award sponsored by Elsevier. This year the jury determined two winners, who were given the opportunity to present their papers during the third session of the workshop. Misha Teplitskiy, University of Chicago, who co-authored his paper “Amplifying the Impact of Open Access: Wikipedia and the Diffusion of Science” with fellow student Grace Lu and supervisor Eamon Duede, showed that open-access and high-impact-factor journals were more likely to be cited in Wikipedia. The paper “Evolution of iSchool Movement (1988-2013): A Bibliometric View,” co-authored by McGill and University of Montreal PhD students Fei Shu and Philippe Mongeon, won the other Best Paper Award. Shu presented the history of the iSchool movement based on more than 4,000 LIS dissertations indexed in ProQuest. He highlighted that interdisciplinarity increased over the years and that computer science gained importance. Elsevier’s sponsorship allowed SIG/MET to award each of the two best student papers with $500 and finance ASIS&T student memberships for all students participating in the workshop.

Processed with Moldiv

Figure 2. Best Student Paper Award winners, from the left, Misha Teplitskiy with SIG/MET chair Isabella Peters; in the picture on the right, Fei Shu and Philippe Mongeon (right) with Peters between them

Altmetrics has been a hot topic of the metrics and scholarly communication community in the past years. This trend is also reflected in the METRICS 2015 program, as the fourth session consisted of four talks on alternative indicators and altmetrics. Stacy Konkiel, Altmetric, presented an exploratory case study based on scientific papers cited in public policy documents. She emphasized that policy documents represent an interesting source for societal impact metrics but that the set of documents covered by Altmetric is still limited to a small number of selected sources. Adèle Paul-Hus, University of Montreal, presented the results of her study co-authored with Philippe Mongeon and Fei Shu. The three PhD students from Montreal identified the Twitter accounts and tweeting activity of faculty members of LIS schools accredited by the American Library Association (ALA) and compared their Twitter followers to their co-authorship networks. They concluded that connections on Twitter reflected mostly institutional affiliations instead of research interests. Jim Pringle, Thomson Reuters, introduced “A New Interest Indicator Based on Researcher Behavior in the Web of Science.” The new usage count was designed to complement the article citation count by a usage indicator to enhance article discovery. Pringle presented interesting disciplinary and journal-based patterns comparing usage and citation counts. The alt(ernative) metrics session concluded with a presentation by Stefanie Haustein, University of Montreal, who, together with her colleague Rodrigo Costas, CWTS, Leiden University, analyzed terms appearing in Twitter bios of users mentioning scientific papers to determine who is tweeting about science. Their preliminary findings suggest that a large share of tweeters identify as academics, which contradicts the assumption that tweets reflect impact on society at-large.

SIG MET pic 3-bestaltmetricspapers

Figure 3. Best Altmetrics Paper Award winners with Stacy Konkiel of Altmetric, from left to right: Fei Shu, Stefanie Haustein, Stacy Konkiel, Adèle Paul-Hus and Philippe Mongeon

For the first time at the SIG/MET workshop, Altmetric and figshare sponsored an altmetrics paper award. The jury decided to award two contributions: the papers by Mongeon, Paul-Hus and Shu and one by Haustein and Costas were selected for Best SIG/MET Altmetrics Paper Awards; each team won $500.

Bradford Demarest, Indiana University, presented his doctoral research on comparative discourse epistemetrics in the session entitled “Broadening the Scope of Bibliometrics.” Analyzing the abstracts and full texts of philosophy and psychology papers, Demarest showed that the two disciplines can be well distinguished based on the adverbs and pronouns used.

The workshop concluded with two presentations showcasing the application and use of metrics. Betsy Martens, University of Oklahoma, presented the Oklahoma Mesonet platform and analyzed more than 650 papers citing the tool. Edwin Henneken, Smithsonian Astrophysical Observatory, demonstrated the new retrieval interface of the Astrophysics Data System (ADS). After the search interface of ADS had remained practically unchanged since 1992, the new platform offers a variety of analytical and visual possibilities to enhance retrieval with a modern interactive design. New functionalities can be applied to any set of results and include a variety of metrics, histograms and time series as well as co-citation and co-author network visualizations.

More information about the SIG and the METRICS 2015 workshop, including abstracts and slides of all presentations, can be found on SIG/MET’s website at www.asis.org/SIG/SIGMET/. For the latest information and news, follow SIG/MET’s mailing list (http://mail.asis.org/mailman/listinfo/sigmetrics) and Twitter account (https://twitter.com/sig_met).


METRICS 2015 Workshop Program

Case studies in bibliometrics

George Lan, Lei Pan and Sophia Katrenko: “Analyzing Interdisciplinary Research Along Multiple Dimensions of Research Impact”

Philippe Mongeon: “The Impact of Funding on Research Collaboration: The Case of Quebec Researchers”

Katherine W. McCain: “Collaboration Patterns in Model Organism Research: Co-Authorship, Acknowledgement and the Starlet Sea Anemone (Nematostella Vectensis)”

Poster session

Hamed Alhoori, Richard Furuta, Sagnik Choudhury, Tarek Kanan, Myrna Tabet, Mohammed Samaka, Edward Fox and Lee Giles: “Mining Altmetrics at Different Levels” – open poster presentation

Virgina A. Dressler: Beyond Google Analytics: Using the “Toolkit for the Impact of Digitized Scholarly Resources to Embed Metrics and Gauge Use of Regional Digital Collections” – peer-reviewed

Shenmeng Xu and Bradley M. Hemminger: “What Flavors Are Different Types of Scholarly Articles? An Investigation of PLOS Publications” – open poster presentation

Relationship between bibliometrics and information retrieval

Kun Lu, Xin Cai, Isola Ajiferuke and Dietmar Wolfram: “Vocabulary Size and Its Effect on Topic Representation for Informetric and Information Retrieval Data Processing”

Rob Koopman, Shengui Wang and Andrea Scharnhorst: “Between Information Retrieval Services and Bibliometrics Research – New Ways of Semantic Browsing and Visual Analytics”

Theresa Velden, Kevin Boyack, Nees Jan van Eck, Wolfgang Glänzel, Jochen Gläser, Frank Havemann, Michael Heinz, Rob Koopman, Andrea Scharnhorst, Bart Thijs and Shenghui Wang: “Same Data, Different Results? On a Comparative Topic Extraction Exercise”

Best student papers – awarded by Elsevier

Misha Teplitskiy, Grace Lu and Eamon Duede: “Amplifying the Impact of Open Access: Wikipedia and the Diffusion of Science”

Fei Shu and Philippe Mongeon: “Evolution of iSchool Movement (1988-2013): A Bibliometric View”

Alt(ernative) metrics

Stacy Konkiel: “Citation Lags for Articles Referenced in Public Policy Documents: An Exploratory Study”

Philippe Mongeon, Adèle Paul-Hus and Fei Shu: “Twitter Activity and Scientific Collaboration of LIS Schools and Faculty Members” – winner of award sponsored by Altmetric and figshare

James Pringle: “A New Interest Indicator Based on Researcher Behavior in the Web of Science”

Stefanie Haustein and Rodrigo Costas: “Identifying Twitter Audiences: Who Is Tweeting About Scientific Papers?” – winner of award sponsored by Altmetric and figshare

Broadening the scope of bibliometrics

Bradford Demarest: “Comparative Discourse Epistemetrics: Research Article Abstracts and Full Texts”

Metrics in use

Betsy Martens, Christopher Fiebrich and Bradley Illston: “Mapping Oklahoma Mesonet Sensor Datastreams”

Edwin Henneken: “Metrics and Visualizations in the Astrophysics Data System”

Stefanie Haustein is chair of ASIS&T Special Interest Group/Metrics (SIG/MET). She is a postdoctoral researcher at the Canada Research Chair on the Transformations of Scholarly Communication at the University of Montreal, Canada. Her current research focuses on social media in scholarly communication and altmetrics and is supported by the Alfred P. Sloan Foundation. Stefanie is a visiting lecturer at Heinrich Heine University Düsseldorf and co-chair of the NISO altmetrics working group on data quality. She can be reached at stefanie.haustein<at>umontreal.ca or found on Twitter via @stefhaustein