In 2017-2018, the Workshop in Methods was directed by Dr. Patricia McManus and featured thematic workshops focused on replication, reproducibility, and transparency, as well as additional workshops. Browse workshops from the 2017-2018 academic year below. All 2017-2018 workshop videos have also been compiled in a playlist on Media Collections Online. You may also browse videos from all years in the full Workshop in Methods collection on Media Collections Online, and access other materials, such as presentation slides, through the Workshop in Methods collection on IUScholarWorks.
Friday, September 1, 2017
Dr. J. Scott Long, "Reproducible Results and the Workflow of Data Analysis"
2-3:30pm, Social Science Research Commons Grand Hall (Woodburn Hall 200)
Many disciplines are paying increasing attention to reproducible results. The fundamental idea is that other scientists should have access to your data and be able to obtain the same results—this is reproducibility. More generally, your results should be robust so that other scientists can confirm your findings using other data. Increasingly journals require authors to provide their data and analysis file before a paper is accepted to verify that that results. Producing reproducible results is highly dependent on your workflow for data analysis. This workflow encompasses the entire process of scientific research: Planning, documenting, and organizing your work; creating, labeling, naming, and verifying variables; performing and presenting statistical analyses; preserving your work; and ending with reproducible results. Most of the work in statistics classes focuses on estimating and interpreting models. In “real world” research projects, these activities may involve less than 10% of the total work. Professor Long’s talk is about the other 90% of the work. An efficient workflow saves time, introduces greater reliability into the steps of the analysis, and generates reproducible results.
Dr. Long is Distinguished Professor and Chancellor’s Professor of Sociology and Statistics at Indiana University.
Materials on IU ScholarWorks (presentation slides, event flyer)
Friday, September 15, 2017
Dr. Emily Meanwell, "Qualitative Research Methods: An Introduction"
Social Science Research Commons Grand Hall (Woodburn Hall 200), 2pm
What is qualitative research? When and how might qualitative research methods help you answer your research questions? This workshop will provide an overview of qualitative research methods, including interviews, focus groups, ethnography, and qualitative content analysis. For each, we will discuss the basics of data collection and analysis, best practices, and illustrative examples. Lastly, we will consider how these methods contrast with, and/or may be combined with, other empirical research methods, including quantitative and survey research methods.
Emily Meanwell is the Director of the Social Science Research Commons and the Study Director for the Sociological Research Practicum. She received her PhD in sociology in 2014 and has used a range of qualitative methods, including interviews, ethnography, and content analysis, in her research.
Materials on IU ScholarWorks (presentation slides, event flyer)
Friday, September 22, 2017
Dr. James Ziliak and Dr. Charles Hokayem, "Kentucky Research Data Center Infosession"
Social Science Research Commons (Woodburn Hall 200), 2pm
The Kentucky Research Data Center (KRDC) is a collaboration between the University of Kentucky and the U.S. Census Bureau established by a grant from the National Science Foundation in 2016. KRDC is part of the nationwide system of Federal Statistical Research Data Centers whose mission is to expand the data infrastructure available to qualified scholars and students with approved projects by providing access to restricted individual- and firm-level data from participating federal statistical agencies. KRDC is maintained by a regional consortium of leading research institutions, including Indiana University. This infosession is designed for IU researchers interested in developing research projects using the KRDC.
James Ziliak is the KRDC Executive Director, Carol Martin Gatton Endowed Chair in Microeconomics, and the Founding Director of the Center for Poverty Research at the University of Kentucky. Charles Hokayem is the KRDC Administrator with the United States Bureau of the Census.
Materials available by email (presentation slides, event flyer)
Video (Media Collections Online)
Friday, September 29, 2017
Dr. Robert Calin-Jageman, "The New Statistics and Open Science: How to Get Started"
Social Science Research Commons (Woodburn Hall 200), 1-3pm
Recently the Association for Psychological Science revised its publication guidelines to reward Open Science practices and to encourage the use of the “New Statistics” as a better alternative to null hypothesis significance testing (NHST). Other journals and professional societies seem to be moving in the same direction, often in collaboration with funding agencies.
This workshop will provide a practical introduction to the New Statistics and some emerging Open Science practices. We will worth through examples from several common research designs. We will also explore resources that can help you adopt these approaches in your own research.
Robert Calin-Jageman is a professor of psychology and the neuroscience program director at Dominican University. He has taught statistics and mentored students in psychological science for 10 years, publishing with 16 undergraduate co-authors (so far). His research focuses on how memories are formed and forgotten. He has also been active in exploring the replicability of psychological science and promoting Open Science. He received his PhD in Biological Psychology from Wayne State University.
Materials on IU ScholarWorks (presentation slides, demo, event flyer)
Video (Media Collections Online)
Introduction to the New Statistics website
Friday, October 13, 2017
Jefferson Davis, "The Grammar of Graphics: An Introduction to ggplot2"
Social Science Research Commons (Woodburn Hall 200), 2-3:30pm
In The Grammar of Graphics, Leland Wilkinson created a systematic way to think about statistical graphics and the presentation of quantitative data. The package ggplot2 by Hadley Wickham implements of Wilkinson's system for the language R.
The talk will discuss the following:
- What are statistical graphs and what are some ways to talk about them?
- Examples of common plot types done in ggplot2.
- Using ggplot2 to display common statistical transformations.
- Grouping and faceting data in ggplot2.
- Using themes to add polish to graphs.
The talk requires no familiarity with ggplot2 or other libraries. Some familiarity with base R, however, will be useful.
Jefferson Davis is a software analyst with Research Analytics. He has worked on several projects that have used R and R libraries. Current projects include work in data visualization, machine learning methods, and large data sets.
Materials on IUScholarWorks (presentation slides, exercise files, event flyer)
Video (Media Collections Online)
Friday, December 1, 2017
Justin Wild, "Structural Equation Modeling in Open-Source Software"
Social Science Research Commons (Woodburn Hall 200), 2-4pm
Structural Equation Modeling (SEM) offers flexible statistical models for the social science researcher. A variety of software packages are available for implementing SEM with researchers’ datasets and are becoming increasingly sophisticated. This talk will briefly outline SEM in comparison with more familiar statistical models (such as linear regression) and review several R packages tailored for the SEM community. In addition, these packages are compared to perhaps the most well-known commercial package available, MPlus.
Justin Wild is a doctoral candidate at Indiana University studying Education Policy Studies, with a concentration in International and Comparative Education, and Inquiry Methodology. He has a Master’s of Arts in African Studies and a Master’s of Public Affairs from Indiana University. His research addresses K-12 language of instruction, language ecology, foreign language learning, evaluation in education, large-scale assessments, multi-method research, and cross-cultural issues. His geographic focus is Tanzania, East Africa.
Materials on IUScholarWorks (presentation slides, exercise files, event flyer)
Video (Media Collections Online)
Friday, January 19, 2018
Dr. J. Scott Long, "Reproducible Results and the Workflow of Data Analysis"
2-3:30pm, Social Science Research Commons Grand Hall (Woodburn Hall 200)
Many disciplines are paying increasing attention to reproducible results. The fundamental idea is that other scientists should have access to your data and be able to obtain the same results—this is reproducibility. More generally, your results should be robust so that other scientists can confirm your findings using other data. Increasingly journals require authors to provide their data and analysis file before a paper is accepted to verify that that results. Producing reproducible results is highly dependent on your workflow for data analysis. This workflow encompasses the entire process of scientific research: Planning, documenting, and organizing your work; creating, labeling, naming, and verifying variables; performing and presenting statistical analyses; preserving your work; and ending with reproducible results. Most of the work in statistics classes focuses on estimating and interpreting models. In “real world” research projects, these activities may involve less than 10% of the total work. Professor Long’s talk is about the other 90% of the work. An efficient workflow saves time, introduces greater reliability into the steps of the analysis, and generates reproducible results.
Dr. Long is Distinguished Professor and Chancellor’s Professor of Sociology and Statistics at Indiana University.
Materials on IUScholarWorks (presentation slides, event flyer)
Friday, January 26, 2018
Dr. Richard Ball and Norm Medeiros, "Documenting Quantitative Research for Transparency and Reproducibility: Principles and Standards"
2-3:30pm (talk) and 4-5pm (hands-on workshop), Social Science Research Commons Grand Hall (Woodburn Hall 200)
This talk will present a set of standards for the replication documentation (data, code, and supporting information) that authors should assemble and make public when they release studies reporting the results of research based on analysis of statistical data. We will begin from first principles: What purposes is replication documentation intended to serve? And what must be true of the contents and organization of the documentation for a study if it is to fulfill those purposes? We will then describe how these general principles are embodied in the particular documentation standards we propose. Further discussion will include: (i) a comparison of our proposed standards with existing guidelines, such as TOPS, DA-RT, the BITSS Manual, and the "data policies" that have been adopted by a number of prominent journals, (ii) using the Open Science Framework (OSF), an on-line file management platform, for assembling and sharing replication documentation, and (iii) the curricular resources that are being produced by Project TIER for teaching and learning reproducible research methods.
Individualized hands-on workshop: After their talk, Norm and Richard will be available to work with anyone interested in a more in-depth and hands-on exploration of any related topics. Ex ante, our agenda is to go into more detail about using OSF--as a tool for managing a reproducible workflow for an individual project (like a doctoral dissertation or any empirical research paper) and/or as a platform to have students use for class exercises and projects. But the session will be informal, flexible, and driven by the preferences of the participants. We will start by having a quick discussion to learn what people are most interested in exploring, and update our ex ante agenda based on this new information. We have a variety of demos and exercises we could present and work through, and we would also be happy to give one-on-one consultation and advice.
Participants planning to attend the hands-on workshop may bring their own laptops or use computers available at the SSRC. Anyone interested in getting advice about managing a current or imminent research project should bring any data, code and related files that they have already assembled for the project.
This workshop will begin about 30 minutes after the conclusion of the talk, and continue until (i) the participants have had enough, (ii) Norm and Richard lose steam, or (iii) IU security comes around to turn out the lights and lock the door (whichever comes first).
Richard Ball is Professor of Economics at Haverford College. His primary teaching areas are game theory and statistical methods, and he supervises several senior theses every year. His research has included theoretical papers on political economy and empirical work on development and social issues. He earned his B.A. at Williams College (self-designed major in cultural anthropology and African studies); his M.S. at Michigan State University (agricultural economics); and his Ph.D. at the University of California, Berkeley (agricultural and resource economics). Richard has studied or worked in Sierra Leone, Chad, Egypt and Côte d'Ivoire.
Norm Medeiros is Associate Librarian at Haverford College (PA) where he oversees the collection management and metadata services division of the Libraries. Norm has been active in the Association for Library Collections & Technical Services (ALCTS) for many years, serving as its president for the 2015-2016 term. He has been co-Director of Project TIER (www.projecttier.org) with Richard Ball since they launched the initiative in 2013.
Materials on IUScholarWorks (event flyer)
Video (Media Collections Online)
Friday, February 2, 2018
Post and Riposte: A Panel Discussion on Open Science, Repositories and Data Sharing in the Social Sciences
2-3:30pm, Social Science Research Commons Grand Hall (Woodburn Hall 200)
Social science journals, funding agencies, and professional organizations increasingly encourage or require the archiving and sharing of data, code, publications, and other research materials for transparency and replicability. Panelists including Dr. Andrew Brown (Applied Health Science), Dr. Timothy Hellwig (Political Science), and Jamie Wittenberg (IU Libraries) will share their insights on the benefits and challenges of open social science scholarship, and their experiences with resources available to support the sharing of data, analysis, and publication files. Following the panel, join us for a question-and-answer period and a broader discussion with the panelists and other university stakeholders about supporting social science data sharing, transparency, and reproducibility efforts at Indiana University.
Materials available by email (event flyer)
Friday, February 9, 2018
Helge-Johannes Marahrens, "Introduction to Python for Social Scientists"
2-4pm, Social Science Research Commons Grand Hall (Woodburn Hall 200)
This workshop is offered in collaboration with the Indiana Intensive Didactic Seminar.
If you prefer to use your own computer for the hands-on portion of the workshop, please arrive 30 minutes early for setup.
Python has become the lead instrument for Data Scientists to collect, clean, and analyze data. As a general purpose programming language, Python is flexible and well-suited to handle large datasets. This workshop is designed for Social Scientists, who are interested in using Python, but have no idea where to start. Our goal is to "de-mystify" Python and to teach Social Scientists how to manipulate and examine data that deviate from the clean, rectangular survey format. Computers with Python pre-loaded are available in the SSRC on a first-come, first-served basis. This workshop is intended for social scientists who are new to programming. No experience required.
Helge-Johannes Marahrens is a second year doctoral student in the department of Sociology at Indiana University, working toward a PhD in Sociology and an MS in Applied Statistics. His research interests include cultural consumption, stratification, and computational social science with a particular focus on Natural Language Processing (NLP).
Materials on IUScholarWorks (presentation slides, code file, event flyer)
Video (Media Collections Online)
Friday, February 16, 2018
Dr. Adia Harvey Wingfield, "Black Professionals at Work: Methodological Approaches for Studying an Underrepresented Population"
3-4:30pm, Social Science Research Commons Grand Hall (Woodburn Hall 200)
Adia Harvey Wingfield is Professor of Sociology at Washington University in St. Louis. She specializes in research that examines the ways intersections of race, gender, and class affect social processes at work, and is an expert on the workplace experiences of minority workers in predominantly white professional settings, and specifically on black male professionals in occupations where they are in the minority. Prior to her talk at the Workshop in Methods, Dr. Wingfield will speak at the Karl F. Schuessler Institute for Social Research (1022 E. Third St.) on "Professional Work in a ‘Postracial' Era: Black Health Care Workers in the New Economy," 12-1:30pm.
Materials on IUScholarWorks (event flyer)
Video (Media Collections Online)
Friday, February 23, 2018
Dr. Olga Scrivner, "Introduction to Web Scraping Using Python"
2-4pm, Social Science Research Commons Grand Hall (Woodburn Hall 200)
In this workshop, you will learn how to extract web data with Beautiful Soup, a Python library for extracting data out of HTML- and XML-structured documents. You will also learn the basics of scraping and parsing data. In this hands-on workshop, we will also be using the DataCamp platform and participants are requested to have a free account with DataCamp prior the workshop.
This workshop is open to anyone, and previous knowledge of Python is not required.
Olga Scrivner is a Research Scientist at Cyberinfrastructure for Network Science Center (https://cns.iu.edu/) at IU and a Corporate Faculty in Data Analytics program at Harrisburg University of Science and Technology. As a CEWIT faculty fellow, she is also working on a number of workshops to expose women to technology and programming.
Materials on IUScholarWorks (presentation slides, handout, event flyer)
Video (Media Collections Online)
Friday, March 23, 2018
Dr. John Kruschke, "Analyzing ordinal data with metric models: What could possibly go wrong?"
2-4pm, Social Science Research Commons Grand Hall (Woodburn Hall 200)
(With co-author Torrin M. Liddell, Research and Statistics Analyst, Public Defender Commission, Indianapolis)
We surveyed all articles in the Journal of Personality and Social Psychology, Psychological Science, and the Journal of Experimental Psychology: General that mentioned the term "Likert," and found that 100% of the articles that analyzed ordinal data did so using a metric model. We demonstrate that analyzing ordinal data as if they were metric can systematically lead to errors. We demonstrate false alarms (i.e., detecting an effect where none exists, Type~I errors) and failures to detect effects (i.e., loss of power, Type~II errors). We demonstrate systematic inversions of effects, for which treating ordinal data as metric indicates the opposite ordering of means than the true ordering of means. We show the same problems --- false alarms, misses, and inversions --- for interactions in factorial designs and for trend analyses in regression. We demonstrate that averaging across multiple ordinal measurements does not solve or even ameliorate these problems. We provide simple graphical explanations of why these mistakes occur. Moreover, we point out that there is no sure-fire way to detect these problems by treating the ordinal values as metric, and instead we advocate use of ordered-probit models (or similar) because they will better describe the data. Finally, although frequentist approaches to some ordered-probit models are available, we use Bayesian methods because of their flexibility in specifying models and their richness and accuracy in providing parameter estimates.
John K. Kruschke is Provost Professor of Psychological and Brain Sciences, and Adjunct Professor of Statistics, at Indiana University in Bloomington, Indiana, USA. He is eight-time winner of Teaching Excellence Recognition Awards from Indiana University. He won the Troland Research Award from the National Academy of Sciences (USA), and the Remak Distinguished Scholar Award from Indiana University. He has been on the editorial boards of various scientific journals, including Psychological Review, the Journal of Experimental Psychology: General, and the Journal of Mathematical Psychology, among others.
Materials on IUScholarWorks (event flyer)
Video (Media Collections Online)
Friday, March 30, 2018
Dr. Victoria Reyes, "Transparency in Ethnography"
2-3:30pm, Social Science Research Commons Grand Hall (Woodburn Hall 200)
With the recent reporting of the reproducibility crisis in psychology and other social sciences, debates and concerns about transparency in quantitative and experimental research has once again taken center stage in academia. Yet how does the move toward open science, and its emphasis on reproducibility and replicability, translate to ethnographic and other forms of qualitative research? In this talk, I discuss the central differences in transparency in quantitative and qualitative research, where the former emphasizes transparency in data analysis and the spotlight on the latter focuses on transparency in data collection. I’ll also discuss how transparency in ethnography goes beyond a default in naming the exact people we study and the exact neighborhood of our research by showing how these decisions are not dichotomous. Some researchers name regions or cities instead of neighborhoods, and/or name public officials, but not their primary participants, while others mask identities of both place and people. These decisions aren't made haphazardly, carelessly, or only as a matter of convention. Instead, qualitative researchers have a long history of basing these, and other, considerations on their goals for the project and on a series of ethical concerns about how to maintain the relative anonymity of the people we study to protect what they tell us, their families, their reputations and any possible unintended consequences. As such, we should think of these decisions around transparency as different tools in our methodological toolkits that we draw on depending on the goals and purpose of each particular research project.
After her talk, Dr. Reyes will also lead a small hands-on session for researchers to workshop transparency issues in their research projects.
Victoria Reyes is an Assistant Professor of Sociology at the University of California, Riverside. She received her PhD from Princeton’s Department of Sociology in January 2015, and was a 2016-2017 Postdoctoral Fellow at the National Center for Institutional Diversity at the University of Michigan. She previously taught in Bryn Mawr College’s Growth and Structure of Cities Department. She studies boundaries; how they are created and remade as well as how they shape inequality in global settings. She has examined these processes as they relate to leisure migration, cultural politics, and legally plural, foreign-controlled places she calls “global borderlands.” Her work has been published or is forthcoming in Social Forces, Ethnography, Theory and Society, City & Community, Poetics, and International Journal of Comparative Sociology, among other outlets. She’s also written for the Monkey Cage at the Washington Post and Inside Higher Ed, and received fellowships from the Institute of International Education (2006-2007 Fulbright Scholar to the Philippines), the National Science Foundation (2009-2012 Graduate Research Fellowship), and the American Sociological Association (2014 cohort, Minority Fellowship Program).
Materials on IUScholarWorks (event flyer)
Video (Media Collections Online)
Friday, April 13, 2018
Dr. Emily Meanwell, "Qualitative Coding: Strategies for Transparency and Reproducibility"
2-4pm, Social Science Research Commons Grand Hall (Woodburn Hall 200)
Coding -- the process of categorizing segments of data (whether text, audio, video, still photos, etc.) -- is an important aspect of qualitative data analysis. Beyond this general definition, however, coding can take multiple forms depending on the goal of the research, the researcher's broad theoretical and/or epistemological approach, the tools and strategies that are employed, and the specific purposes or aims of coding. This workshop will provide an overview of qualitative coding, focusing specifically on reproducibility and transparency in the coding process.
The first part of the workshop provides a general overview of qualitative data coding, a discussion of the purposes and types of coding, and a review of the tools and strategies used for coding (e.g., qualitative data analysis software). In the second part of the workshop, we will delve further into strategies and best practices for reproducibility and transparency in the coding of qualitative data. We will talk about developing and documenting codes, codebooks, and coding procedures; discuss approaches to evaluating the reliability and reproducibility of coding; and discuss strategies for making coding and analysis transparent to others, regardless of whether or not you are able to share your raw data.
This workshop will incorporate a mix of lecture, hands-on activities, and discussion. Attendees are encouraged to bring their own data to work with in individual exercises, and/or to bring a particular research project or research question to keep in mind during hands-on applications.
Emily Meanwell is the Director of the Social Science Research Commons and the Study Director for the Sociological Research Practicum. She received her PhD in sociology in 2014 and has used a range of qualitative methods, including interviews, ethnography, and content analysis, in her research.
Materials on IUScholarWorks (presentation slides, event flyer)