Reproducibility and Replicability in Geography

As the number of scientific research greatly expand in the past few decades, we are seeing certain group of scientists devoting an increasing amount of time and energy on reproducing and replicating existing research. Yet, when these people are making efforts to reproduce or replicate others’ work, what exactly are they doing and what are the underlying purposes of doing so? This question raises the needs for us to examine the significance of reproducibility and replicability (R&R) in the academia as well as to evaluate the benefits and limits. This blog post reflects on the article “Reproducibility and Replicability in the context of contested identities of geography” by Kedron and Sui, where they approached the R&R debate by illustrating its hope and crisis with respect to science in general and geography in specificity.

The hope that R&R has brought to the academia is evident. To reproduce a study, researchers need to access the original data and follow the same procedure, hoping to achieve results consistent with a prior study. To replicate a study, researchers answer the same question but use their own data, hoping to achieve results across studies. With this in mind, both practices are feasible only when the conceptual and procedural transparency of prior research have been maintained. Doing so facilitate the collective production of new knowledge as researchers are held accountable of the original workflow. This allows them to critically examine and refine others’ work, apply them to various disciplines and/or progress further towards any generalizable theories and results.

This reminds us of the movement of adopting open-source softwares in teaching and research, where the collaborative research environment they embodied and the transparent methodologies they appeal for ensure the precise implementation of analytical methods and the identification of errors and limitations. Indeed, the hope of R&R should also be considered in parallel with the open-source movement as adopting open-source softwares eliminate one of the greatest R&R crisis, the lack of code, and detailed description of methods that frequently lead to non-reproducibility.

The discipline of geography is reliant on R&R because geographers are also interested in generalizing and seeking regularities in spatial patterns. As such, geographers reproduce research to eliminate uncertainties of prior findings and replicate research to interpret any characteristics that could differentiate one place from other. Within the framework of using open-source GIS software, researchers share details, such as code chunks, algorithms, workflows, of spatial analyses they have conducted. This made it possible for others to confirm and check the correctness of computation and develop on existing findings, which in turn establishes credibility and novelty of the original research.

Because of the interdisciplinarity inherent in geography, a greater implication of R&R on geography would be the unlimited opportunities to apply studies across disciplines. In other words, the significance of lessons learned from R&R within geography extend beyond the boundaries of geography and benefit knowledge production in other fields (in geology, urban studies, architecture studies, etc.). Hence, if practicing R&R from an open science perspective, geographers can align their research with other dominant scientific inquires.

However, given the hope and benefits of practicing R&R and how it helps to establish the credibility of existing research, should we set up a standard based on the degree of R&R to evaluate the success of a research? AS mentioned by Kedron and Sui, the implication of R&R is complex and multifaced. The crisis of non-reproducibility perpetuates, and the reasons that lead to such non-reproducibility are often multifarious. We are unconscious about many of those reason. For example, many of the exact methods employed in a study may not be detailed in the method section of a study, or the original study itself is designed and reported poorly (NASEM 2019).

To situate our discussion in the context of geography, non-reproducibility and non-replicability are often attributed to the role of place, time, and scale. The spatial and temporal heterogeneity inherent in many geographic research have made it difficult to establish consistent criteria of when replications have been achieved. On top of that, even with openly accessible data and open-source software, any variables operating at different places and scales could complicate the R&R process and eventually render a study non-reproducible. Nevertheless, failing to reproduce and replicate geographic research does not translate to untruth and doing so successfully does not imply truth as in geography, it is always easy to delineate place from place but hard to generalize. At this point, it is more important to reflect on what we can learn from failure to replicate a research: the differences between places or times.

In summary, with the open-source movement as the fundamental pillar, practicing R&R improves existing research, establishes their credibility, and facilitates knowledge production. Yet, while non-reproducibility is a crisis, research that are non-reproducible should not be deemed as unsuccessful. The limit of R&R is its emphasis on identifying regularities in order to systematically explain and interpret results. In geography, it is necessary to not only generalize patterns across space and examine their relation to one another (nomothetic approach), but also to understand the meaning of unique and often cultural, subjective phenomena from a more humanistic approach (idiographic approach).

===

NASEM. 2019. Reproducibility and Replicability in Science. Washington, D.C.: National Academies Press. DOI: 10.17226/25303 Chapter 3, Understanding reproducibility and replicability (pages 31-43)

Sui, D., and P. Kedron. 2021. Reproducibility and Replicability in the Context of the Contested Identities of Geography. Annals of the American Association of Geographers 111 (5):1275–1283. DOI:10.1080/24694452.2020.1806024.

===

Huzhou, China (09/01/20)

How successful could we reproduce this ancient town under the water ~

Back to Main Page