Pro Yearly is on sale from $80 to $50! »

Systematic review and meta-analysis: open synthesis science supports effective decision-making

Cdb758075f7b54ea4ef646898497ecf3?s=47 cjlortie
May 26, 2020

Systematic review and meta-analysis: open synthesis science supports effective decision-making

A summary of the contemporary community of practice for synthesis science including systematic reviews and meta-analyses. The primary goal is to examine how synthesis can be used to inform decision making. Context, synthesis, and transparency tools structure this description of an open science philosophy to approach environmental grand challenges. There are three major implications of adopting this potential workflow for scientific syntheses. 1. Synthesis can confront biases and assumptions and thus change direction of science-based policy. 2. Transparency promotes interactions between the community of practice and our philosophy of science. Sandbox-thinking and tools that support version control, iteration, review, and reuse all promote better scientific synthesis. 3. Policy always uses evidence in some form, and contemporary scientific syntheses can provide a wider breadth of elements to draw from and support stronger evidence implementation. We must endeavor to provide a diverse, accessible set of evidence outcomes from our scientific syntheses to better support policy.

This work supported by this repository: https://github.com/cjlortie/synthesis4decisions and inspired by work with Florencia Miguel and Scott Butterfield in exploring evidence for dryland restoration https://cjlortie.github.io/dryland_restoration_synthesis/#a_synthesis_of_dryland_restoration_techniques

Resources
https://besjournals.onlinelibrary.wiley.com/doi/full/10.1111/1365-2656.12949
https://royalsocietypublishing.org/doi/10.1098/rsbl.2009.0546
https://academic.oup.com/icesjms/article/58/5/937/730862
https://www.environmentalevidence.org/completed-reviews/effects-of-wind-turbines-on-bird-abundance
https://royalsocietypublishing.org/doi/full/10.1098/rspb.2017.0829
https://academic.oup.com/bioscience/article/59/8/699/256190
https://esajournals.onlinelibrary.wiley.com/doi/full/10.1002/ecs2.2367
https://r4ds.had.co.nz
https://bioone.org/journals/freshwater-science/volume-29/issue-1/09-092.1/The-reference-condition--predicting-benchmarks-for-ecological-and-water/10.1899/09-092.1.full

Cdb758075f7b54ea4ef646898497ecf3?s=128

cjlortie

May 26, 2020
Tweet

Transcript

  1. Systematic review and meta-analysis: 
 open synthesis science supports effective

    decision-making @cjlortie
  2. context

  3. currency and ideals

  4. data are the coin of realm, but this is changing

    John Waller, GBIF
  5. John Waller, GBIF

  6. George Box All models are wrong, but some are useful

  7. It is inappropriate to be concerned about mice when there

    are tigers abroad. George Box
  8. Daniel Stouffer 2019 All ecological models are wrong, but some

    are useful
  9. synthesis is one amongst many solutions

  10. we need to know how systems work

  11. but there is a landscape for every issue

  12. shift the focus, blur the background, zoom in

  13. open synthesis science supports effective decision-making zoomed out, big picture

    
 however 
 better thinking, better data, & better tools
 are evolving its capacities
  14. community of practice in synthesis science first principles of inquiry

    still prevail logic relevance common practices scientific principles shared challenges & acceptable solution sets strength of evidence paradigm increasingly open workflows and data
  15. open science opens opportunities to build in new ways reuse,

    replication, realization
  16. how to use synthesis decisions Gavin Stewart 2010 validate ideas

    & assumptions with evidence 
 (extent of what we test and we know) estimate strength of evidence 
 (effect sizes to capture relative difference) identify gaps, bias, and big-picture thinking*
  17. *science will not advance by the mere accumulation of data

    Gavin Stewart 2010
  18. None
  19. implicit bias, compelling case studies, & big data can inadvertently

    misdirect assessment of key drivers and processes that broadly describe patterns
  20. synthesis

  21. Synthesis can confront biases and assumptions and thus change direction

    of science-based policy.
  22. we assumed fisheries can rebound

  23. None
  24. 700 spawner–recruit relationships illustrated that most commercial marine fishes produce

    less than 5 viable young a year at low population sizes
  25. None
  26. None
  27. too little, too late, but we had the data

  28. wind turbines kill birds (and bats)

  29. None
  30. 124 articles, 15 met criteria, 19 datasets

  31. short-term bird abundance studies do not provide robust indicators of

    the potentially deleterious impacts of wind farms on bird abundance small sample sizes are a critical issue
  32. identified acceptable risks and windfarms moved forward in the UK

  33. independent, updated meta in 2017 confirmed we made the right

    call Thaxter et al.
  34. scientific synthesis combines theory, data, and tools to engineer and

    guide knowledge inspired by Carpenter et al. 2009
  35. inspired by Carpenter et al. 2009 practically want to catch

    fishes (evidence) prep (community of practice ideals) then do some cooking (SR/meta)
  36. meta-analyses and systematic reviews are most common but not the

    sole forms of synthesis for decision making environmental science estimated meta frequencies
  37. None
  38. novelty today for synthesis for decision making needs driven rapidly

    increasing diversity in people & evidence cultural changes in acceptance and openness
  39. mode 2 knowledge production

  40. species evidence map

  41. techniques evidence map

  42. viability of techniques for specific species

  43. next step is to assess not just who, what, where,

    when, and why but what works
  44. evidence map of application Miguel et al. in review

  45. meta-analysis of efficacy Miguel et al. in review

  46. develop insights and test application with stakeholders to inform policy

  47. transparency tools

  48. Transparency promotes interactions between community of practice and our philosophy

    of science. Sandbox-thinking and tools that support version control, iteration, review, and reuse all promote better scientific synthesis.
  49. never vote count

  50. draw evidence maps

  51. many beautiful methods to map evidence

  52. develop a workflow consider on open, collaborative set of mechanisms

    1. Search 2. Sort 3. Synthesize 4. Summarize 5. Statistics
  53. tools that enable these steps dialog with stakeholders GitHub (version

    control, distributed work) open, published data in repositories team data checking R and scripts team science code review before writing
  54. workflow thinking borrowed from data science

  55. Wickham & Grolemund prep before cooking is important recipes help

  56. search & sort

  57. synthesize means variance samples become effect sizes

  58. effect sizes are the secret ingredient ratio, index, or weighted

    measure of difference
  59. treatment-control pre-post (time) baseline-changed reference conditions

  60. Hawkins et al. 2010 simplest & luckiest is a control

  61. summarize

  62. statistics

  63. Policy always uses evidence in some form, and contemporary scientific

    syntheses 
 can provide a wider breadth of products to support 
 stronger evidence implementation.
  64. code data maps theory ideas gaps viz models lists capacity

    to repeat