Improving Candidate Selection for Academic Conferences and Beyond

Academic conferences and workshops are the lifeblood of our profession: they provide venues for presenting new results, exchanging knowledge, building new collaborations, and networking with potential future employers, among other things.
As such, conferences are important to the careers of researchers: in particular exclusive, space-limited workshops provide focused, direct contact with other scientists, and presenting at a large international meeting is seen as prestigious. The importance of these meetings also implies that they act as gatekeepers and can reinforce existing power structures while unintentionally excluding qualified candidates. Selection of speakers and attendees is often done during internal discussions of a scientific organizing committee aiming to select the best set of candidates — but what “best” might mean is often not clearly defined and subject to multiple competing constraints. In this talk, I will lay out our recent experiences with participant selection for several space-limited, severely oversubscribed workshops, and introduce Entrofy, an algorithm and open-source software project aimed at making participant selection more transparent and equitable

Transcript

1. Daniela Huppenkothen NYU Center for Cosmology and Particle Physics NYU Center for Data Science ! Tiana_Athriel " dhuppenkothen Improving Candidate Selection for academic and beyond Conferences
3. http://allmalepanels.tumblr.com
4. ~180 applications!
5. • 5 days • 50 researchers • all of astronomy represented • tutorials, lectures, hacking • highly interactive! #AstroHackWeek
6. • 5 days • 50 researchers • all of astronomy represented • tutorials, lectures, hacking • highly interactive! #AstroHackWeek ~180 applications!
7. How do we select the best* • set of participants, • person for the job, • speaker, • panel … ? * and what does “best” even mean?
8. Standard procedures produce a biased set. e.g. Moss-Racusin et al (2012), Azmat & Petrongolo (2014)
9. arXiv: 1502.06326
10. How can we be transparent about our selection procedures?
11. Algorithmic Transparency* *note: humans have no algorithmic transparency whatsoever!
12. Goals: • optimise mix of participants • reduce human bias • transparency • accountability
13. But how???
14. Step 1: Define “best”
15. Astro Hack Week • teach new data science methods • increase connectedness • foster exchange between disciplines • build networks • promote open science
16. Astro Hack Week
17. Diversity beginners — experts: allows learning
18. Diversity undergrads — senior faculty: build networks
19. Diversity academic sub disciplines: foster exchange
20. Diversity diversity in background: new perspectives + ideas
21. AstroHackWeek is only as successful as its participants
22. Step 2: (blind) selection of appropriate candidates
23. Astro Hack Week Selected ~160 out of 180 candidates as appropriate.
24. Python In Astronomy
25. Python In Astronomy • blind scoring of talk abstracts
26. Python In Astronomy • blind scoring of talk abstracts • each committee member gave a yes/no answer for each talk
27. Python In Astronomy • blind scoring of talk abstracts • each committee member gave a yes/no answer for each talk • other members could only see their own response, not their peers’
28. Python In Astronomy • blind scoring of talk abstracts • each committee member gave a yes/no answer for each talk • other members could only see their own response, not their peers’ 20 out of 90 got full scores, 13 slots
29. Step 3: Pre-select • organising committee • keynote speakers • participants with crucial knowledge
30. … and then what ???
31. “You’re a data science fellow. Do data science with this!” — Brian McFee
32. Computers are better at complex optimisation than humans
33. (Some) computer programmes have algorithmic transparency.
34. maths
35. Step 4: Use algorithm to break ties conditioned on the ideal mix of participants
36. Entrofy* https://github.com/dhuppenkothen/entrofy *with Brian McFee
37. Entrofy • inputs: • attributes for each applicant in each relevant category • target fractions for each category • outputs: • optimal set of participants given the input sample and the targets
38. note: pre-selected participants go into algorithm, too!
39. Does it work?
40. transition is not well determined, since the grid used in exploring parameter is fairly coarse (and uses steps of either 0 . 1 or 10 for almost all parameters). 1 10 100 ntrials 0 10 20 30 40 50 Percentage of failures FIG. 3. The percentage of failed Entrofy runs versus the Huppenkothen et al (in prep) Simulations
42. Is it transparent?
43. https://github.com/dhuppenkothen/PyAstro17ParticipantSelection
44. Things we learned
45. automatic participant selection requires clear articulation of workshop goals
46. Goal: diversity in current academic position
48. Goal: accept researchers from all areas of astronomy + solar physics
50. Goal: allow open-source developers to exchange experiences + introduce new developers
52. Goal: select good talks!
54. Goal: bring researchers from underrepresented countries into the community and spread best practices
56. Goal: increase demographic diversity in open-source community
58. Data Is Useful! • better recruitment in following years • better targeting of fundraising efforts
59. • Astro Hack Week • Python in Astronomy • LSST Data Science Fellowship Program • Data on the Mind …
60. Conclusions • selecting candidates is a complex task usually with multiple constraints • reducing human biases should be an objective of any selection procedure • computers are better at complex optimization than humans • Entrofy can be used to break ties in oversubscribed cases • Not just for conferences!
61. Some links • Entrofy: https://github.com/dhuppenkothen/entrofy • In Wired! https://www.wired.com/2017/05/modern- astronomers-teaching-code • PyAstro selection: https://github.com/dhuppenkothen/ PyAstro17ParticipantSelection • Hack Week Paper: https://github.com/uwescience/ HackWeek-Writeup • watch out for the Entrofy paper!
62. Questions?
63. Algorithm 2 lists the full Entrofy algorithm, includ- ing the concave transformation and randomized near-tie breaking. Algorithm 2 The full Entrofy algorithm 1: procedure Entrofy ( S, k, { ai } , { pi } , { wi } , ↵, q ) 2: Let f ( X ) := X i wi min kpi, X x 2 X ai ( x ) ! ↵ 3: Initialize X ; 4: while | X | < k do 5: Let d ( x ) := f ( X, x ) for all x 2 S \ X 6: Let Q := { x | x 2 top- q quantile of d ( x )} 7: Select x uniformly at random from Q 8: X X [ { x } 9: return X , f ( X ) 10: procedure Entrofy-MC ( n, S, k, { ai } , { pi } , { wi } , ↵, q ) 11: for i 2 1 . . . n do 12: X [ i ] , F [ i ] Entrofy ( S, k, { ai } , { pi } , { wi } , ↵, q ) 13: return the X [ i ] with largest F [ i ] participants in 10 and 100, sin will be used fo number of rand tween 1 and 1 in the objectiv combination o leading to a to For each sim jective functio and then com found by Entr the solution we 0, we deemed ure: in this ca solution embed In fig. 2, w Here, we kept fractions (defin applicants) of the target and to explore the