METASCIENCE, SCIENCE POLICY AND ETHICS

Through DARPA's Systematizing Confidence in Open Research and Evidence (SCORE) program, we are developing artificial prediction markets to evaluate the reproducibility of published research claims in the social and behavioral science literatures. Markets are populated by artificial agents; they will be trained and updated within human-expert prediction markets, but deployable offline. This line of work aims to improve our understanding of the robustness of the existing social science literature and highlight paths forward for a more credible, interpretable scholarly record.

In parallel, we are also working with the National Science Foundation's National Center for Science and Engineering Statistics to study trends in research and publishing, integrating data from their annual Survey of Doctoral Recipients. Our broader aim is data-driven policies on scientific processes and funding, with specific focus on underrepresented PhD recipients. 

Recent papers on science policy focus on the ethics of securing our increasingly interdependent cyber-physical critical infrastructure.

  • S. Rajtmajer, C. Griffin, J. Wu, R. Fraleigh, L. Balaji, A. Squicciarini, A. Kwasnica, D. Pennock, M. McLaughlin, T. Fritton, N. Nakshatri, A. Menon, S.A. Modukuri, R. Nivargi, X. Wei, C.L. Giles. A synthetic prediction market for estimating confidence in published work. AAAI 2022 Demonstrations, February 2022.

  • E. Cruz Cortes, S. Rajtmajer, D. Ghosh. Structural Interventions on Automated Decision Making Systems. NeurIPS workshop on Algorithmic Fairness through the Lens of Causality and Robustness (NeurIPS AFCR), December 2021.

  • A. Cwiek, S. Rajtmajer, B. Wyble, V. Honavar, F. Hillary. Feeding the machine: challenges to reproducible predictive modeling in resting-state connectomics. Network Neuroscience, October 2021.

  • N. Nakshatri, A. Menon, C. Lee Giles, S. Rajtmajer, C. Griffin. Design and analysis of a synethetic prediction market using dynamic convex sets. Results in Control and Optimization, September 2021.

  • S. Lanka, S. Rajtmajer, J. Wu, C. Lee Giles. Extraction and evaluation of statistical information from social and behavioral science papers. Workshop on Scientific Knowledge (Sci-K) at The Web Conference, April 2021.

  • S. Modukuri, S. Rajtmajer, A. Squicciarini, J. Wu, L. Giles. Understanding and predicting retractions of published work. AAAI-21 Workshop on Scientific Document Understanding (AAAI-SDU), February 2021. 

  • C. Grady, S. Rajtmajer, L. Dennis. When smart systems fail: the ethics of cyber-physical critical infrastructure risk. IEEE Symposium on Technology and Society (IEEE ISTAS), November 2020.

  • L. Dennis, S. Rajtmajer, C. Grady. Analyzing cyber-physical threats to Pennsylvania dams through a lens of vulnerability. IEEE Symposium on Technology and Society (IEEE ISTAS), November 2020.

  • J. Wu, P. Wang, X. Wei, S. Rajtmajer, C. Lee Giles, C. Griffin. Accurate Acknowledgement Entity Recognition in CORD-19 Papers. Workshop on Scholarly Document Processing and Shared Tasks at The Conference on Empirical Methods in Natural Language Processing (EMNLP SDP), November 2020. 

  • M. Gomez, S. Garcia, S. Rajtmajer, C. Grady, A. Mejia. Fragility of a Multilayer Network of Intrantaional Supply Chains. Applied Network Science, September 2020.

  • S. Rajtmajer and D. Susser. Automated Influence and the Challenge of Cognitive Security. ACM Symposium on the Science of Security (HoTSoS), February 2020.

  • S. Garcia, S. Rajtmajer, C. Grady, P. Mohammadpour, A. Mejia. Performance of a Multiplex Commodity Flow Network in the United States Under Disturbance. International Conference on Complex Networks and their Applications (Complex Networks), December 2019.