Through DARPA's Systematizing Confidence in Open Research and Evidence (SCORE) program, we are developing artificial prediction markets to evaluate the reproducibility of published research claims in the social and behavioral science literatures. Markets are populated by artificial agents; they will be trained and updated within human-expert prediction markets, but deployable offline. This line of work aims to improve our understanding of the robustness of the existing social science literature and highlight paths forward for a more credible, interpretable scholarly record.

In parallel, we are also working with the National Science Foundation's National Center for Science and Engineering Statistics to study trends in research and publishing, integrating data from their annual Survey of Doctoral Recipients. Our broader aim is data-driven policies on scientific processes and funding, with specific focus on underrepresented PhD recipients. 

Recent papers on science policy focus on the ethics of securing our increasingly interdependent cyber-physical critical infrastructure.

​

• S. Koneru, J. Wu, S. Rajtmajer. Can LLMs discern evidence for scientific hypotheses? Case studies in the social sciences. Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING), May 2024.

• C. Wu, X. Wang, J. Carroll, S. Rajtmajer. Reacting to Generative AI: Insights from Student and Faculty Discussions on Reddit. ACM Conference on Web Science (WebSci), May 2024.

• C. Wu, T. Chakravorti, J. Carroll, S. Rajtmajer. Integrating measures of replicability into scholarly search: Challenges and opportunities. ACM Conference on Human Factors in Computing Systems (CHI), May 2024.

• S. Koneru, X. Wei, J. Wu, S. Rajtmajer. Can machine learning algorithms predict publication outcomes? A case study of COVID-19 preprints. AI4SciSci Workshop at the International Conference on Data Mining (ICDM), December 2023.

• K. Ajayi, M. Hasan Choudhury, S. Rajtmajer, J. Wu.  A Study on Reproducibility and Replicability of Table Structure Recognition Methods. International Conference on Document Analysis and Recognition (ICDAR), August 2023.

• ​S. Koneru, M. Smith, D. Guarrera, J. Robinson, S. Rajtmajer. The evolution of scientific literature as metastable knowledge states. PLoS ONE, July 2023.

• K. Ajayi, M. Hasan Choudhury, S. Rajtmajer, J. Wu.  A Study on Reproducibility and Replicability of Table Structure Recognition Methods. International Conference on Document Analysis and Recognition (ICDAR), August 2023.

• T. Chakravorti, R. Fraleigh, T. Fritton, M. McLaughlin, V. Singh, C. Griffin, A. Kwasnica, C. Pennock, C. Lee Giles and S. Rajtmajer. A prototype hybrid prediction market for estimating replicability of published work. The Second International Conference on Hybrid Human-Artificial Intelligence (HHAI), June 2023.

• T. Chakravorti, V. Singh, M. McLaughlin, R. Fraleigh, C. Griffin, A. Kwasnica, D. Pennock, C. Lee Giles and S. Rajtmajer. Artificial prediction markets present a novel opportunity for human-AI collaboration. 22nd International Conference on Autonomous Agents and Multiagent Systems (AAMAS), May 2023. Extended Abstract.

• D. Priestley, J. Staph, S. Koneru, S. Rajtmajer, A. Cwiek, S. Vervoordt, F. Hillary. Establishing ground truth in the traumatic brain injury literature: if replication is the answer, then what are the questions? Brain Communications, December 2022.

• S. Rajtmajer, T. Errington, F. Hillary. How failure to falsify in high-volume science contributes to the replication crisis. eLife, August 2022.

• E. Cruz Cortes, S. Rajtmajer, D. Ghosh. Structural Interventions on Automated Decision Making Systems. ACM Conference on Fairness, Accountability, and Transparency (ACM FAccT), June 2022.

• L. Salasbil, J. Wu, M. Choudhury, W. Ingram, E. Fox, S. Rajtmajer and C. Lee Giles. A Study of Computational Reproducibility using URLs Linking to Open Access Datasets and Software. Sci-K @ The Web Conf, April 2022.

• S. Rajtmajer, C. Griffin, J. Wu, R. Fraleigh, L. Balaji, A. Squicciarini, A. Kwasnica, D. Pennock, M. McLaughlin, T. Fritton, N. Nakshatri, A. Menon, S.A. Modukuri, R. Nivargi, X. Wei, C.L. Giles. A synthetic prediction market for estimating confidence in published work. AAAI 2022 Demonstrations, February 2022.

• E. Cruz Cortes, S. Rajtmajer, D. Ghosh. Structural Interventions on Automated Decision Making Systems. NeurIPS workshop on Algorithmic Fairness through the Lens of Causality and Robustness (NeurIPS AFCR), December 2021.

• A. Cwiek, S. Rajtmajer, B. Wyble, V. Honavar, F. Hillary. Feeding the machine: challenges to reproducible predictive modeling in resting-state connectomics. Network Neuroscience, October 2021.

• S. Lanka, S. Rajtmajer, J. Wu, C. Lee Giles. Extraction and evaluation of statistical information from social and behavioral science papers. Workshop on Scientific Knowledge (Sci-K) at The Web Conference, April 2021.

• S. Modukuri, S. Rajtmajer, A. Squicciarini, J. Wu, L. Giles. Understanding and predicting retractions of published work. AAAI-21 Workshop on Scientific Document Understanding (AAAI-SDU), February 2021. 

• C. Grady, S. Rajtmajer, L. Dennis. When smart systems fail: the ethics of cyber-physical critical infrastructure risk. IEEE Symposium on Technology and Society (IEEE ISTAS), November 2020.

• L. Dennis, S. Rajtmajer, C. Grady. Analyzing cyber-physical threats to Pennsylvania dams through a lens of vulnerability. IEEE Symposium on Technology and Society (IEEE ISTAS), November 2020.

• J. Wu, P. Wang, X. Wei, S. Rajtmajer, C. Lee Giles, C. Griffin. Accurate Acknowledgement Entity Recognition in CORD-19 Papers. Workshop on Scholarly Document Processing and Shared Tasks at The Conference on Empirical Methods in Natural Language Processing (EMNLP SDP), November 2020. 

• S. Rajtmajer and D. Susser. Automated Influence and the Challenge of Cognitive Security. ACM Symposium on the Science of Security (HoTSoS), February 2020.