Machine Learning in the Humanities and Social Sciences: Why It Matters and How to Use It

***Key Takeaways

• **Machine learning is rapidly expanding its role as a methodological complement in the humanities and social sciences — handling unstructured data, bridging explanation and prediction, exploring heterogeneous effects, and supporting LLM-assisted workflows.
• Specific methods such as STM, word embedding, causal forest, computer vision, and LLMs have demonstrated their value through concrete case studies that transform text, image, and spatial data into meaningful social science variables.
• When applying machine learning, systematic validation of measurement validity, sampling bias, normative appropriateness of prediction targets, reproducibility, and ethics must accompany every step.*

1. Introduction

In the humanities and social sciences, machine learning (ML) serves as a methodological complement — one that measures large-scale unstructured data, tests the limits of predictability, and uncovers heterogeneous effects and hidden patterns.

The case for machine learning in the humanities and social sciences rests on several foundations.

• First, data that were once inaccessible through traditional surveys, interviews, or statistical tables — digital text, images, audio, networks, and behavioral logs — have become central research materials.
• Second, abstract concepts such as discourse, sentiment, cultural meaning, urban environment, social status, and political orientation can now be translated into measurable variables from large-scale data.
• Third, “how predictable is a social phenomenon?” has become as important a research question as “why does it occur?”
• Fourth, researchers can now explore heterogeneous effects across subgroups and contexts, rather than being limited to average effects. [Molina, M., & Garip, F (2019)]
• Finally, large language models (LLMs) are increasingly used as auxiliary tools for text coding, hypothesis generation, experiment design, simulation, and literature search.

At the same time, there is growing discussion about the need to validate measurement validity, sampling bias, reproducibility, interpretability, fairness, and ethics whenever machine learning is applied. This is especially important because social science data involve people, institutions, and power relations — which means questions like “what is being predicted?” and “what social construct does the prediction proxy?” matter far more than simple accuracy metrics. [Lazer, D. M., Pentland, A., Wa… (2020)]

2. Why Machine Learning Is Needed

2.1 The Rise of Unstructured Data

Social science and humanities data are no longer confined to survey responses or administrative statistics. News articles, parliamentary speeches, court rulings, social media posts, online communities, images, maps, street photographs, digital archives, academic literature, comments, and network logs have all become research data. Digital text has become a primary medium for observing human interaction, communication, culture, and institutions. [Gentzkow, M., Kelly, B., & Taddy, M (2019)]

Machine learning is not simply a tool for processing large volumes of unstructured data. It functions as a measurement instrument — converting previously unobservable sociocultural concepts into computable variables. For example, topic models estimate the thematic structure of large document collections; word embeddings measure how word meanings and stereotypes shift across time periods; and computer vision extracts information about physical changes in urban environments or neighborhoods from street-level imagery. [Roberts, M. E., Stewart, B. M…. (2014)]

2.2 Combining Explanation-Centered and Prediction-Centered Research

Traditionally, the humanities and social sciences have focused primarily on explanation — answering the question “why does this happen?” Machine learning, by contrast, emphasizes prediction: “how accurately can we forecast this outcome?” Hofman et al. argue that social science should integrate explanatory and predictive models, and that predictive performance can serve as a criterion for assessing how well a theory captures the real world. [Hofman, J. M., Watts, D. J., A… (2021)]

Prediction is especially important in policy research. Questions involving unemployment risk, recidivism risk, school dropout risk, gaps in welfare coverage, health risks, and the likelihood of misinformation spreading are fundamentally predictive in nature — they require prioritizing policy interventions. However, feeding predictions directly into policy decisions can produce discrimination and misjudgment. Predictions should therefore serve as supporting evidence for policy judgment, not as a substitute for it.

2.3 Revealing the Limits of Predictability

Machine learning does not guarantee high accuracy. In fact, some of the most important findings have come from studies showing that even the best data and models cannot reliably predict certain social outcomes. In the Fragile Families Challenge, Salganik et al. had 160 research teams use rich child and family data to predict six life outcomes. Even the top-performing models barely outperformed simple baseline benchmarks. The significance of this study lies in demonstrating — through a large-scale collaborative experiment — that the predictability of complex life outcomes is fundamentally limited. [Salganik, M. J., Lundberg, I.,… (2020)] In short, machine learning in the humanities and social sciences is not only a tool for getting the right answer; it is also a method for revealing the uncertainty inherent in social phenomena, the limits of data, and the explanatory boundaries of theory.

2.4 Exploring Heterogeneous Effects Beyond Average Effects

Classical regression analysis is well-suited to estimating average effects. In practice, however, the same policy or educational program operates differently across subgroups, regions, time periods, and institutional contexts. The important question is not just “is there an effect?” but “for whom, under what conditions, and how large is the effect?” Methods such as causal forest, generalized random forest, and causal tree are useful for identifying which subgroups show larger or smaller treatment effects. This reduces the limitation of traditional statistical analysis, which required researchers to define subgroups in advance. [Athey, S., Tibshirani, J., & Wager, S (2019)]

Brand et al. explain that tree-based machine learning in sociology can reveal effect heterogeneity that the data uncover — not just heterogeneity across subgroups defined a priori. This approach is particularly useful in research on education policy, labor markets, welfare policy, crime prevention, and health inequality. [Brand, J. E., Xu, J., Koch, B., & Geraldo, P (2021)]

2.5 Machine Learning and LLMs as Research Assistants, Coders, and Simulators

Traditional content analysis involved research assistants reading and coding documents manually. While this approach can yield high-quality results, it is costly, slow, and difficult to scale to large corpora. Recent LLMs are increasingly used for text annotation, classification, summarization, frame detection, and argument structure analysis. For instance, hundreds of thousands of news comments can be classified into categories such as “policy criticism,” “party attack,” “hate speech,” “conspiracy theory,” “experience sharing,” and “information provision” — with LLMs handling the initial coding pass and human coders validating and correcting a subsample. This makes content analysis at a scale that was previously impossible due to research staffing and budget constraints now achievable. Bail argues that generative AI has the potential to improve survey research, online experiments, automated content analysis, and agent-based modeling. [Bail, C. A (2024)]

Ziems et al. further note that while LLMs can be used for social phenomenon classification, explanation, annotation, and simulation, they do not always outperform existing fine-tuned models on granular social science tasks, and task-specific validation is required. [Ziems, C., Held, W., Shaikh, O… (2024)] Recent discussions on LLM evaluation emphasize transparency, reliability, validity, and non-determinism. In other words, LLMs can be incorporated into research, but model name, version, prompt, temperature setting, validation data, and inter-rater agreement with human coding must all be documented. [Abdurahman, S., Salkhordeh Zia… (2025)]

3. Application Methods by Research Purpose

4. Research Cases

Case 1. Transforming Text into Social Science Data: STM and Text as Data

Roberts et al.’s Structural Topic Model estimates the topic proportions in a document while incorporating document-level metadata — such as author gender, political orientation, experimental treatment, or time period — directly into the model. This method is well-suited for analyzing open-ended survey responses, policy documents, news articles, parliamentary speeches, and online discussions. Gentzkow, Kelly, and Taddy’s “Text as Data” systematized the full workflow for using text as quantitative data in economics and social science.

In research on criminal investigation policy discourse, for example, this approach can estimate how topics such as “organized crime,” “cybercrime,” “drugs,” “victim protection,” “prevention,” and “harsher punishment” shift over time in congressional records, news articles, and law enforcement reports. External events such as changes in government, major incidents, or legislative revisions can then be included as covariates to connect those events to topic change.

Case 2. Measuring Culture and Stereotypes: Word Embedding

Garg, Schiebinger, Jurafsky, and Zou combined 100 years of text data with demographic records to measure changes in gender and ethnic stereotypes using word embeddings. Hamilton, Leskovec, and Jurafsky used corpora across multiple languages and time periods to analyze statistical patterns of semantic change. Kozlowski, Taddy, and Evans analyzed the cultural meaning structure of class through word embedding space. [Garg, N., Schiebinger, L., Jurafsky, D., & Zou, J (2018)]

By comparing which words appear closest to terms like “worker,” “woman,” “youth,” “criminal,” “immigrant,” and “victim” in newspaper corpora across different periods, researchers can quantitatively track shifts in cultural perceptions of social groups.

Case 3. Policy Decision-Making and Prediction: Bail Decision Research

Kleinberg, Lakkaraju, Leskovec, Ludwig, and Mullainathan reframed the problem of judicial bail decisions as a prediction task: “Will the defendant reappear in court? Will they commit another crime?” The study compared human expert judgment with machine predictions, offering a landmark demonstration of the potential efficiency gains from predictive algorithms in policy contexts. [Kleinberg, J., Lakkaraju, H., … (2018)] This research does not argue for replacing judges with algorithms. Rather, the more important social science questions it raises are: “Where do human judgments systematically err?”, “Which groups bear the burden of algorithmic error?”, and “What is the actual policy objective — reducing recidivism, reducing detention, or ensuring equity?”

Case 4. The Limits of Predicting Life Outcomes: The Fragile Families Challenge

The Fragile Families Challenge was a large-scale collaborative study in which 160 research teams used the same family and child panel data to predict life outcomes. The results carry an important lesson: even with rich data and a variety of ML models, complex social life outcomes were difficult to predict with high accuracy. This case demonstrates that machine learning is not a tool for easily predicting social phenomena — rather, it is a tool for evaluating just how limited our data and theory actually are.

Case 5. Urban and Spatial Research: Street View and Computer Vision

Gebru et al. used automated vehicle recognition from Google Street View images to estimate socioeconomic characteristics of U.S. neighborhoods — including income, race, education level, and voting behavior. Naik et al. analyzed street-level images across time to study physical changes in cities and their predictors. More recent work has extended this to estimating hidden urban profiles from street view imagery. [Gebru, T., Krause, J., Wang, Y… (2017)]

In urban sociology and regional policy research, measuring nationwide changes in the physical environment through fieldwork alone is impractical. Computer vision can extract variables such as pedestrian infrastructure quality, signage, green space, road condition, and building age at scale, enabling their use in studies of regional inequality.

Case 6. LLM-Assisted Text Annotation and Its Limitations

Gilardi, Alizadeh, and Kubli reported that ChatGPT outperformed crowd workers on certain text annotation tasks. However, Egami et al. warn that using LLMs or other surrogate labels directly — even at 80–90% classification accuracy — can introduce bias and incorrect confidence intervals in downstream inference. [Gilardi, F., Alizadeh, M., & Kubli, M (2023)] LLMs are useful tools for reducing the cost of large-scale coding, but in social science publications and policy reports, human-generated gold-standard samples, error analysis, uncertainty calibration, and reproducible prompt documentation are essential.

5. Recommendations for Research Design

When applying machine learning to humanities and social science research, the following steps warrant careful consideration.

• Define the Research Question Distinguish the research purpose first: prediction, explanation, measurement, or causal effect estimation.
• Define Theoretical Concepts Concepts such as “hate speech,” “political polarization,” “social capital,” “cultural status,” and “risk” must be defined by the researcher before the modeling stage.
• Determine the Unit of Analysis Decide on the unit of analysis: document, sentence, user, event, region, institution, network node, relation, or image.
• Construct Labels or Gold-Standard Data Supervised learning and LLM validation require samples coded directly by human annotators. Codebooks, inter-rater reliability scores, and rules for handling disagreements must be documented.
• Select a Model Consider supervised classifiers or LLMs for text classification; topic models for discourse exploration; embeddings for semantic change; causal forests for policy effect heterogeneity; computer vision for image analysis; and network embeddings or GNNs for relational prediction.
• Evaluate and Validate Do not evaluate by accuracy, F1, or AUC alone. Also assess error rates by subgroup, external data validation, out-of-time validation, interpretability, and social costs.
• Interpret and Connect to Theory Do not treat model outputs as direct social facts. Cross-validate against existing theory, historical context, and qualitative data.
• Ensure Ethics and Reproducibility Document data sources, collection dates, preprocessing code, model name, version, prompts, random seed, hyperparameters, and evaluation data.

6. Considerations

The greatest risk lies in mis-specifying the prediction target. Obermeyer et al. showed that a U.S. healthcare algorithm used healthcare spending — rather than medical need — as its prediction target, resulting in systematic underestimation of health risk among Black patients. This illustrates that in social science, “what to predict” is not a technical question but a normative and institutional one. [Obermeyer, Z., Powers, B., Vog… (2019)]

A second problem is fairness models that strip away social context. Selbst et al. argued that when fairness research abstracts away sociotechnical context, it can fail in practice within real institutions. Machine learning in the humanities and social sciences must therefore analyze not only bias within the model, but also the data generation process, institutions, power relations, and the effects of interpretation. [Selbst, A. D., Boyd, D., Fried… (2019)]

7. Conclusion

In the humanities and social sciences, machine learning transforms large-scale unstructured data into theoretically meaningful variables, bridges explanation and prediction, and reveals heterogeneity and limitations that existing theories have missed. But its results must always be combined with the researcher’s theoretical reasoning, contextual interpretation, validity testing, and ethical judgment.

NetMiner | CYRAM Network Science · Social Network Analysis · Graph Analytics https://www.netminer.com

References

• Abdurahman, S., Salkhordeh Ziabari, A., Moore, A. K., Bartels, D. M., & Dehghani, M. (2025). A primer for evaluating large language models in social-science research. Advances in Methods and Practices in Psychological Science, 8(2), 25152459251325174.
• Athey, S., Tibshirani, J., & Wager, S. (2019). Generalized random forests.
• Bail, C. A. (2024). Can generative AI improve social science? Proceedings of the National Academy of Sciences, 121(21), e2314021121.
• Brand, J. E., Xu, J., Koch, B., & Geraldo, P. (2021). Uncovering sociological effect heterogeneity using tree-based machine learning. Sociological Methodology, 51(2), 189–223.
• Garg, N., Schiebinger, L., Jurafsky, D., & Zou, J. (2018). Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences, 115(16), E3635–E3644.
• Gebru, T., Krause, J., Wang, Y., Chen, D., Deng, J., Aiden, E. L., & Fei-Fei, L. (2017). Using deep learning and Google Street View to estimate the demographic makeup of neighborhoods across the United States. Proceedings of the National Academy of Sciences, 114(50), 13108–13113.
• Gentzkow, M., Kelly, B., & Taddy, M. (2019). Text as data. Journal of Economic Literature, 57(3), 535–574.
• Gilardi, F., Alizadeh, M., & Kubli, M. (2023). ChatGPT outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120(30), e2305016120.
• Hofman, J. M., Watts, D. J., Athey, S., Garip, F., Griffiths, T. L., Kleinberg, J., … & Yarkoni, T. (2021). Integrating explanation and prediction in computational social science. Nature, 595(7866), 181–188.
• Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2018). Human decisions and machine predictions. The Quarterly Journal of Economics, 133(1), 237–293.
• Lazer, D. M., Pentland, A., Watts, D. J., Aral, S., Athey, S., Contractor, N., … & Wagner, C. (2020). Computational social science: Obstacles and opportunities. Science, 369(6507), 1060–1062.
• Molina, M., & Garip, F. (2019). Machine learning for sociology. Annual Review of Sociology, 45(1), 27–45.
• Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447–453.
• Roberts, M. E., Stewart, B. M., Tingley, D., Lucas, C., Leder-Luis, J., Gadarian, S. K., … & Rand, D. G. (2014). Structural topic models for open-ended survey responses. American Journal of Political Science, 58(4), 1064–1082.
• Salganik, M. J., Lundberg, I., Kindel, A. T., Ahearn, C. E., Al-Ghoneim, K., Almaatouq, A., … & McLanahan, S. (2020). Measuring the predictability of life outcomes with a scientific mass collaboration. Proceedings of the National Academy of Sciences, 117(15), 8398–8403.
• Selbst, A. D., Boyd, D., Friedler, S. A., Venkatasubramanian, S., & Vertesi, J. (2019, January). Fairness and abstraction in sociotechnical systems. In Proceedings of the Conference on Fairness, Accountability, and Transparency (pp. 59–68).
• Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., & Yang, D. (2024). Can large language models transform computational social science? Computational Linguistics, 50(1), 237–291.