Learning from Machines:

Differentiating US Presidential Campaigns with Attribution and Annotation

Musashi Jacobs-Harukawa (he/him)

University of Oxford

Presented 17 June 2022 at PaCSS

tl;dr

  • Objective: differentiate language of political actors
  • Method: feature attribution on DL language models
  • Advantage: model sequence-level differences
  • Innovation: validation and application
  • Application: 12 Presidential ad campaigns on FB

Task and Challenge

Broad Methodological View

  • Plain description versus descriptive inference
  • Characterization as a form of inference
  • Differentiation as a form of characterization
  • Micro: What makes this ad a Trump ad?
  • Macro: What are Trump ads like?

An Existing Approach

  • Fightin’ Words (Monroe, Colaresi, and Quinn 2008)
  • Example: US legislators abortion debate
    • Republican tokens: kill, babi, mother
    • Democrat tokens: woman, decis, viabil
  • Can infer framing strategy (Chong and Druckman 2007)
  • What about doctor or procedur?

Two Common Steps

  • Discard order
    • Term-Document Matrix: Wordfish, topic models, etc.
    • Local context window: Word2Vec
    • Term-Co-occurrence Matrix: GloVe
  • Output numeric summaries
    • Token weights
    • Document weights

Linking Estimands

  • Everything is Lexical Semantics
  • Theoretical-empirical link via meaning of words
    • Substantive interpretation: word
    • Prevalence/existence: number/statistic

Challenges

  • Validation
    • Numeric reduction removes context
    • Word sense resolution challenging
  • Limits
    • Other linguistic phenomena exist
    • How to model linguistic phenomena produced by order?

Alternative

The Pitch

  • Train BERT(-like) to classify ads by campaign
  • Ask BERT what the differentiating features are
  • Annotate source texts to highlight differentiating language

What’s a BERT? (Devlin et al. 2019)

  • Family of transformer-based deep neural network sequence encoders
  • Deep Learning: complex multilayered neural network
  • Sequence Encoder: text → sequence of vectors

Why ask a BERT?

  • Excel at generating task-specific embeddings with minimal training
  • Transfer Learning: train on large corpus, fine-tune on small corpus
  • Linguistic Knowledge: syntax (Lin, Tan, and Frank 2019), semantics (Ettinger 2020)
  • Superhuman Accuracy: (Linardatos, Papastefanopoulos, and Kotsiantis 2021; Storks and Chai 2021)

How to ask a BERT?

  • Black Box:
    • Large number of parameters (BERT-base has 110 million)
    • Intermediary weights have no straightforward interpretation
  • Feature Attribution:
    • Ask BERT which parts of the document inform its predictions
    • Use feature attribution to get per-token scores

Feature Attribution

Integrated Gradients (Sundararajan, Taly, and Yan 2017)

  • DL analog of decomposing OLS prediction into partial derivatives wrt inputs
  • +ve IG indicates feature “contributes” to +ve class
  • -ve IG indicates feature “contributes” to -ve class
  • 0 IG indicates no contribution
  • Relative magnitudes matter, absolute hard to interpret

How do we know this works?

We care about validation!

  • Statistical: uncertainty measure of IG score
  • Substantive: how can we be know high-IG spans matter?
    • Compare to existing methods
    • Novel validation strategy

Confidence Intervals

Monte Carlo Dropout (Gal and Ghahramani 2016)

  • Most DL architectures include random dropout layers
  • Can be used as computationally efficient bootstrap
  • I generate 30 draws, then bootstrap over these

Validation by Sufficiency

Based on Jain et al. (2020) (FRESH)

  1. Train text classifier
  2. Use IG to extract high-salience regions (rationales)
  3. Train two further models:
    • One only sees rationales
    • Other only sees text-sans-rationales

Strategy Summarized

Model Trained Model Accuracy Strategy Precision Strategy Recall
rationale high high –
text-sans-rationale low – high

How do we use these scores?

  • Tired: Token Weights
    • Can be aggregated
    • Can be compared with established tools
  • Wired: Document Annotation
    • Within-sample characterization
    • Highlight differentiating regions of texts
  • Inspired: Hybrid Qualitative-Quantitative Approach
    • Prepare texts with annotations to guide analysis.

Challenge: “Artefacts”

  • If model accurate, highlighting patterns actually differentiating. But!
  • Model has no sense of “useful” differentiation
    • may just be “artefacts”
  • How much to trust the machine?
    • Solution requires researcher judgment

Application

Data

  • ProPublica Political Facebook Ads (2016-2020, US)
  • Twelve US presidential ad campaigns
  • Fuzzy duplicate removal using Levenshtein similarity

Three Comparisons

Comparsion Purpose N
Trump vs Biden Proof-of-concept, sense check 527
Sanders vs Warren Difficult binary 829
11 Democratic Primary Multiclass 3,165

Parameters

  • Non-destructive: No lowercasing or stopword removal
  • LO/FW pre-lemmatized, IG post-lemmatized for comparison
  • Tokenizer: RoBERTa byte-pair encoding subword tokenization
  • Learning rates \(3e-4\) to \(5e-5\)
  • All choices checked for implementation invariance

Results

Validation 1: Comparison

  • IG has weak correlation with LO and FW
  • Qualitative judgment of diverging results unclear
  • Hard to say what is a better “summary” for this task.

Validation 2: FRESH

  • FRESH strategy shows annotation are relevant information
  • CIs confirm scores sufficiently stable

Rationale Extraction

Output: Annotated Ads

Sanders vs Warren vs Rest

Next Steps

  • Software to make pipeline available to non-technical users
  • Methodology for hybrid AI-augmented qualitative text analysis
  • Further uses for model logic beyond annotation?

References

Chong, Dennis, and James N Druckman. 2007. “Framing public opinion in competitive democracies.” American Political Science Review 101 (4): 637–55.
Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.” In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171–86. Minneapolis, Minnesota: Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1423.
Ettinger, Allyson. 2020. “What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models.” Transactions of the Association for Computational Linguistics 8: 34–48.
Gal, Yarin, and Zoubin Ghahramani. 2016. “Dropout as a bayesian approximation: Representing model uncertainty in deep learning.” In international conference on machine learning, 1050–59. PMLR.
Jain, Sarthak, Sarah Wiegreffe, Yuval Pinter, and Byron C. Wallace. 2020. “Learning to Faithfully Rationalize by Construction.” In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 4459–73. Online: Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.acl-main.409.
Lin, Yongjie, Yi Chern Tan, and Robert Frank. 2019. “Open Sesame: Getting inside BERT’s Linguistic Knowledge.” In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 241–53. Florence, Italy: Association for Computational Linguistics. https://doi.org/10.18653/v1/W19-4825.
Linardatos, Pantelis, Vasilis Papastefanopoulos, and Sotiris Kotsiantis. 2021. “Explainable AI: A Review of Machine Learning Interpretability Methods.” Entropy 23 (1). https://doi.org/10.3390/e23010018.
Monroe, Burt L, Michael P Colaresi, and Kevin M Quinn. 2008. “Fightin’words: Lexical feature selection and evaluation for identifying the content of political conflict.” Political Analysis 16 (4): 372–403.
Storks, Shane, and Joyce Chai. 2021. “Beyond the Tip of the Iceberg: Assessing Coherence of Text Classifiers.” In Findings of the Association for Computational Linguistics: EMNLP 2021, 3169–77. Punta Cana, Dominican Republic: Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.findings-emnlp.272.
Sundararajan, Mukund, Ankur Taly, and Qiqi Yan. 2017. “Axiomatic attribution for deep networks.” In International conference on machine learning, 3319–28. PMLR.