--- title: "Discriminating Gender on Twitter — Burger 2011" description: "The paper addresses automatically inferring the gender of Twitter users." authors: ["John D. Burger", "John Henderson", "George Kim", "Guido Zarrella"] year: 2011 venue: "Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing" source: "https://aclanthology.org/D11-1120/" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/burger-2011-discriminating-gender-twitter/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/burger-2011-discriminating-gender-twitter/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2011](https://md.research.elamur.ai/free/en/papers/2011/) / Discriminating Gender on Twitter # Discriminating Gender on Twitter **Authors:** John D. Burger, John Henderson, George Kim, [Guido Zarrella](https://md.research.elamur.ai/free/en/authors/guido-zarrella/) · **Year:** 2011 · **Venue:** Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/) **Links:** [Publisher's page](https://aclanthology.org/D11-1120/) · [PDF](https://aclanthology.org/D11-1120.pdf) · [OpenAlex](https://openalex.org/W9292421) **Citations:** 542 citations (openalex-snapshot · 2026-10-14) · [89](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary The paper addresses automatically inferring the gender of Twitter users. Twitter profiles carry no demographic fields, and hand-labeled datasets had been small and English-only. The authors used distant supervision to build a large multilingual labeled set. Many Twitter users link from their profile URL to blog platforms whose profiles include a gender chosen from a menu, so the authors copied that label onto the matching Twitter account. This yielded about 184,000 labeled users. They represent each user with simple Boolean word and character n-gram features drawn from tweets, screen name, full name and description, with no language-specific processing. They compared several classifier families, and Balanced Winnow performed best. Using all fields, the classifier reaches 92% accuracy, and tweet text alone gives about 76%. A large Amazon Mechanical Turk study showed that humans, even when aggregated, perform clearly worse than the classifier. The work matters because it shows that cheap, language-independent features combined with automatically harvested labels can beat people at demographic inference at scale. ## Key points - **Corpus and splits.** The source corpus holds about 213M tweets from 18.5M users, sampled since April 2009. The labeled subset is split by user into train (146,925 users), dev (18,380) and test (18,424). - **Label quality and data profile.** - Of 1,000 sampled users, 150 had an explicit gender cue in their description. All 136 of those who also had a blog gender agreed with it. - The data is roughly 55% female. - About 66.7% of training users write mainly in English, followed by Portuguese (14.4%) and Spanish (6.0%). - **Feature space.** There are about 15.6M distinct features in total. Features seen for fewer than three users are discarded, and count-valued features gave no gain over presence/absence. - **Main results.** The all-fields Winnow model scores 92% on dev. The tweet-text-only model scores about 75.5–76%, against a 54.9% all-female baseline. Accuracy also rises as more tweets per user are available. - **Human and self-training comparisons.** - Turkers scored 60.4% per response and 68.7% averaged per worker. - Aggregating workers gave 65.7% with majority vote and 67.3% with Dawid–Skene EM. Fewer than 5% of prolific workers beat the classifier. - Naive self-training lowered accuracy to 90.9%. ## Interesting ideas - **Harvest labels from linked profiles.** Following users' outbound links to structured profiles elsewhere can produce large labeled datasets cheaply. This could work for age, location or language in other social data. - **Cheap label validation.** Spot-checking harvested labels against explicit self-descriptions in a small random sample is an inexpensive sanity check for any distantly supervised dataset. - **Language-agnostic features.** Boolean character n-grams with no tokenization or morphology let one model handle many languages, including unsegmented ones. This is attractive for multilingual or low-resource settings, such as Ukrainian alongside Russian. - **Engineering for scale.** Mapping features to integer codewords once and merging all of a user's texts into a single vector greatly cut memory needs. This trick suits user-level classification over huge corpora. - **Fair human comparisons.** Comparing the model with each annotator on exactly the items that annotator saw prevents easy subsets from inflating human scores. Aggregating crowd labels with EM, rather than only majority vote, is a useful companion baseline. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) ## Cited by Cited by 89 papers of our library cite it (of the 140884 whose bibliographies we read); 5 of them have a page here, 84 not yet. - [Ethical Research Protocols for Social Media Health Research](https://md.research.elamur.ai/free/en/papers/benton-2017-ethical-research-protocols-social-media/) - [Toward Gender-Inclusive Coreference Resolution](https://md.research.elamur.ai/free/en/papers/cao-2020-gender-inclusive-coreference-resolution/) - [Toward Gender-Inclusive Coreference Resolution: An Analysis of Gender and Bias Throughout the Machine Learning Lifecycle*](https://md.research.elamur.ai/free/en/papers/cao-2021-gender-inclusive-coreference-resolution-analysis/) - [What to do about bad language on the internet](https://md.research.elamur.ai/free/en/papers/eisenstein-2013-what-do-about-bad-language/) - [Adversarial Removal of Demographic Attributes from Text Data](https://md.research.elamur.ai/free/en/papers/elazar-2018-adversarial-removal-demographic-attributes-text/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{burger2011discriminating, title = {{Discriminating Gender on Twitter}}, author = {Burger, John D. and Henderson, John and Kim, George and Zarrella, Guido}, booktitle = {Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing}, year = {2011}, url = {https://aclanthology.org/D11-1120/} } ``` - [← previous Design of Arabic Diacritical Marks](https://md.research.elamur.ai/free/en/papers/hssini-2011-design-arabic-diacritical-marks/) - [next → Domain Adaptation via Pseudo In-Domain Data Selection](https://md.research.elamur.ai/free/en/papers/axelrod-2011-domain-adaptation-pseudo-domain-data/) - [↑ back to papers of 2011](https://md.research.elamur.ai/free/en/papers/2011/) ## References 19 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Shlomo Argamon, Moshe Koppel, James W. Pennebaker, and Jonathan Schler. 2007. Mining the blogosphere: Age, gender, and the varieties of self-expression. First Monday, 12(9), September. [find](https://scholar.google.com/scholar?q=Shlomo+Argamon%2C+Moshe+Koppel%2C+James+W%2E+Pennebaker%2C+and+Jonathan+Schler%2E+2007%2E+Mining+the+blogosphere%3A+Age%2C+gender%2C+and+the+varieties+of+self%2Dexpression%2E+First+Monday%2C+12%289%29%2C+September%2E) 2. John D. Burger and John C. Henderson. 2006. An exploration of observable features related to blogger age. In Computational Approaches to Analyzing Weblogs: Papers from the 2006 AAAI Spring Symposium. AAAI Press. [find](https://scholar.google.com/scholar?q=John+D%2E+Burger+and+John+C%2E+Henderson%2E+2006%2E+An+exploration+of+observable+features+related+to+blogger+age%2E+In+Computational+Approaches+to+Analyzing+Weblogs%3A+Papers+from+the+2006+AAAI+Spring+Symposium%2E+) 3. Chris Callison-Burch and Mark Dredze. 2010. Creating speech and language data with Amazon’s Mechanical Turk. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, CSLDAMT ’10. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chris+Callison%2DBurch+and+Mark+Dredze%2E+2010%2E+Creating+speech+and+language+data+with+Amazon%E2%80%99s+Mechanical+Turk%2E+In+Proceedings+of+the+NAACL+HLT+2010+Workshop+on+Creating+Speech+and+Language+Data+with+Ama) 4. Chih-Chung Chang and Chih-Jen Lin, 2001. LIBSVM: a library for support vector machines. Software available at [http://www.csie.ntu.edu.tw/](http://www.csie.ntu.edu.tw/) ∼ cjlin/libsvm. [↗](http://www.csie.ntu.edu.tw/) 5. A.P. Dawid and A.M. Skene. 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Journal of the Royal Statistical Society. Series C (Applied Statistics), 28(1). [find](https://scholar.google.com/scholar?q=A%2EP%2E+Dawid+and+A%2EM%2E+Skene%2E+1979%2E+Maximum+likelihood+estimation+of+observer+error%2Drates+using+the+EM+algorithm%2E+Journal+of+the+Royal+Statistical+Society%2E+Series+C+%28Applied+Statistics%29%2C+28%281%29%2E) 6. Jacob Eisenstein, Brendan O’Connor, Noah A. Smith, and Eric P. Xing. 2010. A latent variable model for geographic lexical variation. In Conference on Empirical Methods on Natural Language Processing. [find](https://scholar.google.com/scholar?q=Jacob+Eisenstein%2C+Brendan+O%E2%80%99Connor%2C+Noah+A%2E+Smith%2C+and+Eric+P%2E+Xing%2E+2010%2E+A+latent+variable+model+for+geographic+lexical+variation%2E+In+Conference+on+Empirical+Methods+on+Natural+Language+Processing%2E) 7. Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H. Witten. 2009. The WEKA data mining software: An update. SIGKDD Explorations, 11(1). [find](https://scholar.google.com/scholar?q=Mark+Hall%2C+Eibe+Frank%2C+Geoffrey+Holmes%2C+Bernhard+Pfahringer%2C+Peter+Reutemann%2C+and+Ian+H%2E+Witten%2E+2009%2E+The+WEKA+data+mining+software%3A+An+update%2E+SIGKDD+Explorations%2C+11%281%29%2E) 8. Bill Heil and Mikolaj Jan Piskorski. 2009. New Twitter research: Men follow men and nobody tweets. Harvard Business Review, June 1. [find](https://scholar.google.com/scholar?q=Bill+Heil+and+Mikolaj+Jan+Piskorski%2E+2009%2E+New+Twitter+research%3A+Men+follow+men+and+nobody+tweets%2E+Harvard+Business+Review%2C+June+1%2E) 9. Susan C. Herring, Inna Kouper, Lois Ann Scheidt, and Elijah L. Wright. 2004. Women and children last: The discursive construction of weblogs. In L. Gurak, S. Antonijevic, L. Johnson, C. Ratliff, and J. Reyman, editors, Into the Blogosphere: Rhetoric, Community, and Culture of Weblogs. [http://blog.li…](http://blog.li…) [↗](http://blog.li…) 10. David Huffaker. 2004. Gender similarities and differences in online identity and language use among teenage bloggers. Master’s thesis, Georgetown University. [http://cct.georgetown.edu/thesis/DavidHuffaker.pdf](http://cct.georgetown.edu/thesis/DavidHuffaker.pdf). [↗](http://cct.georgetown.edu/thesis/DavidHuffaker.pdf) 11. Panagiotis G. Ipeirotis, Foster Provost, and Jing Wang. 2010. Quality management on Amazon Mechanical Turk. In Proceedings of the Second Human Computation Workshop (KDD-HCOMP 2010). [find](https://scholar.google.com/scholar?q=Panagiotis+G%2E+Ipeirotis%2C+Foster+Provost%2C+and+Jing+Wang%2E+2010%2E+Quality+management+on+Amazon+Mechanical+Turk%2E+In+Proceedings+of+the+Second+Human+Computation+Workshop+%28KDD%2DHCOMP+2010%29%2E) 12. Nick Littlestone. 1988. Learning quickly when irrelevant attributes abound: A new linear-threshold algorithm. Machine Learning, 2, April. [find](https://scholar.google.com/scholar?q=Nick+Littlestone%2E+1988%2E+Learning+quickly+when+irrelevant+attributes+abound%3A+A+new+linear%2Dthreshold+algorithm%2E+Machine+Learning%2C+2%2C+April%2E) 13. Andrew Kachites McCallum. 2002. MALLET: A machine learning for language toolkit. [http://mallet.cs.umass.edu](http://mallet.cs.umass.edu). [↗](http://mallet.cs.umass.edu) 14. Claire Cain Miller. 2010. Why Twitter’s C.E.O. demoted himself. New York Times, October 30. [http://www.nytimes.com/2010/10/31/technology/31ev.html](http://www.nytimes.com/2010/10/31/technology/31ev.html). [↗](http://www.nytimes.com/2010/10/31/technology/31ev.html) 15. Arjun Mukherjee and Bing Liu. 2010. Improving gender classification of blog authors. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, Cambridge, MA, October. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Arjun+Mukherjee+and+Bing+Liu%2E+2010%2E+Improving+gender+classification+of+blog+authors%2E+In+Proceedings+of+the+2010+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+Cambridge%2C+MA%2C+October%2E+) 16. Sasa Petrovic, Miles Osborne, and Victor Lavrenko. 2010. The Edinburgh Twitter corpus. In Computational Linguistics in a World of Social Media. AAAI Press. Workshop at NAACL. [find](https://scholar.google.com/scholar?q=Sasa+Petrovic%2C+Miles+Osborne%2C+and+Victor+Lavrenko%2E+2010%2E+The+Edinburgh+Twitter+corpus%2E+In+Computational+Linguistics+in+a+World+of+Social+Media%2E+AAAI+Press%2E+Workshop+at+NAACL%2E) 17. Delip Rao, David Yarowsky, Abhishek Shreevats, and Manaswi Gupta. 2010. Classifying latent user attributes in Twitter. In 2nd International Workshop on Search and Mining User-Generated Content. ACM. [find](https://scholar.google.com/scholar?q=Delip+Rao%2C+David+Yarowsky%2C+Abhishek+Shreevats%2C+and+Manaswi+Gupta%2E+2010%2E+Classifying+latent+user+attributes+in+Twitter%2E+In+2nd+International+Workshop+on+Search+and+Mining+User%2DGenerated+Content%2E+ACM%2E) 18. Jonathan Schler, Moshe Koppel, Shlomo Argamon, and James Pennebaker. 2006. Effects of age and gender on blogging. In Computational Approaches to Analyzing Weblogs: Papers from the 2006 AAAI Spring Symposium. AAAI Press, March. [find](https://scholar.google.com/scholar?q=Jonathan+Schler%2C+Moshe+Koppel%2C+Shlomo+Argamon%2C+and+James+Pennebaker%2E+2006%2E+Effects+of+age+and+gender+on+blogging%2E+In+Computational+Approaches+to+Analyzing+Weblogs%3A+Papers+from+the+2006+AAAI+Spring+Sym) 19. Robin Wauters. 2010. Only 50% of Twitter messages are in English, study says. TechCrunch, February 1. [http://techcrunch.com/2010/02/24/twitter-languages/](http://techcrunch.com/2010/02/24/twitter-languages/). [↗](http://techcrunch.com/2010/02/24/twitter-languages/)