Lit Review 1: Data Sets & Cleaning
- Constrained Decoding for Neural NLG from Compositional Representations in Task-Oriented DialogueIn Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 831–844, 2019.
Generating fluent natural language responses from structured semantic representations is a critical step in task-oriented conversational systems. Avenues like the E2E NLG Challenge have encouraged the development of neural approaches, particularly sequence-to-sequence (Seq2Seq) models for this problem. The semantic representations used, however, are often underspecified, which places a higher burden on the generation model for sentence planning, and also limits the extent to which generated responses can be controlled in a live system. In this paper, we (1) propose using tree-structured semantic representations, like those used in traditional rule-based NLG systems, for better discourse-level structuring and sentence-level planning; (2) introduce a challenging dataset using this representation for the weather domain; (3) introduce a constrained decoding approach for Seq2Seq models that leverages this representation to improve semantic correctness; and (4) demonstrate promising results on our dataset and the E2E dataset.
@inproceedings{balakrishnan-etal-2019-constrained, author = {Balakrishnan, Anusha and Rao, Jinfeng and Upasani, Kartikeya and White, Michael and Subba, Rajen}, title = {Constrained Decoding for Neural {NLG} from Compositional Representations in Task-Oriented Dialogue}, booktitle = {Proceedings of the 57th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics}, year = {2019}, address = {Florence, Italy}, doi = {10.18653/v1/P19-1080}, editor = {Korhonen, Anna and Traum, David and M{\`a}rquez, Llu{\'i}s}, month = {July}, pages = {831--844}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/P19-1080/} } - ArgRewrite V.2: an annotated argumentative revisions corpusLanguage Resources and Evaluation, 56(3), pp. 881–915, 2022.
Analyzing how humans revise their writings is an interesting research question, not only from an educational perspective but also in terms of artificial intelligence. Better understanding of this process could facilitate many NLP applications, from intelligent tutoring systems to supportive and collaborative writing environments. Developing these applications, however, requires revision corpora, which are not widely available. In this work, we present ArgRewrite V.2, a corpus of annotated argumentative revisions, collected from two cycles of revisions to argumentative essays about self-driving cars. Annotations are provided at different levels of purpose granularity (coarse and fine) and scope (sentential and subsentential). In addition, the corpus includes the revision goal given to each writer, essay scores, annotation verification, pre- and post-study surveys collected from participants as meta-data. The variety of revision unit scope and purpose granularity levels in ArgRewrite, along with the inclusion of new types of meta-data, can make it a useful resource for research and applications that involve revision analysis. We demonstrate some potential applications of ArgRewrite V.2 in the development of automatic revision purpose predictors, as a training source and benchmark.
@article{kashefi-etal-2022-argrewrite, author = {Kashefi, Omid and Afrin, Tazin and Dale, Meghan and Olshefski, Christopher and Godley, Amanda and Litman, Diane and Hwa, Rebecca}, title = {{A}rg{R}ewrite V.2: an annotated argumentative revisions corpus}, journal = {Language Resources and Evaluation}, year = {2022}, archiveprefix = {arXiv}, doi = {10.1007/s10579-021-09567-z}, eprint = {2206.01677}, number = {3}, pages = {881--915}, publisher = {Springer}, url = {https://link.springer.com/article/10.1007/s10579-021-09567-z}, volume = {56} } - Adding Chinese Captions to ImagesIn Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval (ICMR), pp. 271–275, 2016.
@inproceedings{li-etal-2016-adding, author = {Li, Xirong and Lan, Weiyu and Dong, Jianfeng and Liu, Hailong}, title = {Adding {C}hinese Captions to Images}, booktitle = {Proceedings of the 2016 {ACM} on International Conference on Multimedia Retrieval ({ICMR})}, year = {2016}, address = {New York, New York, USA}, doi = {10.1145/2911996.2912049}, pages = {271--275}, publisher = {Association for Computing Machinery}, url = {https://dl.acm.org/doi/10.1145/2911996.2912049} } - The Teams Corpus and Entrainment in Multi-Party Spoken DialoguesIn Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1421–1431, 2016.
When interacting individuals entrain, they begin to speak more like each other. To support research on entrainment in cooperative multi-party dialogues, we have created a corpus where teams of three or four speakers play two rounds of a cooperative board game. We describe the experimental design and technical infrastructure used to collect our corpus, which consists of audio, video, transcriptions, and questionnaire data for 63 teams (47 hours of audio). We illustrate the use of our corpus as a novel resource for studying team entrainment by 1) developing and evaluating team-level acoustic-prosodic entrainment measures that extend existing dyad measures, and 2) investigating relationships between team entrainment and participation dominance.
@inproceedings{litman-etal-2016-teams, author = {Litman, Diane and Paletz, Susannah and Rahimi, Zahra and Allegretti, Stefani and Rice, Caitlin}, title = {The Teams Corpus and Entrainment in Multi-Party Spoken Dialogues}, booktitle = {Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing}, year = {2016}, address = {Austin, Texas}, doi = {10.18653/v1/D16-1149}, editor = {Su, Jian and Duh, Kevin and Carreras, Xavier}, month = {November}, pages = {1421--1431}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/D16-1149/} } - Cree Corpus: A Collection of nêhiyawêwin ResourcesIn Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6354–6364, 2022.
Plains Cree (nêhiyawêwin) is an Indigenous language that is spoken in Canada and the USA. It is the most widely spoken dialect of Cree and a morphologically complex language that is polysynthetic, highly inflective, and agglutinative. It is an extremely low resource language, with no existing corpus that is both available and prepared for supporting the development of language technologies. To support nêhiyawêwin revitalization and preservation, we developed a corpus covering diverse genres, time periods, and texts for a variety of intended audiences. The data has been verified and cleaned; it is ready for use in developing language technologies for nêhiyawêwin. The corpus includes the corresponding English phrases or audio files where available. We demonstrate the utility of the corpus through its community use and its use to build language technologies that can provide the types of support that community members have expressed are desirable. The corpus is available for public use.
@inproceedings{teodorescu-etal-2022-cree, author = {Teodorescu, Daniela and Matalski, Josie and Lothian, Delaney and Barbosa, Denilson and Demmans Epp, Carrie}, title = {{C}ree Corpus: A Collection of n{\^e}hiyaw{\^e}win Resources}, booktitle = {Proceedings of the 60th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)}, year = {2022}, address = {Dublin, Ireland}, doi = {10.18653/v1/2022.acl-long.440}, editor = {Muresan, Smaranda and Nakov, Preslav and Villavicencio, Aline}, month = {May}, pages = {6354--6364}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2022.acl-long.440/} } - Negative language transfer in learner English: A new datasetIn Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3129–3142, 2021.
Automatic personalized corrective feedback can help language learners from different backgrounds better acquire a new language. This paper introduces a learner English dataset in which learner errors are accompanied by information about possible error sources. This dataset contains manually annotated error causes for learner writing errors. These causes tie learner mistakes to structures from their first languages, when the rules in English and in the first language diverge. This new dataset will enable second language acquisition researchers to computationally analyze a large quantity of learner errors that are related to language transfer from the learners’ first language. The dataset can also be applied in personalizing grammatical error correction systems according to the learners’ first language and in providing feedback that is informed by the cause of an error.
@inproceedings{farias-wanderley-etal-2021-negative, author = {Farias Wanderley, Leticia and Zhao, Nicole and Demmans Epp, Carrie}, title = {Negative language transfer in learner {E}nglish: A new dataset}, booktitle = {Proceedings of the 2021 Conference of the {N}orth {A}merican Chapter of the {A}ssociation for {C}omputational {L}inguistics: Human Language Technologies}, year = {2021}, address = {Online}, doi = {10.18653/v1/2021.naacl-main.251}, editor = {Toutanova, Kristina and Rumshisky, Anna and Zettlemoyer, Luke and Hakkani-Tur, Dilek and Beltagy, Iz and Bethard, Steven and Cotterell, Ryan and Chakraborty, Tanmoy and Zhou, Yichao}, month = {June}, pages = {3129--3142}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2021.naacl-main.251/} } - Describing a Knowledge BaseIn Proceedings of the 11th International Conference on Natural Language Generation, pp. 10–21, 2018.
We aim to automatically generate natural language descriptions about an input structured knowledge base (KB). We build our generation framework based on a pointer network which can copy facts from the input KB, and add two attention mechanisms: (i) slot-aware attention to capture the association between a slot type and its corresponding slot value; and (ii) a new table position self-attention to capture the inter-dependencies among related slots. For evaluation, besides standard metrics including BLEU, METEOR, and ROUGE, we propose a KB reconstruction based metric by extracting a KB from the generation output and comparing it with the input KB. We also create a new data set which includes 106,216 pairs of structured KBs and their corresponding natural language descriptions for two distinct entity types. Experiments show that our approach significantly outperforms state-of-the-art methods. The reconstructed KB achieves 68.8% - 72.6% F-score.
@inproceedings{wang-etal-2018-describing, author = {Wang, Qingyun and Pan, Xiaoman and Huang, Lifu and Zhang, Boliang and Jiang, Zhiying and Ji, Heng and Knight, Kevin}, title = {Describing a Knowledge Base}, booktitle = {Proceedings of the 11th International Conference on Natural Language Generation}, year = {2018}, address = {Tilburg University, The Netherlands}, doi = {10.18653/v1/W18-6502}, editor = {Krahmer, Emiel and Gatt, Albert and Goudbeek, Martijn}, month = {November}, pages = {10--21}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/W18-6502/} } - Decay No More: A Persistent Twitter Dataset for Learning Social MeaningIn Workshop Proceedings of the 16th International AAAI Conference on Web and Social Media (ICWSM), 2022.
With the proliferation of social media, many studies resort to social media to construct datasets for developing social meaning understanding systems. For the popular case of Twitter, most researchers distribute tweet IDs without the actual text contents due to the data distribution policy of the platform. One issue is that the posts become increasingly inaccessible over time, which leads to unfair comparisons and a temporal bias in social media research. To alleviate this challenge of data decay, we leverage a paraphrase model to propose a new persistent English Twitter dataset for social meaning (PTSM). PTSM consists of 17 social meaning datasets in 10 categories of tasks. We experiment with two SOTA pre-trained language models and show that our PTSM can substitute the actual tweets with paraphrases with marginal performance loss.
@inproceedings{zhang-etal-2022-decay, author = {Zhang, Chiyu and Abdul-Mageed, Muhammad and Nagoudi, El Moatez Billah}, title = {Decay No More: A Persistent {T}witter Dataset for Learning Social Meaning}, booktitle = {Workshop Proceedings of the 16th International {AAAI} Conference on Web and Social Media ({ICWSM})}, year = {2022}, doi = {10.36190/2022.92}, url = {https://workshop-proceedings.icwsm.org/abstract?id=2022_92} } - Talrómur: A large Icelandic TTS corpusIn Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa), pp. 440–444, 2021.
We present Talrómur, a large high-quality Text-To-Speech (TTS) corpus for the Icelandic language. This multi-speaker corpus contains recordings from 4 male speakers and 4 female speakers of a wide range in age and speaking style. The corpus consists of 122,417 single utterance recordings equating to approximately 213 hours of voice data. All speakers read from the same script which has a high coverage of possible Icelandic diphones. Manual analysis of 15,956 utterances indicates that the corpus has a reading mistake rate no higher than 0.25%. We additionally present results from subjective evaluations of the different voices with regards to intelligibility, likeability and trustworthiness.
@inproceedings{sigurgeirsson-etal-2021-talromur, author = {Sigurgeirsson, Atli and Gunnarsson, {\TH}orsteinn and {\"O}rn{\'o}lfsson, Gunnar and Magn{\'u}sd{\'o}ttir, Eyd{\'i}s and {\TH}{\'o}rhallsd{\'o}ttir, Ragnhei{\dh}ur and J{\'o}nsson, Stef{\'a}n and Gu{\dh}nason, J{\'o}n}, title = {{T}alr{\'o}mur: A large {I}celandic {TTS} corpus}, booktitle = {Proceedings of the 23rd {N}ordic Conference on Computational Linguistics ({N}o{D}a{L}i{D}a)}, year = {2021}, address = {Reykjavik, Iceland (Online)}, editor = {Dobnik, Simon and {\O}vrelid, Lilja}, month = {May 31--2 June}, pages = {440--444}, publisher = {Link{\"o}ping University Electronic Press, Sweden}, url = {https://aclanthology.org/2021.nodalida-main.50/} } - The Little Prince in 26 Languages: Towards a Multilingual Neuro-Cognitive CorpusIn Proceedings of the Second Workshop on Linguistic and Neurocognitive Resources, pp. 43–49, 2020.
We present the Le Petit Prince Corpus (LPPC), a multi-lingual resource for research in (computational) psycho- and neurolinguistics. The corpus consists of the children’s story The Little Prince in 26 languages. The dataset is in the process of being built using state-of-the-art methods for speech and language processing and electroencephalography (EEG). The planned release of LPPC dataset will include raw text annotated with dependency graphs in the Universal Dependencies standard, a near-natural-sounding synthetic spoken subset as well as EEG recordings. We will use this corpus for conducting neurolinguistic studies that generalize across a wide range of languages, overcoming typological constraints to traditional approaches. The planned release of the LPPC combines linguistic and EEG data for many languages using fully automatic methods, and thus constitutes a readily extendable resource that supports cross-linguistic and cross-disciplinary research.
@inproceedings{stehwien-etal-2020-little, author = {Stehwien, Sabrina and Henke, Lena and Hale, John and Brennan, Jonathan and Meyer, Lars}, title = {The Little Prince in 26 Languages: Towards a Multilingual Neuro-Cognitive Corpus}, booktitle = {Proceedings of the Second Workshop on Linguistic and Neurocognitive Resources}, year = {2020}, address = {Marseille, France}, editor = {Chersoni, Emmanuele and Devereux, Barry and Huang, Chu-Ren}, isbn = {979-10-95546-52-8}, language = {eng}, month = {May}, pages = {43--49}, publisher = {European Language Resources Association}, url = {https://aclanthology.org/2020.lincr-1.6/} } - Criteria for Useful Automatic Romanization in South Asian LanguagesIn Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 6662–6673, 2022.
This paper presents a number of possible criteria for systems that transliterate South Asian languages from their native scripts into the Latin script, a process known as romanization. These criteria are related to either fidelity to human linguistic behavior (pronunciation transparency, naturalness and conventionality) or processing utility for people (ease of input) as well as under-the-hood in systems (invertibility and stability across languages and scripts). When addressing these differing criteria several linguistic considerations, such as modeling of prominent phonological processes and their relation to orthography, need to be taken into account. We discuss these key linguistic details in the context of Brahmic scripts and languages that use them, such as Hindi and Malayalam. We then present the core features of several romanization algorithms, implemented in a finite state transducer (FST) formalism, that address differing criteria. Implementations of these algorithms have been released as part of the Nisaba finite-state script processing library.
@inproceedings{demirsahin-etal-2022-criteria, author = {Demirsahin, Isin and Johny, Cibu and Gutkin, Alexander and Roark, Brian}, title = {Criteria for Useful Automatic {R}omanization in {S}outh {A}sian Languages}, booktitle = {Proceedings of the Thirteenth Language Resources and Evaluation Conference}, year = {2022}, address = {Marseille, France}, editor = {Calzolari, Nicoletta and B{\'e}chet, Fr{\'e}d{\'e}ric and Blache, Philippe and Choukri, Khalid and Cieri, Christopher and Declerck, Thierry and Goggi, Sara and Isahara, Hitoshi and Maegaard, Bente and Mariani, Joseph and Mazo, H{\'e}l{\`e}ne and Odijk, Jan and Piperidis, Stelios}, month = {June}, pages = {6662--6673}, publisher = {European Language Resources Association}, url = {https://aclanthology.org/2022.lrec-1.718/} }
Lit Review 2: Grammars
- Recurrent Neural Network GrammarsIn Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 199–209, 2016.
We introduce recurrent neural network grammars, probabilistic models of sentences with explicit phrase structure. We explain efficient inference procedures that allow application to both parsing and language modeling. Experiments show that they provide better parsing in English than any single previously published supervised generative model and better language modeling than state-of-the-art sequential RNNs in English and Chinese.
@inproceedings{dyer-etal-2016-recurrent, author = {Dyer, Chris and Kuncoro, Adhiguna and Ballesteros, Miguel and Smith, Noah A.}, title = {Recurrent Neural Network Grammars}, booktitle = {Proceedings of the 2016 Conference of the {N}orth {A}merican Chapter of the {A}ssociation for {C}omputational {L}inguistics: Human Language Technologies}, year = {2016}, address = {San Diego, California}, doi = {10.18653/v1/N16-1024}, editor = {Knight, Kevin and Nenkova, Ani and Rambow, Owen}, month = {June}, pages = {199--209}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/N16-1024/} } - Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at ScaleTransactions of the Association for Computational Linguistics, 10, pp. 1423–1439, 2022.
We introduce Transformer Grammars (TGs), a novel class of Transformer language models that combine (i) the expressive power, scalability, and strong performance of Transformers and (ii) recursive syntactic compositions, which here are implemented through a special attention mask and deterministic transformation of the linearized tree. We find that TGs outperform various strong baselines on sentence-level language modeling perplexity, as well as on multiple syntax-sensitive language modeling evaluation metrics. Additionally, we find that the recursive syntactic composition bottleneck which represents each sentence as a single vector harms perplexity on document-level language modeling, providing evidence that a different kind of memory mechanism—one that is independent of composed syntactic representations—plays an important role in current successful models of long text.
@article{sartran-etal-2022-transformer, author = {Sartran, Laurent and Barrett, Samuel and Kuncoro, Adhiguna and Stanojevi{\'c}, Milo{\v{s}} and Blunsom, Phil and Dyer, Chris}, title = {Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale}, journal = {Transactions of the Association for Computational Linguistics}, year = {2022}, address = {Cambridge, MA}, doi = {10.1162/tacl_a_00526}, editor = {Roark, Brian and Nenkova, Ani}, pages = {1423--1439}, publisher = {MIT Press}, url = {https://aclanthology.org/2022.tacl-1.81/}, volume = {10} } - Automatic Learning of Context-Free GrammarIn Proceedings of the 18th Conference on Computational Linguistics and Speech Processing, pp. 53–62, 2006.
In this paper we study the problem of learning context-free grammar from a corpus. We investigate a technique that is based on the notion of minimum description length of the corpus. A cost as a function of grammar is defined as the sum of the number of bits required for the representation of a grammar and the number of bits required for the derivation of the corpus using that grammar. On the Academia Sinica Balanced Corpus with part-of-speech tags, the overall cost, or description length, reduces by as much as 14% compared to the initial cost. In addition to the presentation of the experimental results, we also include a novel analysis on the costs of two special context-free grammars, where one derives only the set of strings in the corpus and the other derives the set of arbitrary strings from the alphabet.
@inproceedings{chen-etal-2006-automatic, author = {Chen, Tai-Hung and Tseng, Chun-Han and Chen, Chia-Ping}, title = {Automatic Learning of Context-Free Grammar}, booktitle = {Proceedings of the 18th Conference on Computational Linguistics and Speech Processing}, year = {2006}, address = {Hsinchu, Taiwan}, editor = {Wang, Yih-Ru and Gao, Zaho-Ming}, month = {September}, pages = {53--62}, publisher = {The Association for Computational Linguistics and Chinese Language Processing (ACLCLP)}, url = {https://aclanthology.org/O06-1004/} } - Multilingual Grammar Induction with Continuous Language IdentificationIn Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 5728–5733, 2019.
The key to multilingual grammar induction is to couple grammar parameters of different languages together by exploiting the similarity between languages. Previous work relies on linguistic phylogenetic knowledge to specify similarity between languages. In this work, we propose a novel universal grammar induction approach that represents language identities with continuous vectors and employs a neural network to predict grammar parameters based on the representation. Without any prior linguistic phylogenetic knowledge, we automatically capture similarity between languages with the vector representations and softly tie the grammar parameters of different languages. In our experiments, we apply our approach to 15 languages across 8 language families and subfamilies in the Universal Dependency Treebank dataset, and we observe substantial performance gain on average over monolingual and multilingual baselines.
@inproceedings{han-etal-2019-multilingual, author = {Han, Wenjuan and Wang, Ge and Jiang, Yong and Tu, Kewei}, title = {Multilingual Grammar Induction with Continuous Language Identification}, booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing ({EMNLP}-{IJCNLP})}, year = {2019}, address = {Hong Kong, China}, doi = {10.18653/v1/D19-1576}, editor = {Inui, Kentaro and Jiang, Jing and Ng, Vincent and Wan, Xiaojun}, month = {November}, pages = {5728--5733}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/D19-1576/} } - An Open Source Persian Computational GrammarIn Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12), pp. 1686–1693, 2012.
In this paper, we describe a multilingual open-source computational grammar of Persian, developed in Grammatical Framework (GF) ― A type-theoretical grammar formalism. We discuss in detail the structure of different syntactic (i.e. noun phrases, verb phrases, adjectival phrases, etc.) categories of Persian. First, we show how to structure and construct these categories individually. Then we describe how they are glued together to make well-formed sentences in Persian, while maintaining the grammatical features such as agreement, word order, etc. We also show how some of the distinctive features of Persian, such as the ezafe construction, are implemented in GF. In order to evaluate the grammar’s correctness, and to demonstrate its usefulness, we have added support for Persian in a multilingual application grammar (the Tourist Phrasebook) using the reported resource grammar.
@inproceedings{virk-abolahrar-2012-open, author = {Virk, Shafqat Mumtaz and Abolahrar, Elnaz}, title = {An Open Source {P}ersian Computational Grammar}, booktitle = {Proceedings of the Eighth International Conference on Language Resources and Evaluation ({LREC}'12)}, year = {2012}, address = {Istanbul, Turkey}, editor = {Calzolari, Nicoletta and Choukri, Khalid and Declerck, Thierry and Do{\u{g}}an, Mehmet U{\u{g}}ur and Maegaard, Bente and Mariani, Joseph and Moreno, Asuncion and Odijk, Jan and Piperidis, Stelios}, month = {May}, pages = {1686--1693}, publisher = {European Language Resources Association (ELRA)}, url = {https://aclanthology.org/L12-1614/} } - A Computational Grammar of GaIn Proceedings of the first workshop on Resources for African Indigenous Languages, pp. 36–44, 2020.
The paper describes aspects of an HPSG style computational grammar of the West African language Ga (a Kwa language spoken in the Accra area of Ghana). As a Volta Basin Kwa language, Ga features many types of multiverb expressions and other particular constructional patterns in the verbal and nominal domain. The paper highlights theoretical and formal features of the grammar motivated by these phenomena, some of them possibly innovative to the formal framework. As a so-called deep grammar of the language, it hosts a rich lexical structure, and we describe ways in which the grammar builds on previously available lexical resources. We outline an environment of current resources in which the grammar is part, and lines of research and development in which it and its environment can be used.
@inproceedings{hellan-2020-computational, author = {Hellan, Lars}, title = {A Computational Grammar of {G}a}, booktitle = {Proceedings of the first workshop on Resources for {A}frican Indigenous Languages}, year = {2020}, address = {Marseille, France}, editor = {Mabuya, Rooweither and Ramukhadi, Phathutshedzo and Setaka, Mmasibidi and Wagner, Valencia and van Zaanen, Menno}, isbn = {979-10-95546-60-3}, language = {eng}, month = {May}, pages = {36--44}, publisher = {European Language Resources Association (ELRA)}, url = {https://aclanthology.org/2020.rail-1.7/} } - Identifying Grammar Rules for Language Education with Dependency Parsing in GermanIn Proceedings of the Fifth International Conference on Dependency Linguistics (Depling, SyntaxFest 2019), pp. 100–111, 2019.
We propose a method of determining the syntactic difficulty of a sentence, using syntactic patterns that identify grammatical rules on dependency parses. We have constructed a novel query language based on constraint-based dependency grammars and a grammar of German rules (relevant to primary school education) with patterns in our language. We annotated these rules with a difficulty score and grammatical prerequisites and built a matching algorithm that matches the dependency parse of a sentence in CoNLL-U format with its relevant syntactic patterns. We achieved 96% precision and 95% recall on a manually annotated set of sentences, and our best results on using parses from four parsers are 88% and 84% respectively
@inproceedings{metheniti-etal-2019-identifying, author = {Metheniti, Eleni and Park, Pomi and Kolesova, Kristina and Neumann, G{\"u}nter}, title = {Identifying Grammar Rules for Language Education with Dependency Parsing in {G}erman}, booktitle = {Proceedings of the Fifth International Conference on Dependency Linguistics (Depling, {S}yntax{F}est 2019)}, year = {2019}, address = {Paris, France}, doi = {10.18653/v1/W19-7712}, editor = {Gerdes, Kim and Kahane, Sylvain}, month = {August}, pages = {100--111}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/W19-7712/} } - Modeling Incremental Language Comprehension in the Brain with Combinatory Categorial GrammarIn Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pp. 23–38, 2021.
Hierarchical sentence structure plays a role in word-by-word human sentence comprehension, but it remains unclear how best to characterize this structure and unknown how exactly it would be recognized in a step-by-step process model. With a view towards sharpening this picture, we model the time course of hemodynamic activity within the brain during an extended episode of naturalistic language comprehension using Combinatory Categorial Grammar (CCG). CCG has well-defined incremental parsing algorithms, surface compositional semantics, and can explain long-range dependencies as well as complicated cases of coordination. We find that CCG-derived predictors improve a regression model of fMRI time course in six language-relevant brain regions, over and above predictors derived from context-free phrase structure. Adding a special Revealing operator to CCG parsing, one designed to handle right-adjunction, improves the fit in three of these regions. This evidence for CCG from neuroimaging bolsters the more general case for mildly context-sensitive grammars in the cognitive science of language.
@inproceedings{stanojevic-etal-2021-modeling, author = {Stanojevi{\'c}, Milo{\v{s}} and Bhattasali, Shohini and Dunagan, Donald and Campanelli, Luca and Steedman, Mark and Brennan, Jonathan and Hale, John}, title = {Modeling Incremental Language Comprehension in the Brain with {C}ombinatory {C}ategorial {G}rammar}, booktitle = {Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics}, year = {2021}, address = {Online}, doi = {10.18653/v1/2021.cmcl-1.3}, editor = {Chersoni, Emmanuele and Hollenstein, Nora and Jacobs, Cassandra and Oseki, Yohei and Pr{\'e}vot, Laurent and Santus, Enrico}, month = {June}, pages = {23--38}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2021.cmcl-1.3/} }
Lit Review 3: Language Models
- Show Some Love to Your n-grams: A Bit of Progress and Stronger n-gram Language Modeling BaselinesIn Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4113–4118, 2019.
In recent years neural language models (LMs) have set the state-of-the-art performance for several benchmarking datasets. While the reasons for their success and their computational demand are well-documented, a comparison between neural models and more recent developments in n-gram models is neglected. In this paper, we examine the recent progress in n-gram literature, running experiments on 50 languages covering all morphological language families. Experimental results illustrate that a simple extension of Modified Kneser-Ney outperforms an lstm language model on 42 languages while a word-level Bayesian n-gram LM (Shareghi et al., 2017) outperforms the character-aware neural model (Kim et al., 2016) on average across all languages, and its extension which explicitly injects linguistic knowledge (Gerz et al., 2018) on 8 languages. Further experiments on larger Europarl datasets for 3 languages indicate that neural architectures are able to outperform computationally much cheaper n-gram models: n-gram training is up to 15,000x quicker. Our experiments illustrate that standalone n-gram models lend themselves as natural choices for resource-lean or morphologically rich languages, while the recent progress has significantly improved their accuracy.
@inproceedings{shareghi-etal-2019-show, author = {Shareghi, Ehsan and Gerz, Daniela and Vuli{\'c}, Ivan and Korhonen, Anna}, title = {Show Some Love to Your n-grams: A Bit of Progress and Stronger n-gram Language Modeling Baselines}, booktitle = {Proceedings of the 2019 Conference of the {N}orth {A}merican Chapter of the {A}ssociation for {C}omputational {L}inguistics: Human Language Technologies, Volume 1 (Long and Short Papers)}, year = {2019}, address = {Minneapolis, Minnesota}, doi = {10.18653/v1/N19-1417}, editor = {Burstein, Jill and Doran, Christy and Solorio, Thamar}, month = {June}, pages = {4113--4118}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/N19-1417/} } - Implicit n-grams Induced by RecurrenceIn Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1624–1639, 2022.
Although self-attention based models such as Transformers have achieved remarkable successes on natural language processing (NLP)tasks, recent studies reveal that they have limitations on modeling sequential transformations (Hahn, 2020), which may promptre-examinations of recurrent neural networks (RNNs) that demonstrated impressive results on handling sequential data. Despite manyprior attempts to interpret RNNs, their internal mechanisms have not been fully understood, and the question on how exactly they capturesequential features remains largely unclear. In this work, we present a study that shows there actually exist some explainable componentsthat reside within the hidden states, which are reminiscent of the classical n-grams features. We evaluated such extracted explainable features from trained RNNs on downstream sentiment analysis tasks and found they could be used to model interesting linguistic phenomena such as negation and intensification. Furthermore, we examined the efficacy of using such n-gram components alone as encoders on tasks such as sentiment analysis and language modeling, revealing they could be playing important roles in contributing to the overall performance of RNNs. We hope our findings could add interpretability to RNN architectures, and also provide inspirations for proposing new architectures for sequential data.
@inproceedings{sun-lu-2022-implicit, author = {Sun, Xiaobing and Lu, Wei}, title = {Implicit n-grams Induced by Recurrence}, booktitle = {Proceedings of the 2022 Conference of the {N}orth {A}merican Chapter of the {A}ssociation for {C}omputational {L}inguistics: Human Language Technologies}, year = {2022}, address = {Seattle, United States}, doi = {10.18653/v1/2022.naacl-main.117}, editor = {Carpuat, Marine and de Marneffe, Marie-Catherine and Meza Ruiz, Ivan Vladimir}, month = {July}, pages = {1624--1639}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2022.naacl-main.117/} } - TimeLMs: Diachronic Language Models from TwitterIn Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 251–260, 2022.
Despite its importance, the time variable has been largely neglected in the NLP and language model literature. In this paper, we present TimeLMs, a set of language models specialized on diachronic Twitter data. We show that a continual learning strategy contributes to enhancing Twitter-based language models’ capacity to deal with future and out-of-distribution tweets, while making them competitive with standardized and more monolithic benchmarks. We also perform a number of qualitative analyses showing how they cope with trends and peaks in activity involving specific named entities or concept drift. TimeLMs is available at github.com/cardiffnlp/timelms.
@inproceedings{loureiro-etal-2022-timelms, author = {Loureiro, Daniel and Barbieri, Francesco and Neves, Leonardo and Espinosa Anke, Luis and Camacho-Collados, Jose}, title = {{T}ime{LM}s: Diachronic Language Models from {T}witter}, booktitle = {Proceedings of the 60th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics: System Demonstrations}, year = {2022}, address = {Dublin, Ireland}, doi = {10.18653/v1/2022.acl-demo.25}, editor = {Basile, Valerio and Kozareva, Zornitsa and Stajner, Sanja}, month = {May}, pages = {251--260}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2022.acl-demo.25/} } - Metaphors in Pre-Trained Language Models: Probing and Generalization Across Datasets and LanguagesIn Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2037–2050, 2022.
Human languages are full of metaphorical expressions. Metaphors help people understand the world by connecting new concepts and domains to more familiar ones. Large pre-trained language models (PLMs) are therefore assumed to encode metaphorical knowledge useful for NLP systems. In this paper, we investigate this hypothesis for PLMs, by probing metaphoricity information in their encodings, and by measuring the cross-lingual and cross-dataset generalization of this information. We present studies in multiple metaphor detection datasets and in four languages (i.e., English, Spanish, Russian, and Farsi). Our extensive experiments suggest that contextual representations in PLMs do encode metaphorical knowledge, and mostly in their middle layers. The knowledge is transferable between languages and datasets, especially when the annotation is consistent across training and testing sets. Our findings give helpful insights for both cognitive and NLP scientists.
@inproceedings{aghazadeh-etal-2022-metaphors, author = {Aghazadeh, Ehsan and Fayyaz, Mohsen and Yaghoobzadeh, Yadollah}, title = {Metaphors in Pre-Trained Language Models: Probing and Generalization Across Datasets and Languages}, booktitle = {Proceedings of the 60th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)}, year = {2022}, address = {Dublin, Ireland}, doi = {10.18653/v1/2022.acl-long.144}, editor = {Muresan, Smaranda and Nakov, Preslav and Villavicencio, Aline}, month = {May}, pages = {2037--2050}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2022.acl-long.144/} } - SATLab at SemEval-2022 Task 4: Trying to Detect Patronizing and Condescending Language with only Character and Word N-gramsIn Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pp. 490–495, 2022.
A logistic regression model only fed with character and word n-grams is proposed for the SemEval-2022 Task 4 on Patronizing and Condescending Language Detection (PCL). It obtained an average level of performance, well above the performance of a system that tries to guess without using any knowledge about the task, but much lower than the best teams. To facilitate the interpretation of the performance scores, the F1 measure, the best level of performance of a system that tries to guess without using any knowledge is calculated and used to correct the F1 scores in the manner of a Kappa. As the proposed model is very similar to the one that performed well on a task requiring to automatically identify hate speech and offensive content, this paper confirms the difficulty of PCL detection.
@inproceedings{bestgen-2022-satlab, author = {Bestgen, Yves}, title = {{SATL}ab at {S}em{E}val-2022 Task 4: Trying to Detect Patronizing and Condescending Language with only Character and Word N-grams}, booktitle = {Proceedings of the 16th {I}nternational {W}orkshop on {S}emantic {E}valuation ({S}em{E}val-2022)}, year = {2022}, address = {Seattle, United States}, doi = {10.18653/v1/2022.semeval-1.67}, editor = {Emerson, Guy and Schluter, Natalie and Stanovsky, Gabriel and Kumar, Ritesh and Palmer, Alexis and Schneider, Nathan and Singh, Siddharth and Ratan, Shyam}, month = {July}, pages = {490--495}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2022.semeval-1.67/} } - chrF++: words helping character n-gramsIn Proceedings of the Second Conference on Machine Translation, pp. 612–618, 2017.
Character n-gram F-score (chrF) is shown to correlate very well with human relative rankings of different machine translation outputs, especially for morphologically rich target languages. However, its relation with direct human assessments is not yet clear. In this work, Pearson’s correlation coefficients for direct assessments are investigated for two currently available target languages, English and Russian. First, different β parameters (in range from 1 to 3) are re-investigated with direct assessment, and it is confirmed that β = 2 is the optimal option. Then separate character and word n-grams are investigated, and the main finding is that, apart from character n-grams, word 1-grams and 2-grams also correlate rather well with direct assessments. Further experiments show that adding word unigrams and bigrams to the standard chrF score improves the correlations with direct assessments, though it is still not clear which option is better, unigrams only (chrF+) or unigrams and bigrams (chrF++). This should be investigated in future work on more target languages.
@inproceedings{popovic-2017-chrf, author = {Popovi{\'c}, Maja}, title = {chr{F}++: words helping character n-grams}, booktitle = {Proceedings of the Second Conference on Machine Translation}, year = {2017}, address = {Copenhagen, Denmark}, doi = {10.18653/v1/W17-4770}, editor = {Bojar, Ond{\v{r}}ej and Buck, Christian and Chatterjee, Rajen and Federmann, Christian and Graham, Yvette and Haddow, Barry and Huck, Matthias and Yepes, Antonio Jimeno and Koehn, Philipp and Kreutzer, Julia}, month = {September}, pages = {612--618}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/W17-4770/} } - Climbing towards NLU: On Meaning, Form, and Understanding in the Age of DataIn Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5185–5198, 2020.
The success of the large neural language models on many NLP tasks is exciting. However, we find that these successes sometimes lead to hype in which these models are being described as “understanding” language or capturing “meaning”. In this position paper, we argue that a system trained only on form has a priori no way to learn meaning. In keeping with the ACL 2020 theme of “Taking Stock of Where We’ve Been and Where We’re Going”, we argue that a clear understanding of the distinction between form and meaning will help guide the field towards better science around natural language understanding.
@inproceedings{bender-koller-2020-climbing, author = {Bender, Emily M. and Koller, Alexander}, title = {Climbing towards {NLU}: {On} Meaning, Form, and Understanding in the Age of Data}, booktitle = {Proceedings of the 58th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics}, year = {2020}, address = {Online}, doi = {10.18653/v1/2020.acl-main.463}, editor = {Jurafsky, Dan and Chai, Joyce and Schluter, Natalie and Tetreault, Joel}, month = {July}, pages = {5185--5198}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2020.acl-main.463/} }
Lit Review 4: HMMs
- Bridging Pre-trained Language Models and Hand-crafted Features for Unsupervised POS TaggingIn Findings of the Association for Computational Linguistics: ACL 2022, pp. 3276–3290, 2022.
In recent years, large-scale pre-trained language models (PLMs) have made extraordinary progress in most NLP tasks. But, in the unsupervised POS tagging task, works utilizing PLMs are few and fail to achieve state-of-the-art (SOTA) performance. The recent SOTA performance is yielded by a Guassian HMM variant proposed by He et al. (2018). However, as a generative model, HMM makes very strong independence assumptions, making it very challenging to incorporate contexualized word representations from PLMs. In this work, we for the first time propose a neural conditional random field autoencoder (CRF-AE) model for unsupervised POS tagging. The discriminative encoder of CRF-AE can straightforwardly incorporate ELMo word representations. Moreover, inspired by feature-rich HMM, we reintroduce hand-crafted features into the decoder of CRF-AE. Finally, experiments clearly show that our model outperforms previous state-of-the-art models by a large margin on Penn Treebank and multilingual Universal Dependencies treebank v2.0.
@inproceedings{zhou-etal-2022-bridging, author = {Zhou, Houquan and Li, Yang and Li, Zhenghua and Zhang, Min}, title = {Bridging Pre-trained Language Models and Hand-crafted Features for Unsupervised {POS} Tagging}, booktitle = {Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2022}, year = {2022}, address = {Dublin, Ireland}, doi = {10.18653/v1/2022.findings-acl.259}, editor = {Muresan, Smaranda and Nakov, Preslav and Villavicencio, Aline}, month = {May}, pages = {3276--3290}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2022.findings-acl.259/} } - Scaling Hidden Markov Language ModelsIn Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1341–1349, 2020.
The hidden Markov model (HMM) is a fundamental tool for sequence modeling that cleanly separates the hidden state from the emission structure. However, this separation makes it difficult to fit HMMs to large datasets in modern NLP, and they have fallen out of use due to very poor performance compared to fully observed models. This work revisits the challenge of scaling HMMs to language modeling datasets, taking ideas from recent approaches to neural modeling. We propose methods for scaling HMMs to massive state spaces while maintaining efficient exact inference, a compact parameterization, and effective regularization. Experiments show that this approach leads to models that are much more accurate than previous HMMs and n-gram-based methods, making progress towards the performance of state-of-the-art NN models.
@inproceedings{chiu-rush-2020-scaling, author = {Chiu, Justin and Rush, Alexander}, title = {Scaling Hidden {M}arkov Language Models}, booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing ({EMNLP})}, year = {2020}, address = {Online}, doi = {10.18653/v1/2020.emnlp-main.103}, editor = {Webber, Bonnie and Cohn, Trevor and He, Yulan and Liu, Yang}, month = {November}, pages = {1341--1349}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2020.emnlp-main.103/} } - Spectral modification for recognition of children’s speech undermismatched conditionsIn Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa), pp. 94–100, 2021.
In this paper, we propose spectral modification by sharpening formants and by reducing the spectral tilt to recognize children’s speech by automatic speech recognition (ASR) systems developed using adult speech. In this type of mismatched condition, the ASR performance is degraded due to the acoustic and linguistic mismatch in the attributes between children and adult speakers. The proposed method is used to improve the speech intelligibility to enhance the children’s speech recognition using an acoustic model trained on adult speech. In the experiments, WSJCAM0 and PFSTAR are used as databases for adults’ and children’s speech, respectively. The proposed technique gives a significant improvement in the context of the DNN-HMM-based ASR. Furthermore, we validate the robustness of the technique by showing that it performs well also in mismatched noise conditions.
@inproceedings{kathania-etal-2021-spectral, author = {Kathania, Hemant Kumar and Kadiri, Sudarsana Reddy and Alku, Paavo and Kurimo, Mikko}, title = {Spectral modification for recognition of children{'}s speech undermismatched conditions}, booktitle = {Proceedings of the 23rd {N}ordic Conference on Computational Linguistics ({N}o{D}a{L}i{D}a)}, year = {2021}, address = {Reykjavik, Iceland (Online)}, editor = {Dobnik, Simon and {\O}vrelid, Lilja}, month = {May 31--2 June}, pages = {94--100}, publisher = {Link{\"o}ping University Electronic Press, Sweden}, url = {https://aclanthology.org/2021.nodalida-main.10/} } - Tagging a Norwegian Dialect CorpusIn Proceedings of the 22nd Nordic Conference on Computational Linguistics, pp. 350–355, 2019.
This paper describes an evaluation of five data-driven part-of-speech (PoS) taggers for spoken Norwegian. The taggers all rely on different machine learning mechanisms: decision trees, hidden Markov models (HMMs), conditional random fields (CRFs), long-short term memory networks (LSTMs), and convolutional neural networks (CNNs). We go into some of the challenges posed by the task of tagging spoken, as opposed to written, language, and in particular a wide range of dialects as is found in the recordings of the LIA (Language Infrastructure made Accessible) project. The results show that the taggers based on either conditional random fields or neural networks perform much better than the rest, with the LSTM tagger getting the highest score.
@inproceedings{kasen-etal-2019-tagging, author = {K{\r{a}}sen, Andre and N{\o}klestad, Anders and Hagen, Kristin and Priestley, Joel}, title = {Tagging a {N}orwegian Dialect Corpus}, booktitle = {Proceedings of the 22nd {N}ordic Conference on Computational Linguistics}, year = {2019}, address = {Turku, Finland}, editor = {Hartmann, Mareike and Plank, Barbara}, month = {September–October}, pages = {350--355}, publisher = {Link{\"o}ping University Electronic Press}, url = {https://aclanthology.org/W19-6140/} } - Joint Rumour Stance and Veracity PredictionIn Proceedings of the 22nd Nordic Conference on Computational Linguistics, pp. 208–221, 2019.
The net is rife with rumours that spread through microblogs and social media. Not all the claims in these can be verified. However, recent work has shown that the stances alone that commenters take toward claims can be sufficiently good indicators of claim veracity, using e.g. an HMM that takes conversational stance sequences as the only input. Existing results are monolingual (English) and mono-platform (Twitter). This paper introduces a stance-annotated Reddit dataset for the Danish language, and describes various implementations of stance classification models. Of these, a Linear SVM provides predicts stance best, with 0.76 accuracy / 0.42 macro F1. Stance labels are then used to predict veracity across platforms and also across languages, training on conversations held in one language and using the model on conversations held in another. In our experiments, monolinugal scores reach stance-based veracity accuracy of 0.83 (F1 0.68); applying the model across languages predicts veracity of claims with an accuracy of 0.82 (F1 0.67). This demonstrates the surprising and powerful viability of transferring stance-based veracity prediction across languages.
@inproceedings{lillie-etal-2019-joint, author = {Lillie, Anders Edelbo and Middelboe, Emil Refsgaard and Derczynski, Leon}, title = {Joint Rumour Stance and Veracity Prediction}, booktitle = {Proceedings of the 22nd {N}ordic Conference on Computational Linguistics}, year = {2019}, address = {Turku, Finland}, editor = {Hartmann, Mareike and Plank, Barbara}, month = {September–October}, pages = {208--221}, publisher = {Link{\"o}ping University Electronic Press}, url = {https://aclanthology.org/W19-6122/} }
Lit Review 5: PCFGs & Adding Grammar to More Advanced Models
- Unsupervised PCFG Induction for Grounded Language Learning with Highly Ambiguous SupervisionIn Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pp. 433–444, 2012.
“Grounded” language learning employs training data in the form of sentences paired with relevant but ambiguous perceptual contexts. Börschinger et al. (2011) introduced an approach to grounded language learning based on unsupervised PCFG induction. Their approach works well when each sentence potentially refers to one of a small set of possible meanings, such as in the sportscasting task. However, it does not scale to problems with a large set of potential meanings for each sentence, such as the navigation instruction following task studied by Chen and Mooney (2011). This paper presents an enhancement of the PCFG approach that scales to such problems with highly-ambiguous supervision. Experimental results on the navigation task demonstrates the effectiveness of our approach.
@inproceedings{kim-mooney-2012-unsupervised, author = {Kim, Joohyun and Mooney, Raymond}, title = {Unsupervised {PCFG} Induction for Grounded Language Learning with Highly Ambiguous Supervision}, booktitle = {Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning}, year = {2012}, address = {Jeju Island, Korea}, editor = {Tsujii, Jun{'}ichi and Henderson, James and Pa{\c{s}}ca, Marius}, month = {July}, pages = {433--444}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/D12-1040/} } - Neural Bi-Lexicalized PCFG InductionIn Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 2688–2699, 2021.
Neural lexicalized PCFGs (L-PCFGs) have been shown effective in grammar induction. However, to reduce computational complexity, they make a strong independence assumption on the generation of the child word and thus bilexical dependencies are ignored. In this paper, we propose an approach to parameterize L-PCFGs without making implausible independence assumptions. Our approach directly models bilexical dependencies and meanwhile reduces both learning and representation complexities of L-PCFGs. Experimental results on the English WSJ dataset confirm the effectiveness of our approach in improving both running speed and unsupervised parsing performance.
@inproceedings{yang-etal-2021-neural, author = {Yang, Songlin and Zhao, Yanpeng and Tu, Kewei}, title = {Neural Bi-Lexicalized {PCFG} Induction}, booktitle = {Proceedings of the 59th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)}, year = {2021}, address = {Online}, doi = {10.18653/v1/2021.acl-long.209}, editor = {Zong, Chengqing and Xia, Fei and Li, Wenjie and Navigli, Roberto}, month = {August}, pages = {2688--2699}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2021.acl-long.209/} } - Character-based PCFG Induction for Modeling the Syntactic Acquisition of Morphologically Rich LanguagesIn Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 4367–4378, 2021.
Unsupervised PCFG induction models, which build syntactic structures from raw text, can be used to evaluate the extent to which syntactic knowledge can be acquired from distributional information alone. However, many state-of-the-art PCFG induction models are word-based, meaning that they cannot directly inspect functional affixes, which may provide crucial information for syntactic acquisition in child learners. This work first introduces a neural PCFG induction model that allows a clean ablation of the influence of subword information in grammar induction. Experiments on child-directed speech demonstrate first that the incorporation of subword information results in more accurate grammars with categories that word-based induction models have difficulty finding, and second that this effect is amplified in morphologically richer languages that rely on functional affixes to express grammatical relations. A subsequent evaluation on multilingual treebanks shows that the model with subword information achieves state-of-the-art results on many languages, further supporting a distributional model of syntactic acquisition.
@inproceedings{jin-etal-2021-character-based, author = {Jin, Lifeng and Oh, Byung-Doh and Schuler, William}, title = {Character-based {PCFG} Induction for Modeling the Syntactic Acquisition of Morphologically Rich Languages}, booktitle = {Findings of the {A}ssociation for {C}omputational {L}inguistics: {EMNLP} 2021}, year = {2021}, address = {Punta Cana, Dominican Republic}, doi = {10.18653/v1/2021.findings-emnlp.371}, editor = {Moens, Marie-Francine and Huang, Xuanjing and Specia, Lucia and Yih, Scott Wen-tau}, month = {November}, pages = {4367--4378}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2021.findings-emnlp.371/} } - LSTMs Can Learn Syntax-Sensitive Dependencies Well, But Modeling Structure Makes Them BetterIn Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1426–1436, 2018.
Language exhibits hierarchical structure, but recent work using a subject-verb agreement diagnostic argued that state-of-the-art language models, LSTMs, fail to learn long-range syntax sensitive dependencies. Using the same diagnostic, we show that, in fact, LSTMs do succeed in learning such dependencies—provided they have enough capacity. We then explore whether models that have access to explicit syntactic information learn agreement more effectively, and how the way in which this structural information is incorporated into the model impacts performance. We find that the mere presence of syntactic information does not improve accuracy, but when model architecture is determined by syntax, number agreement is improved. Further, we find that the choice of how syntactic structure is built affects how well number agreement is learned: top-down construction outperforms left-corner and bottom-up variants in capturing non-local structural dependencies.
@inproceedings{kuncoro-etal-2018-lstms, author = {Kuncoro, Adhiguna and Dyer, Chris and Hale, John and Yogatama, Dani and Clark, Stephen and Blunsom, Phil}, title = {{LSTM}s Can Learn Syntax-Sensitive Dependencies Well, But Modeling Structure Makes Them Better}, booktitle = {Proceedings of the 56th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)}, year = {2018}, address = {Melbourne, Australia}, doi = {10.18653/v1/P18-1132}, editor = {Gurevych, Iryna and Miyao, Yusuke}, month = {July}, pages = {1426--1436}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/P18-1132/} } - Have Attention Heads in BERT Learned Constituency Grammar?In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, pp. 8–15, 2021.
With the success of pre-trained language models in recent years, more and more researchers focus on opening the “black box” of these models. Following this interest, we carry out a qualitative and quantitative analysis of constituency grammar in attention heads of BERT and RoBERTa. We employ the syntactic distance method to extract implicit constituency grammar from the attention weights of each head. Our results show that there exist heads that can induce some grammar types much better than baselines, suggesting that some heads act as a proxy for constituency grammar. We also analyze how attention heads’ constituency grammar inducing (CGI) ability changes after fine-tuning with two kinds of tasks, including sentence meaning similarity (SMS) tasks and natural language inference (NLI) tasks. Our results suggest that SMS tasks decrease the average CGI ability of upper layers, while NLI tasks increase it. Lastly, we investigate the connections between CGI ability and natural language understanding ability on QQP and MNLI tasks.
@inproceedings{luo-2021-attention, author = {Luo, Ziyang}, title = {Have Attention Heads in {BERT} Learned Constituency Grammar?}, booktitle = {Proceedings of the 16th Conference of the {E}uropean Chapter of the {A}ssociation for {C}omputational {L}inguistics: Student Research Workshop}, year = {2021}, address = {Online}, doi = {10.18653/v1/2021.eacl-srw.2}, editor = {Sorodoc, Ionut-Teodor and Sushil, Madhumita and Takmaz, Ece and Agirre, Eneko}, month = {April}, pages = {8--15}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2021.eacl-srw.2/} }
Lit Review 6: Broader Issues & Understanding Human Use of Language
- Towards Afrocentric NLP for African Languages: Where We Are and Where We Can GoIn Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3814–3841, 2022.
Aligning with ACL 2022 special Theme on “Language Diversity: from Low Resource to Endangered Languages”, we discuss the major linguistic and sociopolitical challenges facing development of NLP technologies for African languages. Situating African languages in a typological framework, we discuss how the particulars of these languages can be harnessed. To facilitate future research, we also highlight current efforts, communities, venues, datasets, and tools. Our main objective is to motivate and advocate for an Afrocentric approach to technology development. With this in mind, we recommend what technologies to build and how to build, evaluate, and deploy them based on the needs of local African communities.
@inproceedings{adebara-abdul-mageed-2022-towards, author = {Adebara, Ife and Abdul-Mageed, Muhammad}, title = {Towards Afrocentric {NLP} for {A}frican Languages: Where We Are and Where We Can Go}, booktitle = {Proceedings of the 60th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)}, year = {2022}, address = {Dublin, Ireland}, doi = {10.18653/v1/2022.acl-long.265}, editor = {Muresan, Smaranda and Nakov, Preslav and Villavicencio, Aline}, month = {May}, pages = {3814--3841}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2022.acl-long.265/} } - From Tool to Companion: Storywriters Want AI Writers to Respect Their Personal Values and Writing StrategiesIn Proceedings of the 2022 ACM Designing Interactive Systems Conference (DIS '22), pp. 1209–1227, 2022.
@inproceedings{biermann-etal-2022-tool, author = {Biermann, Oloff C. and Ma, Ning F. and Yoon, Dongwook}, title = {From Tool to Companion: Storywriters Want {AI} Writers to Respect Their Personal Values and Writing Strategies}, booktitle = {Proceedings of the 2022 {ACM} Designing Interactive Systems Conference ({DIS} '22)}, year = {2022}, address = {New York, NY, USA}, doi = {10.1145/3532106.3533506}, pages = {1209--1227}, publisher = {Association for Computing Machinery}, url = {https://dl.acm.org/doi/10.1145/3532106.3533506} } - Dead or Murdered? Predicting Responsibility Perception in Femicide News ReportsIn Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1078–1090, 2022.
Different linguistic expressions can conceptualize the same event from different viewpoints by emphasizing certain participants over others. Here, we investigate a case where this has social consequences: how do linguistic expressions of gender-based violence (GBV) influence who we perceive as responsible? We build on previous psycholinguistic research in this area and conduct a large-scale perception survey of GBV descriptions automatically extracted from a corpus of Italian newspapers. We then train regression models that predict the salience of GBV participants with respect to different dimensions of perceived responsibility. Our best model (fine-tuned BERT) shows solid overall performance, with large differences between dimensions and participants: salient focus is more predictable than salient blame, and perpetrators’ salience is more predictable than victims’ salience. Experiments with ridge regression models using different representations show that features based on linguistic theory similarly to word-based features. Overall, we show that different linguistic choices do trigger different perceptions of responsibility, and that such perceptions can be modelled automatically. This work can be a core instrument to raise awareness of the consequences of different perspectivizations in the general public and in news producers alike.
@inproceedings{minnema-etal-2022-dead, author = {Minnema, Gosse and Gemelli, Sara and Zanchi, Chiara and Caselli, Tommaso and Nissim, Malvina}, title = {Dead or Murdered? Predicting Responsibility Perception in Femicide News Reports}, booktitle = {Proceedings of the 2nd Conference of the {A}sia-Pacific Chapter of the {A}ssociation for {C}omputational {L}inguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)}, year = {2022}, address = {Online only}, doi = {10.18653/v1/2022.aacl-main.79}, editor = {He, Yulan and Ji, Heng and Li, Sujian and Liu, Yang and Chang, Chua-Hui}, month = {November}, pages = {1078--1090}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2022.aacl-main.79/} } - “It’s a Whole Vibe”: testing evaluations of grammatical and ungrammatical AAE on TwitterLinguistics Vanguard, 7(1), 2021.
This study focuses on listener perceptions of African American English (AAE) on Twitter, examining both grammatical and ungrammatical usages, as well as how these perceptions may be affected by the race of the speaker and rater. We conducted an experimental survey designed to address the following questions: 1. Does avatar race affect perception of the grammaticality of AAE? 2. Are differences between grammatical and ungrammatical AAE discernible to naive raters of different races? 3. How are various social attributes evaluated for avatars of different races and different linguistic varieties? Results indicate that participants generally do not downgrade avatars who use grammatical AAE on ratings of grammaticality or personal characteristics. However, participants of all races disprefer ungrammatical uses of AAE, with black raters being especially sensitive to ungrammatical AAE. These findings have implications for sociolinguistics in that they demonstrate that participants across racial backgrounds may differentiate grammatical versus ungrammatical AAE online, and that contrary to expectations based on previous literature, AAE is not universally downgraded in these contexts. However, results also indicate that the use of AAE still negatively impacts listeners’ perceptions of speakers as educated, demonstrating that some widespread biases against AAE-speakers persist in an online context.
@article{holliday-tano-2021-whole, author = {Holliday, Nicole and Tano, Marie}, title = {``It{'}s a Whole Vibe'': testing evaluations of grammatical and ungrammatical {AAE} on {T}witter}, journal = {Linguistics Vanguard}, year = {2021}, doi = {10.1515/lingvan-2020-0095}, number = {1}, publisher = {De Gruyter}, url = {https://www.degruyter.com/document/doi/10.1515/lingvan-2020-0095/html}, volume = {7} } - Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 PapersIn Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 7297–7306, 2021.
This paper presents the first large-scale meta-evaluation of machine translation (MT). We annotated MT evaluations conducted in 769 research papers published from 2010 to 2020. Our study shows that practices for automatic MT evaluation have dramatically changed during the past decade and follow concerning trends. An increasing number of MT evaluations exclusively rely on differences between BLEU scores to draw conclusions, without performing any kind of statistical significance testing nor human evaluation, while at least 108 metrics claiming to be better than BLEU have been proposed. MT evaluations in recent papers tend to copy and compare automatic metric scores from previous work to claim the superiority of a method or an algorithm without confirming neither exactly the same training, validating, and testing data have been used nor the metric scores are comparable. Furthermore, tools for reporting standardized metric scores are still far from being widely adopted by the MT community. After showing how the accumulation of these pitfalls leads to dubious evaluation, we propose a guideline to encourage better automatic MT evaluation along with a simple meta-evaluation scoring method to assess its credibility.
@inproceedings{marie-etal-2021-scientific, author = {Marie, Benjamin and Fujita, Atsushi and Rubino, Raphael}, title = {Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Papers}, booktitle = {Proceedings of the 59th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)}, year = {2021}, address = {Online}, doi = {10.18653/v1/2021.acl-long.566}, editor = {Zong, Chengqing and Xia, Fei and Li, Wenjie and Navigli, Roberto}, month = {August}, pages = {7297--7306}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2021.acl-long.566/} } - Quantifying Discourse Support for Omitted PronounsIn Proceedings of the Fifth Workshop on Computational Models of Reference, Anaphora and Coreference, pp. 1–12, 2022.
Pro-drop is commonly seen in many languages, but its discourse motivations have not been well characterized. Inspired by the topic chain theory in Chinese, this study shows how character-verb usage continuity distinguishes dropped pronouns from overt references to story characters. We model the choice to drop vs. not drop as a function of character-verb continuity. The results show that omitted subjects have higher character history-current verb continuity salience than non-omitted subjects. This is consistent with the idea that discourse coherence with a particular topic, such as a story character, indeed facilitates the omission of pronouns in languages and contexts where they are optional.
@inproceedings{zhang-etal-2022-quantifying, author = {Zhang, Shulin and Li, Jixing and Hale, John}, title = {Quantifying Discourse Support for Omitted Pronouns}, booktitle = {Proceedings of the Fifth Workshop on Computational Models of Reference, Anaphora and Coreference}, year = {2022}, address = {Gyeongju, Republic of Korea}, editor = {Ogrodniczuk, Maciej and Pradhan, Sameer and Nedoluzhko, Anna and Ng, Vincent and Poesio, Massimo}, month = {October}, pages = {1--12}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2022.crac-1.1/} } - Language (Technology) is Power: A Critical Survey of “Bias” in NLPIn Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5454–5476, 2020.
We survey 146 papers analyzing “bias” in NLP systems, finding that their motivations are often vague, inconsistent, and lacking in normative reasoning, despite the fact that analyzing “bias” is an inherently normative process. We further find that these papers’ proposed quantitative techniques for measuring or mitigating “bias” are poorly matched to their motivations and do not engage with the relevant literature outside of NLP. Based on these findings, we describe the beginnings of a path forward by proposing three recommendations that should guide work analyzing “bias” in NLP systems. These recommendations rest on a greater recognition of the relationships between language and social hierarchies, encouraging researchers and practitioners to articulate their conceptualizations of “bias”—i.e., what kinds of system behaviors are harmful, in what ways, to whom, and why, as well as the normative reasoning underlying these statements—and to center work around the lived experiences of members of communities affected by NLP systems, while interrogating and reimagining the power relations between technologists and such communities.
@inproceedings{blodgett-etal-2020-language, author = {Blodgett, Su Lin and Barocas, Solon and Daum{\'e} III, Hal and Wallach, Hanna}, title = {Language (Technology) is Power: A Critical Survey of ``Bias'' in {NLP}}, booktitle = {Proceedings of the 58th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics}, year = {2020}, address = {Online}, doi = {10.18653/v1/2020.acl-main.485}, editor = {Jurafsky, Dan and Chai, Joyce and Schluter, Natalie and Tetreault, Joel}, month = {July}, pages = {5454--5476}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2020.acl-main.485/} } - Gendered Mental Health Stigma in Masked Language ModelsIn Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 2152–2170, 2022.
Mental health stigma prevents many individuals from receiving the appropriate care, and social psychology studies have shown that mental health tends to be overlooked in men. In this work, we investigate gendered mental health stigma in masked language models. In doing so, we operationalize mental health stigma by developing a framework grounded in psychology research: we use clinical psychology literature to curate prompts, then evaluate the models’ propensity to generate gendered words. We find that masked language models capture societal stigma about gender in mental health: models are consistently more likely to predict female subjects than male in sentences about having a mental health condition (32% vs. 19%), and this disparity is exacerbated for sentences that indicate treatment-seeking behavior. Furthermore, we find that different models capture dimensions of stigma differently for men and women, associating stereotypes like anger, blame, and pity more with women with mental health conditions than with men. In showing the complex nuances of models’ gendered mental health stigma, we demonstrate that context and overlapping dimensions of identity are important considerations when assessing computational models’ social biases.
@inproceedings{lin-etal-2022-gendered, author = {Lin, Inna and Njoo, Lucille and Field, Anjalie and Sharma, Ashish and Reinecke, Katharina and Althoff, Tim and Tsvetkov, Yulia}, title = {Gendered Mental Health Stigma in Masked Language Models}, booktitle = {Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing}, year = {2022}, address = {Abu Dhabi, United Arab Emirates}, doi = {10.18653/v1/2022.emnlp-main.139}, editor = {Goldberg, Yoav and Kozareva, Zornitsa and Zhang, Yue}, month = {December}, pages = {2152--2170}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2022.emnlp-main.139/} }