Correlations Between Positive or Negative Utterances and Basic Acoustic Features of Voice: a Preliminary Analysis

Łukasz Stolarski

doi:10.18778/1731-7533.20.2.03

Authors

Łukasz Stolarski Jan Kochanowski University in Kielce, Poland

DOI:

https://doi.org/10.18778/1731-7533.20.2.03

Keywords:

sentiment analysis, acoustic features, feature selection

Abstract

The major aim of this paper is to establish possible correlations between continuous sentiment scores and four basic acoustic characteristics of voice. In order to achieve this objective, the text of “A Christmas Carol” by Charles Dickens was tokenized at the sentence level. Next, each of the resulting text units was assessed in terms of sentiment polarity and aligned with the corresponding fragment in an audiobook. The results indicate weak but statistically significant correlations between sentiment scores and three acoustic features: the mean F0, the standard deviation of F0 and the mean intensity. These findings may be useful in selecting optimal acoustic features for model training in multimodal sentiment analysis. Also, they are essential from a linguistic point of view and could be applied in studies on such language phenomena as irony.

References

Abbasi, Ahmed, Hassan, Ammar and Dhar, Milan.2014. Benchmarking Twitter Sentiment Analysis Tools. In LREC, Vol. 14, 26–31.

Aldeneh, Zakaria, Khorram, Soheil, Dimitriadis, Dimitrios and Provost, Emily Mower. 2017. Pooling acoustic and lexical features for the prediction of valence. In Proceedings of the 19th ACM International Conference on Multimodal Interaction, 68–72. ACM. https://doi.org/10.1145/3136755.3136760 DOI: https://doi.org/10.1145/3136755.3136760

Audacity Team. 2014. Audacity(R): Free audio editor and recorder (version 2.0.5) [computer software].

Aue, Anthony and Gamon, Michael. 2005. Customizing sentiment classifiers to new domains: A case study. In Proceedings of recent advances in natural language processing (RANLP), Vol. 1, 2–10. Citeseer.

Austin, John Langshaw. 1962. How to do things with words. Oxford: Clarendon Press.

Bezooijen, Renée. 1984. Characteristics and recognizability of vocal expressions of emotion. Dordrecht, Netherlands: Foris Publications. https://doi.org/10.1515/9783110850390 DOI: https://doi.org/10.1515/9783110850390

Boersma, Paul and Weenink, David. 2014. Praat, a system for doing phonetics by computer (version 5.4.01) [computer software]. Amsterdam: University of Amsterdam.

Borth, Damian, Ji, Rongrong, Chen, Tao, Breuel, Thomas and Chang, Shih-Fu. 2013. Large-scale visual sentiment ontology and detectors using adjective noun pairs. In Proceedings of the 21st ACM international conference on Multimedia, 223–232. ACM. https://doi.org/10.1145/2502081.2502282 DOI: https://doi.org/10.1145/2502081.2502282

Breitenstein, Caterina, Van Lancker, Diana and Daum, Irene. 2001. The contribution of speech rate and pitch variation to the perception of vocal emotions in a German and an American sample. Cognition and Emotion, 15(1), 57–79. https://doi.org/10.1080/02699930126095 DOI: https://doi.org/10.1080/0269993004200114

Cambria, Erik, Poria, Soujanya, Bajpai, Rajiv and Schuller, Björn W. 2016. SenticNet 4: A Semantic Resource for Sentiment Analysis Based on Conceptual Primitives. In COLING, 2666–2677.

Chen, Minghai, Wang, Sen, Liang, Paul Pu, Baltrušaitis, Tadas, Zadeh, Amir and Morency, Louis-Philippe. 2017. Multimodal sentiment analysis with word-level fusion and reinforcement learning. In Proceedings of the 19th ACM International Conference on Multimodal Interaction, 63–171. ACM. https://doi.org/10.1145/3136755.3136801 DOI: https://doi.org/10.1145/3136755.3136801

Choi, Yejin and Cardie, Claire. 2008. Learning with compositional semantics as structural inference for subsentential sentiment analysis. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (pp. 793–801). Association for Computational Linguistics. https://doi.org/10.3115/1613715.1613816 DOI: https://doi.org/10.3115/1613715.1613816

Coleman, Robert F. and Williams, Robert. 1979. Identification of emotional states using perceptual and acoustic analyses. In Transcript of the 8th Symposium: Care of the Professional Voice, Part I. The Voice Foundation, New York.

Collier, William G. and Hubbard, Timothy L. 1998. Judgments of happiness, brightness, speed, and tempo change of auditory stimuli varying in pitch and tempo. Psychomusicology, 17(1/2), 36–55. https://doi.org/10.1037/h0094060 DOI: https://doi.org/10.1037/h0094060

Collier, William G. and Hubbard, Timothy L. 2001. Musical scales and evaluations of happiness and awkwardness: Effects of pitch, direction, and scale mode. American Journal of Psychology, 114(3), 355–375. https://doi.org/10.2307/1423686 DOI: https://doi.org/10.2307/1423686

Davitz, Joel R. 1964. Auditory correlates of vocal expressions of emotional meaning. The Communication of Emotional Meaning, 101–112.

Diniz, Joao P., Bastos, Lucas, Soares, Elias, Ferreira, Miller, Ribeiro, Filipe and Benevenuto, Fabrıcio. 2016. ifeel 2.0: A multilingual benchmarking system for sentence-level sentiment analysis.

Eldred, Stanley H. and Price, Douglas B. 1958. A linguistic evaluation of feeling states in psychotherapy. Psychiatry, 21(2), 115–121. https://doi.org/10.1080/00332747.1958.11023120 DOI: https://doi.org/10.1080/00332747.1958.11023120

Eyben, Florian, Weninger, Felix, Gross, Florian and Schuller, Björn. 2013. Recent developments in opensmile, the munich open-source multimedia feature extractor. In Proceedings of the 21st ACM international conference on Multimedia, 835–838. ACM. https://doi.org/10.1145/2502081.2502224 DOI: https://doi.org/10.1145/2502081.2502224

Fairbanks, Grant and Pronovost, Wilbert. 1939. An experimental study of the pitch characteristics of the voice during the expression of emotion. Speech Monographs, 6(1), 87–104. https://doi.org/10.1080/03637753909374863 DOI: https://doi.org/10.1080/03637753909374863

Fonagy, Ivan.1978. A new method of investigating the perception of prosodic features. Language and Speech, 21(1), 34–49. https://doi.org/10.1177/002383097802100102 DOI: https://doi.org/10.1177/002383097802100102

Frick, Robert W. 1985. Communicating emotion: The role of prosodic features. Psychological Bulletin, 97(3), 412–429. https://doi.org/10.1037/0033-2909.97.3.412 DOI: https://doi.org/10.1037/0033-2909.97.3.412

Goldman, Jean-Philippe. 2011. EasyAlign: an automatic phonetic alignment tool under Praat. In Proceedings of Interspeech, 3233–3236. https://doi.org/10.21437/Interspeech.2011-815 DOI: https://doi.org/10.21437/Interspeech.2011-815

Gonçalves, Pollyanna, Araújo, Matheus, Benevenuto, Fabrício and Cha, Meeyoung. 2013. Comparing and combining sentiment analysis methods. In Proceedings of the first ACM conference on Online social networks (pp. 27–38). ACM. https://doi.org/10.1145/2512938.2512951 DOI: https://doi.org/10.1145/2512938.2512951

Govindaraj, Sureshkumar and Gopalakrishnan, Kumaravelan. 2016. Intensified sentiment analysis of customer product reviews using acoustic and textual features. ETRI Journal, 38(3), 494–501. https://doi.org/10.4218/etrij.16.0115.0684 DOI: https://doi.org/10.4218/etrij.16.0115.0684

Hargreaves, William A., Starkweather, John A. and Blacker, K. H. 1965. Voice quality in depression. Journal of Abnormal Psychology, 70(3), 218–220. https://doi.org/10.1037/h0022151 DOI: https://doi.org/10.1037/h0022151

Höffe, Wilhelm L. 1960. Über Beziehungen von Sprachmelodie und Lautstärke. Phonetica, 5(3–4), 129–159. https://doi.org/10.1159/000258054 DOI: https://doi.org/10.1159/000258054

Huddleston, Rodney. 1988. English grammar: An outline. Cambridge University Press. https://doi.org/10.1017/CBO9781139166003 DOI: https://doi.org/10.1017/CBO9781139166003

Huron, David. 2008. A comparison of average pitch height and interval size in major-and minor-key themes: Evidence consistent with affect-related pitch prosody. Empirical Musicology Review, 3(2), 59–63. https://doi.org/10.18061/1811/31940 DOI: https://doi.org/10.18061/1811/31940

Huron, David, Yim, Gary, and Chordia, Parag. 2010. The effect of pitch exposure on sadness judgments: An association between sadness and lower than normal pitch. In Proceedings of the 11th International Conference on Music Perception and Cognition, 63–66.

Huttar, George L. 1968. Relations between prosodic variables and emotions in normal American English utterances. Journal of Speech, Language, and Hearing Research, 11(3), 481–487. https://doi.org/10.1044/jshr.1103.481 DOI: https://doi.org/10.1044/jshr.1103.481

Hutto, C. J. and Gilbert, Eric. 2014. Vader: A Parsimonious Rule-Based Model for Sentiment Analysis of Social Media Text. In Eighth International AAAI Conference on Weblogs and Social Media (ICWSM-14), 216–255. Ann Arbor, MI. https://doi.org/10.1609/icwsm.v8i1.14550 DOI: https://doi.org/10.1609/icwsm.v8i1.14550

Kaiser, L. 1962. Communication of affects by single vowels. Synthese, 14(4), 300–319. https://doi.org/10.1007/BF00869311 DOI: https://doi.org/10.1007/BF00869311

Kappas, Arvid, Hess, Ursula and Scherer, Klaus R. 1991. Voice and emotion. In R. S. Feldman and B. Rim (eds.), Fundamentals of nonverbal behavior (pp. 200–238). Paris, France: Editions de la Maison des Sciences de l’Homme.

Kennedy, Alistair and Inkpen, Diana. 2005. Sentiment classification of movie and product reviews using contextual valence shifters. In Proceedings of the Workshop on the Analysis of Informal and Formal Information Exchange during Negotiations. Ottawa, Ontario, Canada.

Kennedy, Alistair and Inkpen, Diana. 2006. Sentiment classification of movie reviews using contextual valence shifters. Computational Intelligence, 22(2), 110–125. https://doi.org/10.1111/j.1467-8640.2006.00277.x DOI: https://doi.org/10.1111/j.1467-8640.2006.00277.x

Kiritchenko, Svetlana and Mohammad, Saif M. 2016a. Happy Accident: A Sentiment Composition Lexicon for Opposing Polarity Phrases. In Proceedings of the 10th edition of the Language Resources and Evaluation Conference (LREC). Portoro, Slovenia. https://doi.org/10.18653/v1/N16-1128

Kiritchenko, Svetlana and Mohammad, Saif M. 2016b. Sentiment composition of words with opposing polarities. In Proceedings of the 15th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), pp. 1102–1108. San Diego, California. https://doi.org/10.18653/v1/N16-1128 DOI: https://doi.org/10.18653/v1/N16-1128

Kiritchenko, Svetlana, Zhu, Xiaodan and Mohammad, Saif M. 2014. Sentiment analysis of short informal texts. Journal of Artificial Intelligence Research, 50, 723–762. https://doi.org/10.1613/jair.4272 DOI: https://doi.org/10.1613/jair.4272

Ladd, D. Robert, Silverman, Kim E.A., Tolkmitt, Frank, Bergmann, Günther and Scherer, Klaus R. 1985. Evidence for the independent function of intonation contour type, voice quality, and F0 range in signaling speaker affect. The Journal of the Acoustical Society of America, 78(2), 435–444. https://doi.org/10.1121/1.392466 DOI: https://doi.org/10.1121/1.392466

Lee, Akinobu and Kawahara, Tatsuya. 2009. Recent development of open-source speech recognition engine julius. In Proceedings: APSIPA ASC 2009: Asia-Pacific Signal and Information Processing Association, 2009 Annual Summit and Conference, 131–137. Asia-Pacific Signal and Information Processing Association, 2009 Annual ….

Leinonen, Lea, Hiltunen, Tapio, Linnankoski, Ilkka and Laakso, Maija-Liisa. 1997. Expression of emotional–motivational connotations with a one-word utterance. The Journal of the Acoustical Society of America, 102(3), 1853–1863. https://doi.org/10.1121/1.420109 DOI: https://doi.org/10.1121/1.420109

Li, Bryan, Dimitriadis, Dimitrios and Stolcke, Andreas. 2019. Acoustic and Lexical Sentiment Analysis for Customer Service Calls. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 5876–5880. IEEE. https://doi.org/10.1109/ICASSP.2019.8683679 DOI: https://doi.org/10.1109/ICASSP.2019.8683679

Liu, Bing. 2012. Sentiment analysis and opinion mining. Synthesis Lectures on Human Language Technologies, 5(1), 1–167. https://doi.org/10.1007/978-3-031-02145-9 DOI: https://doi.org/10.2200/S00416ED1V01Y201204HLT016

Liu, Bing and Zhang, Lei. 2012. A survey of opinion mining and sentiment analysis. In Mining text data, 415–463. Springer. https://doi.org/10.1007/978-1-4614-3223-4_13 DOI: https://doi.org/10.1007/978-1-4614-3223-4_13

Mairesse, François, Polifroni, Joseph and Di Fabbrizio, Giuseppe. 2012. Can prosody inform sentiment analysis? experiments on short spoken reviews. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2012 5093–5096. IEEE. https://doi.org/10.1109/ICASSP.2012.6289066 DOI: https://doi.org/10.1109/ICASSP.2012.6289066

McAuliffe, Michael, Socolof, Michaela, Mihuc, Sarah, Wagner, Michael and Sonderegger, Morgan. 2017. Montreal Forced Aligner: Trainable Text-Speech Alignment Using Kaldi. In Interspeech, 498–502. https://doi.org/10.21437/Interspeech.2017-1386 DOI: https://doi.org/10.21437/Interspeech.2017-1386

Mohammad, Saif M., Kiritchenko, Svetlana and Zhu, Xiaodan. 2013. NRC-Canada: Building the state-of-the-art in sentiment analysis of tweets. ArXiv Preprint ArXiv:1308.6242.

Mohammad, Saif M. and Turney, Peter D. 2010. Emotions evoked by common words and phrases: Using Mechanical Turk to create an emotion lexicon. In Proceedings of the NAACL HLT 2010 workshop on computational approaches to analysis and generation of emotion in text, 26–34. LA, California: Association for Computational Linguistics.

Mohammad, Saif M. and Turney, Peter D. 2013. Crowdsourcing a word–emotion association lexicon. Computational Intelligence, 29(3), 436–465. https://doi.org/10.1111/j.1467-8640.2012.00460.x DOI: https://doi.org/10.1111/j.1467-8640.2012.00460.x

Pang, Bo and Lee, Lillian. 2008. Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval, 2(1–2), 1–135. https://doi.org/10.1561/1500000011 DOI: https://doi.org/10.1561/1500000011

Peng, Zeshan. 2017. Acoustic feature-based sentiment analysis of call center data (PhD Thesis). University of Missouri–Columbia.

Pereira, Moisés Henrique Ramos, Pádua, Flávio Luis Cardeal, Pereira, Adriano César Machado, Benevenuto, Fabrício and Dalip, Daniel Hasan. 2016. Fusing audio, textual, and visual features for sentiment analysis of news videos. In Tenth International AAAI Conference on Web and Social Media.

Pérez-Rosas, Verónica, Mihalcea, Rada and Morency, Louis-Philippe. 2013. Utterance-level multimodal sentiment analysis. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 973–982.

Plutchik, Robert. 1980. Emotion: A psychoevolutionary synthesis. Harper and Row.

Plutchik, Robert. 1997. The circumplex as a general model of the structure of emotions and personality. In R. Plutchik and H. R. Conte (eds.), Circumplex models of personality and emotions, 7–45. Washington, DC, US: American Psychological Association. https://doi.org/10.1037/10261-001 DOI: https://doi.org/10.1037/10261-000

Plutchik, Robert. 2000. Emotions in the practice of psychotherapy: Clinical implications of affect theories, Vol. 13. Washington, DC, US: American Psychological Association. https://doi.org/10.1037/10366-000 DOI: https://doi.org/10.1037/10366-000

Plutchik, Robert. 2001a. Integration, differentiation, and derivatives of emotion. Evolution and Cognition, 7(2), 114–125.

Plutchik, Robert. 2001b. The nature of emotions. American Scientist, 89(4), 344–350.

R Development Core Team. 2018. R: A language and environment for statistical computing (version 3.4.4) [computer software]. Vienna, Austria. https://doi.org/10.1511/2001.4.344 DOI: https://doi.org/10.1511/2001.4.344

Razak, Aishah Abd, Abidin, Mohd Izani Zainal and Komiya, Ryoichi. 2003. Emotion pitch variation analysis in Malay and English voice samples. In The 9th Asia-Pacific Conference on Communications 2003, Vol. 1, 108–112.

Ribeiro, Filipe N., Araújo, Matheus, Gonçalves, Pollyanna, Gonçalves, Marcos André and Benevenuto, Fabrício. 2016. Sentibench-a benchmark comparison of state-of-the-practice sentiment analysis methods. EPJ Data Science, 5(1), 1–29. https://doi.org/10.1140/epjds/s13688-016-0085-1 DOI: https://doi.org/10.1140/epjds/s13688-016-0085-1

Rosas, Verónica Pérez, Mihalcea, Rada and Morency, Louis-Philippe. 2013. Multimodal sentiment analysis of spanish online videos. IEEE Intelligent Systems, 28(3), 38–45. https://doi.org/10.1109/MIS.2013.9 DOI: https://doi.org/10.1109/MIS.2013.9

Scherer, Klaus R. 1986. Vocal affect expression: A review and a model for future research. Psychological Bulletin, 99(2), 143–165. https://doi.org/10.1037/0033-2909.99.2.143 DOI: https://doi.org/10.1037/0033-2909.99.2.143

Schuller, Björn, Batliner, Anton, Seppi, Dino, Steidl, Stefan, Vogt, Thurid, Wagner, Johannes, … Kessous, Loic. 2007. The relevance of feature type for the automatic classification of emotional user states: low level descriptors and functionals. In Eighth Annual Conference of the International Speech Communication Association. https://doi.org/10.21437/Interspeech.2007-612 DOI: https://doi.org/10.21437/Interspeech.2007-612

Schuller, Björn, Steidl, Stefan and Batliner, Anton. 2009. The interspeech 2009 emotion challenge. In Tenth Annual Conference of the International Speech Communication Association. https://doi.org/10.21437/Interspeech.2009-103 DOI: https://doi.org/10.21437/Interspeech.2009-103

Sheikh, Imran, Dumpala, Sri Harsha, Chakraborty, Rupayan and Kopparapu, Sunil Kumar. 2018. Sentiment analysis using imperfect views from spoken language and acoustic modalities. In Proceedings of Grand Challenge and Workshop on Human Multimodal Language (Challenge-HML), 35–39. https://doi.org/10.18653/v1/W18-3305 DOI: https://doi.org/10.18653/v1/W18-3305

Skinner, E. Ray. 1935. A calibrated recording and analysis of the pitch, force and quality of vocal tones expressing happiness and sadness. Communications Monographs, 2(1), 81–137. https://doi.org/10.1080/03637753509374833 DOI: https://doi.org/10.1080/03637753509374833

Sobin, Christina and Alpert, Murray. 1999. Emotion in speech: The acoustic attributes of fear, anger, sadness, and joy. Journal of Psycholinguistic Research, 28(4): 347–365. https://doi.org/10.1023/A:1023237014909 DOI: https://doi.org/10.1023/A:1023237014909

Sperber, Dan and Wilson, Deirdre. 1986. Relevance: Communication and Cognition. Oxford: Basil Blackwell.

Stolarski, Łukasz. 2018. Lack of effects of gender on the reading rate of long texts. Sociolinguistic Studies, 12(3–4), 461–479. https://doi.org/10.1558/sols.32924 DOI: https://doi.org/10.1558/sols.32924

Stolarski, Łukasz. 2020. The influence of character’s gender and the basic emotions of ‘happiness’ and ‘sadness’ on voice pitch in the reading of fiction. Brno Studies in English, 46(1), 49–89. https://doi.org/10.5817/BSE2020-1-3 DOI: https://doi.org/10.5817/BSE2020-1-3

Stolarski, Łukasz. 2021. Comparison of key statistical instruments used in lexicon-based tools for sentiment analysis in the English language. Token: A Journal of English Linguistics, 13: 219–248.

Taboada, Maite, Brooke, Julian, Tofiloski, Milan, Voll, Kimberly and Stede, Manfred. 2011. Lexicon-based methods for sentiment analysis. Computational Linguistics, 37(2): 267–307. https://doi.org/10.1162/COLI_a_00049 DOI: https://doi.org/10.1162/COLI_a_00049

Thelwall, Mike. 2017. Heart and soul: Sentiment strength detection in the social web with SentiStrength (summary book chapter). In J. Holyst (ed.), Cyberemotions: Collective emotions in cyberspace, 119–134. Berlin, Germany: Springer. https://doi.org/10.1007/978-3-319-43639-5_7 DOI: https://doi.org/10.1007/978-3-319-43639-5_7

Thelwall, Mike and Buckley, Kevan. 2013. Topic-based sentiment analysis for the social web: The role of mood and issue-related words. Journal of the Association for Information Science and Technology, 64(8), 1608–1617. https://doi.org/10.1002/asi.22872 DOI: https://doi.org/10.1002/asi.22872

Thelwall, Mike, Buckley, Kevan and Paltoglou, Georgios. 2012. Sentiment strength detection for the social web. Journal of the Association for Information Science and Technology, 63(1), 163–173. https://doi.org/10.1002/asi.21662 DOI: https://doi.org/10.1002/asi.21662

Thelwall, Mike, Buckley, Kevan, Paltoglou, Georgios, Cai, Di, and Kappas, Arvid. 2010. Sentiment strength detection in short informal text. Journal of the American Society for Information Science and Technology, 61(12): 2544–2558. https://doi.org/10.1002/asi.21416 DOI: https://doi.org/10.1002/asi.21416

Thelwall, Mike, Buckley, Kevan, Paltoglou, George, Skowron, Marcin, Garcia, David, Gobron, Stephane, … Holyst, Janusz A. 2013. Damping sentiment analysis in online communication: discussions, monologs and dialogs. In International Conference on Intelligent Text Processing and Computational Linguistics (pp. 1–12). Berlin, Germany: Springer. https://doi.org/10.1007/978-3-642-37256-8_1 DOI: https://doi.org/10.1007/978-3-642-37256-8_1

Traunmüller, Hartmut and Eriksson, Anders. 1995. The frequency range of the voice fundamental in the speech of male and female adults. Unpublished Manuscript.

Wallbott, Harald G. and Scherer, Klaus R. 1986. Cues and channels in emotion recognition. Journal of Personality and Social Psychology, 51(4), 690–699. https://doi.org/10.1037/0022-3514.51.4.690 DOI: https://doi.org/10.1037/0022-3514.51.4.690

Wöllmer, Martin, Weninger, Felix, Knaup, Tobias, Schuller, Björn, Sun, Congkai, Sagae, Kenji and Morency, Louis-Philippe. 2013. Youtube movie reviews: Sentiment analysis in an audio-visual context. IEEE Intelligent Systems, 28(3), 46–53. https://doi.org/10.1109/MIS.2013.34 DOI: https://doi.org/10.1109/MIS.2013.34

Wu, Wei, Zheng, Thomas Fang, Xu, Ming-Xing, and Bao, Huanjun. 2006. Study on speaker verification on emotional speech. In Proceedings of Ninth International Conference on Spoken Language Processing, INTERSPEECH, 2102–2105. Pittsburgh, Pennsylvania. https://doi.org/10.21437/Interspeech.2006-191 DOI: https://doi.org/10.21437/Interspeech.2006-191

Zadeh, Amir, Chen, Minghai, Poria, Soujanya, Cambria, Erik, and Morency, Louis-Philippe. 2017. Tensor fusion network for multimodal sentiment analysis. ArXiv Preprint ArXiv:1707.07250. https://doi.org/10.18653/v1/D17-1115 DOI: https://doi.org/10.18653/v1/D17-1115

Zhu, Xiaodan, Kiritchenko, Svetlana and Mohammad, Saif M. 2014. NRC-Canada-2014: Recent Improvements in the Sentiment Analysis of Tweets. In SemEval@ COLING, 443–447. https://doi.org/10.3115/v1/S14-2077 DOI: https://doi.org/10.3115/v1/S14-2077

Zuberbier, Erika. 1957. Zur Schreib-und Sprechmotorik der Depressiven. Zeitschrift Für Psychotherapie Und Medizinische Psychologie, 7, 239–249.