hig.sePublications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard-cite-them-right
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • sv-SE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • de-DE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Text mining Twitter social media for Covid-19: Comparing latent semantic analysis and latent Dirichlet allocation
University of Gävle, Faculty of Engineering and Sustainable Development, Department of Computer and Geospatial Sciences.
2020 (English)Independent thesis Basic level (degree of Bachelor), 10 credits / 15 HE creditsStudent thesis
Abstract [en]

In this thesis, the Twitter social media is data mined for information about the covid-19 outbreak during the month of March, starting from the 3’rd and ending on the 31’st. 100,000 tweets were collected from Harvard’s opensource data and recreated using Hydrate. This data is analyzed further using different Natural Language Processing (NLP) methodologies, such as termfrequency inverse document frequency (TF-IDF), lemmatizing, tokenizing, Latent Semantic Analysis (LSA) and Latent Dirichlet Allocation (LDA). Furthermore, the results of the LSA and LDA algorithms is reduced dimensional data that will be clustered using clustering algorithms HDBSCAN and K-Means for later comparison. Different methodologies are used to determine the optimal parameters for the algorithms. This is all done in the python programing language, as there are libraries for supporting this research, the most important being scikit-learn. The frequent words of each cluster will then be displayed and compared with factual data regarding the outbreak to discover if there are any correlations. The factual data is collected by World Health Organization (WHO) and is then visualized in graphs in ourworldindata.org. Correlations with the results are also looked for in news articles to find any significant moments to see if that affected the top words in the clustered data. The news articles with good timelines used for correlating incidents are that of NBC News and New York Times. The results show no direct correlations with the data reported by WHO, however looking into the timelines reported by news sources some correlation can be seen with the clustered data. Also, the combination of LDA and HDBSCAN yielded the most desireable results in comparison to the other combinations of the dimnension reductions and clustering. This was much due to the use of GridSearchCV on LDA to determine the ideal parameters for the LDA models on each dataset as well as how well HDBSCAN clusters its data in comparison to K-Means.

Place, publisher, year, edition, pages
2020. , p. 46
Keywords [en]
Data mining, Text mining, artificial intelligence, Natural language processing, Latent Semantic Analysis, Latent Dirichlet Allocation, KMeans, HDBSCAN, Dimension reduction
National Category
Engineering and Technology
Identifiers
URN: urn:nbn:se:hig:diva-32567OAI: oai:DiVA.org:hig-32567DiVA, id: diva2:1438923
External cooperation
Consid AB
Subject / course
Computer science
Educational program
Högskoleingenjör
Presentation
2020-06-04, Zoom, 13:47 (Swedish)
Supervisors
Examiners
Available from: 2020-06-11 Created: 2020-06-11 Last updated: 2025-10-02Bibliographically approved

Open Access in DiVA

sheikha_hassan_thesis(962 kB)1466 downloads
File information
File name FULLTEXT01.pdfFile size 962 kBChecksum SHA-512
9fe796c05b191071e1e342d5871400920b4805913473532e9b05d9118ea947655e8976f04be818ed9efb10261548b0e0b604dd97535ec64ee638c119c258af58
Type fulltextMimetype application/pdf

Search in DiVA

By author/editor
Sheikha, Hassan
By organisation
Department of Computer and Geospatial Sciences
Engineering and Technology

Search outside of DiVA

GoogleGoogle Scholar
Total: 1469 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

urn-nbn

Altmetric score

urn-nbn
Total: 4467 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard-cite-them-right
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • sv-SE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • de-DE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf