hig.sePublications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard-cite-them-right
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • sv-SE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • de-DE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Evaluating AI Judgements: A Case Study of LLMs in Product Development
University of Gävle, Faculty of Engineering and Sustainable Development, Department of Computer and Geospatial Sciences, Decision, Risk and Policy Analysis.
2025 (English)Independent thesis Advanced level (degree of Master (One Year)), 10 credits / 15 HE creditsStudent thesis
Abstract [en]

This thesis investigates how Large Language Models (LLMs) can be evaluated using Multi-Criteria Decision Analysis (MCDA) tools. Instead of employing AI for direct decision-making, this study reimagines the role of MCDA tools as a crucial lens into the reasoning processes of LLMs that otherwise lack transparency. Adopting a pragmatist approach, priority is placed on practical insights for decision-makers considering large-scale automation of decisions using LLMs. To really test the LLMs, this case study focuses on a product development related scenario, where the LLMs are asked to evaluate products that have a complex real-world regulatory and practical context which need to be understood for informed decisions. Two approaches are used to vary how the models are prompted: one asks sequentially for pairwise comparisons, while the other one specifies that the models should use the AHP methodology and asks for all the pairwise comparisons to be provided at once. The analysis examines the coherence of the LLM's responses using AHP and case specific expectations to see not only whether the responses are consistent but also whether they correspond with the varying real-world requirements. The study yielded mixed results with LLMs not performing uniformly across measures that were used. The analysis of AHP data showed that decision-making performance and consistency are not equivalent, and performance needs to be considered from a broader range of dimensions including how well the outcomes match the expectations in a specific decision-making context. Overall, none of the state-of-the-art LLMs tested performed well enough in terms of referencing relevant regulatory information or ranking alternatives according to the needs of different customer groups. The study did result in several metrics which can be used for LLM evaluation purposes and show that more recent LLMs have substantial computational capabilities which may be useful for multi-criteria decision-making.

Place, publisher, year, edition, pages
2025. , p. 63
National Category
Other Engineering and Technologies
Identifiers
URN: urn:nbn:se:hig:diva-47090OAI: oai:DiVA.org:hig-47090DiVA, id: diva2:1965527
Subject / course
Decision, risk and policy analysis
Educational program
Decision, risk och policy analysis - master’s programme (one year)
Supervisors
Examiners
Available from: 2025-06-09 Created: 2025-06-09 Last updated: 2025-10-02Bibliographically approved

Open Access in DiVA

fulltext(1899 kB)365 downloads
File information
File name FULLTEXT01.pdfFile size 1899 kBChecksum SHA-512
84fa2dbab1facbab9b4bc549d79b0f469e88f462720bdc8e385f0a6572af571f520c4a5e7dc6aa506af320aece29534a4e1ba152021d2351d0ae4bbc6c5db3ae
Type fulltextMimetype application/pdf

By organisation
Decision, Risk and Policy Analysis
Other Engineering and Technologies

Search outside of DiVA

GoogleGoogle Scholar
Total: 368 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

urn-nbn

Altmetric score

urn-nbn
Total: 474 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard-cite-them-right
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • sv-SE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • de-DE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf