This thesis investigates how Large Language Models (LLMs) can be evaluated using Multi-Criteria Decision Analysis (MCDA) tools. Instead of employing AI for direct decision-making, this study reimagines the role of MCDA tools as a crucial lens into the reasoning processes of LLMs that otherwise lack transparency. Adopting a pragmatist approach, priority is placed on practical insights for decision-makers considering large-scale automation of decisions using LLMs. To really test the LLMs, this case study focuses on a product development related scenario, where the LLMs are asked to evaluate products that have a complex real-world regulatory and practical context which need to be understood for informed decisions. Two approaches are used to vary how the models are prompted: one asks sequentially for pairwise comparisons, while the other one specifies that the models should use the AHP methodology and asks for all the pairwise comparisons to be provided at once. The analysis examines the coherence of the LLM's responses using AHP and case specific expectations to see not only whether the responses are consistent but also whether they correspond with the varying real-world requirements. The study yielded mixed results with LLMs not performing uniformly across measures that were used. The analysis of AHP data showed that decision-making performance and consistency are not equivalent, and performance needs to be considered from a broader range of dimensions including how well the outcomes match the expectations in a specific decision-making context. Overall, none of the state-of-the-art LLMs tested performed well enough in terms of referencing relevant regulatory information or ranking alternatives according to the needs of different customer groups. The study did result in several metrics which can be used for LLM evaluation purposes and show that more recent LLMs have substantial computational capabilities which may be useful for multi-criteria decision-making.