The Effect of Model Configuration on HHEM Scores

PDF documents serve an important role in sharing and protecting information in today’s digital world. However, obtaining useful information from these pdfs documents can be difficult. Summarizing pdf documents enables users to quickly extract key information and gain a deeper understanding of the document’s content. Text summarization is a critical Natural Language Processing (NLP) task with applications ranging from information retrieval to content generation. Leveraging Large Language Models (LLMs) has shown remarkable promise in enhancing summarization techniques. While, automatically-generated summaries were riddled with artifacts such as grammar errors, repetition, and hallucination. Hallucination in text summarization refers to the phenomenon where the model generates information that is not supported by the input source document. Hallucination poses significant obstacles to the accuracy and reliability of the generated summaries. Detecting these hallucinations of LLMs for pdf summarization is a critical issue to evaluate summarization factual consistency rate. In the proposed project, we introduce LLM-based application called Cloudilic-HHEM that contains the following contributions: Enable users for chatting with different uploaded pdfs to extract useful and meaningful information, Summarizing pdf documents by different LLMs Like GPT 3.5, Google Gemini and LLAMA 2, Using Vectara-HHEM model to detect the score of hallucination of the used LLM in text summarization, Using dynamic temperatures when calling LLMs to compute the relative of hallucination score with the temperature parameter of LLM, The project has been presented by good stremlit GUI for user experience.