Accelerating Financial Intelligence: Generative AI for Automated Document Summarization

The company is a fully owned subsidiary of one of India’s leading stock exchanges. It is an analytics, news and data provider that sells various products to financial organizations and analysts.

THE CHALLENGE

The company’s primary product which combines data from over 200 sources, is used for research, risk management and asset management of treasury sales, equity trading, commodity trading, corporate treasury, fixed income trading etc. Their products are built on key financial documents such as annual reports, annual general meeting (AGM) statements, analyst call transcripts and published quarterly results for both listed, as well as unlisted companies. Currently, the stock exchange has over 2,600 companies that are listed on it. The financial products are also based on reports from prominent ratings agencies like Credit Rating Information Services Of India Limited (CRISIL) and ICRA Limited etc.

Several direct competitors had introduced Artificial Intelligence (AI) powered products for automatically generating summaries from essential financial documents. These products combine Intelligent Document Processing (IDP) and trained Large Language Models (LLMs) to provide precise summaries that are important to investors and other finance professionals. Since the company did not have any existing technical solutions to generate similar summaries from financial documents, they had to depend on their analysts to manually create custom reports, which was a very time consuming and expensive process.

The company needed an analytics partner to develop a new Generative AI solution built on open-source technologies that would automatically create summaries from the wide variety of financial documents, with a very high level of accuracy.

THE SOLUTION

The Prescience evaluated different LLMs to build a Proof of Concept (PoC) which prepared summarizations for a limited set of financial documents that were PDF files. Based on this prototype, the team designed and developed a full-fledged solution that consisted of the

1. Preprocessing Phase
2. Summarization Agent
3. Evaluation Agent

Figure: Solution Overview

For the Preprocessing Phase, the solution uses Optical Character Recognition (OCR) deep learning models to extract text from each PDF file that is being processed. A custom chunking strategy was developed to handle the different file types. For Analyst Call Transcripts which consist of discussions as well as questions and answers, specific rules were built to identify questions and their responses, to build unique chunks. For other documents like Annual Reports, the headers and paragraphs are used to create chunks and context. The solution uses vector embeddings to store and organize data to perform semantic similarity using the Approximate Nearest Neighbour (ANN) search.

For the Summarization Agent, in addition to the standard themes that are extracted by the LLM, the solution creates 15 other themes that are specific to the contents of the financial document. Natural Language Processing (NLP) is leveraged to convert conversations into the third person format. If multiple similar themes and summaries are identified, then they are automatically appended together.

For the Evaluation Agent, since off the shelf LLM frameworks were not able to handle the complex use cases, a custom evaluation framework was developed. This can handle hallucinations with a set of pre-defined business rules. The solution adopts a two-step validation process to keep the findings of the evaluation model in check. If any summaries are flagged as potential hallucinations, users can review the contents and edit it as part of the agent evaluation process.

The team also benchmarked the performance of the overall summarization solution against that of industry leading LLMs like GPT-4o and GPT-3.5 Turbo.

The technologies used for this engagement were,

1. Deepdoctection for OCR
2. Python for chunking
3. E5-large for vector embeddings
4. Meta Llama 2 7B Chat for theme and summary generation
5. Meta Llama 3 8B for evaluation agent
6. PostgreSQL for database
7. Pgvector for vector databases
8. Flask for application programming interface (API)
9. React for user interface

THE IMPACT

With this Generative AI solution, the company is now able to generate accurate summaries from different types of financial documents with hundreds of pages of content, in a matter of minutes. Automating this solution has helped the company to save millions of rupees annually. It will lead to higher levels of customer satisfaction and a greater adoption of their financial products.