About This Project

This project began when I had a vague recollection of a data point in a consultant report on the Gas Pipeline from more than a decade ago. Confirming that hunch was like finding a needle in a digital haystack.

With extensive documentation from the Alaska Gas Pipeline Project—including hundreds of public reports spanning tens of thousands of pages of text and charts—relying on Ctrl+F keyword search was inefficient and daunting. So, I decided to create a better option.

Retrieval Augmented Generation (RAG) systems leverage the power of Large Language Models (LLMs) to conduct much more sophisticated and comprehensive similarity searches. While the technical details are complex, this production version implements a four-stage process: OCR processing of hundreds of PDFs, intelligent document chunking that preserves hierarchical structure, conversion to vector embeddings that are stored in a vector database, and finally connection to an LLM that synthesizes and translates search results into coherent natural language responses. This version runs on AWS for scalability and is most appropriate for public documents like these. For scenarios involving confidential documents, I also created a fully local version that operates without external connections.