cavepediav2/poller/README.md

poller

Processes documents and indexes them to be searched.

Every 5 minutes, this polls for new documents as follows:

1. Moves any documents from s3://cavepediav2-import to s3://cavepediav2-files and updates the metadata table.
   • This table has a split column, indicating if the file has been split into individual pages.
2. Checks the metadata table for any unsplit files, then splits them and stores the pages in s3://cavepediav2-pages and creates an row in the embeddings table for each page.
3. Checks claude for any OCR batches that have finished, then stores the results in the embeddings table.
4. Checks the embeddings table for un-OCR'd pages and batches them in groups of 1000 to be OCR'd by claude.
   • Only 1 batch is created per 5 minutes, as it can be easy to overload the server hosting the files.
   • A temporary public S3 file link is generated using a presigned s3 url.
5. Checks the embeddings table for any rows that have been OCR'd, but do not have embeddings generated, then generates embeddings with cohere.
   • No batching is used with cohere.