Have you recently become interested in building an in-house ChatGPT? Looking to index all your enterprise information through vector search and embeddings? It's a good idea – a generative AI assistant is essential to unlocking the full potential and productivity of everyday work. However, slapping a chatbot over your corpus won't give you the results you're looking for.
Rather, a sustainable and dependable solution requires a search platform that can consistently feed the model the right information, every time. That requires a robust, comprehensive, and nuanced crawler that:
- Prevents staleness
- Is resilient
- Vertically & horizontally scalable
Register for this on-demand Tech Talk, where Alice Wang, Founding Engineer at Glean, dives into what makes a great crawler and how we built ours.
