Discovery
robots.txt
User-agent: *
sitemaps.xml
<urlset>
Ingestion
HTML / JSON-LD
{"@type": "Article"}
Filter Scripts
<script> ✕
Indexing
Clean Text
[chunk_01, ...]
Vector DB
[0.012, -0.438, ...]