AI Dictionary

Technical meaning. llms.txt is a plain-text file placed at the root of a website that gives AI language models a concise, curated summary of the site: who or what it is about, the most important pages, and key facts. It is a newer, proposed standard, written specifically for large language models (the “LLM” in the…

Technical meaning. A web crawler (also called a bot or spider) is an automated program that visits web pages and reads their content. A small file called robots.txt, kept at the root of a website, tells those crawlers which parts of the site they are allowed to read. “AI crawler access” means crawlers run by…