LLMs.txt

LLMsTxtTools lets an agent discover and read documentation from an llms.txt index, optionally loading it into Knowledge.

LLMsTxtTools reads llms.txt files. The format is a standardized way for websites to publish an LLM-friendly documentation index. The toolkit operates in two modes depending on whether you pass a Knowledge instance.

Agentic Mode

Without knowledge, the agent reads the index and decides which pages to fetch.

1from kern.agent import Agent
2from kern.models.openai import OpenAIResponses
3from kern.tools.llms_txt import LLMsTxtTools
4
5agent = Agent(
6    model=OpenAIResponses(id="gpt-5.4"),
7    tools=[LLMsTxtTools()],
8    instructions=[
9        "First use get_llms_txt_index to see what pages are available.",
10        "Then use read_llms_txt_url to fetch only the pages relevant to the question.",
11    ],
12    markdown=True,
13)
14
15agent.print_response(
16    "Using the llms.txt at https://kern.ndx.rocks/llms.txt, "
17    "find and read the documentation about how to create an agent with tools",
18    stream=True,
19)

Knowledge Mode

Pass a Knowledge instance and the toolkit exposes read_llms_txt_and_load_knowledge, which ingests the indexed pages into your knowledge base for retrieval.

1from kern.knowledge.knowledge import Knowledge
2from kern.tools.llms_txt import LLMsTxtTools
3
4tools = LLMsTxtTools(knowledge=knowledge)

Toolkit Params

Parameter	Type	Default	Description
`knowledge`	`Optional[Knowledge]`	`None`	When set, switches to Knowledge mode (load pages instead of returning).
`max_urls`	`int`	`20`	Maximum number of pages to read from an index.
`timeout`	`int`	`60`	HTTP timeout in seconds.
`skip_optional`	`bool`	`False`	Skip pages listed under the index's optional section.
`allowed_hosts`	`Optional[List[str]]`	`None`	SSRF guard. Only fetch from these hosts. `None` allows any host.

Toolkit Functions

Function	Mode	Description
`get_llms_txt_index`	Agentic	Fetch and parse an llms.txt index.
`read_llms_txt_url`	Agentic	Read a single page referenced by the index.
`read_llms_txt_and_load_knowledge`	Knowledge	Read indexed pages and load them into Knowledge.

All functions have sync and async variants.

LLMs.txt

Agentic Mode

Knowledge Mode

Toolkit Params

Toolkit Functions

Developer Resources