Add support for llms.txt

[thepushkarp]

BearBlog has good support for SEO via Sitemap and robots.txt. Recently, GEO (Generative Engine Optimization) has focused a lot on adding a /llms.txt for easier discoverability by LLMs when doing web search. Would be good to have control over what they see by adding support for editing a custom llms.txt, maybe under Settings > Advanced settings

Cassidoo has a good blog about it too.

[thepushkarp]

@HermanMartinus LMK if this is in plans. I can pick it up and create a PR if this is something we'd want to integrate into the platform

[HermanMartinus]

I'll take a look at this this week 👍

[HermanMartinus]

I just took a look through the logs and there are no requests for /llms.txt across all blogs by any of the major LLM providers. Currently all of the LLM searches don't request it or even /robots.txt or /sitemap.xml. It seems that they go directly to the pages they're looking for via their own search indexes.

I'm going to hold off on implementing llms.txt until it becomes an LLM search standard. There's nothing forcing these companies to adhere to it or robots.txt so I think it unlikely that it will actually be necessary or useful.

I do appreciate the suggestion and I'll keep my eye on the logs to see whether this pattern does change in the future.

[thepushkarp]

makes sense, thanks for looking into the logs and getting back