Provide Articles in a machine-readable format

[lukasjuhrich]

(This is a question / a feature request)

Is there any way to present the nlab articles in a different format than a complete HTML website with all its surroundings?

I wanted to write a chatbot which looks up or searches nlab articles, but realized that I would

• either have to do a lot of ugly scraping of the HTML output[1], but even if that worked, it is usually unstable and either does not for 100% of the articles and is not stable under structural changes to the HTML page
• on the other hand, fetch /source/<article-name>, and try to parse the instiki syntax (if there is even a spec for this) myself. However, completely building such a parser seems overkill.

1. Is it possible to provide the pages in a different format (whatever this may be) , or
2. is there a parser for the instiki syntax – or whatever derivative of this is used in the nlab – and
3. would it even be possible to provide a REST-API?

[1] For instance, it's not quite clear when the first actual part of the article begins – usually, we have some <h1 id="contents">, but that need not exist.

[williamsonrichard]

Hi, thanks for raising this.

A) Could you explain in more detail the kind of format you would like?

B) I suspect that what you are looking for is currently not available. The parser (which is mostly not the original one in Instiki, it is home-grown) takes markdown more or less directly to HTML. However, if I know what you are looking for (i.e. given your answer to A)), I may be able to suggest a way for you to extract the information you need.

C) I can certainly provide a REST API if we have the kind of thing that you are looking for, but as in B), I suspect that is not currently the case.

[lukasjuhrich]

I'm looking for some way to programmatically search for articles, and, given the best match, present a link and the short description of the article (e.g. the definition or whatever summary the author chose to use as an introduction).

I intend to realize this as a discord bot.

For the searching part, I initially thought of fetching all pages, extracting the structurally relevant content, and building an index from that, but using Mediawiki for that may even be a better approach (AFAICS, it is literally built for managing metadata in a broad definition of the word).

However, the extraction of the summary still remains the relevant part. Whether that is realized in a REST API or by different means should not matter too much.

(Apologies for the late response, I fear this project doesn't enjoy the highest priority at the moment)