Yozora

See Yozora documentation (or https://yozorajs.github.io) for more details.

yozora.demo.mp4

中文文档

🎉 Why is it named "Yozora"?

Yozora is the romanization of the Japanese word 「よぞら」, taken from the lyrics of 『花鳥風月』 by the band 世界の終わり.

This project is a monorepo that aims to implement a highly extensible, pluggable Markdown parser. Based on the idea of middleware, the core algorithm @yozora/core-parser schedules tokenizers (such as @yozora/tokenizer-autolink) to complete the parsing tasks. More accurately, Yozora is an algorithm that parses Markdown or its extended syntax into an abstract syntax tree (AST).

✨ Features

• 🔖 Fully supports all the rules mentioned in the GFM specification, and has passed almost all test cases created based on the examples in the specification (except for https://github.github.com/gfm/#example-657, as there is no plan to support native HTML tags in the React Renderer for Yozora AST, so I'm a little lazy to do the tag filtering. If you need it, you can do the filtering yourself).

  See @yozora/parser-gfm or @yozora/parser-gfm-ex for further information.

• 🚀 Robust.

  • All code is written in TypeScript, with strict static type checking.

  • ESLint and Prettier constrain coding styles to avoid error-prone problems such as hacky syntax and shadowed variables.

  • Tested with Vitest and a large number of test cases.

• 💚 Tidy: No third-party runtime dependencies.

• ⚡️ Efficient.

  • The parsing complexity is the length of the source content multiplied by the number of tokenizers, which has reached the lower bound of theoretical complexity.

  • The parser API supports streaming input (using generators/iterators), and supports parsing while reading (currently only block-level data is supported).

  • Array creation and concatenation are handled carefully. Arrays are reused as much as possible during the entire matching phase, and array indexes delineate matching ranges. Several strategies are also applied to reduce repeated matching and parsing operations.

• 🩹 Compatibility: The parsed syntax tree is compatible with the one defined in mdast.

  Even if some data types are not compatible in the future, it is easy to traverse the AST for adaptation and modification through the API provided in @yozora/ast-util.

• 🎨 Extensibility: Yozora comes with a plugin system that allows it to schedule tokenizers through an internal algorithm to complete the parsing tasks.

  • It's easy to create and integrate custom tokenizers.

  • All tokenizers can be mounted or unmounted freely.

    Some tokenizers for data types not mentioned in GFM have been implemented in this repository, such as @yozora/tokenizer-admonition, @yozora/tokenizer-footnote, etc. All of them are built into @yozora/parser by default; you can uninstall them at will if you don't like them.

Usage

• @yozora/parser: (Recommended) A Markdown parser with rich built-in tokenizers.

  import YozoraParser from '@yozora/parser'
  
  const parser = new YozoraParser()
  parser.parse('source content')

• @yozora/parser-gfm: A Markdown parser that supports the GFM specification. Built-in tokenizers support all grammars mentioned in the GFM specification (excluding the extended grammar mentioned in the specification, such as table).

  import GfmParser from '@yozora/parser-gfm'
  
  const parser = new GfmParser()
  parser.parse('GitHub Flavored Markdown content')

• @yozora/parser-gfm-ex: A Markdown parser that supports the GFM specification. Built-in tokenizers support all grammars mentioned in the GFM specification (including the extended grammar mentioned in the specification, such as table).

  import GfmExParser from '@yozora/parser-gfm-ex'
  
  const parser = new GfmExParser()
  parser.parse('GitHub Flavored Markdown content with extensions')

• Convert an AST into markup content

  import { DefaultMarkupWeaver } from '@yozora/markup-weaver'
  
  const weaver = new DefaultMarkupWeaver()
  weaver.weave({
    "type": "root",
    "children": [
      {
        "type": "paragraph",
        "children": [
          {
            "type": "text",
            "value": "emphasis: "
          },
          {
            "type": "strong",
            "children": [
              {
                "type": "text",
                "value": "foo \""
              },
              {
                "type": "emphasis",
                "children": [
                  {
                    "type": "text",
                    "value": "bar"
                  }
                ]
              },
              {
                "type": "text",
                "value": "\" foo"
              }
            ]
          }
        ]
      }
    ]
  })
  // => emphasis: **foo "*bar*" foo**

Overview

• Parsers

  Parser	Description
  @yozora/parser	A Markdown parser with rich built-in tokenizers
  @yozora/parser-gfm	A Markdown parser with built-in tokenizers to fully support GFM (without GFM extensions)
  @yozora/parser-gfm-ex	A Markdown parser with built-in tokenizers to fully support GFM and GFM extensions

• Weavers

  Weaver	Description
  @yozora/markup-weaver	Weave AST into markup content.

• Tokenizers

  Tokenizer	Description
  @yozora/tokenizer-admonition	Resolve admonitions
  @yozora/tokenizer-autolink	Resolve GFM Autolinks
  @yozora/tokenizer-autolink-extension	Resolve GFM Autolinks (extension)
  @yozora/tokenizer-blockquote	Resolve GFM blockquotes
  @yozora/tokenizer-break	Resolve GFM hard line breaks and GFM soft line breaks
  @yozora/tokenizer-definition	Resolve GFM link reference definitions
  @yozora/tokenizer-delete	Resolve GFM strikethrough (extension)
  @yozora/tokenizer-ecma-import	Resolve ECMAScript import statements
  @yozora/tokenizer-emphasis	Resolve GFM emphasis and strong emphasis
  @yozora/tokenizer-fenced-code	Resolve GFM fenced code blocks
  @yozora/tokenizer-footnote	Resolve footnotes
  @yozora/tokenizer-footnote-definition	Resolve footnote definitions
  @yozora/tokenizer-footnote-reference	Resolve footnote references
  @yozora/tokenizer-heading	Resolve GFM ATX headings
  @yozora/tokenizer-html-block	Resolve GFM HTML blocks
  @yozora/tokenizer-html-inline	Resolve GFM raw HTML
  @yozora/tokenizer-image	Resolve GFM images
  @yozora/tokenizer-image-reference	Resolve GFM reference images
  @yozora/tokenizer-indented-code	Resolve GFM indented code blocks
  @yozora/tokenizer-inline-code	Resolve GFM code spans
  @yozora/tokenizer-inline-math	Resolve inline formulas
  @yozora/tokenizer-link	Resolve GFM links
  @yozora/tokenizer-link-reference	Resolve GFM reference links
  @yozora/tokenizer-list	Resolve GFM lists (with GFM list items and GFM task list items)
  @yozora/tokenizer-math	Resolve block formulas
  @yozora/tokenizer-paragraph	Resolve GFM paragraphs
  @yozora/tokenizer-setext-heading	Resolve GFM setext headings
  @yozora/tokenizer-table	Resolve GFM tables
  @yozora/tokenizer-text	Resolve GFM textual contents
  @yozora/tokenizer-thematic-break	Resolve GFM thematic breaks

• Utils

  Package	Description
  @yozora/ast	Yozora Markdown AST types and constants
  @yozora/ast-util	Utility functions to handle Yozora Markdown AST
  @yozora/character	Utility functions to handle characters encoded in ASCII and Unicode.
  @yozora/core-parser	Types and utility functions for building a Yozora parser.
  @yozora/core-tokenizer	Types and utility functions for building a Yozora tokenizer.
  @yozora/invariant	A simple invariant function

💡 FAQ

• How to use Yozora with Gatsby?

  • Try the @yozora/gatsby-transformer and @yozora/gatsby-images

• How to implement a custom tokenizer?

  • Copy an existing tokenizer under tokenizers/ as a starting point.

  • Check @yozora/core-tokenizer for implementation details of tokenizer.

  • For testing, refer to @yozora/test-util and an existing tokenizer's __test__ directory.

  • Check @yozora/core-parser and @yozora/parser for information on how to integrate a custom tokenizer.

  See CONTRIBUTING.md for the full step-by-step guide. It's also recommended to refer to the existing tokenizers implementation to create a custom one.

💬 Contact

• GitHub issues

📄 License

Yozora is MIT licensed.

Related

• ✨光和尘一直想要一个清爽博客: Why this project was written.
• @yozora/react-markdown: A library that renders Yozora AST into React components.
• @yozora/html-markdown: A library that renders Yozora AST into HTML strings.