TypeError: Cannot read properties of undefined (reading 'tokenizePlaceholders') with `swcJsLoader: true,`

[thde]

As v2.2.0 has been released with support for @docusaurus/faster, I tried to enable it for our repository, but it fails with the following error if swcJsLoader: true, is enabled:

TypeError: Cannot read properties of undefined (reading 'tokenizePlaceholders')

It's not clear to me if the issue is at redocusaurus'es side or on the side of docusaurus.

The Spec can be found here. The full stack trace looks like this:

[ERROR] Error: Unable to build website for locale en.
    at tryToBuildLocale (***/docs.nine.ch/node_modules/@docusaurus/core/lib/commands/build/build.js:78:15)
    at async ***/docs.nine.ch/node_modules/@docusaurus/core/lib/commands/build/build.js:34:9
    at async mapAsyncSequential (***/docs.nine.ch/node_modules/@docusaurus/utils/lib/jsUtils.js:21:24)
    at async Command.build (***/docs.nine.ch/node_modules/@docusaurus/core/lib/commands/build/build.js:33:5) {
  [cause]: Error: Docusaurus static site generation failed for 1 paths:
  - "/api/"
      at generateStaticFiles (***/docs.nine.ch/node_modules/@docusaurus/core/lib/ssg/ssg.js:129:15)
      at async executeSSG (***/docs.nine.ch/node_modules/@docusaurus/core/lib/ssg/ssgExecutor.js:29:23)
      ... 6 lines matching cause stack trace ...
      at async Command.build (***/docs.nine.ch/node_modules/@docusaurus/core/lib/commands/build/build.js:33:5) {
    [cause]: AggregateError
        at generateStaticFiles (***/docs.nine.ch/node_modules/@docusaurus/core/lib/ssg/ssg.js:130:20)
        at async executeSSG (***/docs.nine.ch/node_modules/@docusaurus/core/lib/ssg/ssgExecutor.js:29:23)
        at async buildLocale (***/docs.nine.ch/node_modules/@docusaurus/core/lib/commands/build/buildLocale.js:60:31)
        at async runBuildLocaleTask (***/docs.nine.ch/node_modules/@docusaurus/core/lib/commands/build/build.js:93:5)
        at async ***/docs.nine.ch/node_modules/@docusaurus/core/lib/commands/build/build.js:74:13
        at async tryToBuildLocale (***/docs.nine.ch/node_modules/@docusaurus/core/lib/commands/build/build.js:70:9)
        at async ***/docs.nine.ch/node_modules/@docusaurus/core/lib/commands/build/build.js:34:9
        at async mapAsyncSequential (***/docs.nine.ch/node_modules/@docusaurus/utils/lib/jsUtils.js:21:24)
        at async Command.build (***/docs.nine.ch/node_modules/@docusaurus/core/lib/commands/build/build.js:33:5) {
      [errors]: [
        Error: Can't render static file for pathname "/api/"
            at generateStaticFile (***/docs.nine.ch/node_modules/@docusaurus/core/lib/ssg/ssg.js:167:15)
            at processTicksAndRejections (node:internal/process/task_queues:105:5)
            at runNextTicks (node:internal/process/task_queues:69:3)
            at process.processImmediate (node:internal/timers:459:9)
            at async ***/docs.nine.ch/node_modules/p-map/index.js:57:22 {
          [cause]: TypeError: Cannot read properties of undefined (reading 'tokenizePlaceholders')
          Please report this to https://github.com/markedjs/marked.
              at ***/docs.nine.ch/build/__server/assets/js/f0ad3fbb.0f5fb192.js:49942:40
              at Object.run (***/docs.nine.ch/build/__server/assets/js/f0ad3fbb.0f5fb192.js:50972:6)
              at Object.highlight (***/docs.nine.ch/build/__server/assets/js/f0ad3fbb.0f5fb192.js:50881:12)
              at jt (***/docs.nine.ch/build/__server/assets/js/f0ad3fbb.0f5fb192.js:52600:11348)
              at Object.highlight (***/docs.nine.ch/build/__server/assets/js/f0ad3fbb.0f5fb192.js:52600:52821)
              at Er.code (***/docs.nine.ch/build/__server/assets/js/f0ad3fbb.0f5fb192.js:52600:41553)
              at Sr.parse (***/docs.nine.ch/build/__server/assets/js/f0ad3fbb.0f5fb192.js:52600:44695)
              at parse (***/docs.nine.ch/build/__server/assets/js/f0ad3fbb.0f5fb192.js:52600:43979)
              at ***/docs.nine.ch/build/__server/assets/js/f0ad3fbb.0f5fb192.js:52600:49196
              at Ar (***/docs.nine.ch/build/__server/assets/js/f0ad3fbb.0f5fb192.js:52600:49316)
        }
      ]
    }
  }
}

[rohit-gohri]

It's not clear to me if the issue is at redocusaurus'es side or on the side of docusaurus.

Might be worth reporting it there too (you can link this issue), since it's still experimental and your normal build works (I guess?)

[thde]

Yeah, makes sense. Will do 👍🏼

[snowliver]

Hi @thde @rohit-gohri ,
same issue on some of our api docs. Has something to do with markdown codeblocks in the description field of a parameter.

But after i followed the first line after the "please report to marked...", i landed somewhere in the prism.languages.php definitions.

So, I got it working with adding the additional language php into the docusaurus.config.js as described here : https://docusaurus.io/docs/markdown-features/code-blocks#supported-languages

prism: {
   additionalLanguages: ['php']
}

Also not happening, if you do a docusaurus build --dev.

But - not sure if that makes things easier to permantly fix.

[thde]

That's it! If you follow the generated code in the stack trace you end up in ***/build/__server/assets/js/f0ad3fbb.8489e1b5.js:50825:40:

Prism.hooks.add('after-tokenize', function (env) {
   Prism.languages['markup-templating'].tokenizePlaceholders(env, 'php');
});

After adding php to additionalLanguages , the build worked for me too.

[rohit-gohri]

Is this still an issue? If it is, then maybe someone can add php to list of languages like scala was added as it caused some issues too -

redocusaurus/packages/docusaurus-theme-redoc/src/redocData.ts

Line 31 in 4cafbe9

additionalLanguages: ['scala'],

[marsh73]

not sure if this is related but.. when trying to use the future: {experimental_faster} I'm also seeing

ReferenceError: Prism is not defined which seems to be getting referenced from redocusaurus. My config block looks like

prism: {
        theme: lightCodeTheme,
        darkTheme: darkCodeTheme,
        additionalLanguages: [
          "bash",
          "python",
          "json",
          "javascript",
          "docker",
          "jsx",
        ],
      },

and the reference to Prism seems to be coming from the docusaurus-theme-redoc.

[snowliver]

Is this still an issue? If it is, then maybe someone can add php to list of languages like scala was added as it caused some issues too -

redocusaurus/packages/docusaurus-theme-redoc/src/redocData.ts

Line 31 in 4cafbe9

additionalLanguages: ['scala'],

Hi @rohit-gohri ,

tried today with docusaurus 3.8.0 and redocusaurus 2.5.0 .
Issue still exists. And I wasn't able to get your suggested workaround working. (but that's most likely my fault.)

@marsh73 i think you are using an outdated config => https://docusaurus.io/docs/migration/v3#prism-react-renderer-v20, so not the same issue.

[mwood23]

not sure if this is related but.. when trying to use the future: {experimental_faster} I'm also seeing

ReferenceError: Prism is not defined which seems to be getting referenced from redocusaurus. My config block looks like

prism: {
        theme: lightCodeTheme,
        darkTheme: darkCodeTheme,
        additionalLanguages: [
          "bash",
          "python",
          "json",
          "javascript",
          "docker",
          "jsx",
        ],
      },

and the reference to Prism seems to be coming from the docusaurus-theme-redoc.

I was getting this myself. Wrote a small plugin to workaround it:

import { LoadContext, Plugin } from '@docusaurus/types';

/**
 * There is a problem inside of redocusaurus that causes prism to not be
 * properly injected into the page. This plugin fixes that.
 */
export default function fixPrismPlugin(context: LoadContext): Plugin {
  return {
    name: 'fix-prism',
    // Docusaurus passes a 3rd `currentBundler` param when using the modern
    // faster pipeline. We preferentially pick its `ProvidePlugin` to remain
    // agnostic, and fall back to the classic `webpack` one.
    configureWebpack(_config, _isServer, { currentBundler }) {
      return {
        plugins: [
          new currentBundler.instance.ProvidePlugin({
            Prism: ['prismjs'],
          }),
        ],
      };
    },
  };
}

[rohit-gohri]

@mwood23 would be great if you can contribute the fix here - https://github.com/rohit-gohri/redocusaurus/blob/main/packages/docusaurus-theme-redoc/src/index.ts#L16

so no custom plugin is needed