docfx `memberLayout` option affects processing speed for ManagedRef templates by increasing RawModel size and number

[banchan86]

Description

Recently starting from bonsai-rx/docs#114 we have starting enabling this property in docfx.json which reduces API/Reference page clutter and sorts the TOC into types. This is intended to be a first step towards standardizing a new template for our API/Reference pages.

"memberLayout": "separatePages"

However this option exacerbates the underlying issue in #23.

Example

For instance, using the same example of the Bonsai.Core docs illustrated in #23 with this command:

dotnet docfx --exportRawModel  

• When memberLayout is not defined:

It generates ~1000 RawModel files, each ~70MB with >1.8 million lines of code. This command took 30 minutes to build.

• When memberLayout: separatePages is defined:

It generates ~4000 RawModel files, each ~90MB with >2.4 million lines of code. I did not let this command finish, but I estimate it would have taken 2-3 hours.

Enabling this option makes the issue even apparent for smaller repositories like Bonsai.ML where it was not an issue before. Using this command

dotnet docfx --serve

No template, no separatePages - 30 seconds

Template, no separatePages - 45 seconds

Template, separatePages - 2 minutes 30 seconds.

Solution

I believe that while we should enable it for now as we do not have a standardized template yet, we should disable it once we implement the ManagedReference templates.

As this option also changes the RawModel that is used by the javascript processing scripts, we could also plan for the future and not build ManagedReference templates on top of it, as we would need to adapt those scripts if we phase it out. I believe that everything that is done with that option can be achieved with the ManagedReference templates.

[glopesdev]

@banchan86 thanks for the investigation! I was wondering that one thing which may be possible to do in order to bring the separate pages to be at least the same as without could be to avoid sharing models of individual members.

I noticed that the JavaScript function controlling sharing takes a model as the input:

exports.getOptions = function (model) {
    return {
        isShared: true
    };
}

Up to now we have just been sharing everything by default, but I wonder if there is some attribute or property of the model object which we could use to know whether a particular model is a separate member page. If this were possible we could use that as a test to avoid sharing the individual member pages and therefore hopefully bring the separate members build to match the non-separate one.

[banchan86]

Your suggestion seemed like it would be an easy fix, so I tried it out on the Bonsai.ML repository. It seems like all the extra separatePages RawModel files have model._isOverload defined and set to true (it's a lil bit weird to me) so something like this works:

exports.getOptions = function (model) {
    if (!model._isOverload) {
        return {
            isShared: true
        };
    } 
        return {
            isShared: false
        };
}

While it does trim the RawModel size, it has less of an impact on performance as the number of RawModels is still the same. This is what I found when I tested dotnet docfx --exportRawModel:

• no separatePages, full_share - 30 secs, , 90 raw.json files, individual raw.json ~8MB, total size 722MB
• separatePages, full_share - 56 secs, 198 raw.json, individual raw.json ~9MB, total size 1.69GB
• separatePages, trimmed_share - 52 secs, 198 raw.json, individual raw.json ~7MB, total size 1.42GB

This might have a bigger performance increase

• with custom templates (this was timed with just the default modern template and not custom templates)
• for bigger repositories

So its still worth keeping this in mind if we want to keep using separatePages and its a good potential fix.

sorry I went down a bit of a rabbit hole with this 😅