Recursive external JSON Schema refs can overflow resolveSubtree

[Richardpwe]

Summary

resolveSubtree can run into a stack overflow when resolving recursive external JSON Schema references.

This is related to earlier resolver issues such as #1362, but the case below is reproducible on current master with recursive external JSON Schema documents.

Problem

Swagger UI can hang or throw:

RangeError: Maximum call stack size exceeded

when rendering a schema that references an external JSON Schema document which recursively references itself through another external document.

The schema shape is valid JSON Schema recursion, for example:

object.json -> basic-types.json -> object.json

A real-world example is OGraf's GDD schema, where an object schema references basic types, and object/array child schemas can reference the object schema again.

Minimal shape

An OpenAPI 3.0.3 schema contains:

components:
  schemas:
    RenderTargetSchema:
      type: object
      properties:
        gdd:
          $ref: '#/components/schemas/ShallowGDDObjectSchema'

    ShallowGDDObjectSchema:
      allOf:
        - $ref: 'http://mock.swagger.test/gdd/object.json'

/gdd/object.json references /gdd/basic-types.json, and /gdd/basic-types.json references /gdd/object.json again from nested items / properties.

Expected behavior

resolveSubtree should terminate safely when it reaches a recursive external $ref.

At the recursive boundary, it should preserve a $ref or otherwise avoid expanding the same external reference path forever.

Actual behavior

The resolver can continue expanding the recursive external references until Swagger UI hangs or throws a stack overflow.