Resource handler returned message: "Too Many Requests (Service: ApiGateway, Status Code: 429, Request ID: <omitted>, Extended Request ID: null)

[martin-dessureault]

Keep getting this 429 error when trying to deploy aws api gateway models documentation. If I remove the models, the deployment is successful. If I add the models (46 total), I always get the error :(

Is there a way to configure serverless or the documentation plugin to handle this? Maybe there is a limit I can increase in AWS?

Versions:

     Operating System:          darwin
     Node Version:              14.4.0
     Framework Version:         2.51.0 (local)
     Plugin Version:            5.4.3
     SDK Version:               4.2.3
     Components Version:        3.13.2

     serverless-aws-apigateway-documentation:  1.1.7

Stacktrace:

 Serverless Error ----------------------------------------

  ServerlessError: An error occurred: UsersNotesResponseModel - Resource handler returned message: "Too Many Requests (Service: ApiGateway, Status Code: 429, Request ID: <omitted>, Extended Request ID: null)" (RequestToken: <omitted>, HandlerErrorCode: InternalFailure).
      at ../node_modules/serverless/lib/plugins/aws/lib/monitorStack.js:129:23
      at processTicksAndRejections (internal/process/task_queues.js:97:5)
      at async AwsDeploy.update (../node_modules/serverless/lib/plugins/aws/lib/updateStack.js:144:5)

[jweyrich]

Same problem here.

[christopherchubb]

I have personally submitted two identical tickets to my AWS support channel about this exact issue and both times they have accepted that it's broken, suggested a "workaround" that would completely refactor the deployment mechanism and then closed the tickets due to inactivity. It's been broken for 3+ years. The underlying issue is that Cloudformation sends ALL model requests to API Gateway at the same time and it has no fallback and retry logic when it gets a 429. CF does rate limiting for every other service that it creates, but apparently not API Gateway models.

I did get them to tell me that the rate is 10/s with a bucket of 40. So if you have more than 40 models, expect this to fail at some point.

[logicalor]

The workaround I used for this was to add another plugin - serverless-plugin-split-stacks.

All of the model files are placed in a models directory under the project root, and split by subdirectory according to the type of model. For example, Employee models would be under models/employee. I then refer to these files from the documentation config:

- name: EmployeeContactHome
  description: Home Contact Details of An Employee
  contentType: "application/json"
  schema: ${file(models/employee/EmployeeContactHome.v1.yaml)}

Then I use the following stacks-map.js to define stacks for the models of each resource type:

const fs = require('fs')
const modelPath = 'models'

/**
 * Lists nodes in a directory, optionally include files or directories
 * @param {*} path
 * @param {*} dirs
 * @param {*} files
 * @returns
 */
const listNodes = (path, dirs = false, files = true) => {
    path = fs.realpathSync(path)
    return fs.readdirSync(
        path,
        { withFileTypes: true}
    )
    .filter(
        (node) => {
            if (dirs && node.isDirectory()) {
                return true
            }
            if (files && !node.isDirectory()) {
                return true
            }
            return false
        }
    ).map(
        (node) => {
            return node.name
        }
    )
}

/**
 * Map model files (in the modelPath directory) to stack names based on their directory path
 * @param {*} path
 * @param {*} map
 */
const mapModelFilesToStacks = (path, map) => {
    const files = listNodes(path, false, true).map(
        (file) => {
            return file.replace(/\..+$/, '') + 'Model'
        }
    )
    if (files.length > 0) {
        stackName = 'API' + path.split(/[\/-]/).map(
            (part) => {
                return part.charAt(0).toUpperCase() + part.slice(1)
            }
        ).join('')
        for (const file of files) {
            map[file] = stackName
        }
    }
    const dirs = listNodes(path, true, false).map((dir) => {
        return path + '/' + dir
    })
    for (const dir of dirs) {
        mapModelFilesToStacks(dir, map)
    }
}

const modelFileStackMap = {}
mapModelFilesToStacks(modelPath, modelFileStackMap)

/**
 * This is run for every resource in the CloudFormation template that Serverless
 * generates.  It can be used to direct individual resources to different nested
 * CloudFormation stacks
 * @param {*} resource The resource object itself
 * @param {*} logicalId The key of the resource in the CloudFormation stack
 * @returns
 */
module.exports = (resource, logicalId) => {

    // A model's resource id should match one of the mapped model file to stack keys
    if (logicalId in modelFileStackMap) {
        // Set the resource's stack based on the mapped stack
        return {destination: modelFileStackMap[logicalId]}
    }
};

In my serverless.yml, my splitStacks configuration is like so:

  splitStacks:
    perFunction: false
    perType: false
    perGroupFunction: false
    # This limits concurrency of stack deployment to the given value.  Note that
    # turning this on can lead to resource dependency conflicts in the generated
    # CloudFormation templates.
    stackConcurrency: 2

The end result is a stack with all of my base resources, and extra stacks for the models for each resource type.

Hope this helps.

[christopherchubb]

Thank you for your post. We also use split stacks and put models into their own stack, and I hadn't thought about splitting them up into sub-directories so that each subdirectory was it's own split stack. I'll check the upper limit on how many substacks I can have in a stack, but that might very well be an easy way to refactor in place to solve this.