[Feature] Configurable rate limiting for Hasura GraphQL endpoint to prevent event log exhaustion

[yg-ht]

Feature Description

Add a configurable rate limiting mechanism for writes to the Hasura GraphQL endpoint.

This is intended to prevent misbehaving automation, tooling, or integrations from writing very large numbers of repeated records in a short period. In the observed failure mode, the main application data remains relatively unaffected, but the Hasura event log and related tables grow extremely quickly. This can fill the available disk and cause GhostWriter to stop functioning. Arguably, it may be beneficial for a rate limit to be applied to all endpoints.

Are you intending to implement this feature?

No, not in the immediate future.

I may be able to help test a proposed change or configuration, but I am likely to struggle for time to implement this properly in the near term. I am raising this first to ask for input on the best place in the stack to implement it.

Current Behavior

At present, a client or tool can submit a very large number of repeated writes through the Hasura GraphQL endpoint.

If a tool behaves unexpectedly, it can create millions of repeated records. The primary database content may not be badly affected, but the Hasura event_log and related tables can grow very rapidly. This can increase the database size substantially, fill the disk, and cause GhostWriter to fail.

The resulting failure mode is not obvious at first. GhostWriter stops working because the disk is full, but it takes time to identify that the root cause is rapid growth in the Hasura event log tables, clean up the affected records, recover disk space, and restore service.

Desired Behavior

GhostWriter should provide, or document, a supported way to limit excessive write activity through the Hasura GraphQL endpoint.

Ideally this should be configurable so different deployments can tune the behaviour to their own environment. For example, it should be possible to define limits by one or more of the following, depending on what is practical in the current stack:

• Source IP address.
• User.
• Role.
• API token or client identity.
• GraphQL operation type.
• Write-heavy mutations specifically.

When a client exceeds the configured limit, the request should be rejected or delayed with a clear error response. The event should also be logged clearly enough that an operator can quickly understand that rate limiting has occurred.

Use Case

This would help protect GhostWriter deployments from accidental self-inflicted denial of service conditions caused by automation or integrations.

A realistic example is a locally developed CLI or integration tool that enters a bad state and repeatedly writes the same or similar records. Even if the application data can tolerate this, the supporting event log tables can grow quickly enough to exhaust disk space.

The benefit is operational resilience. A rate limiter would reduce the chance that one misbehaving client can consume all available storage, break the GhostWriter service, and force manual database cleanup.

It would also make diagnosis easier. A clear rate limit error and associated log entry would be much easier to understand than discovering the problem indirectly after the service fails due to disk exhaustion.

Implementation Suggestions

I am not certain which part of the stack is best suited to this, so input would be useful before implementation.

Possible approaches include:

1. Reverse proxy rate limiting, for example through nginx or Traefik, if all GraphQL traffic is guaranteed to pass through that layer.
2. Hasura-level controls, if there is a native or supported way to limit mutation frequency by role, user, token, or client identity.
3. Application-level throttling before writes reach Hasura.
4. Database-level safeguards, such as deduplication, uniqueness constraints, or defensive cleanup of repeated event records.
5. A documented deployment-level configuration with safe defaults and guidance for operators.

My current assumption is that reverse proxy rate limiting may be the quickest deployment-level control. However, it may not be sufficient if local CLI tooling, internal containers, or trusted services can reach Hasura directly without passing through the proxy.

A stack-level solution that protects all mutation paths would be more robust.

Additional Information

The observed impact is:

• Millions of repeated records can be written in a short period.
• Hasura event log and related tables grow very quickly.
• Database backups can become much larger if they run during or after the event.
• Disk space can be exhausted.
• GhostWriter can stop functioning.
• Recovery requires manual investigation, database cleanup, disk space recovery, and service restoration.

A useful fix would probably include:

• A documented way to limit repeated GraphQL writes.
• Configuration examples for a typical GhostWriter deployment.
• Guidance on the preferred enforcement point: reverse proxy, Hasura, application code, database, or a combination.
• Sensible default recommendations.
• Clear logging when requests are throttled or rejected.
• Minimal impact on normal interactive GhostWriter usage.

[chrismaddalena]

I will look into this. It's a good idea, especially as we've just reworked the API tokens and there's an ever-increasing use of LLMs with Ghostwriter.

Out of curiosity and to help me understand your context, when have you experienced this issue? Was it with a custom integration?

[yg-ht]

Hey Chris,

Yup. We have our own tooling that interacts with the GraphQL endpoints. We are implementing some better constraints on our tooling side, but, I thought it would be a good idea for GhostWriter to be more defensive in this area as well.

[chrismaddalena]

Rate limiting will be helpful, but I think we want more than just a rate limit. We can do that as the initial implementation. The next step is guarding against possible repeated requests from a script or LLM. For example, if something tries to upload the same evidence file multiple times we probably don't want to accept that. If the same record is being sent more than once, we probably want to return the existing record or deny it.

In the meantime, we can address the event log issue quite easily. I'm updating the Docker configurations to limit logs to 30MBs of logs across a maximum of three files. Across all containers, that's a maximum of ~240MB in total. That should have been in there anyway.