Scaling Issues

In certain circumstances, your application may hit some scaling issues that necessitate workarounds. These workarounds should be treated as temporary and should be revisited in the future as Gatsby scales to support larger applications with hundreds of thousands of pages.

However — until we get to that point, some workarounds are useful to consider if they unblock your team from deploying, developing, etc.

Just looking for possible solutions? Jump ahead

What is a scaling issue?

A scaling issue is evident if your application is unable to build due to an error or if it is extremely slow in some lifecycle, e.g. develop or build. For example:

• An “Out of Memory” occurs when building in CI
• The develop lifecycle takes 10x as long as build

and more. An example of a scaling error thrown to the console may look something like:

When can a scaling issue arise?

A scaling issue can arise in varied cases, but typically something like:

• A Gatsby application with ~100K+ pages
  • See this issue for an example
• Extremely large json files sourced with gatsby-transformer-json
• Extremely large GraphQL queries, which are stored in memory in the develop lifecycle
  • See this issue for an example

If you are seeing errors or slowness and your Gatsby app matches one of the above use-cases, it’s very likely you are hitting some scaling issues.

Possible solutions to scaling issues

It’s difficult to pin down exactly how to fix a scaling issue. We have some recommendations and workarounds that may work for your application.

Note: the application of these techniques should be considered analogous to applying a bandage. A bandage solves the underlying issue, but at some indeterminate point in the future the underlying issue may be healed! In much the same way—treat these techniques as temporary and re-visit in the future if underlying scaling issues in Gatsby have since been resolved.

GATSBY_DB_NODES

In preparation for future versions of Gatsby, we’ve enabled experimental support for a different mechanism for the persistence of nodes: Loki. It’s challenging to assess whether this could lead to unforeseen issues without breaking changes, so we’ve exposed it behind a flag while we continue to assess the impact to Gatsby applications.

Loki allows us to opt-in to possibly more performant internal operations and it may resolve your scaling issues. If it does—please let us know! To opt-in to this experimental feature:

Switch off type inference for SitePage.context

When using the createPages API to pass large amounts of data to pages via context (which is generally not recommended), Gatsby’s type inference can become slow. In most cases, it is not actually necessary to include the SitePage.context field in the GraphQL schema, so switching off type inference for the SitePage type should be safe: