--
Following our previous post about GraphQL we’d like to tell you a little story about how we fixed the problem of setting up a GraphQL server for a REST API using code generation.
We built a TypeScript-interfaces based code generator, to generate GraphQL and JSON schemas on top of the Amplience REST API, so that Amplience could join our GraphQL gateway through federation.
We rebuilt the generator to take a GraphQL schema as the source of truth, which gave us more control over the schema itself. This worked better and we could use the graphql-codegen framework.
Also, we could use it to generate Terraform configuration from that.
This resulted in an almost completely automated way of creating a GraphQL server for Amplience.
All of the modules are published open source (MIT).
At Lab Digital creating advanced and future proof digital solutions that fully meet our clients needs, often requires innovative thinking and inventing things that don’t exist yet. This is the approach that led us to building Schema Generate and GraphQL Codegen Amplience — tools that allow us to use a GraphQL server for a REST API, in a largely automated way.
Having decided to build a MACH architecture for our client, GraphQL was the chosen best practice to build the back-end for the front-end as you can read in our previous blog post. One of the components Amplience CMS, however does not yet support GraphQL out of the box. This meant we needed to implement GraphQL support for it — ideally without reinventing the API ourselves completely.
We believe that a composable architecture should have a single “backend for frontend” that frontends connect to as this helps us keep our frontends simple and expose a unified data interface from the back-end.
Our approach to this is to use GraphQL in conjunction with Apollo Federation. GraphQL decouples the frontend from backend and provides a single endpoint to query an API from. Federation allows this endpoint to be powered by multiple back-end systems — Amplience being one of them.
To find out more about how we leverage Federation for building a ‘single graph’ for composable architectures read our post GraphQL Federation & Composable Architecture: a Cloud Native love story.
Other CMSs have GraphQL APIs available — we simply federate their schema through our GraphQL gateway and we’re good to go. Amplience CMS had no GraphQL server, so we had to design and create our own and basically wrapping it around Ampliences REST API.
To manage content types and content type schemas Amplience uses JSON schema definitions that you upload to its API. So in order to create a GraphQL server, leveraging these schemas provided a good starting point for keeping our GraphQL server in sync with Amplience itself; as long as both would use the same schemas we would be fine.
Despite a number of pros to using JSON schemas, there are a few unavoidable cons however; there is no extensive error reporting in the case of invalid JSON data or structure, for a start. And semantics and referencing to other JSON schemas can be quite unclear. Additionally JSON schemas are typically written by hand, making them tedious and error-prone.
These cons quickly introduced a lot of unwanted repetition and errors. Similar code had to be rewritten in Typescript and then once more with GraphQL. In the short run this wasn’t much of an issue. But in the long run, maintenance would have been tricky. Writing JSON schemas kinda sucks if we’re honest.
For a lack of existing tooling our solution was to develop our own schema generator. The generator would be able to take in Typescript Interfaces or a GraphQL schema and generate the other, as well the Amplience JSON schemas. This would allow us to have custom-generated content types for the JSON schema as opposed to manually writing it out.
Since the end goal of our plugin was to generate the Amplience JSON schemas we had to choose between generating the Typescript types and JSON schema from a GraphQL schema, or generating the GraphQL and JSON schemas from a Typescript file with interfaces. Due to time constraints and the direction the project was already headed in, we decided on the latter. Schema Generate was born.
Schema Generate allows us to generate the GraphQL schema from the Typescript file with interfaces that output the GraphQL types and the Amplience JSON schemas, resulting in a single definition of the schema in Typescript.
The GraphQL schema generator takes Typescript files with interfaces as input and generates GraphQL types and JSON schemas as a result.
If we have a Typescript file as follows:
This will generate the following GraphQL types:
From the same Typescript file the plugin generates the full JSON schema needed for Amplience as well:
As you can see the JSON file is far more verbose in comparison to either of the two files. And we love not having to deal with creating these files manually or maintaining and editing them by hand.
The core of this functionality is also not too difficult, and we can achieve most of it in around 100 lines of code in this index.ts file.
The schema-generate plugin was a little unpolished and exploratory, but it worked quite well and was a great success for the project. It had one fault though: it used the TypeScript interfaces as the “source” from which everything else was generated. We needed a more robust solution, and in particular more control over the GraphQL schema — in our experience having complete control over this is quite important, which makes it less suitable for generating it. Schema-first, if you will.
We found that generating code (particularly Typescript) from a GraphQL schema is a problem already solved by GraphQL Code Generator. This is the GraphQL “community standard” code generation tool, generating language-agnostic code out of your GraphQL schema and operations through SDL — the GraphQL Schema Definition Language — as the name implies, also a way of defining schemas.
So we chose to implement a new plug-in that took GraphQL SDL as the source of truth and generated anything from that instead of using TypeScript types.
Our successive Codegen plugin allowed us to generate our TypeScript Types and JSON schemas from GraphQL.
Since we had our Codegen plugin doing most of the heavy lifting, we figured we could pass on this problem too and have it generate Terraform files, which we use to manage the infrastructure-side of Amplience.
Being able to generate Terraform code from the same GraphQL schema goes a long way towards setting up an automated infrastructure experience for Amplience and other platforms that don’t yet support GraphQL.
Now our improved Codegen plugin employs a “schema-first” design by generating Typescript, Amplience JSON, and Amplience-Terraform resources from a single GraphQL schema, over which we have full control.
This is what the GraphQL SDL file looks like; based on this schema all other files (JSON, TypeScript & Terraform) are generated:
Here is an example of the generated Terraform configuration that ties together everything we need:
For more on the Amplience Terraform provider we’ve created, read here.
We consider this second approach the way to go, as this is how the entire GraphQL-codegen ecosystem approaches code generation. Whereas Typescript is designed for implementing functionality, GraphQL is designed to describe the form of data, making it the better tool for the job in this case.
From a development perspective generating code from a GraphQL schema is easier, because the GraphQL schema model is more practical than the Typescript AST libraries. From a conceptual perspective the schema should be leading (“schema-first”) and GraphQL is a better schema language than Typescript.
GraphQL Codegen Amplience is now available. You can use it to manage all your Amplience content types and infrastructure through the declaration of just a single GraphQL schema and use that to expose Amplience content through GraphQL. Did someone say “Amplience-as-code”?
Even if Amplience were to build their own GraphQL server, which we know they have in development, we would use a similar process.
When Amplience have their GraphQL implementation depending on their implementation, our process would probably look something like this:
Create our own basic GraphQL schema definition
Generate JSON schemas, TypeScript types and Terraform code for Amplience from that.
These JSONs will result in another GraphQL schema generated by Amplience themselves (which will be more or less the same).
Extend the Amplience-created GraphQL in our GraphQL-Federation component with our own custom logic for things like resolving product references.
In this case, our GraphQL schema would not be used by the API, but would be used as a Content Type design language where we generate all other configurations from.
This workflow is not unique — for example, AWS Amplify uses a similar strategy. They use it to create the necessary resources in AWS behind the scenes, such as database resources.
If you’re also using Amplience you’ll probably find our open source generators and plugins useful directly. Instead of fighting with JSON you can generate a GraphQL server. Then you can enjoy the speed and comfort of writing in TypeScript or GraphQL and simply install and run the tools to get the output you need.
But in a way, the generic problem is more interesting. If you’re interfacing with a REST API for a platform that doesn’t (yet) support GraphQL, you’ll probably be able to apply very similar principles. Instead of feeling trapped by interfaces provided to you by other companies or teams you have the option of building your own middleware tooling and using that instead.
This helps for scalability too. Often it’s hard to justify spending time building tooling when you can “just do it manually” at first. Building tooling has an initial upfront cost, but allows you to keep manual work under control scale long term.
In our case we estimated that having to build and maintain Amplience JSON Schema manually would have gotten more and more difficult and time-consuming as our projects grew. By instead using some extra engineering time at the start of the project we could drastically cut down on the amount of time we needed to spend building JSON schemas, Terraform code, and TypeScript types by hand (and fixing the resulting issues that arose from this error-prone process).
We chose to open source these modules so that we could share them with our colleagues, clients and competitors to encourage healthy competition and innovation.
To summarise: some things we’ve learned in working with Amplience and configuration generation include:
Use a GraphQL schema-first approach. While often you can choose which part of the configuration to use as the source of truth, the power and flexibility of GraphQL makes it the clear “winner” in standards and languages that have some overlap.
Keep things DRY. If you have multiple configuration files that represent the same, or overlapping, resources, make sure that you can define things exactly once, and generate all variants from there.
You don’t have to be limited by what you’re given. Tools like Amplience do not support GraphQL out of the box, but there’s no need to be limited by this. As we do, it’s simple enough to generate a GraphQL server on top of an existing REST API.
Code generation is not magic or scary . We’ve exploited the benefits of code generation in many of our projects: we generate SDKs based on specs, and now the entire Amplience-GraphQL server, its JSON config, as well as the required Terraform configuration. And MACH composer is pretty much a code generator as well. Code generation might be daunting to start with, but it gives you enormous value (not to mention consistency) in the long-term.
We believe that giving engineers the autonomy to build the tooling they need is the highest priority. At Lab Digital, being passionate about crafting elegant solutions, we’re continuously inventing new, improved ways of working and applying new technologies in the process, while also publishing quite some open source modules.
Come join us!
Kors van loon is a code wizard at Lab Digital. He loves diving in to the type systems of programming languages if he’s not already painting while listening to music with rock organs.