Motivation

As time goes on, lots of things become outdated sooner or later, and documentation is no exception. But when documentation becomes outdated, it automatically loses its essence – to be consulted for valuable information. In modern software development, where Application teams are continuously adding features to their services and Platform teams are continuously evolving the platform and infrastructure to accommodate all the workloads, it is extremely easy for documentation to get out of date just in a couple of weeks.

This is something the Empathy.co Platform Engineering team experienced firsthand when we realized there was an issue with how Development teams learn how to work with the platform provided for them. We noticed that there wasn't a clear path for everyone to follow when using the platform tools. While we had some documentation available that we felt was sufficient, we realized it wasn't as complete or as maintainable as we thought. So, we had to find a solution.

Design Thinking

The following items were identified as the main pain points of the problem:

• The documentation was spread across various places and different tools.
• Some documentation was out of date, meaning it was no longer useful and was sometimes confusing to readers.
• With the fast growth of the company, new colleagues didn't even know where various pieces of documentation were located.
• The overall process was unmaintainable and not future-proof.

Scope

There are several types of documentation, and not all of them should be handled the same way, nor do they have the same lifecycle. Assuming that not all the roles at the company have the same skillset, like fluent writing of Markdown documents and working with Git, we concentrated our efforts specifically on technical documentation. That way, the skillset of those interacting with the documentation was more clearly defined.

More specifically, we focused on the documentation about the Internal Development Platform (IDP, for short) that Platform Engineering needs to share with the Application teams, like CI/CD processes, monitoring, logging, etc. Since the nature of the IDP is to help developers by using certain tools and practices, we like to refer to this documentation as Developer Workflow (DW).

Centralization

Aiming to centralize the technical documentation in a single place and tool, we decided to deploy Backstage to the platform. Backstage is a tool developed by Spotify that, among other features, enables the creation of documentation sites out of the Markdown documentation stored in a Git repository, using MkDocs.

Hosting the documentation in Git repositories also provides the benefits of using Git, like versioning, tracking, code reviewing, etc.

Content Up To Date

Unfortunately, there's no tool or magic wand that keeps the documentation up to date. It is a continuous effort, so it should be included in teams' ways of working. From our experience, it is very helpful to take into consideration the time it will take to create or update the documentation when estimating a given task. A task shouldn't be marked as finished if the documentation hasn't been reviewed and updated.

In this specific case, it was hard work reviewing all the documents that needed to be updated, created, or deleted accordingly. It is always better to keep the documentation up to date by making small updates frequently, rather than updating a large amount of content infrequently.

Visibility

As stated above, one of the main pain points of the previous solution was that the stakeholders weren't aware of where to find certain articles within the documentation. Adding Backstage helped centralize all the technical documentation into a single place, but the problem wasn't yet solved. Backstage was still unfamiliar to some people, mostly those who recently joined the company. We also received feedback about the search experience not working as well as we had hoped, so we decided to create a separate site to host only the Developer Workflow documentation and reduce all the surrounding elements that could prevent users from finding the necessary documents.

We didn't want to split the documentation again, so we decided to keep it in Backstage and the standalone website, using the exact same source of truth for both to avoid duplicating the content. This part of the solution will be explored more in-depth, later.

Maintainability

Documentation evolves over time, and as part of that evolution, things are likely to break. A few of the issues that can occur are:

• Broken hyperlinks
• Tables of contents not matching page sections
• Navigation bars showing items that no longer exist or not showing items that should be shown

All of these items were successfully addressed just by using a few MkDocs plugins, which will be explained in more detail in the next section.

Not only does the content have to be maintainable, so does the overall process. In order to account for this in the design, it is key to keep the logic as simple as possible (following the KISS Principle), so teams managing the process can easily update the documentation, keep the process up to date, and evolve the whole solution accordingly. Keeping it as simple as possible also makes it easier to transfer ownership to another team and accelerate the onboarding process of new team members, so they understand the workflow and how to support the solution.

The Solution

Now that the context and requirements are clear, it is time to look at how to implement them into the solution. It mainly consists of a series of agreements in the way of working, in conjunction with a series of best practices and MkDocs plugins.

Documentation Source: Git Repository

Since Backstage aggregates all the MkDocs documentation sites from several Git repositories into a single documentation portal, we decided to treat the Developer Workflow documentation as another component in Backstage. This repository also contains the Terraform code to build a simple Cloudfront + S3 static web hosting and a simple Github action to build the DW standalone website. This enabled us to have the exact same documentation in two places, fed by the same source.

Focus on Maintainability: MkDocs Plugins

All documentation structure, navigation bars, and page-linking related items were addressed using MkDocs theme features and additional MkDocs plugins.

Since Backstage uses a slightly modified version of MkDocs Material Theme, we had to explore what theme features were compatible with Backstage and used the following:

mkdocs.yaml

theme:
  name: material
  features:
    - navigation.sections
    - navigation.expand
    - navigation.indexes
    - navigation.top

• navigation.sections: Enabled using the file structure of the docs folder to automatically generate the navigation sidebar. This prevents an out-of-date sidebar, as it is automatically rendered based on the content.
• navigation.expand: Automatically expands all the sections from the navigation sidebar. The motivation for this was to give the user a bird's-eye view of all the documents at once, reducing the need to search – except for when looking for more specific content.
• navigation.indexes: Enables using the section titles as pages, which is great for overview pages, offering a general view of what the section is about.
• navigation.top: Adds a "Back to top" button when the scroll is not at the very top of the page, which is especially useful when navigating large pages.

In addition to the MkDocs Material theme features, we also used the following MkDocs plugins. Note that to make it work both in Backstage and standalone, the plugins must be installed in Backstage, as well.

Note that the order in which the plugins are listed is the order in which they are executed when building the documentation site. This is important to keep in mind when adding or removing them, as the results may vary.

mkdocs.yaml

plugins:
  - techdocs-core
  - search
  - awesome-pages
  - macros
  - glightbox:
      touchNavigation: false
      loop: false
      effect: zoom
      width: 100%
      height: auto
      zoomable: true
      draggable: false
  - webcontext:
      context: catalog/default/component/developer-workflow
  - alias

• techdocs-core: A Backstage built-in plugin, it adds the basic documentation site functionality to Backstage. The rest of the plugins and extensions must be compatible with this one in order for all of them to work properly. The plugin also needs to be installed in the standalone version of the DW documentation site to make sure all the features used are compatible with Backstage.
• search: A built-in MkDocs plugin that adds indexing and search capabilities to the site.
• awesome-pages: Enables the creation of a .pages file inside a folder to tune folder-specific settings like navigation, sorting, hiding, etc., among other things. We mostly use it to customize section names that have the same name as the folders, by default. For example, a folder called docs/04-ci-cd could be tuned to be rendered as CI/CD - Build and Deploy just by adding a .pages file with the following content:

.pages

title: CI/CD - Build and Deploy

This makes it possible to use the name of the folder for sorting, but with a more user-friendly name for the sections.

• macros: A plugin that enables using Jinja2 templating and variables within MkDocs Markdown pages, which is great for automating the rendering of certain blocks within the documentation.

For example, let's say there's a reference to a team email address in several pages of the documentation. If the email address changes for some reason, it would have to be updated on all the pages where it appears. With the MkDocs macros plugin, it can be set as a variable in the extra section inside mkdocs.yaml and be expanded as many times on as many pages as needed.

mkdocs.yaml

extra:
  team_email_address: myteam@mycompany.org
  slack_handler: @myteam

contact-info.md

Contact us:

-   Email: **{{ team_email_address }}**
-   Slack Handler: **{{ slack_handler }}**

That's just a simple example, but it can be extended to other uses. For example, it could be used to type all the links to the tools of the Internal Development Platform once and then refer to those links from multiple pages. Additionally, it could be used generate a page with all the links, for faster access.

mkdocs.yaml

extra:
  tool_urls:
    tool_one:
      test: https://tool-one.test.company.org
      stage: https://tool-one.stage.company.org
      prod: https://tool-one.prod.company.org

    tool_two:
      test: https://tool-two.test.company.org
      stage: https://tool-two.stage.company.org
      prod: https://tool-two.prod.company.org

tools.md

{% for tool in (tool_urls | sort) %}
### **{{ tool | replace("_"," ") | upper }}**
	
{% for environment in (tool_urls[tool] | sort) %}
	- **{{ environment | replace("_"," ") | upper }}**: {{ tool_urls[tool][environment] }}
{% endfor %}
	
{% endfor %}

The rendering the above would look like this:

• glightbox: A plugin that adds basic zoomable image functionality, a feature that MkDocs doesn't provide by default.
• webcontext: A plugin that aids in maintaining compatibility with Backstage. As Backstage renders each documentation site for each of the registered components or services, it generates the links relative to the component base URL ${BACKSTAGE_URL}/catalog/default/component/${COMPONENT_NAME}, causing links between pages to break on one of the websites. This is configured to match Backstage context and is dynamically set to / as part of the automation to deploy the standalone website.
• alias: A plugin that makes it possible to use aliases for each documentation page. This hardens the internal links between pages, as the links point to an alias or to a relative path in the filesystem, instead of to the fully qualified URL. Using this plugin, when a file is renamed or moved to a different folder, the internal links don't break.

Lastly, let's look at all of the Markdown extensions used. All of them are included in the techdocs-core plugin. Almost all of them are using the default behaviour, we just set them explicitly in the mkdocs.yaml to avoid misbehavior if a default value changes at some point.

mkdocs.yaml

markdown_extensions:
  - admonition
  - pymdownx.highlight:
      # Enables linking to a specific line in a code block.
      anchor_linenums: true 
  - pymdownx.inlinehilite
  - pymdownx.snippets
  - pymdownx.superfences
  - attr_list
  - pymdownx.emoji:
      emoji_index: !!python/name:materialx.emoji.twemoji
      emoji_generator: !!python/name:materialx.emoji.to_svg
  - def_list
  - pymdownx.tasklist
  - pymdownx.details
  - footnotes

Working Agreements

In order to support the solution with minimal effort, we decided to establish some practices around the Developer Workflow documentation. The following are some of the most important practices that should be checked as part of a code review:

• All folders must contain an index.md file with an overview of the whole section.
• Avoid using lots of images. If something can be described with text, it should be. This may feel counterproductive, but keeping images and screenshots up to date in GitOps documentation is a tough task.
• Diagrams should be in SVG format. SVG files are written as text, can be embedded directly in MkDocs, and can be raw-edited without keeping a separate file that requires exporting to an image format.
• All documentation pages must have an alias and all links between documentation pages must point to the page alias.
• All the DW documents must be of the interest of the users of the platform. Documents that are more in-depth and intended for Platform Administrators must be hosted in a different repository and served through Backstage.

Preview Documentation Changes in Localhost

Working with Backstage is pretty difficult, when it comes to previewing how rendered markdown files will look. Even though there are various tools and extensions that enable previewing Markdown files from the IDE, those are not capable of rendering with the exact same plugins that MkDocs is going to use when building the site. On the other hand, working directly with MkDocs and knowing that all the plugins, features and extensions are compatible with Backstage, it becomes quite easy and simple to render the site in localhost, even with real-time updates. To do so, the official MkDocs material Docker image needs to be used as the base image before plugins are added to it.

Dockerfile

FROM squidfunk/mkdocs-material:8.4.2

RUN python -m pip install --upgrade pip \
    && pip install mkdocs-alias-plugin==0.4.0 \
       mkdocs-awesome-pages-plugin==2.8.0 \
       mkdocs-macros-plugin==0.7.0 \
       mkdocs-techdocs-core==1.1.4 \
       mkdocs-webcontext-plugin==0.1.0 \
       mkdocs-glightbox==0.1.7

Docker Build and Run commands

# Docker Build
docker image build -t mkdocs-material-local .

# Docker Run
# Run this command inside the folder containing the mkdocs.yaml
docker container run --rm -it -p 8000:8000 -v ${PWD}:/docs mkdocs-material-local

The next step is to fire a browser up and navigate to http://localhost:8000. The rendered site should appear, and it should be automatically updated as the files are updated.

Deploying to Production

All roads come to an end and, in this case, the road ends when the Developer Workflow documentation site is deployed to a Production environment, where it can be accessed by the public. We use a simple setup of Cloudfront + S3, using a Lambda Edge to authenticate the site using Google Workspaces login, but that's out of the scope of this post. Since there are lots of different ways to host a static website, just choose the one that you are most comfortable with or is best for your organization.

In order to keep the deployment also as simple as possible, we just have a Github actions workflow that performs four main steps:

1. Build the Docker image with the MkDocs Material and its plugins.
2. Tune the mkdocs.yaml file a bit for the standalone deployment. For example, the webcontext plugin.
3. Run a Docker container using the just-built MkDocs Material image and invoke the build command, that renders the website and generates the static site in a folder named  site.
4. Copy the contents of the site folder to the static web hosting. In our case, this is the Amazon S3 bucket. Here, we also create a Cloudfront invalidation to the distribution, to prevent the users from waiting until the Cloudfront cache expires.

Github Workflow: publish-website.yaml

name: Publish Website
on: [push]

jobs:
  cicd:
    runs-on: [self-hosted]
    # Omitted for readability

    steps:
    - name: Checkout
      uses: actions/checkout@v3

    - name: Build docker images
      run: docker image build -t mkdocs-material-custom:local .

    - name: Parse mkdocs.yaml
      run: |
        # Install yq tool
        wget https://github.com/mikefarah/yq/releases/download/v4.21.1/yq_linux_386 -O ./yq && chmod +x ./yq
        # Use default webcontext and disable use_directory_urls to make it work with CloudFront
        ./yq '.plugins[select(. == "webcontext")].[].context = "/" | .use_directory_urls = false' -i mkdocs.yaml

    - name: Build mkdocs site
      run: docker container run -v $PWD:/docs mkdocs-material-custom:local build

    # Omitted for readability

    - name: Deploy
      if: github.ref == 'refs/heads/main'
      run: |
        aws s3 sync site s3://xxxxxx.company.org --delete --cache-control max-age=3600
        aws cloudfront create-invalidation --distribution-id XXXXXXXXXXXXXX --paths '/*'

The following are screenshots of the real result, so you can see how DW documentation looks both as a standalone MkDocs website and when integrated into Backstage.

References

• MkDocs documentation: https://www.mkdocs.org
• MkDocs Material documentation: https://squidfunk.github.io/mkdocs-material
• List of MkDocs plugins: https://github.com/mkdocs/mkdocs/wiki/MkDocs-Plugins
• Backstage documentation: https://backstage.io/docs/features/techdocs/techdocs-overview
• Diagrams made with https://www.diagrams.net