Moving to a mono-repo: Part 2, The Destination

The adventure continues, diving deeper into Empathy.co’s experience moving to a mono-repo to go open source with Interface X. If you haven’t read Part 1 of this blog series, start there for the challenges we faced. And now on to the successful migration…​

Migrating to a mono-repo

After deciding to use Lerna, we had to develop our migration strategy. If you recall from Part 1, we had several different projects. Each project used different dependency versions, different tools…

The first step we took was creating an empty git repository and initialising a node project inside it. After that, we could install Lerna and create a new project with the lerna init command. Once we had done that, we could start importing our projects into this mono-repo.

Importing the projects

We identified a project without any internal dependencies and without pending work to do. Importing it was as easy as running lerna import <project-directory> --preserve-commitcommand. This command is used to import a git repository into the mono-repo that we had just created. Because we didn’t want to lose the original commits authors and dates, we used the preserve commit flag. And just running that command, we had imported an external project into the mono-repo. So far so good.

Once we had imported a project, we decided to try to sync, clean, and standardise its dependencies. So we upgraded a few dependencies and configured the linter, as some previous projects were still using TSLint, which had been deprecated in favour of ESLint. And finally, we prepared the package.json and a few markdown files for this new open-source era.

Our strategy was migrating the projects with the fewest dependencies one by one. This made this migration process very easy and smooth.

GitHub workflow

To reduce the git friction, we also stop for a moment to choose a better workflow. In the past, we used GitFlow. GitFlow is great, but we weren’t really benefiting from it; we were only getting the disadvantage, which is simply more branches to handle.

We decided to give GitHub flow a try. This is an easy flow with only a main branch. Developers create branches from the main branch, commit code, submit a pull request, and this pull request is merged back to the main branch when approved by the maintainers.

The CI

One of the greatest benefits of using a mono-repo is that we can easily test the integration between every one of our packages every time we push a commit. Previously, someone could upload a change into one of our repositories without realising it was a breaking change that was affecting another package.

But now with the mono-repo, this is another story. Every project inside the mono-repo has to support its sibling development version. Thankfully, GitHub Actions have no limit with open source projects, which allows us to run different actions, executing several steps to ensure that these projects work properly together.

Pull requests

Every time someone uploads a new pull request, two workflows are triggered. They basically do the same actions: installing, building, testing and linting all the projects inside the mono-repo, but they do so with different code states.

One job is for the actual last commit of the branch exactly as it is. If this source branch of the pull request is not updated with the last main branch changes, it will be built without them. This should help the developer know if their code is working properly.

The other job tries to merge the pull request into the target branch and ensures everything keeps working fine there. This allows us to know if the new code will work properly when integrated into the main branch.

Alpha releases

We wanted to automatise the alpha releases, and fortunately, GitHub Actions provide a way for triggering workflows just by clicking a button. In our case, this job — after installing, building, testing and linting — runs lerna publish to bump the affected projects versions, update its changelog, and finally publish them to the public npm registry.

If successful, these changes are then pushed directly to the main branch. We don’t have to invest time maintaining our changelog or deciding which is the new version for a project. Just by clicking a button, the workflow will be triggered, and if everything goes well, in a few minutes the affected projects will be released.

Stable releases

Like the alpha workflow, we wanted to automatise this as much as we could. However, as we are now talking about a stable release, we wanted to ensure it was properly reviewed by some of the maintainers. So rather than directly push the changes to the main branch and publish the packages to the npm registry, we decided to create a pull request from the manually dispatched workflow.

Github Actions make this pretty easy to do, as it has hooks for triggering jobs when a pull request is merged. Once a pull request from a release branch is merged into the main one, this job will be triggered, and each affected package will be published.

The destination

After all the migration, and a few months working using a mono-repo, we can talk about our experience with this new way to work.

Improvements

There are lots of improvements in our workflow. Using a mono-repo allowed us to automatise some of our most boring and monotonous tasks, which in the end, turns out to be more efficient while developing.

Faster development

Now we are able to iterate faster. Lerna allows us to link the mono-repo projects between themselves, and perform atomic changes to all the packages.

If we want to add a new feature, we can touch as many projects as we want — provided we keep the changes within the same scope and the pull requests at a manageable size. Reviewers will have the full picture when reviewing, and its job is facilitated thanks to Lerna.

No more duplicated dependencies

When Lerna bootstraps the mono-repo, it hoists the shared dependencies to the root node_modules folder. If two projects are using conflicting versions, it warns you about it.

Managing these dependencies is much easier also. As all the libraries we usually work with are in the same repository, you can simply update the dependencies you want in one pull request.

Versioning

Lerna has 2 different ways of versioning projects. The standard mode forces each package to have the same version, something similar to what Angular does with its ecosystem.

The independent mode essentially allows having different versions in each package. We chose this mode, as our packages have different purposes and we want them to evolve freely.

When invoking the versioning process, Lerna decides which projects should have a new version and which it should be, and generates a new section in each project changelog using the commits that affected them since the last release.

Breaking changes

If a developer adds a change that modifies the public API of a project, and another package in the mono-repo is affected by this change, they are the one that has to adapt the package, as they know best how to solve this problem.

The code that arrives at the main branch is guaranteed to always work, and every mono-repo package has to support the last version of its siblings.

Open source

Using a mono-repo to handle open source projects with multiple libraries has become a common pattern. For example, RollupJS plugins use Pnpm, React uses Yarn, VueJS CLI uses Lerna…

It is much easier for the developers to have all related packages into one single repository, and if working with these packages is as similar as possible, even better.

Homogeneity

Thanks to having all projects together, it is much easier to homogenise them. Common tasks like upgrading dependencies, extracting common utilities, testing, linting, building… are now easier to do in the same way.

Faster releases

With Lerna and the new CI, releasing is incredibly easy and fast. Just by running lerna publish, you can have all the projects that have changed since the last release updated with a new version, a new changelog entry, released into the NPM registry, and with the code pushed into your git repository. How awesome is that?

We complicated this release process a little bit more, as we still want to review and ensure that the new versions work properly, especially for stable releases. But overall, this still is a big improvement over our previous release workflow.

New problems

Every solution to a problem comes with some cost. We solved the problems that we wanted to solve, but new ones appeared.

Slower CI

The CI is now slower than before. It has to install the dependencies of every package, build them, and lint and test each one of the packages, which takes some time.

This is the cost of ensuring that all of our packages are able to work together properly. However, this is not a significant issue right now for us, and we can always invest some time into trying to optimise our jobs steps with techniques like caching.

Bigger pull requests

Being able to touch any of our packages from a single branch is tempting, but we have to remember that big pull requests do not work properly. The bigger the pull request is, the easier it is to introduce a new bug, and the harder it will be for reviewers to understand everything in it.

We have to try to limit the scope of our pull requests, and learn to divide our tasks into small steps that guarantee that the mono-repo keeps working properly. In theory, this seems pretty easy to do, but in practice it is challenging.

New tool

Lerna is a tool, and like all tools, you have to learn how to best use it. For anyone who is not used to working with Lerna, it adds another layer of friction for the mono-repo. Fortunately, Lerna is well documented, and its surface API is homogeneous, making it very easy to get started.

Conclusions

Lerna has proven to be a tool that works really well for managing our mono-repo. It clearly was designed to do this job, and it does it brilliantly. Its different commands made our migration process very easy, and keep helping us each day.

I consider the mono-repo a big improvement over our previous way of organising projects and working. When you have to deal with lots of different projects with dependencies between them every day, a mono-repo is a much more comfortable and efficient way to do it.

If you are curious to know more about how we configured our Interface X Mono-repo, please have a look and feel free to leave us any questions or suggestions.