Optimizing a codebase with gradle modules
Introduction
Gradle multi modules (or multi projects) is a feature of Gradle that helps with modularization of software projects in the JVM world.
As the projects is growing, it is wise to start splitting it into smaller components so its management becomes easier. The reason for this is that when the codebase grows problems with the build process and collaboration start emerging.
To solve this, a lot of teams start breaking the components into microservices which brings a new set of problems. Here i will talk about breaking the code into multiple gradle modules and what benefits that brings.
I will share some benefits that i have seen from this feature and more specifically for the following view points:
- Domain design
- Build speed of the project
For the purpose of this post i will use as an example an ecommerce application.
Domain design
Lets say we have an ecommerce application where:
- customers create orders using our web interface
- we submit payments using a payment processor
- we have to send invoices
Packages
Normally if would want to structure our code in following domains, we would create the root directory and add each of the packages inside it.
com.examplecom.example.invoicecom.example.paymentscom.example.notificationcom.example.order
This indeed does work and can take us far but we will have a problem when the project grows. We know that gradle relies on its cache in order to avoid executing tasks. When we change the source code, gradle has to evaluate if it should recompile already compiled sources.
What happens when the package gets bigger and we add more members that change the source code?
Having a single module means that all the developers will work in the same spots. This results into gradle having to throw out its cached outputs and reevauluate everything very frequently which increases the compile times and has a higher risk of conflicts since the changes get concentrated in smaller set of source files.
Modules
We can leverage the multi module functionality to solve the problem:
ecomerce-app
|-order
|-invoice
|-payments
|-notification
This effectively moves each package to a dedicated module and now we have to start defining the relationships amongst them.
Our orders module needs to submit payments and send emails to the customers, so we depend on those modules:
// order/build.gradle.kts
dependencies {
implementation(":payments")
implementation(":notification")
}
The invoice module on the other hand does not need the payments module but it does need the notifications module.
// invoice/build.gradle.kts
dependencies {
implementation(":notification")
}
With this approach we now have clear boundaries between modules which gives us the following benefit:
The modules and their relationships draw an outline of the architecture of the project by sketching what are the components of the project and how they related to each other.
It is now visible that we have an invoicing module but also a payment module which have different responsibilities. Having them as packages inside a single module does not make the distinction clear as someone might think that the payments package is part of invoicing for example.
Build speed
I mentioned in the introduction that the structure of the project with a single module affects the build speed of the project.
The reason for this is that gradle works with a graph underneath. Having all the code in a single module means that gradle has fewer nodes in its graph which makes it harder for it to not recalculate the graph each time.
Adding modules allows gradle to build a better graph of the project, since a module adds new nodes to the graph which it can use to skip tasks using its caching powers.
Of course there are cases where gradle cannot skip a task. In those cases it will benefit from multi modules because it will be able to execute their tasks in parallel as we will see below.
Compilation
Lets take the example from above, we have the following scenario:
|<-------[order]
[payments]<-------|
|<-------[invoice]
If we have a change on the payment module, gradle will be able to compile both order and invoice in parallel
this requires this parallel property to be enabled
Cache
We have 2 cases where gradle skips work for a task
- task is up to date (this happens when the module has no changes)
- task was served from the cache (this happens when the module is served from gradle’s cache)
The second case is the one that benefits most by using a module. Thats due to the fact that we reduce the chance of a cached output to be overwritten by a recompilation.
Modules achieve that by constructing better boundaries so gradle can know what to compile and what not to.
Tests
Similarly with the compilation task, tests also benefit for the multi module since they can be also run in parallel.
This requires you to have correct (integration) tests when you use stateful services (databases, caches etc). Since the tests are run in parallel you can run into conflicts if your tests are not written in a way that can handle parallel database insertions for example.
An additional benefit of modules is that we can have tests that are specific to modules. Take as an example the case of the payment module.
Since it interacts with payment providers we might want to have end to end tests with those providers to verify the correctness of our code.
End to end tests though are generally slower than unit and integration tests. Having modules allows us to define a new test suite inside the payment module, which we can run in a different workflow in our (CI) pipeline:
- Normal pipeline that gets triggered on every commit
build:
steps:
- ./gradlew compileKotlin
- ./gradlew test
- Payment end to end pipeline that gets triggered every night
build:
steps:
- ./gradlew payment:compileKotlin
- ./gradlew payment:e2eTest
Conclusion
Gradle is often critized for its complexity and not without reason. Despite that, it has quite some powerful features such as the modules as long as you keep things under control.
Modules is one of its features that has very clear benefits but it requires effort to setup.
During the process of modularizing a project you might reach to a point where the way you split the modules does not work out, for example by ending with cyclical dependencies between modules.
Usually you end up with these cyclical dependencies due to:
- The modules you want to split should in fact be the same module
- The modules you want to connect should not be connected directly but through a third module.
As you can see, resistance from the build system can help with the modularization of your application by identifying flaws in your domain setup.
I personally believe that this effort pays off very fast and its worth to go through. For hobby projects it probably is not useful but for project where multiple people are working together and the codebase is several years old, i find the investment to be worth it.