A journey on the growing complexity of a project scripting tasks. Introducing a task runner to address common developer pain and frustrations.
In my previous life as a C++ developer good old Makefiles did the job just fine. When I started coding in Javascript, Gulp and Grunt were all the rage, but now people seem to be satisfied with npm run alone. So why should anyone need yet another task runner?
Every time I start a Node.js project I define basic scripts to open my editor, build the code and run tests.
"scripts": {
"code": "code my-project.code-workspace",
"build": "tsc",
"test:w": "vitest watch",
"test:ci": "vitest --run --allowOnly=false"
},
The main value of this approach is that everyone knows npm scripts are a thing, and will look into a package.json file or just type npm run to get an overview of what is available. We had Gulp and Grunt for a while, but it seems like the added complexity was not worth it. Still there are many limitations: no comments, no dependencies, only one liners, etc.
Most languages have their own expected task runner, but there is one universal solution that shows up when the tasks at hand grows large: Shell scripts!
├── bin
│ ├── build--code.sh
│ ├── build--docker.sh
│ └── publish.sh
└── package.json
With this we can code more complex tasks and get some space to write comments. Did you notice the double dash looks like BEM naming? That’s quite handy for self documenting scripts.
Now that I can write all the comments I need, but I want so much more than that. Here comes some Task Folders for DevOps.
├── bin
│ └── build--code.sh
├── package.json
└── tasks
├── build--docker
│ ├── _build
│ ├── _issues
│ ├── app-start.sh.ejs
│ └── index.sh
└── publish
├── _data
├── index.md
└── index.sh
Creating a whole folder might seem overblown, but in some cases it is totally justified.
The bin folder tends to be a great middle ground compromise. The one for build—code can stay there since it will not get more complicated, but the ones under ./tasks show 2 unique cases
tasks/build—docker needs asset files like app-start.sh.ejs to prepare the container along its own build folder before the final docker build
tasks/publish takes commenting to a whole new level with a dedicated markdown file. When I had to publish my first npm package it was not just running a plain npm publish. I started publishing in GitHub because it was a private package, added a GitHub action to auto publish packages, had to think the branching strategy triggers a new private release. Later it became public and I had to decide about a dual publish strategy considering the official npm repository. It was not a one day task, the need itself was constantly evolving too. Planning for it and solving the task was a multi step evolution that needed some journaling and note taking.
Which place could be better for those notes other than the folder for task itself? Jeez, I do not even need to remember how or where I created those notes, they are next to the place where I have to get the job done. It was also a place were I started curating a SOP (Standard Operating Procedure) about how I should proceed with every package I publish. Also an index to track the packages I am responsible for. As you can imagine, several notes forked from this folder alone. Some were kept for the project, others for my private collection of learning notes. But all came interconnected nicely together just linking boring UUIDs instead relying on a data greedy app.
Do you know that feeling of “I did this before, where was it?”, well, that is no longer the case since I can quickly cross link snippets of code that do a similar job.
There is also the irony that now publishing a package is so simple and straightforward, that once I mastered the task and relocated the notes, I no longer have a real need for a whole tasks/publish folder.
Naming the files index.sh is also a convention to identify the main script to run the task, but it could also be index.py or index.ts. Using a full programming language (aka. not Bash/Zsh) is useful when the tasks are so complex that there is a need to reuse code with a shared library.
Having folders named _build and _data is also another convention recognized by TaskFolders. It makes easier highlighting special folders with a recurring or expected job.
Having multiple script systems in place can be confusing when trying to get an overview of the tasks you can run: there are more places to explore.
The scripts within the bin will not auto-magically be available everywhere in your environment.
Once you break down a project into multiple packages, some nice to have scripts for every project will not be globally available. For example, as soon as I visit a package folder in a monorepo and want to start my editor with npm run code it will not work. Npm run scripts only considers the first package.json it can find, so there is no easy way to reuse scripts.