lucasew's blog posts

Release automation and testing on CI

Fri, 19 Sep 2025 00:00:00 GMT

I am the kind of guy that feels annoyed when doing the same thing twice by process, like fill the same task on two places, so I often look for a way to automate. Also, I do not like to do repetitive stuff often like checking for dependencies on my not so active projects.

For this repetitive stuff of checking for dependencies I rely on the renovate bot[^renovate_bot] but often the users don’t clone the project to use the software, so I had to find a way to setup releases and build artifacts to them. As I am using GitHub Actions because I am on GitHub and I am most comfortable using it, mostly because of choice inertia to be honest.

[^renovate_bot]: Renovate: GitHub

I may translate the renovate post from portuguese[^renovate-portuguese] to english soon.

[^renovate-portuguese]: Renovate post in portuguese: Post

Going back to the release problem, what I did is to basically move it all to an Autorelease workflow. In my GitHub, most of the projects that are not just “Deploy to Vercel” are having that or being archived.

The specific approaches actually depend on the things used to develop that specific software, but it’s mostly the same.

Basically the repository has the following items:

mise.toml: Has the toolchain versions and sometimes the commands to build the repository.
renovate.json: Renovate config file, out of the standard often there are only some automerge rules to merge patch updates and stuff like that. It merges automatically those cases when CI is green by default.
make_release: Changes version.txt, or some other files if needed, to update the version of the project. Normally it’s run by CI. In some Python projects this file is inside the project main module.
Dockerfile: Some of the projects package a service. This file mostly have a standard multi-stage docker build to generate the project’s container.
.dockerignore and .gitignore: Has the output directory of builds and other stuff, so it doesn’t get committed by the workflow. It sucks when it happens!
.github/workflows/autorelease.yaml: The one and only workflow required.

How the workflow is set up

I’ll take the workflow from ts-proxy[^ts-proxy-repo]. I think it’s one of the reference implementations and often the one that I copy over to new projects when I am setting it up on another project, even when the project is not made in Golang.

[^ts-proxy-repo]: ts-proxy: GitHub.

name: Autorelease

on:
  push:
    branches:
    - main
    tags:
    - '*'
  workflow_dispatch:
    inputs:
      new_version:
        description: 'New tag version'
        default: 'patch'
  schedule:
    - cron: '0 2 * * 6' # saturday 2am
jobs:
  autorelease:
    env:
      REGISTRY: ghcr.io
      IMAGE_NAME: ${{ github.repository }}
      USERNAME: ${{ github.actor }}
    runs-on: ubuntu-latest
    permissions:
      contents: write
      packages: write
      attestations: write
      id-token: write
      pull-requests: write
    steps:

    - uses: actions/checkout@v5
    - name: Setup git config
      run: |
        git config user.name actions-bot
        git config user.email actions-bot@users.noreply.github.com

    - uses: jdx/mise-action@v3

    - name: Create Pull Request if there is new stuff from updaters
      uses: peter-evans/create-pull-request@v7
      id: pr_create
      with:
        commit-message: Updater script changes
        branch: updater-bot
        delete-branch: true
        title: "Updater: stuff changed"
        body: |
          Changes caused from update scripts

    - name: Stop if a pull request was created
      env:
        PR_NUMBER: ${{ steps.pr_create.outputs.pull-request-number }}
      run: |
        if [[ ! -z "$PR_NUMBER" ]]; then
          echo "The update scripts changed something and a PR was created. Giving up deploy." >> $GITHUB_STEP_SUMMARY
          exit 1
        fi

    - name: Build binaries
      env:
        TAG: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
      run: |
        echo "::group::Build container"
        docker build -t "$TAG:latest" .
        echo "::endgroup::"

        mkdir build -p && ls && pwd

        echo "# Built targets" >> $GITHUB_STEP_SUMMARY
        export CGO_ENABLED=0
        go tool dist list | grep -v wasm | while IFS=/ read -r GOOS GOARCH; do
          echo "::group::Build $GOOS/$GOARCH"
          GOOS=$GOOS GOARCH=$GOARCH go build -v -o build/ts-proxyd-$GOOS-$GOARCH ./cmd/ts-proxyd && (echo "- $GOOS/$GOARCH" >> $GITHUB_STEP_SUMMARY) || true
          echo "::endgroup::"
        done

    - name: Make release if everything looks right
      env:
        NEW_VERSION: ${{ github.event.inputs.new_version }}
      run: |
        if [[ ! -z "$NEW_VERSION" ]]; then
          NO_TAG=1 ./make_release "$NEW_VERSION"
          echo "New version: $(cat version.txt)" >> $GITHUB_STEP_SUMMARY
          echo "RELEASE_VERSION=$(cat version.txt)" >> $GITHUB_ENV
        fi

    - name: Create release
      if: env.RELEASE_VERSION != ''
      env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          TAG: ${{ env.RELEASE_VERSION }}
          TITLE: Release ${{ env.RELEASE_VERSION }}
      run: |
        gh release create "$TAG" \
          --title "$TITLE" \
          --generate-notes \
          --notes-start-tag $(gh release list --limit 1 --json tagName -q .[].tagName) 

    - uses: svenstaro/upload-release-action@v2
      if: env.RELEASE_VERSION != ''
      with:
        repo_token: ${{ secrets.GITHUB_TOKEN }}
        file: build/*
        tag: ${{ env.RELEASE_VERSION }}
        overwrite: true
        file_glob: true

    - name: Login to registry
      uses: docker/login-action@v3
      if: env.RELEASE_VERSION != ''
      with:
        registry: ${{ env.REGISTRY }}
        username: ${{ env.USERNAME }}
        password: ${{ github.token }}

    - name: "Build and publish container"
      env:
        TAG: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
      if: env.RELEASE_VERSION != ''
      run: |
        VERSION="$(cat version.txt)"
        docker tag "$TAG:latest" "$TAG:$VERSION"
        docker push "$TAG:$VERSION"
        docker push "$TAG:latest"

First things first, if you want to take a workflow like this, you must in the project settings enable an option so GitHub Actions can create PRs. It will not explode if you don’t do that, but it’s an avoidable workflow failure. Not gonna lie, I often forget that when setting up new projects!

As you may have noticed, this bad boy has a lot of triggers.

The push triggers are for commits on the main branch, or master if your project is older. I still have some of these out there. The tag trigger is so the deployment of the new release happens properly on a new release.
The workflow_dispatch trigger is there so I can trigger a new release via the web interface. If ts-proxy receives a Tailscale update from renovate I don’t want to have to ssh to somewhere to run ./make_release patch. I can do it all on my phone, via the browser. By default, if I don’t specify, the release will be of type patch, like in a “x.y.z” scenario it would bump the z. It can be major, minor, patch or a specific version but this last case I never used after I set up this named bump system.
The schedule is basically to run consistency checks to review on saturday morning. Some projects that still use Nix would have their flakes updated even if there is no other activity in the repository.

Now for the jobs part, there is only one. No need for more!

First, there is the env part, where I setup some variables to use below.

The steps part I can separate in the following phases:

Environment setup: sets up Git to be able to commit the consistency check changes and have all that is required to run the consistency checks and the build itself.
Consistency checks: some programs have code generation dependencies or just need to keep lock files up to date, this is the phase that runs those code generation tools, this includes the Nix flake bump.
Pull request creation: if the consistency check step left the Git tree in a dirty state, this part commits this state and creates a PR, if not, the process continues.
Build: this is the phase where the actual build happens, in Go projects, the build happens both for binaries and containers. Nothing is pushed yet in this phase.
Conditional release: if the workflow was launched by the user, the workflow_dispatch route, it will run the make_release without creating a tag and create a release with an automatically generated changelog. Also, this is the part that the artifacts and containers are uploaded. Artifacts become release artifacts and containers are tagged and pushed. If there is no release, the workflow skips those steps.

Now I am using mostly Mise to setup the build environment. This is not the first iteration. I used Nix before and had good results with it, but Nix has a little annoying problem: It forces you to go all-in. Also, it doesn’t allow one to use a specific version of packages without adding a significant amount of complexity. And as Nix is mostly Linux specific it limits adoption of the software being built by it to other platforms. Mise doesn’t have this problem. I used it to setup the JRE on Windows for a Java program I am doing to a company, and it worked really well. Also, the environment setup of those projects is mostly simple enough that Mise could do it in a much simpler and faster way.

This system could be very likely ported to another CI system, and that may be a future project, but the way GitHub Actions and Renovate works is simple enough for this to not be urgent.

And that’s it. That’s how I am shipping software faster and with less annoyances. Want me to set it up for you? Let me know at lucas@lew.tec.br!

How I entered the state of flow reviewing PRs in nixpkgs

Tue, 24 Dec 2024 00:00:00 GMT

For those that don’t know, I am a Linux user, specifically a NixOS user (and I expect memes incoming about this) for over 4 years, 2020-07-37 to be exact, and eventually in the Nix world you will need to engage in nixpkgs somehow.

I have over 100 PRs there, some of them adding modules, some of them adding packages, some maintaining packages and some failed experiments (that some of them I don’t consider exactly failed).

One struggle I had with reviewing PRs is being a bit too superficial, like I can’t test stuff properly. Having whiterun now (Ryzen 5600G) helped a lot building tests because riverwood (i5 7200U) is a little too slow for builds.

From a few months to now we (the NixOS Brazil group) experimented with implementations of GitHub Actions workflows to build stuff for us. Most of the work came from me and thiagokokada. We first wanted a way to clean up space from the GitHub Actions runtime. This is his implementation and this is my implementation. My implementation is much more aggressive and ignorant working on files themselves because, as the name says, I only care about Nix. If it’s not strictly essential for the workflow runtime I am deleting it. He may have tweaked it more since I visit the last time tho.

About the automatic review system, I had built, at first, a workflow that runs in my dotfiles repository, and he committed to his nixpkgs fork. Different approaches, basically the same results.

There is nothing stolen in either repo. It’s all public property :)

Then he commented this.

I was just entered end of year collective vacations (this is a thing here at $WORK) and I was kinda bored looking for a side project when at home.

Seemed like a perfect fit.

At first their idea was to write a cronjob that would getUpdates Telegram then react to messages, then I told them about Cloudflare Workers.

I did some stuff with Cloudflare Workers already, and I loved how fast and responsive things are. No cold starts. At the end of the day it’s all a JS bundle. Would be fun. This was in 2024-12-19.

Lucão é uma máquina de falar coisa aleatória e implementação duvidosa :P (translation: Lucão is a random talk and doubtful implementation machine :P) ~thiagokokada, 2024 - Link

He is not wrong.

I was defending my idea like a good father of ideas would do, showing the concerns, how the technology deals with common problems in the scope.

And I was alone because I was basically the only one that did know about this technology, and that’s when nixpkgs-reviewd was born.

Most of the work happened in two days. Basically the challenge was the following:

A route handler that would treat webhooks from Telegram
That route handler would parse Telegram commands and react accordingly and if the command is well formed, it would trigger a GitHub Actions workflow dispatch workflow with arguments from that command.
That workflow would trigger either one or three jobs that would run nixpkgs-review in each target I can setup. One == only linux. Three == linux + darwin.
Each job would post the result independently.

This approach had some issues, especially around spammy behavior. I needed to iterate on this because some people got angry about that, and I am fine with it. It was a bit rude but, let’s be honest, I deserved it.

After many iterations and bug fixes the current iteration only posts two comments. One to warn that the workflow started, so the people on the PR can get real time information about the state of the build, and another when everything is done, that would post a comment almost the same as the one nixpkgs-review itself would post.

Yeah, there were suggestions around using multi-target nixpkgs-review then setting up workflows as remote builders, but this would cause some issues like:

Remote builders have to have ssh authentication.
Remote builders would have to be running for as long as the slowest build is running.
To connect the builders, there would need to have some kind of overlay networking that is fast, so they can reach each other.
I can get a very close result with much less complexity involved.

Also, while doing this I learned a GitHub Actions thing I really loved. You can add user-friendly summaries that appear when the workflow finishes by writing it to $GITHUB_STEP_SUMMARY. This helped me to resolve the spammy behavior by bringing the details of a run in the summary so if someone wants to know something about the run, the workflow link is there, so you only need to visit it.

How it helped me to achieve flow?

Nixpkgs has some patterns around the PR list.

Most of them are r-ryantm automated PRs, so I could only skip them unless they tag me for being a maintainer.
There are some first time contributors that bring more out of shape PRs that need some iterations until buildable state is achievable.
Other PRs may include manual bumps and backports, miscellaneous fixes, people adding update scripts to let the bot cook the next updates and stuff like that, and refactoring.

The flow came from the following:

I search for PRs that are not created by r-ryantm and not stale: is:open is:pr -author:r-ryantm -label:"2.status: stale"
Open a few in new tab. I only open PRs that already have the rebuild tags in there.
If it looks functionally good, I run the bot. If there is darwin rebuilds I also run the bot with the +darwin flag.
When the bot notifies me that the PR is ready I then look at the results, try to help debug the package and if it’s alright I check if stuff is in the right place, commit names OK and if everything seems mergeable I try to bring more people to look at it either by adding the link to “PRs ready for review” or adding the reviewers manually.
I get an email for each round trip, and email is asynchronous communication, so I look when I am ready.

Note that none of the steps requires me to wait, stay on the computer and react fast. I can put a dozen PRs to review then touch some grass.

I can do most of the work in my phone. Right now only pragmatic tests, like the one I did with chime, require me to stay at the computer, and those I can queue up and when I go to the computer those are basically all cached.

And that’s it. By dogfooding a tool I created based on shoulders of giants I’ve never reviewed so many nixpkgs PRs.

BTW Merry Christmas to you all, and I hope you have a 2025 as wonderful as 2024 was to me!

Hashbang hacking for fun and, maybe in the future, profit

Wed, 17 Jul 2024 00:00:00 GMT

This is the story of how I created a thing to make self contained scripts. It’s basically a Nix flakes based workaround, so the environment of a script can be automatically provisioned before the actual script starts without having to install stuff globally.

I am a Nix user, and as a Nix user, I tend to put Nix in everything. At least everything that I think it makes sense.

Hashbangs are the UNIX-like way to make a text file executable, so it becomes a program, and a pain point I have is that there is so much boilerplate to have something simple to start hacking on something. In a non-Nix distro one would have to install many packages and pray for them to not conflict with each other, build some kind of boilerplate like a package.json for a Node application, or a requirements.txt for a Python application, or a pom.xml, or Gradle, for a Java project. What if I do it a little different?

For the ones that don’t know, Nix allows one to setup temporary environments in a way that you don’t need to install stuff globally. Think of it like a Python virtualenv but using a content addressed storage, so, if you use the exact same packages in more than one project, these are only downloaded once. And if there is a native extension in the middle that requires compilation it can just be downloaded using Nix’s substituters (aka binary caches) if available. And if stuff needs to be compiled you can setup remote builders to a beefier machine so the compilation happens somewhere else and you just get the chewed result from that machine.

Because stuff changes, this ephemeral shell thing changed too. Right now in Nix there is a old way (nix-shell, nix-env and stuff) and the new way (flakes, nix command, right now all behind a feature flag) and some stuff doesn’t map exactly from the old world to the new world. An example is how development shell works. In nix-shell the environment is built using the pkgs.mkShell function, but in nix develop it either assumes that this pkgs.mkShell is already applied or just add the binaries to the PATH environment variable ignoring all the other details such as shell hooks.

Also, nix-shell allows it to be used in hashbangs but the old way is very reliant on the NIX_PATH environment variable, that is basically system state, so there is a limbo in the middle. What if I could use the power of flakes and bring it to a nix-shell-esque style?

That’s what I tried to do in nix-flake-shell.

Basically one adds a hashbang running nix run to the repository and adds a bunch of directives using comments, that would be ignored by the language but interpreted using a Nix script. For it to be recognized the line must only have a #!nix-flake-shell somewhere.

This way one can declare fetchers in a way that the result path is available in an environment variable passed to the payload application, what is the interpreter, which flake inputs to use, which extra packages to bring with all it’s bells and whistles and propagated stuff and Nix dark magic for stuff to be available.

The tests folder has examples of usage of the script. If you know some more exoteric languages and want to contribute with examples feel free to try and play with it, if you found some rough edges or suggestions please feel free to open an issue or a pull request.

BTW this is me exploring the potential of this workaround:

Do not say hello only

Wed, 11 Aug 2021 00:00:00 GMT

Introduction

This post is based in dedicated pages about the subject, such as:

https://no-hello.com/

Very likely, someone sends you here using a link in some sort of chat, like Slack, IRC, WhatsApp, Facebook Messenger, and maybe Telegram after you entered the chat and only sent a “hello”, or a “good afternoon” or any greeting waiting for a response and is curious why this someone sent a link to here.

Did I do anything wrong?

Well, yes and no:

It’s polite to start a conversation with an introduction.
When you arrive only saying “hello” or greeting and waits an answer, you either force the other to ignore your greeting or answer trying to understand the reason of the contact.

Let’s imagine Alice and Fred. Alice is working in something difficult, and it’s highly concentrated in the task while Fred wants to ask a question to Alice.

[fred] Good morning Alice, I have a question!
[alice] Good morning, what is your question?
[fred] * finally unrolls *

In this example, Alice got the message from Fred, stopped what she was doing and asked how she could help. Only when she asked back Fred started typing (at least in WhatsApp, they often send audios, preferentially when I do not have a headphone) and Alice tries to resume what she was doing while Fred types and asks the actual question, which makes Alice lose the focus in her task and very likely will take some time until she goes back to the flow again.

Now with another example:

[fred] Good morning Alice, * unrolls *?
[alice] * answers *

Details to be noted:

Fred did finish typing when Alice sees the message.
Alice can instantly see the question and answer it directly, or with some waiting: She can delay the response until she achieves a checkpoint in her task where she doesn’t need to maintain the task context in her head.

Another level, isn’t it?

How I deal with this stuff

Priority is always text, if the person didn’t yet ask I send a “send it” when I am ready.
If you want me to make some effort to answer fast, do not send audios.
Phone in silent 24/7 with floating notifications off. I look when I take it, and I kinda take it a lot.
If you want the fish it’s because you can pay.
I always try to answer with a link. The site generally explains better than I could explain at the moment. I created this blog to reference my own posts out there hehehe.
Silence is more efficient than rudeness.
Want to exchange some ideas? Nice, I won’t bite. I am often chill when I am not in much hurry, just cut the bs.