Prologue

Welcome - or welcome back - to my Dev Log! In my first coding session for PlatePal, I completed the full setup for my backend and frontend codebases, and documented all of it in Part I of the PlatePal Dev Log . Since I want to stretch the development process over a longer time and have a ton of features planned, I want to reduce repetitive work as much as possible - which is where the magic keyword DevOps comes in. I took a DevOps course at university, which was kind of unhelpful, and was tasked with some smaller automation tasks at work once in a while. However, I’m still very new to the field, especially when working with GitHub Actions instead of GitLab CI/CD.

Nevertheless, I’ve set a few goals for myself that I know will streamline the development process as a whole:

  • Frequent testing of all code, in both frontend and backend
  • Automatic deployment of the backend
  • Automatic building of an APK (I will mainly test the frontend on Android for now)

I might add automatic deployment for the frontend web view at some point, but for now I don’t want to spend too much time on automation, as I am still unsure how far I will take the web view within this first development time bracket.

Overview & General Progress (TLDR)

Tasks I worked on during this session:

  • ✅ Server setup
    • I have a ready-to-go VPS with IONOS, which I want to use for this project. Git, Java, Maven etc. were already installed for a previous project
    • Had to upgrade to Java 17
  • ✅ Backend CI/CD
    • Docker compose to run PostgreSQL database and spring boot application
    • Update and restart docker container on push to main
    • Run tests on push
  • ✅ Frontend CI/CD
    • Very basic CI/CD for now, might add deployment for web view later on
    • Test (npm install) on push, build apk on git tag creation

I will try to lay out the steps I took to make everything work - but be aware that this was not a straightforward process for me. I am very new to (Dev)Ops tasks and have only very little experience with networking, server setups, and Docker. The way I solved my problems might work now, but I cannot tell whether they will fall apart at some point.

Backend CI/CD

Running the tests

As a very first step, I checked whether the one and only test in my Spring Boot project ran successfully - which I should have done more thoroughly. I always have my postgres server running locally - but not on my server. Which means the test ran successfully when I tried it on my development machine, but I later ran into issues with the tests when I realized that loading the context required a configured DataSource. And I had absolutely no idea how to configure a non-permanent database for this purpose.

After a bit of research I figured I wanted to solve this with an embedded database, which can be done by adding the h2 dependency:


<dependency>
    <groupId>com.h2database</groupId>
    <artifactId>h2</artifactId>
</dependency>

I then added the @DataJpaTest annotation to my test class, which will in return autoconfigure a “throwaway” test database at the newly provided embedded DataSource. This test database will then go through the migrations given by liquibase and will be deleted after completing all tests.

During this process I ran into some confusing issues with the test classpath not recognising the changelog-master.xml file I had written for liquibase. The given spring.liquibase.change-log property from /main/src/application.yml was not read properly, which I was able to solve by running mvn clean install in my project.

Deploying manually

After a bit of trial-and-error I figured that I wanted to deploy the backend application as a docker container on my VPS. I had done this before for a different project, so I was able to re-use some of my configurations.

As a first step I had to add a Dockerfile, which describes how my application is supposed to be run within the container. By running mvn clean install earlier, I already had a .jar file ready to be executed.

FROM openjdk:17-jdk-slim-buster
WORKDIR /opt/app
ARG JAR_FILE=target/platepal-0.0.1-SNAPSHOT.jar
COPY ${JAR_FILE} app.jar
ENTRYPOINT ["java","-jar","app.jar"]

Once I change my version number naming in the pom.xml I will have to change the Dockerfile as well, but for now this will do.

I also added a docker-compose.yml to run a postgres container along with the Spring Boot application and allow communication between the both of them.

version: "3.7"
services:
  api_service:
    build: .
    restart: always
    ports:
      - "8081:8080"
    depends_on:
      - postgresql_db
    links:
      - postgresql_db:postgresql_db
    secrets:
      - postgres_password
  postgresql_db:
    image: "postgres:latest"
    restart: always
    ports:
      - "5433:5432"
    volumes:
      - ./data:/var/lib/postgresql/data
    environment:
      POSTGRES_DB: platepal
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
    secrets:
      - postgres_password
secrets:
  postgres_password:
    file: ./postgres_password.txt

The backend application within the docker container is available on the port 8080, but that port is already taken on my VPS, which is why I forward it to 8081. Same goes for the PostgreSQL container: I already have a postgres server running on this machine, so I forwarded the postgres container to 5433.

Both containers use the postgres_password secret, which I stored in a secured file. I ran into quite a few issues with this, because docker is pretty complicated, and docker swarms/secrets were even more of a mystery to me. I would have preferred to store the secret within a docker swarm instead of a file I wrote myself - but I could not get the hang of it. Because of this I also tried out docker stack deploy (as described in this StackOverflow question ) instead of docker compose, but went back to my initial solution.

I also added the volumes configuration to the postgres container, to make sure that all data would be persisted, even when restarting the container.

Additionally, to make this work, I had to edit the application.yml I had previously added all the local configuration to. I split the file to add a local profile with the existing config and used the new configurations as the default.

spring:
  config:
    import: optional:configtree:/run/secrets/
  datasource:
    url: jdbc:postgresql://postgresql_db:5432/platepal
    username: postgres
    password: ${postgres_password}
  liquibase:
    change-log: classpath:/db/changelog/changelog-master.xml
---
spring:
  config:
    activate:
      on-profile: local
  datasource:
    url: jdbc:postgresql://localhost:5432/platepal
    password: ${PG_PASSWORD}

Notice, that the default datasource url contains the name of the postgres docker container that is defined in the docker-compose.yml.

Before automating the deployment, I tested all of these configurations manually on my VPS. To do so, I cloned the git repository into a desired directory, added all of the changes via SCP (WinSCP is such a great tool!), and ran

$ mvn clean package -DskipTests
$ docker compose build
$ docker compose up

With that, the application was up an running and I was able to curl to http://localhost:8081/login on the VPS, to get the default login screen by spring security as a response. I then opened the 8081 port of my VPS via the IONOS dashboard and was able to access the same page via the corresponding IP from my other devices as well. At some point I will probably add a domain name to this address.

Automating the deployment

With the manual deployment done, I was able to automate the whole procedure via GitHub actions. I considered two different processes:

  1. Installing a GitHub runner on my VPS and building and deploying the docker container within that runner. I didn’t like this option all too much because I had to give the runner a few too many access rights to my VPS for my liking. If you still want to check this out, take a look at the github runner documentation:
  2. Using a github action to ssh to my VPS and execute the necessary commands to deploy a new version. This is the solution I decided to use - because it was the simplest way to transform my manual deployment process into an automation.

I looked through quite a few tutorials and documentations for this, so I’ll link you the most helpful ones:

As a result I ended up with a workflow like this:

name: CI/CD
on:
  workflow_dispatch:
  push:
    branches:
      - master
jobs:
  deploy:
    name: Deploy
    runs-on: ubuntu-latest
    steps:
      - name: Deploy to VPS
        uses: appleboy/ssh-action@master
        with:
          host: ${{ secrets.SSH_HOST }}
          username: ${{ secrets.SSH_USERNAME }}
          key: ${{ secrets.SSH_PRIVATE_KEY }}
          passphrase: ${{ secrets.SSH_PASSPHRASE }}
          script: |
            cd ${{ secrets.PROJECT_PATH }}
            git pull origin master
            /opt/maven/bin/mvn clean package -DskipTests
            docker compose up --build -d --remove-orphans
            docker image prune -a -f

What this does is the following:

  • Whenever changes are pushed to the master branch of the backend repository, the workflow is started
  • With the given ssh configuration, the github runner connects to my VPS
  • It then changes its path into the project directory I created earlier
  • All changes are pulled, then built with maven
  • The docker containers are rebuilt and redeployed
  • Old docker images are deleted
Note: I used /opt/maven/bin/mvn instead of just mvn here, because the per ssh connected user did not seem to have the maven path variable set.

Automated testing

A big part of CI/CD is the continuous testing of any written code. However - testing is a weak spot of mine. I rarely write unit tests (even at work) and have no experience with test-driven development whatsoever. I used to always have an eye on test coverage and code quality at the company I previously worked at - but that was mainly because they used a SonarQube build breaker and I really enjoyed the corresponding tools and interfaces by Sonar (so much so, that I installed SonarLint in most of my IDEs now).

I considered adding a sonar check/build breaker as a GitHub action, but quickly realized that I would have to self-host the tool and was not willing to do so. Therefore, I added a simple mvn test run to a test workflow and called it a day.

name: Tests
on: [ push ]
jobs:
  check:
    name: Unit tests
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v1
      - name: Set up JDK
        uses: actions/setup-java@v1
        with:
          java-version: '17'
      - name: Cache Maven packages
        uses: actions/cache@v1
        with:
          path: ~/.m2
          key: ${{ runner.os }}-m2-${{ hashFiles('**/pom.xml') }}
          restore-keys: ${{ runner.os }}-m2
      - name: Run Tests
        run: mvn -B test

Frontend CI/CD

I kept the frontend deployment/testing automation a lot simpler, mainly because I hadn’t decided on a web deployment yet. I might add the website view deployment to my VPS at some point as well, but for now I simply wanted to automate the APK build. I will mostly test on android in the near future, so this will probably be sufficient.

Therefore, I added two workflows - again one for testing, and one for building.

name: Tests
on: [ push ]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Install npm dependencies
        run: |
          npm install

For testing purposes npm install is executed whenever new changes are pushed to the repository. For the APK build gradle is used, which lies within the /android directory. The build workflow is only executed on tag creation. This way I can decide for myself when I want to build a new APK.

name: react-native-android-build-apk
on:
  push:
    tags:
      - '*'
jobs:
  build-android:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Install npm dependencies
        run: |
          npm install          
      - name: Build Android Release
        run: |
          cd android && ./gradlew assembleRelease          
      - name: Upload Artifact
        uses: actions/upload-artifact@v1
        with:
          name: app-release.apk
          path: android/app/build/outputs/apk/release/

With this I ran into two issues:

  1. The /.gradlew file was not executable. This is a known issue , which I solved by executing git update-index --chmod=+x gradlew in the android directory, and then pushing the config change to origin.
  2. Because I renamed the index.js entryfile to add a web view, no entryfile could be found by gradle. I added entryFile = file("../../index.native.js") to the react section in the /android/app/build.gradle file to fix this.

To try the workflow out I created a pre-release with a tag in GitHub and built a first APK. With that, I had completed all the CI/CD I had planned to implement at this early stage.