## Linux Kernel Hacking

Ever since I started using Linux in 7th grade, I have been fascinated by and fervently
believed in the Free Software Movement and the projects that emerged as a result. One
of these projects is the Linux Kernel. Started as a small hobby project and to solve
a real problem by developer Linus Torvalds, it is now the most widely used operating
system in the world. While nowadays it is mostly funded and developed by corporations,
non-trivial amounts of development are done by hobbyists and hackers. Some people work
on simple drivers for pieces of hardware that they have that are not yet supported,
while others are trying to bring [Linux to Apple hardware](https://asahilinux.org/ "Asahi Linux").

Naturally, I wanted to join the community of kernel hackers (or contributors). I
found Greg Kroah-Hartman's [video at FOSDEM](https://www.youtube.com/watch?v=LLBrBBImJt4) showing how to write and submit a kernel patch (a fancy word for a piece of code).
For those who do not know, Greg KH is Linus's right-hand man. Greg does it all, and
is one of the most respected members of the kernel community. Also, funnily enough,
while this video was made more than 15 years ago, it is still 100% accurate and relevant.
For better or worse, not many projects can say their development process has remained unchanged for 15 years.

After watching the video, I searched through the directories of projects I was interested
in. At the time, I was familiarizing myself with SELinux, so I looked through the source
tree to try to grok how everything worked and learn the development process. Thankfully,
I was fortunate enough that [Paul Moore](https://paul-moore.com/about), the maintainer of the SELinux subsystem, was
kind and patient enough to take [my code](https://web.git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=65d96351b135c2a694814520a6031f3721020ba9 "My SELinux Contribution").
While it did nothing but appease the static code analyzers, it was exciting for 14 year
old me! I had contributed to the most prolific operating system in the world. Due to
SELinux's use in Android, my code runs on billions of devices around the world. ;)

A few weeks after that, I started poking around in the `staging/` directory. This location
folder holds all the drivers that have been submitted and accepted upstream but are not yet
parts of the core kernel. They remain in `staging/` until the code meets style guides and
features are more fully flushed out. This is where most beginner contributors start. I
fixed a few documentation errors in the [comedi driver](https://web.git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=f6a1a425cbb74f4836f2e21e0110330c65e33ad2).
Greg KH, who I mentioned above, maintains the `staging/` directory. This means that he
was the one who responded to my emails and accepted my code. Pretty neat!

## The Kernel Self Protection Project (KSPP)

Most of my earlier contributions were pretty insignificant. However, I have found a way
to meaningfully contribute needed and valuable security fixes to the Linux Kernel via the Kernel Self Protection Project.
Founded by Google employee [Kees Cook](https://github.com/kees "Kees Cook's GitHub"), the project is dedicated to making the kernel
more secure and more resilient to attacks. Specifically, his work focuses on making
C safer by default and fixing bugs long before they are ever discovered. [Gustavo Silva](https://embeddedor.com/ "Gustavo's website")
is another very active contributor. In fact, he has reviewed some of my code, which
is neat considering his presentations are responsible for my involvement in the KSPP. Thanks, Gustavo!

While I may not have the knowledge to rewrite a driver from scratch to be more
performant or have the skills to reverse engineer Apple hardware, I am capable
of critical thinking and analyzing small sections of code in detail. This is what
the KSPP is about. Most of the changes within the project
are small changes: a few lines of code here or there. Some are as simple as changing a single
character from the `strncpy` function to `strscpy`! But there is a purpose behind
each patch.

There are over 100 open issues on the [KSPP's GitHub tracker](https://github.com/KSPP/linux/issues).
Each one focuses on a different part of the Kernel. While some issues focus on
working towards the enabling of a compiler flag that hardens the kernel, some are
design changes and code changes. One of the easier ones to learn about and understand
is the migration from [open coded arithmetic to helper functions](https://github.com/KSPP/linux/issues/162).
Essentially, instead of using `kzalloc(nr * sizeof(*int), ...)` we should use
`kcalloc(nr, sizeof(*int), ...)`. This change not only increases code readability,
but there are built in protections within `kcalloc` against overflows due to casting or memory weirdness.
This is a really easy way to contribute and make the code verifiably safer and better.

The past few weeks I have sent a litany of patches on to the mailing lists changing
code in the [erofs](https://web.git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=a78c5c2269e39544fab4140105f55fca4502e915), gpu/drm, scheduler, thermal, and sound subsystems. This has been an excellent way
to practice using tools like `b4`, `git rebase`, writing good commit messages, and
learning the netiquette of email-based workflows. I even found a set of code to
practice sending multiple patches, or a series, at once. I sent [4 patches](https://lore.kernel.org/lkml/20250227-amd-display-v1-0-52a060a78d08@ethancedwards.com/) for
various AMD drivers in `gpu/drm/amd`.

## Getting started

For those who are interested, I recommend watching the above video and reading
the [official kernel documentation](https://docs.kernel.org/process/submitting-patches.html) on contributing.
After cloning the source code, run `git grep -n kzalloc | grep ' \* ' | grep -v ':\s\+\*'`
and find instances of code that can be improved! Sometimes while searching
for KSPP issues to fix, I come across ways to make the code better in other ways.
For example, in the sound subsection I removed more than 30 lines of code by
[combining two functions](https://web.git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e576e7843c0d65b82d4092e5b386d9fbf5bc10c3).

So, my final advice for readers who are interested in learning to hack on the
kernel is to jump in! It takes some persistence, especially when learning the
tooling, but the effort is worth it. Find a subsystem that is interesting
and exciting and start poking around. Think of ways to improve code, break it,
fix it, and do it all over again. It is incredibly gratifying and rewarding
to have one's first contribution merged! If this little blog post has inspired
anyone to get started and submit their first patch, please CC me on the email
and I would be more than happy to review it and provide feedback.


Learning Kernel hacking and increasing security, courtesy of the Kernel Self Protection Project!

The KSPP and Linux Kernel Hacking

## Background

I started using Nix and NixOS around four years ago because I appreciated the declarative
nature and features it brought. At the time,
I had an internship with a local web-hosting company and one of the problems we faced was
our bring-up process for provisioning new servers. One of the company's goals at the time
was simplifying the process so that an admin could run one command and we would have a
functional system that could integrate securely with our Docker Swarm.

We explored various tools like simple Bash scripts, more complex Python scripts, and even
Ansible (which is really just a glorified Python script), but each of them presented the
same issue: state. We did not have an easy way to keep track of when things changed.
Further, when we did push a change with Ansible or modified one of our scripts, we could
not guarantee that the change was made on each system without manually checking them
ourselves.

After some research and tinkering, I proposed that we try NixOS. We gave it a try and in a few weeks had switched
over most of the systems. While the company eventually switched back over to Debian, it
proved to be an invaluable experience using Nix in enterprise systems. Its (mostly)
stateless configuration options were exactly what I needed. The configuration files became
the single source of truth for our systems, and we did not have to worry about forgotten
packages and services becoming security vulnerabilities. If it was not in the file, it
was not on the system. This idea was only strengthened when we implemented [impermanence](https://grahamc.com/blog/erase-your-darlings/).

At the same company, we had a variety of custom Docker images for products developed
in-house and others with some slight modifications for popular pieces of software like
WordPress. One of the tasks I was assigned was to create a CI (Continuous Integration) system that built and
pushed the Docker images to a container registry. In about a day, I figured out how to
integrate with GitLab CI and push to our instance's GitLab Container Registry. This was
my first taste of using CI to automate the DevOps process.

Nowadays, I use GitHub Actions (GHA) for a variety of jobs. For example, this website and the
APIs that support it are built by GitHub Actions. Each code repository has a Dockerfile
and GHA workflow file that builds an OCI image and pushes it to a registry.

While I was at this internship, I also started migrating my personal devices over to
NixOS. I went pretty deep down the rabbit hole. However, I started high school a few
months later, and with sports, extracurriculars, and homework, I mostly stopped
tinkering with NixOS and largely stepped away from Linux and the Open Source community.
A lot has changed since then!

As my high school graduation approaches and I start to have more free time, I have picked up some of
my old hobbies again. The past three months I have dived back into the world of NixOS and
resumed my contributions to [nixpkgs](https://github.com/nixos/nixpkgs). Additionally,
I declared "nix bankruptcy" and rebuilt my [configuration](https://github.com/ethancedwards8/dotfiles/tree/master/.nix)
from scratch.

## Integrating GitHub Actions with my NixOS and nix-darwin configurations

This whole time, I had been thinking about how I could integrate my NixOS system and
nix-darwin (which offers NixOS-like reproducibility on MacOS) system configurations with
GitHub Actions like nixpkgs does. I thought I could benefit from automatic builds
that are cached by [Cachix](https://cachix.org), especially when building my Raspberry
Pi's configuration. It turns out, I'm not the only person who has thought of this. Domen Kožar,
founder of Cachix, utilizes GitHub's free CI system to [build and apply](https://github.com/domenkozar/homelab/blob/main/.github/workflows/deploy.yml)
his system configurations to his computer automatically. While I do not need the CD (Continuous Deployment)
service offered by Cachix, I am a customer of Cachix's caching service. I decided to model my CI after his.

GitHub Action Workflows and Jobs are defined by YAML files in the `.github/workflows` directory.
Below is the code that creates the jobs that run on the GitHub Actions runners. I use the
`x86_64-linux`, `aarch64-darwin` (MacOS), and `aarch64-linux` runners that are defined below. They
build my NixOS, nix-darwin, and NixOS-arm systems respectively. Apparently, the ARM runners are
[a recent addition](https://github.com/orgs/community/discussions/148648#) to GitHub Actions and are experimentally supported.

<SyntaxHighlighter language="yaml">
{`
name: Build & Push

on:
  push:
    branches:
      - master

jobs:
  build:
    name: Build and Push Config
    strategy:
      matrix:
        os: [ubuntu-latest, macos-latest, ubuntu-24.04-arm]
        include:
          - os: ubuntu-latest
          - os: macos-latest
          - os: ubuntu-24.04-arm
    runs-on: \${{ matrix.os }}
`}
</SyntaxHighlighter>

After the runners are defined, we have to declare what they are actually going to do.
Below is the code that installs Nix into the GHAs runner environment. Cachix has an
action that nicely integrates with GHA to quickly install Nix and configure
pushing/pulling from Cachix.

<SyntaxHighlighter language="yaml">
{`
    steps:
      - run: sudo rm -rf /opt&
      - uses: actions/checkout@v4
      - uses: cachix/install-nix-action@08dcb3a5e62fa31e2da3d490afc4176ef55ecd72 # v30
      - uses: cachix/cachix-action@v15
        with:
          name: ethancedwards8
          authToken: "\${{ secrets.CACHIX_AUTH_TOKEN }}"
`}
</SyntaxHighlighter>

Next is the actual building of each system. Obviously, MacOS cannot build Linux systems
and vice-versa (at least without the use of emulation/virtual machines, but that is
too much overhead when I can just use multiple runners). Consequently, I use conditional
statements to only build x86 targets on the x86 runner, Darwin targets on the MacOS runner,
and aarch64 targets on the ARM runner. The targets are the different Nix flake outputs
defined in my [configuration's `flake.nix`](https://github.com/ethancedwards8/dotfiles/blob/master/.nix/flake.nix).
At the very end of the file, Cachix will push the result of any of the builds to the
caching service, allowing my downstream computers to pull the results from them, saving
time and energy. Essentially, I am offloading the building of configuration files, USB
ISOs, development environments, and custom packages to GitHub.

<SyntaxHighlighter language="yaml">
{`
      - name: x86_64-linux Build
        if: matrix.os == 'ubuntu-latest'
        run: nix build -L .nix#{nixvm,usb,archpc,devShell.x86_64-linux}

      - name: aarch64 Build
        if: matrix.os == 'ubuntu-24.04-arm'
        run: nix build -L .nix#{nixrpi,devShell.aarch64-linux}

      - name: MacOS Build
        if: matrix.os == 'macos-latest'
        run: nix build -L .nix#{mbair,devShell.aarch64-darwin}

      - name: Deploy
        run: |
          cachix push ethancedwards8 ./result*
`}
</SyntaxHighlighter>

The biggest benefit I get from this setup is the building of a USB ISO that has my configurations
on it. It builds each time I make a change to my setup or update my inputs, meaning I always have
the freshest software and newest configurations that I can flash to a flash drive to use as a
portable Linux machine. Instead of having to build it manually each time I need it, GHA automatically
builds it and Cachix stores it. When I need it, I run `nix build` locally and it is retrieved from
Cachix and downloaded into my local Nix store.

While my use case is somewhat academic and not super practical, there are instances
where I, and others, have used GitHub Actions to automate and accelerate the development process
in software development. Popular new terminal emulator [Ghostty](https://github.com/ghostty-org/ghostty)
written in Zig is an excellent example. Their developers use Nix to build and create development
environments that are built by GitHub Actions and cached. This allows users and programmers to
do work faster instead of waiting for things to compile or Nix to finish evaluating.


Learning to take advantage of GitHub action's powerful CI system with Nix!

Building Nix Systems with GitHub Actions and Cachix

## Background

A few years ago, I started learning about the Pretty Good Privacy (PGP) security model. Essentially, it is a
collection of tools that allow you to encrypt, sign, and authenticate using cryptographic technologies.
Common uses include signing Git commits and tags, sending encrypted emails, publishing signed messages, and more.
I was really into online privacy at the time, so I thought this was something I should learn more about.
The whole idea is that an individual has two types of keys: public and private. The public key can
and should be shared with the rest of the world. It is your identifier. Private keys are, like the name suggests, private.
Messages are encrypted with your public key and decrypted with your private key. Everyone has a master key. Some individuals
choose to allow their master key to also serve as Authentication, Signing, and Encryption keys, but this is generally
considered insecure. The most important job of the master key, however, is creating subkeys and revocation certificates.
Subkeys can be assigned any combination of Authentication, Signing, or Encryption roles. Revocation certificates allow you
to permanently proclaim that your key is no longer in use and disable it. This can be done for a number of reasons like
having your private key compromised, but also for more benign reasons like you wanted to upgrade to the newest
cryptographic algorithm. The most common practice is to create the master key so that it only has the ability to
create new subkeys, and disallow it from Encrypting, Signing, or Authenticating, leaving that up to three individual subkeys.
There are also identities. Multiple identities (a name and email address, with an optional comment) can be stored in a single key.

A graphic may help us understand this. As you can see below, this is my current master key with a C. Below it are three subkeys
each with a capability (S, E, or A). My identity is also there. Don't be intimidated by those strings of hexadecimal characters.
Those are just the key IDs.

![My subkeys](/blogimages/keyoutput.png)

I immediately wanted to make my own keys. So I did. I followed [Dr. Duh's guide](https://github.com/drduh/YubiKey-Guide/tree/84c9d9654d73ad679aa8554b0819f93f397c61a8), which I highly recommend.
Unfortunately, I was a little bit overzealous when I did this. Like I mentioned earlier, it is generally considered a good practice
to create subkeys for each of the capabilities (S, E, or A). I decided I wanted a separate set of subkeys for each of my devices.
This led to me creating **nine** subkeys. I did not realize it at the time, but they would not play nice with each other.
Also, it was just ugly. Each time I listed my keys to use them or share them, all nine subkeys would appear. I also created three
identities, one for each of my email addresses. Finally, I put comments in my identities, a practice that is [generally frowned upon.](https://web.archive.org/web/20160306174622/https://debian-administration.org/users/dkg/weblog/97)
Afterwards, a few of my close friends and I signed each other's keys, verifying that their identities matched their keys. This
creates the "Web-of-Trust" that PGP is famous for.

One feature of PGP that I have not mentioned yet is its permanence. Once a key is uploaded to the internet, like most things, it is
there **forever**. While you can request that keyservers delete your key, they are not obligated to honor that request and it is
likely that thousands of other people around the world already have downloaded a copy. However, it is possible to revoke a key,
declaring to everyone that it should no longer be used. Because of this, I was hesitant to stop using this key.
For the past four years, I have used my original key, 0xF93DDAFA26EF2458, despite the multiple identities and excessive number of keys.
But today, I revoked my old one and created a new one.

## The New Key, 0x34C04305D581DBF

Armed with experience and a bit more knowledge, I created a new key with ID 0x34C04305D581DBF. A few paragraphs ago I briefly
mentioned signatures. In order to verify that this new key belongs to me and it is not someone masquerading as me, I signed
my new key with my old one. As you can see below, my new key is signed by itself (standard practice) as well as signed by
my old key. Immediately after, I sent the revocation of my old key to the keyservers. Therefore, if you trusted my old key
then you should trust my new one.

![Signatures on new key](/blogimages/keysignatures.png)

Another change I made was which algorithm I used. When I made my original key in 2020, it utilized RSA4096. It takes advantage
of some properties of prime numbers which make it very secure. However, for my new key, I chose to use ed25519, an algorithm
that uses elliptic curve cryptography. While my understanding on this topic is limited, ed25519 is just as secure as RSA4096,
but much faster and has shorter keys.

That being said, if you used my old key and have it in your keyring, please remove it and updated it with my current one
with fingerprint 2E51 F618 39D1 FA94 7A73  00C2 34C0 4305 D581 DBFE.


GPG Key Fun

## Hello World! How does this site work?

This is my blog, which features some pretty cool stuff. For example, code highlighting, quote changers, and an about page.

Code highlighting!
<SyntaxHighlighter language="javascript">
{`
const doStuff = () => {
	return console.log('hey');
}
`}
</SyntaxHighlighter>

It is built on NextJS, React, and Express (although not much express is used.). I also use some pretty cool smaller libraries for various different functionalities.

Blog posts are written in MDX.

Additionally, the cool quote changer on the main page is powered by an [API](https://github.com/ethancedwards8/ethancedwards-quotes/) that I wrote! Powered by Python and Flask.

Although at the time of writing, the site may not be the prettiest, I am confident that it will look good in enough time.

You can view the site source code on [GitHub](https://github.com/ethancedwards8/ethancedwards.com), feel free to submit pull requests if you see anything you can improve on!

## A bit about me

Right now, I am a full time student, working concurrently on both my high school and college degrees. Despite being a student, I do work as consulting and engineering for various small businesses around me. I also write for [StorageReview](https://storagereview.com) occasionally. I have been using Linux full time since 7th grade, and my current distro of choice is [Arch Linux](https://archlinux.org) with LVM on LUKS. My daily driver is a FrameWork laptop, and I have a desktop computer with a Ryzen 2600X alongside 16G of RAM. These machines however are dwarfed in power by my homelab, which consists of a cluster of Raspberry Pis ;). My homelab is currently hosting my site, but it may move locations in the future, but that's for another post.

I also have hobbies outside of tech. I am a cuber, reader, language learner, and mathematics lover. I am teaching myself Spanish through [Refold](https://refold.la), of which I am a huge proponent. This topic also deserves its own post. I plan on writing here a lot, although, everyone says that.

I am also super excited to talk about the books I am reading. I mostly read fiction, primarily Sci-Fi and Fantasy. But I also read lots of non-fiction, ranging from the history of an obscure religion to how electronics work. I love learning and I will read anything that you put in front of me.

## Bye!

This is the end of my first blog post, I hope you enjoyed reading it.


List of posts:

The KSPP and Linux Kernel Hacking

Ethan Carter Edwards

March 9, 2025

Building Nix Systems with GitHub Actions and Cachix

Ethan Carter Edwards

March 2, 2025

GPG Key Fun

Ethan Carter Edwards

December 15, 2024

First Post

Ethan Carter Edwards

June 27, 2022