Go Virtual Filesystem —The Image File Format (failed attempt 1)

There’s no code snippets this week — just a lot of disaster.

This basically describes how this week went.

The end is nigh and version 1 is almost ready to be packaged, shipped and exit our lives. This project really has gone much smoother than I ever imagined writing a filesystem could’ve ever been. It makes me wonder if I did it properly, you know what I mean? Like if I did it “correctly”. More on that existential dread some other time; let’s tally up what we did in the past seven days.

As the foreshadowing in the heading implies, I was unsuccessful this week at making a filesystem image and I have no idea why. I know what I was trying to do and it just didn’t work. File writing in a single stream-like manner I found is a lot easier in C because in C all bytes are equal but in Go some bytes are more equal than others.

So, here’s what happened.

The Overview — Our Filesystem Image Format

The idea I’m going with for now is to have a structure of contiguous data chunks where each of these chunks represent a single file. It goes something like this:

type fileBlock struct {
   name string 
   rootPath string
   fileSize int
   content []bytes
}

Each filesystem image would be a bunch of packed structures next to one another in the form of the above; the idea being that when an image is loaded, a loop would unpack each “struct” and rebuild the filesystem using the information packed in each.

Here’s how it all went wrong.

Writing Byte after Byte

I decided to keep things super simple, or at least what I assumed would be simple, and stay away from packing libraries of all kind because this is an exciting domain that I’d almost certainly get carried away with. There’s so many libraries I could use just for the hell of it from FlatBuffers to ProtocolBuffer to the simple struc Go package but I decided to save this for a separate project entirely — and I have a deadline.

I scrapped all of those and opted to just write the data old school-like; open a file, write the data, write a “separator” in the form of a pattern of bytes, write some more data and so on until it’s done.

The packed data would look something like this in a file:

[filename]\x88\x88[rootPath]\x11\x22\x33\x44[data]\x44\x33\x22\x11[filename]...

Simple enough in theory. The unpacker would loop through this really long byte array, determine what value it needs to unpack based on the previous separator bytes and unpack and build it into the filesystem, be it a file or directory. \x88\x88 implies the rootPath is to be unpacked after the separator, \x11\x22\x33\x44 tells us that the most recent filename’s file’s data comes next and \x44\x33\x22\x11 tells us that we’ll be working on a brand new file structure at that point. What could possibly go wrong.

The Strategy

It goes without saying that this didn’t go as planned. Mostly for reasons I haven’t quite grasped. My separators for some reason didn’t seem to ever be in the correct places. And because I opted to use a method of unpacking that didn’t rely on custom data structures like structs, I couldn’t really see what I was doing so debugging was a nightmare. That and the code became so convoluted that I couldn’t tell what I was doing so to hell with it I said; no code snippets with week. I’m gonna start again and use the rest of this post to talk about what I’ll be experimenting with for the next seven days.

And I’ll be working with…

Protocol Buffers

Protocol buffers (or protobuf) is a nifty opensource library from Google designed to make packing data really easy. I was put off by a lot of the setup that was associated to it’s C/C++ counterpart what with it’s need for it’s own compiler but that’s just projecting and letting my past decide my future so head first into protobuf I go. I’ve never used them before but now have the perfect excuse to.

I guess my follow up article will be going into how I end up using protobuf to handle my filesystem image’s structure.

In Conclusion — No Progress Was Made

So this is my first “nothing was done and I totally failed” update and I’m not sure if these are worth really documenting.

My consistent flow was broken by my recent failure which, admittedly, I didn’t expect to happen as things were going so well. So live and learn I suppose.

There’s no repo link this week for our progress. Last week’s triumph is still as far as we are.

Protocol Buffers can be found here: https://developers.google.com/protocol-buffers/docs/gotutorial

Onwards to completing our image.