An abrupt end to an otherwise not so epic journey.
Okay so admittedly I didn’t really know how to round this off. I had one component left to go (the image maker/reader) but then a bunch of life events threw me off my flow in a pretty big way. I lost a bunch of the carefully curated momentum I had going for myself, then proceeded to spend the three or so months ignoring my problems and plough through Persona 3, 4, 5 and 5 Strikers. Was it worth it? It totally was, those games are fantastic. Would I do it again? Probably. What happens to me now? I’m currently thinking of my next lengthy project. Though before moving onto it I wanted to round this one off. I wasn’t able to finish it but I did want to give it closure, I was also on the final step of Ver1 so it would be remiss of me to just not end it.
Taking Stock
The Go Virtual Filesystem. Ah, what a project. It was reasonably ambitious for my first project in Go, I’ll admit that wholeheartedly. But also at the same time, it made the concept of writing a filesystem turn out to be not so hard because of all the hand-holding the language does for you more than it would it, say, C. I think overall to get the best learning experience in terms of writing a filesystem, it would be best to do it in C because the concept of manually having to control buffers, inodes, flushes to disc and so on I think are important learning details if your goal is strictly educational. And since it’s been a long time, I’d like to spend this article giving a brief overview of each component, what I’d like to think I learnt from it and hopefully you did too and what I think could be built on top of it should I ever revisit this project.
1. User management
Creating our Foundation : https://itnext.io/a-virtual-filesystem-in-go-creating-our-foundation-9af62b0e82db
Our First Feature : https://alysonn.medium.com/a-virtual-filesystem-in-go-our-first-features-dea8699d7d2d
This was the first of the components I worked on. It consisted of the laughably simple “user registration” screen that simply prompted you for a name and sent you on your way. There’s absolutely no security in this system; anyone could make anything on it. Just pop your name in and hit enter and you’re good to go.
It could potentially be expanded upon with a sort of IAM system (Identity and Access Management). Think a system that ties a certain privilege level to a single user and allows any directories with matching privilege levels to be accessible. Maybe the user also has a sort of access tag; and this tag allows them to make specific files that are only visible to users that have the same tag.
2. Shell creation
And a Shell Too I guess: https://alysonn.medium.com/go-virtual-filesystem-and-a-shell-too-i-guess-25d4f357d8ad
I’m reminded of the nightmare I endured when having to design a shell from scratch when I was studying and this entire section can be used to relive that horrifying experience.
My initial shell is directly built alongside the filesystem with built-in functionality for shell navigation (cd), file listing (ls), clearing screen and a few other niceties.
A great extension to this would be to find a way to decouple the shell even further. Maybe this makes it easier to integrate completely new, filesystem specific commands as separate programs (?). Another neat idea could be to make admin shell specific tools; maybe throw some user database manipulation functionality to add, edit and remove users from our bootleg IAM system.
Some additions can be purely cosmetic too; a colored prompt similar to zsh or file scrolling.
Admittedly because of how I built the VFS, I couldn’t think of any practical way to make the shell its own standalone object with an identity separate from the entire system. I really wanted to though, I thought it would be kinda cool having two separate projects in one. I quite enjoyed writing a shell back in the day, when it wasn’t making me cry and spend the last of my dinner money for the week on shameful amounts of McDonald’s comfort food.
3. Filesystem Initialization
Completing Filesystem Initialization: https://itnext.io/virtual-filesystem-in-go-completing-filesystem-initialization-2470e3bd8243
I promise the actual contents of this subject are a lot clearer than that lazy heading suggests. This was the the meat of the challenge; creating our files and folders in the correct order as copied from the host filesystem. This part I found incredibly interesting because this is the detail that made Go’s beautiful simplicity completely obvious to me in a way it wasn’t entirely up until this point. I’d been dreading the tedium of handling nested structs inside arrays that had structs that had more arrays.
The idea went like this; there was a single struct for a single directory. Inside this struct were an array of file structs; these structs held info on editable files; think their contents, file sizes, unique hashes and so on. Another list of structs inside a directory structs were (you guessed it) an array of directory structs; these represented the subdirectories found inside a directory.
Frightening as this seemed at the time, my thinking was filtered through the lens of attempting this in C. And in C, we have pointers and memory allocation and manual memory management, details that made any endeavor trickier than it ought to be. But no, this was Go. Doing this sorta just.. well.. it just worked. I don’t really know what else to say.
And now reader, here comes the plot twist reader so brace yourself.
This next part was possibly a massive mistake.
I won’t add any further details about what I’d do to improve this section; we’ll get to that later on. Because the only way this detail could be improved is to be erased from existence.
4. Writing a Text Editor
Adding a Text Editor: https://itnext.io/virtual-filesystem-in-go-completing-filesystem-initialization-2470e3bd8243
This turned out to be an incredibly interesting problem, especially because I work on Windows. This component took an incredibly long time because of host system incompatibility issues I kept running into.
The plan went as such; go online, find a Vim like text editor (for it is always Vim), take its code and bury it deep into the confines of the filesystem’s depths and presto, it would work like magic. So far so peaches and gravy. I even found a Github repo for a project called kilo-in-go that was the famous Vim-like Kilo editor written in Go.
Things were going well. Perhaps a little too well?
I’d go into the source code and start forcefully shoving it into my filesystem as fast as I could hoping to catch my deadline. My first official hurdle; the Kilo editor uses very very very native system calls unique to Unix-like operating systems. The long story short about these system calls (or syscalls) is that they helped with controlling things like interrupts (your Ctrl+Cs and Ctrl+Ss) as well as the behavior of that flickering prompt you see in your terminal which would be used as prompt inside the editor (I’ll link to more info on syscalls at the end of this article, it’s quite the read). Windows didn’t share any with Unix systems. I couldn’t even find these equivalents because who does systems programming in Windows I guess… :(.
My final strategy was to use the web browser as my text editor which ended up being the method I went with. I had to quickly learn to throw together a website in Go overnight, threw in a bit of JavaScript, found a text editor JS framework, got Ctrl+S for saving working and overall tried to make it the most Sublime experience I could (see what I did there :D?).
I guess the only improvement I can honestly think of would be to make different text editor options based on your filesystem. I’d still integrate the kilo-in-go editor and use that when on a Unix system and then the web version when on Windows. I still think the Kilo one would’ve been easier to manage because I could directly work with the code and affect its integration directly whereas the web version, I sorta felt that I just had to deal with what I got. I really felt like my filesystem gave me attitude and it kinda hurt my feelings and I couldn’t do anything to punish it back and I thought that was unfair.
5 Filesystem Image File
The Image File Format: https://alysonn.medium.com/go-virtual-filesystem-the-image-file-format-failed-attempt-1-4d015a5e800f
And this is where our journey comes to an end. Equal parts in fault to my suddenly turbulent personal life, equal parts I really should’ve thought all of this through a bit more.
And now we’ll talk further about the disastrous mistake to make the filesystem’s structure from recursive structs in arrays inside structs. I’d been struggling with this the most because I could safely pack all my data into an image file. The packing was simple enough; first pack all files, then save the name of the directory we’re in, enter the array of directories, do the same to each in the array recursively until we reach the very bottom of the tree.
Unpacking was always the problem.
Everything I tried never seemed to work mainly because the structure I’d given the packed files (namely first a directory, then a list of files, then a directory deep, then it’s list of files, then an even deeper directory and so on). I’d packed them in a depth-first approach where a breadth first would’ve probably been easier to work with. The main issue was controlling how we’d go backwards.
I’ll give a more concrete example.
Say the directory we’re currently in is /first/second/third/fourth levels deep and then right after that, we have a directory first/fifth level right after that. We’d have to go back two directories. It gets worse because, sure, I could’ve just split the strings, and then count how many directories back I had to travel, but all of this was in a single bytestream; basically right after the end of this string would be contents of a file.
I found this a lot easier to think about rather than to pull off, especially because of the mess that the recursive structure I’d given my filesystem caused in my brain. It led to messy, unmaintainable code that I’d get confused working with way too often.
I also feel that because of the size of the redesign I’d want to give the filesystem, it would be better off to just move on to a version 2 with a better design and a few more of the shiny stuff I’d want to have originally added. It also helped that this was the final component to the first version in the first place so I feel no guilt :D.
And That’s It
I’m a little bummed out that I never got it working perfectly but I’d want to redo it anyway. It reached a point that I simply wasn’t enjoying it very much. My “short” hiatus from the project stifled my momentum and I’ve been struggling to get back to feeling enthusiastic about this project for a while since so I’ve decided to leave it here.
Filesystems are pretty cool and I feel a little guilty for not having implemented it without the complexities of inodes and disc blocks though for now, my curiosity about the topic’s been satisfied.
Thank you reader for sticking with these articles through my sudden erroneous schedule. I’ll be cleaning up the filesystem repo in the coming days and will begin work on my next project soon.
Hopefully this time my planning is a bit better ‘:D.
Edit: so I don’t know what I want to make next. I’ll try to work on smaller projects I can finish in short periods of time to keep my momentum fresh and pushes frequent.
I’ll also be polishing up the VFS repo I made to round it off: https://github.com/AlysonBee/GoVirtualFilesystem
Cheers
Alyson
Here’s a few helpful resource I consulted for my doomed expedition down this dreaded path:
Syscall (System call) stuff : https://en.wikipedia.org/wiki/System_call
Kilo-in-go: https://github.com/bediger4000/kilo-in-go
Inodes: https://en.wikipedia.org/wiki/Inode
Great books on filesystems:
Practical Filesystem Design : https://www.amazon.com/Practical-System-Design-Dominic-Giampaolo/dp/1558604979 — This one I found would’ve been useful if I’d actually gone through the entire thing but I mostly just used it as reference material to ensure I was using my lingo correctly.
UNIX System Programming : https://www.amazon.com/Unix-System-Programming-Keith-Haviland/dp/0201877589--This one I used mainly for the shell stuff and also, admittedly, read it a long time ago but the principals about how shells work and make use of command line tools is helpful.
