A Go project to get me started in Go and because filesystems are cool.
So today I decided I was gonna get started in pursuing a glamorous and innovative career contributing to opensource. I’ve had my eye on Go for a long time but never really committed to really learning it, believing that C was the only low level language for me. However, there are so many interesting projects in Go across Github to sink one’s teeth into and even Go itself is like the highly intuitive and functional offspring of Python and C — the only two languages I could really do any damage in before now. It’s got that beautifully clean and easy management of data akin to Python with the pointer-y and efficiency of C — sort of; I don’t want to be quoted on that. It also doesn’t enforce OOP which is a massive plus.
So here I am, reader, ready to jump into Go by building a super awesome project that I was absolutely gonna get to writing eventually/totally not kidding… but then didn’t. I had plans to cut my teeth on a virtual filesystem in C but then I spent a year building a C compiler instead, that I then went on and refactored from Java into C (so now the C compiler was written in C) and now the language makes me feel a bit ill when I have to think of writing in it.
I thought writing a filesystem and then documenting it would be a great idea. I’d learn Go, try my hand at tutorial writing and finally cross that filesystem project off my to-do list. Well, I say filesystem, we’ll be taking a lot of liberties. The benefits of it being virtual is that all that paging and memory mapping stuff can be glazed over if not outright ignored.
So for any of you interested in following along, sit back, relax, grab your text editors and let’s set up our story because every good struggle and great protagonist needs an equally diabolical and convincing villain. So, let’s… Go (see what I did there)?
Now, The Project:
Every great project needs a great problem to stand triumphantly atop so to make things interesting, our filesystem will do a few things differently.
So here’s the origin of the problem we’ll be tackling; imagine you’re sitting there, needing to make a small copy of a small part of your filesystem to send to a friend or coworker. You could just make your changes, zip up your files and send them through. However, what if you need to make edits while still keeping a previous version of all those files and folders you’ve now tainted? That’s a lot of Ctrl+Zs you have to do to go back on all those changes. So our filesystem will address this nonexistent issue by allowing us to edit a virtual copy of a directory and all its subdirectories, all in virtual space and instantly save a copy of this edited “mini-filesystem” ready to be sent off to a friend or to the cloud. This is the premise on which everything we do will be built upon.
The ideas aren’t anything revolutionary, but apart from writing a filesystem tutorial, we’ll get our hands dirty with a lot of Go’s syntax; from linked list implementation to hashtables, data structures and the flavorful standard library I’ve heard so many good things about and with a bit of luck, we’ll have a sort of working filesystem. As a safety precaution, I recommend you build and test this code in a folder that doesn’t have a lot of subdirectories. Running this in a folder that has the Dota 2 binary would probably kill most of your RAM, so safety first. But more on that when we actually begin.
The Prep Material
As I’m writing this, I’m still preparing and getting comfortable with a lot of the techniques I’ll need to understand to pull this off in Go by doing short exercises in the language. Mainly to teach myself some concepts but also to test small versions of the larger implementation details to come; things like using a linked list to mimic directory traversal, using key/value maps to mimic the file/contents relationship and so on. Basically, there’s quite a lot I don’t think I know and I’m worried I might not be able to pull off a lot of my ideas so this practice run is a necessary detour.
The first exercise was basic linked list manipulation and directory traversal; I figure this is a great first step. This has to be completely crystal clear from the very offset or bad things can happen.
It wasn’t so bad; intimidating at first as my brain had to resist the urge to just “do it in C” and be done with it.
The code for my efforts looked something like this.
package main
import "fmt"
// a bidirectional list node that holds a single int
type list struct {
number int
next *list
prev *list
}
// newList initializes the list with a new number
func newList(number int) *list {
return &list{
number: number,
}
}
// pushNumber appends numbers to the end of the list with each call
func (numbers *list) pushNumber(number int) error {
currentNode := numbers
for currentNode.next != nil {
currentNode = currentNode.next
}
currentNode.next = newList(number)
currentNode.next.prev = currentNode
return nil
}
// goBack traverses backwards by 1 node step
func goBack(numbers *list) *list {
if numbers.prev == nil {
fmt.Println("Can't go back")
} else {
numbers = numbers.prev
}
return numbers
}
// goForward traverses forward with by 1 node step.
func goForward(numbers *list) *list {
if numbers.next == nil {
fmt.Println("Can't go forward")
} else {
numbers = numbers.next
}
return numbers
}
// printList prints out each number at each node
func (numbers *list) printList() {
currentNode := numbers
for currentNode != nil {
fmt.Println("%+v ", currentNode.number)
currentNode = currentNode.next
}
}
// testing it all out
func main() {
listing := newList(1)
listing.pushNumber(2)
listing.pushNumber(3)
listing.pushNumber(4)
listing.pushNumber(5)
listing = goForward(listing)
listing = goForward(listing)
listing = goBack(listing)
listing.printList()
}One function for each of our inevitable operations; creating a node; each node is a single directory, and a traversal system for going to and from nodes or directories.
Second on the list was basic file I/O and prompt manipulation. This will translate into making a digital copy of a directory and all its subdirectories when the filesystem is exited. Writing it to a file to save your work. And the shell loop for user input.
package mainimport (
"bufio"
"fmt"
"bytes"
"os"
)func main() { var file_content bytes.Buffer /* initialize an empty buffer */reader := bufio.NewReader(os.Stdin) // initialize stdin reading
for {
fmt.Printf("$>")
text, _ := reader.ReadString('\n')// read from stdin and stop when newline is reached if text == "\r\n" { // if input just has a Windows formatted newline, continue
continue
}
fmt.Printf(text)
file_content.WriteString(text[:len(text)]) // append content to the buffer
fmt.Println(file_content.String())
}
}
I suspect creating the filesystem part will be a lot easier than the actual loading and saving of information, but we’ll burn that bridge when we cross it.
Lastly, to round things off, a basic overview of the file and directory traversal will look something like this; Directories arranged as structs with their own map values. Their key/value pairs being the file names and contents respectively, and lastly an array of directory structs that will be the directories inside these directories.
package mainimport "fmt"// Dummy payload that represents a directory example
type list struct {
path string // represents the directory name.
files map[string]string // a map that represents the files in this directory
directories []list // a sub list containing all the directories in this directory
}// newNumber creates a directory - this function is poorly named
func newNumber(name string) *list {
return &list{
path: name,
}
}// addDirectory adds a new directory to the current directory
func (fs *list) addDirectory(dir_name string) error {
new_dir := list{path: dir_name}
fs.directories = append(fs.directories, new_dir)
return nil
}
// createFile creates a file and its contents in a directory
func createFile(file string, content string, value map[string]string) map[string]string {
value[file] = content
return value
}// A few implementation tests
func main() {
fs := newNumber("./")
placeholder := make(map[string]string)
fs.files = placeholder fs.files = createFile("hello.txt", "hello world", fs.files)
fs.files = createFile("gone", fs.files)
fmt.Println(fs.files)
fs.addDirectory("Desktop")
fs.addDirectory("Users")
fmt.Println(fs.path)
fmt.Println(fs.directories[1])
fs.directories[0].addDirectory("NewWorkFolder") fmt.Println(fs.directories[0])
}
The missing pieces aren’t as horrendous as I first imagined they’d be, looking through C tinted glasses. I was bracing for the absolute worst but with all my ideas working out so far, it seems like this is doable.
I’ll be documenting each step in a tutorial-esque fashion after this brief overview of the road so far.
I suspect for the longest time I’ll run into the trap of coding C in Go, but here’s to hoping I’ll be able to grow out of it as we go along.
And that’s it for our first entry. I’ll probably need to do a few more of these short exercises but I figure I won’t document the rest and just skip to beginning the project with what I have. I’ll see you soon with a part 2 as I get started on laying the groundwork and actually start coding the project out.
Till next time.
