Byfrost Source Code Visualizer in Development 1— An Indexing Graph

This title could be better but this does have a lot to do with graphs so it’s at least not false advertising.

But first, a picture of a Bifrost-looking thing!

This follows on from my initial article where I announced my source code path visualiser here; Introducing Byfrost-A Source Code Path Visualizer.

So Byfrost’s development has finally kicked off to a lot of fanfare and unearned praised showered upon yours truly across Reddit and right here on Medium! I love the excitement around this tool and so I figured I’d keep you all updated with the harrowing, palpitation-inducing tragedies of trying to build an indexer from scratch. So sit back, relax and peer behind the curtain and in absolutely no time, you too might have an idea of what contributing to this beautiful nightmare will be when it comes time for the theme park to open.

So, what have I been up to in the past two weeks since dropping that bombshell? A lot and then suddenly nothing at all and then everything at once! I’m pleased to announce that we’ve got a working backend!! 🎉 The functionality so far:

Progress!!

UI needs a lot of positioning work but the core is up and running!

Before we hop into the nitty gritties, let’s go over some backpeldles quick: I won’t be supporting C first after all. I said this a bit in what I hesitate to call Q&As on Reddit that this tool would support C as a first language and now I’ve decided that I will indeed not be doing that after all. 🤔. It turns out C is a lot harder to parse than I originally thought. It’s deceptive because the lack of structures made me assume this would be easy to do but that was a lie.

Another is that this would be a plugin. Yes and also no. It’ll be a web app but also a plugin but also a web app because I kinda need this immediately and I haven’t learnt plugin development. But you’ll be able to pick your poison someday.

Okay let’s get started!

What I’ll be discussing in today’s update

I’ve become extremely productive since intially announcing this little project that an insane amount of ground has been covered. Because of this, I’ll be keeping things as “overview” as possible to get the ideas behind the indexer across without turning this into a novel.

I’ll be going over:

• Overall architecture of the indexer — sort of (EDIT: in retrospect I shelved a lot of the detail because this article became too long)
• The indexing approach and relating function labels to respective functions

The system’s written with a Golang backend and an Angular frontend. Why Go and Angular? It’s the languages of my day job so it’s the only stack I could pick up in a timely manner. I mean I’m a Go engineer by trade and wow, that Angular sure did a number on my will to live lemme tell ya. Though I made use of an Angular frontend, I accidentally made my frontend rendering structures framework-agnostic, mainly because I couldn’t get a lot of HTML on-the-fly rendering done with Angular’s many directives and builtin Angular “stuff” packed in. I’ll go into detail about this in the Approaches section below.

The Frontend — The Battle With Angular

It’s as epic as it looks

Oh boy, did this part kick me in the teeth multiple times. I’m not a frontend engineer at all and had to learn the craft to a somewhat competent degree in a matter of weeks to pull this off and I’m gonna say it; frontend is harder than backend — HOLD ON! LEMME EXPLAIN! PUT THE FRYING PAN DOWN BRIAN!! (That’s probably not your name but I figured I’d freak out all the readers named “Brian”).

I find frontend akin to telling a painter to paint the Monalisa from scratch and exclusively following instructions you’ve poorly written on murky newspaper while drunk and suffering from fresh heartbreak. I was bewildered by diffiicult most of the simpler looking things ended up being. With on-the-fly large scale HTML rendering, turns out you need a bit of trickery here and there to not spaghetti your code into oblivion.

The Approaches

…hmmm….

Here I’ll go over the three approaches I took, why they failed and eventually why the last one was successful. This will cover a lot of the struggles I had with Angular.

Approach 1: Runtime Component Generation

The first approach that made intuitive sense to me was make an Angular component that would represent a single code display panel. Something like this;

export class ByFrost implements whatever {
  ...
}

And this would hold everything. It would have variables for that would make up a single program panel.

Something like;

let filePath: string = "<a (click)="openFile()">main.c</a>"
let sourceCode: string = "int main(int argc char **argv)\n{\n\t<a (click)=\"newComponent()\">printf</a>(\"hello world\");\n\treturn (0);...

The filePath being… well… the file path and the sourceCode … you get the idea.

Only problem with this approach is that it turns out those directives (the (click)="openFile" ) and the other one can’t be rendered during runtime like that and are compile-time exclusive niceties. See; those are the Angular “stuff” I mentioned earlier (directives or something). What they do is that they’d bind those a tags to those target functions which would trigger in the background when clicked. openFile would trigger a function called openFile that you specify to trigger when clicked. Same story with printf . For printf a new component that contains its source code would be generated and appended to the original ByFrost component. That was the idea; open the file when the path is clicked; and create a new function component when a function is clicked.

Super easy. How hard could it be, really?

Oh the foreshadowing…

This, my friends, turns out is a very very very veeeeeerryyy hard and is an incredibly bad idea.

Turns out you can’t actually do this. I discovered after countless hours of crying over this that runtime compilation of those Angular tags/directives/etc really isn’t possible. Not really. And I went all in on this approach; partly because it made me feel like a real compiler engineer and I could vindicate all the years I’ve spent trying to be one. I was even looking into Angular’s Just-in-Time compiler to see if I could force it to compile these at runtime. And I’d have gone into Angular source code to force a fix too (and rendered my entire Angular setup trash as it would break the CLI and everything that makes Angular not terrible) but I stopped and decided to spend the rest of the afternoon watching Top Gun: Maverick in cinema instead. Great movie.

Approach 2: Using innerHTML for Rendering

This is close to what I eventually ended up settling on but my first attempts had similar problems. I wasn’t generating new components this time. No, this time, I’d just have a massive list of HTML strings representing code display panels that would output in an Angular ngFor loop.

This was somewhat of an improvement; things were rendering but I still had the issue of my renders not compiling the Angular functions and tags.

I was completely at my wits end. After all the fanfare; all the years of struggling to build my indexing tool. Was the road finally over?

Approach 3: Prepare All Frontend Functionality in the Backend

This new way of thinking about this came about when talking to another fellow backend engineer. I ran into a few hiccups that I won’t go into too much detail with here as this article’s gonna get pretty long. The approach here consisted of removing the usage of Angular components in the HTML altogether; this was never gonna work because dynamic HTML rendering is an absolute must for this and those never compiled. Removing the Angular “stuff” worked out pretty well actually. The final final problem with this approach was that rendering HTML source code as a super long string and then managing the relationship dynamic between functions and the functions that those parent functions called got messy pretty quickly. I couldn’t get any of the function calls to remember what function their parent was and who they were related to.

Take this simple example:

void first() {
   toRepeat();
}
void second() {
   toRepeat();
}

Once toRepeat is clicked to be expanded, without a heirarchical structure in the HTML, there was no way to see which function toReturn ‘s call came from. It had no parent. A lost, aimless child, wandering in darkness within darkness…

However, racking my brain over this final problem, I was hit by a stroke of genius…

The Backend — Filesystem, My Dearly Beloved

Ahh… this brings back memories.

Old-school readers might remember my old Go Virtual Filesystem project from about a year ago. That taught me a lot about handling heirarchical relationships in a tree data structure pretty well in Go.

This was exactly that kind of problem. The bodies of functions were directories and the functions those function bodies called were subdirectories.

This is how I’d do it. This is how I’d establish relationships between functions and their children function calls.

With this new paradigm, this program:

int  main() {
  helloworld("test");
  read_file("test2");
}
char  helloworld(char i) {
  testing(42);
  return 'c';
}
int read_file(char *filename) {
 helloworld(filename[0]);
 return (0);
}
int testing(int i) {
 return (0);
}

Would be translated into the following tree structure:

Each function would have a unique path attached to it denoting that function heirarchy. read_file ‘s unique path would be main.read_file and its call to helloworld would have main.read_file.helloworld as its path and so on.

The Go structure supporting this functionality would look like this;

type Display struct {
 name    string    //  read_file
 path    string    //  main.read_file
 active    bool    //  when function expands, this is true.
 rawTokens []*Token
  // Display{
  //   name:"hellowworld", 
  //   path: "main.read_file.hellowworld"
  // ...  }
  childDisplays map[string]*Display
}

The name being the name of current function, like main with the childDisplays holding their own displays for the functions main calls; in our example, the displays for read_file and helloworld . active is used for collapsing logic. When a function is clicked, this value will determine if it’s to be expanded or collapsed. Finally, we have rawTokens . These hold all the tokens in an array that will visually make up the current function and hold UI config information that Angular will interpret to make it pretty.

rawTokens is built up from the lexer and parsing stages but serve an extra purpose of determining which labels are clickable in the frontend and, in future, will hold styling information for the syntax highlighting I have to build by hand (you’ll see why in a bit). So that’ll be covered in a future article when we go over that.

With this, each function link held by the pathvalue above will, in the frontend, attach a unique ID to each function label call that, when clicked, would instantly identify exactly which path has been expanded on. This also makes accurate function path collapsing possible too. So when you click on an already expanded function, clicking it again will set active to false and when this happens, a recursive looping of all that current function’s children will be parsed and all set to false as well, triggering Angular to delete them in the UI.

With that final tweak, this also allows each of the panels to be their own clean, standalone selves and take on the following shape;

I like this design for each of the function panels because this means that, from a UI perspective, each individual block could be styled without them stepping all over each other. It also makes adding and remove code blocks increidbly easy to do because HTML DIV bodies aren’t related to each other (no nesting of children or anything like that).

This also makes the frontend framework-agnostic to a large extent. As no Angular-specific structures are being used, this could easily be ported to React or even JQuery (heaven forbid). I didn’t know this would happen when I initially set out but it’s a happy accident. It’ll make styling a lot easier too.

And that wraps up the road so far. Next I’ll cover some honourable mentions that I wasn’t able to find natural place for in the article.

Some Honourable Mentions

This ended up being the hardest thing I think I’ve ever done. Frontend’s turning out to be quite a monster and the amount of roadblocks I’ve been running into is quite something.

For some honourable mentions, I’d like to cover some of those; problems I ran into and strategies I tried to get over them. I’ll also go over the future of integrating new language support and turning all of this from a web app to a plugin.

• Drawing Lines — this is by far the one thing that splitting up the frontend HTML displays ended benefitting immensely. Initially I was going to use some hacky svg approach of drawing them manually by hand — perhaps calculate where each panel would appear in real time and then draw some choppy line towards it. But using traditional JavaScript with Angular calling the shots is something I still struggle with. Doing simple things like referencing HTML elements with getElementByIdand copying code off of StackOverflow (because of course I did) for targeting the line destinations just never worked. I suspected it was traditional JS steppnig all over Angular. Thankfully there’s a great library called linear-line for Angular that did the trick. Using without the relationship issue of bundling HTML DIVs to parents saved me a lot of suffering. And then there was importing libraries into Angular… kill me now… 💀.
• Syntax Highlighting — Okay so I’m genuinely offended by this one because this was supposed to be an easy low-hanging fruit; go online download some of that sweet Ace editor action I used for my Go virtual filesystem’s text editor, switch it to readonly and bam, early lunch. Unfortunately not — turns out adding Ace disables all my buttons so clicking on functions no longer expanded anything. Not to worry though, Google has a fancy library called Prettify except it’s garbage! Seriously, how is highlighting this hard to do! This didn’t go so well either. Though in all fairness, I’m sure the library works just fine if it’s not be forced into Angular so unceremoniously or something. This time round I kept on running into syntax highlighting issues of my text not changing color and sometimes not “switching on” here and there so screw it. I’m first-principles-ing this bad boi. In C’s case if we think of Vim’s highlighting; datatypes are green, control flow statements are yellow, literals are pink, so I’m making my own syntax highlighting and it’s gonna be a dream to use. I have a spec in the works for how this is gonna work because I have no choice but to get it right for launch day. How hard could it be…
• The Language API — This is something that’s also gonna need a lot of attention and a some formal spec too. So far, adding languages is very very difficult because they’re all so different. I decided to drop C and focus on Go as it has a native parser library for itself. I thought C would be easier because of its lack of structures but it’s deceptively messy and I couldn’t find an opensource parser I could dismantle for my needs. Plugging in all languages under the sun’s gonna be a monstrous task. It’d be great to get help from compiler developers on this aspect.
• Making it a plugin — For now, I’ll be keeping it a web app for the first version while polishing up a plugin for VSCode because that’s hard…

And that’s the long and short for now. I got a lot of work done over the past three weeks and covered way too much ground to be able to unpack all of it in a single article and even still, this article’s easily the longest I’ve ever written.

The Roadmap

As for what’s left on the roadmap before a first initial alpha release we’ve got:

• Syntax highlighting — only one theme for now as I flesh out the library and approach to syntax highlighting.
• Go porting — this tool will only be available for Go when it drops. Go’s the language I use in my daily life and it has parser support. This healthy support will also help in determining a sort of standard for the language API or service or whatever.
• Code panel positioning and sizing — This is slightly tricky as right now, all panels are seemingly the same size and flow down the page. Making them spread out rightwards in a more “readable” style is the final feature to deliver on Byfrost.

And that’s all I have for the first Byfrost in Development post.

I’ll be posting more updates and preparing an alpha to send out for some early testing. If you’re keen to give it a spin, I’ll set up somewhere you can sign-up to give it a Go (see what I did there).

And of course, a Github repo’s still on the way.

Chat soon! 👋

Alyson