Port dagzet to rust

My lua implementation was always designed to just be a prototype, and nowadays it's getting quite heavy. Now that I'm a bit more familiar with the language, I don't think it'd be too difficult to build it in Rust, tbh. I think the standard library has enough rich data types that it should be pretty straightforward.

My hope is that the lua implementatoin is trivial enough that I can bring it over to rust without too many complications. I will outline some of the broad strokes steps required to get this program up and running.

While writing these thoughts

The first thing I'll need to be able to do is read a file from disk, possibly entirely into memory. The dagzet parser works line by by line, so if there's some iterator abstraction that allows me be to do this, great.

For parsing commands, I need to be able to read the first three characters of each line to determine the command code. Commands are two characters followed by a space, with the rest of the line being being arguments.

Commands need to map to functions which can parse the argument data of that line, and potentially append or modifiy a rust data struct representing the graph being built up somehow. But, just setting up a convenient way to map commands to functions would be great. There will be quite a few, and I often find myself wanting to add more commands to meet my needs. I made use of Lua tables to create a look-up table of callbacks. Hopefully I can do a similar thing in Rust without too much fuss?

I need to be able to split data up by spaces. Lua does not have a built-in split() like you'd see in other languages. I'm hoping Rust standard library has one somewhere (this functionality should be standard not outsourced to a crate right? right?!)

I need a top-level data struct that can be populated with information that is parsed. I the lua implementation, I used tables: usually as a hashmap or array structure, sometimes an array of array. IIRC Rust STD has hashmaps, and vectors I think should be enough. I can't foresee too many ownership issues due to how imperative this is, but who knows.

I need to implement topological sort (Kahn's algorithm). My lua approach used node IDs instead of references, so I think this is going to be mostly Rust-friendly to port. Still, I get the feeling that I may be forgetting something that will be a pain point with Rust.

When it has been determined that the graph contains no cycles, generate the SQLite code to standard output. This seems pretty straight forward to me, more or less going through the generated struct (read-only) and printing equivalent SQLite code.

Going to try to use neovim for this now.

tangent: trying to get auto-import working. Found: <<neovim/nvim_cmp>>.

how to get this working with lazy, README only has vim-plug?

Wait, it's already installed, according to the Lazy control panel. What does InsertEnter mean?

Okay, I get the recommendations, but I don't know how to insert it.

Got it! Typing "File" then hitting ctrl-y does it.

I have it parsing command codes in the test file, and there is some placeholder stuff where I can eventually do stuff with those commands.

Eventually this will be populated with data from the commands.

Being able to add in-line descriptions of struct contents that get rendered to rust documentation would be helpful.

I have a incremental TDD approach to porting this, which feels nice.

I'm making use of Result and putting return codes into a single Enum. This is just mirroring how I'd do it in C. Hopefully it's idiomatic enough in Rust.

Turns out it doesn't have a lot to say. Voxbox, on the other hand, is a mess.

added a lines command. now I'm very tired.

Connect adds a pair of edges to the graph.

These are set to be local connections. ID values work just fine here.

I'm curious about knowing how I did cx? Was that another structure? Am I checking cycles there?

cx is another structure and loops/cycles are not= being checked for.

My original dagzet implementation reperesents the dagzet connections as strings that are only checked at the end of parsing when all the nodes have been added. Only then does it attempt to resolve the symbols. This makes for a more permissive parser.

My dagzet parser does the topological sort with the strings. I think it would be better to resolve those chunks into local ID values before the sort.

This is intentional. Verifications are defferred until everything is parsed. Verification is beyond the responsibility of connection.

There is a linear check for already existing connections, and a test is put into place to make sure this is all properly caught.

Adding TODOs for that.

I've implemented enough today.

It's got a good scope, it's got data structures, and I'm putting in the time to incrementally test as I go.

This is also a migration project. If I do this right, I should be able to drop in the dagzet program and have it replace my adhoc lua code for my knowledge tree generator, which I'm using to power the knowledge graph here.

hashmaps are used as the data structure to ensure that a node isn't created twice.

if I wanted to get a node name from a string, how would I do that? Right now, the answer seems to be enumerate through all the keys and find an answer. I could also just make a separate inverse lookup table out of a vector.

It's a memory hit, and there's room for the eventual possibility that the tables will go out of sync. But, it's good enough for now. I imagine I'll be doing this lookup operatoin quite a few times I don't want to do a linear sweep every time.

Variation on a theme, I suspect. Unplugging and getting lunch now.

My goal is to use this rust port as a drop-in replacement for my current dagzet program. So, a good thing to do would be to write a program that scrapes all the current commands used in my RC dagzet instance.

I also have more than enough here to begin working on implementing topological sort and SQLite code generation.

This is more difficult than I expected. A connection remark needs to reference a connection (the last connection made). So, how does one reasonably accomplish this?

Since the connections are a pair, one approach is to use a 2-dimensional hashmap of values. It feels like a lot more space is used up here. It also duplicates some of the logic in the connections hashmap.

Another thing to do is to somehow reference the connection. An actual Rust memory reference could cause the borrow checker to be grumpy. Using unique ID values to reference each connection, and having that be a key in a hash table could work. This could become a bookkeeping problem if I ever added the ability to remove connections, but there are no plans to add such a feature.

Connection ID can be their position in the "connections" vector.

Since there is no way to selecting a specific connection, it is reasonable to assume that the last connection will always be implicitely be what remarks are being written to. So, the ID will always be length of the vector (minus one for the index).

It was silently erroring. I needed to add a namespace.

I initially thought I was ready for topsort and cycle checking. But unknown node resolution should happen first.

An edge list should be created from the connections for the topsort. But those should all be resolved ID values. "Check unknown" nodes should return a set of nodes that do not exist in the node.

Rust does have a set in their library! Huzzah! Called HashSet. That's what I need.

The trick here is that the connections need to be resolved, which is another step to test independently.

A connection list, after it has been verified, should be turned into an edge list. This function is still allowed to error out. This edge list then gets passed in to the cycle checker.

Note: Topsort makes use of sets that will expand and grow. I expect to dynamically generate these inside the function.

Generating an edgelist should come first. Panic on missing nodes for now. that can be fixed later.

The top-level cycle checker should be a method called check_for_cycles. The original implementation was able to get some information on these cycles.

Looks like the original topsort populates a "loops found" list. I will do the same, only it'll be a HashSet. On success it will return an Ok, otherwise an error code.

Implementation will come next. Signing off for now.

Most of this boils down to string generation. It'd be good to have some intermediate structures before that.

Being able to generate sqlite schemas is a good start. A table as named values, each with a type.

It would be nice to have a consistent ordering of these names, for things like generating insert statements.

A table itself also has a distinct name.

stringify() could be a behavior that types and tables implement.

Ah, stringify is already a behavior the standard libary implements.

The best thing would be to have some struct map to the VALUES part in an INSERT statement. A method on the table, such as sqlize_insert, could then take in a pointer to a struct implementing that row that's able to generate values.

Trying to un-confuse myself: supposing I had table A. I'd want to make it so A.sqlize_insert(row) would generate an insert statement from data in row. Another table B attempting to call B.sqlize_insert(row) would get a compiler error due a type mismatch.

Generics for trait? Like this trait can only work when the type is A, not B.

The table schema itself needs to be a concrete type, which it is not right now. If that happens, then I can make a row type for that table.

Define the interface that I want to see in a test, then work backwards from there.

See: <<rust/phantom_types>>.

Real quick stuff.

Lots of elaboration needed, but it's a great start. Things work.

I'm trying to rework the nodes table code into something more re-usable.

It feels like it's just starting to slip into something out of my grasp.

This makes me feel better. It has an easy code smell.

Make this an interface somehow?

you know what they are just going to be functions. refactor later.

I'm hoping to use the trick in C where the file handle can be standard out as well as a file.

Let's try the connections table.

Most of the hard stuff is accomplished I think.

I just need enough to generate my RC dagzet and replace the lua implementation.

The results of my shell-ing:

$ cat *.dz | awk '{print $1}' | grep -v "^$" | sort | uniq -c
    127 co
     50 cr
     15 cx
     59 fr
     38 gr
    165 hl
    357 ln
    280 nn
     40 ns
     96 rm
      1 sn
     15 td
     76 tg
      4 zz

Done: co, cr, ln, gr, ln, nn, ns.

TODO: cx, fr, hl, rm, sn, td, tg, zz

Wow that's more than I expected. And there's going to be a non-trivial amount of time troubleshooting the SQLite generated code because of course it's not going to work on the first try.

More slow, incremental testing?

No, the errors are going to come from incorrectly generated SQLite code, and SQL code validation is outside the scope of dagzet.

I need to generate that once, and cache it in dagzet.

The connections can remain as strings.

Oh yeah, that's right. This has some shorthand behavior as well. '$' is used to reference the last file. This might take up the rest of my morning before I break.

Some of the slowness comes from building up the test correctly. I'm doing this one right, becaues I've already done it wrong.

Comments in place, testing a handful of edge cases. Now I just need to implement incrementally. That's for another time.

Okay, nevermind.

This one is slightly more interesting because of how the data is represented in SQLite vs how it's entered in dagzet. From a dagzet point of view, tags are a list of entries associated with a node. In SQLite, they boil down into pairs.

Tags should be able to be called more than once.

I think I'm only going to have time for the initial test before I break for lunch. Maybe.

I'm not sure why it's not getting the correct number of tags? Oh well, will have to look at it later.

I was misinterpretting the boolean result of the hashset insert method. Negating that fixed stuff.

tags finished.

I think for the most part these could be thought of as integration tests.

Nevermind. I just won't fight it right now.

Installed with =cargo install --path .". Replaced my lua program with my rust program. Looked around on the generated ouput. It seems to be right so far? Publishing to see what will happen.

Yeah, it says it is fine. so that's pretty cool.

I've done enough for one session. It's almost there.

It should have crashed. Make it crash, then I'm done for now.

oooh. the "dagzet" tool and "gendb.sh" are different. I gotta replace it there.

I need better error reporting. It seems like this version of dagzet has stricter parsing (you can't have duplicate tags, for example, whereas my old implementation could).

The test I had for it was too small.

"cx" is an external connection, and it allows you to make connections to nodes that are external

A partial implementation of cx is all that is needed for now. No aliases, no shorthands. It's all just full paths.

Any nodes used by cx get stored in a list (set?). When unknown nodes are checked, if it can't find the node in the locally created nodes, it'll check the nodes here.

Connections will work the same way since it utilizes a SQL command. The burden is on the dagzetter to ensure that nodes get created before they are connected with cx. This is different from co, which follows a more declarative style.

At this point, any unknown nodes should have checked for.

Is there ever a situation where an edge would have an unknown node missed by the unknown node error check?

It's a sqlite error. I need to figure out what the error is.

I need to use db:errmsg

I didn't configure the column name correctly.

I'm going to have to call it quits after this.

It is random too, probably due to randomness in lua tables.

Something is going wrong in the SQLite table generation. When I place debug prints in the shortname generator, I get a ton of stuff.

My goal is to take a look at the generated table outputs of both.

This is probably due to the randomized hashmap structure in Rust.

But, the node IDs are the same, monotonically increasing unique values.

There seems to be duplicates. I generated and sorted the connections alphabetically and found these.

The rust version also has inconsistent connections as well.

It is consistently this. I wonder if "cx" is partially to blame. disabling all the cxs now.

There are 15 cx connections made, and that accounts for the missing connections.

Now, let's see if the error goes away.

I think I really mean it this time. It generates the data without crashing and everything. Peaking around, it seems to produce the data just fine.