Incoming Progress

Creating some really cool projects over the course of the past week.

introduction

Hello everyone! With another few weeks behind me, an exciting amount of progress has come to the chifi projects. This progress is slowly inching us toward the ultimate goal of a multi-user collaborative notebook editor. Along with making progress on the project, I have also had the opportunity to learn a good bit more and design some new types of systems I have never designed before — such as my own low-level network protocol. I have gotten a lot done on within most chifi ecosystems in preparation for what will become the final steps in deploying my project.

toolips ecosystem updates

First and foremost, I built a small update into Toolips and also patched its ToolipsServables dependency. The ToolipsServables update focuses on fixing a singular incorrect replacement in rep_in , as well as updating some Component aliases and adding the dateinput Component . The Toolips update focuses on building upon 0.3.10 ‘s TCP server functionality, as well as expanding on functions like new_app to better facilitate server extensions going forward.

There has also been a new release of ToolipsCrawl ; the Toolips -based web-crawling system, which I simplified significantly in this version to great success. Eventually, there will also need to be a new, breaking, ToolipsUDP release. A lot of the Toolips 0.3.11 update moves things from ToolipsUDP into Toolips itself, so a new version will be essential to import them instead of creating them.

Finally, thanks to the next project we are going to discuss, I have finally taken some initiative and started the ToolipsORM module. As of right now, alongside ChiDB , ToolipsORM is capable of performing a number of authenticated queries and has a system that will be incredibly easy to build more query commands into. For the first release of ToolipsORM , my main objective is to support the ChiDB ff system. Future releases of ToolipsORM will include more drivers, and like most other Toolips things the package is hyper-extensible. We will take a look at using the package after we cover the data-base server.

chidb

[embed]GitHub - ChifiSource/ChiDB.jl: toolips + algebra streams = database server toolips + algebra streams = database server. Contribute to ChifiSource/ChiDB.jl development by creating an account on…github.com

It is official, I have finally decided to sink my teeth into creating my own data-base server. Why? First and foremost, I want a bit more flexibility and the way this is going to come together eventually will make having the data-base server in Julia a huge advantage. Secondly, the main reason I usually do things is because it is fun, and presents an educational oppurtunity. Even if you never finish a project you still learn a lot from the problems you solved on your way to finishing the project. Starting the project takes interest and creativity, finishing the project takes everything.

Despite the relative complexity of a data-base server, I have gotten pretty far on the project and I am happy with its current state — even if it will still require a few more hours to finish a working version. This data-base server’s unique feature is that it streams all of its data live from the disk. Schema is represented as directories and data is represented as typed feature-files. To create a data-base, we create a directory and then start ChiDB , providing that path.

mkdir newdb
cd newdb
julia

julia> using ChiDB
Precompiling ChiDB...
  2 dependencies successfully precompiled in 2 seconds. 50 already precompiled.

julia> ChiDB.start(pwd())
┌ Warning: ChiDB is not yet fully functional or ready for production use.
└ @ ChiDB ~/dev/packages/chifi/ChiDB/src/ChiDB.jl:362
[ Info: this version is primarily being used for testing, at the moment. This project is a work-in-progress.
[ Info: ChiDB server started for the first time at /home/emmac/dev/packages/chifi/newdb
[ Info: admin login: (duxzszibsrkfeympqkbmgxqrisjhmnvb) admin pnkdtyqeobsrjkuy
[ Info: pem: bzlmzqwzqmohlflehzsatrxwuxopacbz
[ Info: loaded dbuser admin
[ Info: pnkdtyqeobsrjkuy

We will get an info printout listing our admin login details. Once we login as admin , we can create new users and set their login details using query commands. This is the only time that admin ‘s password will be shown, so make sure to grab it here. Because we started our server in an empty directory, our data-base has no tables or columns. To get a list, we will perform our first query, l . I will be using Toolips ORM for this:

using ToolipsORM
orm = ToolipsORM.ORM("127.0.0.1":8005, ToolipsORM.FF, "admin", "pnkdtyqeobsrjku>
connect!(orm)
@info query(String, orm, 'l')

┌ Info: 
└ empty data-base (0 columns)

We currently have an empty data-base, we can query to add schema:

# create table `people`
julia> @info query(String, orm, 't', "people")
# join column `name` to `people`
@info query(String, orm, 'j', "people", "name", String)

julia> @info query(String, orm, 'l')
┌ Info: 
│ people (1 columns)
└

This will also create new files inside of our database directory.

|10:12 AM emmac @ ~/dev/packages/chifi/newdb > tree .
.
├── db
│   ├── history.txt
│   ├── key.pem
│   ├── secrets.txt
│   └── users.txt
└── people
    └── name.ff

• We also could have created this schema by making these files and directories before starting.

Now I will make another column, called age :

@info query(String, orm, 'j', "people", "age", Integer)

Now let’s add a row:

julia> @info query(String, orm, 'a', "people", "emma!;25")
┌ Info: 
└ added row

From here, we could easily query back that data:

julia> @info query(String, orm, 'i', "people/name", "emma")
┌ Info: 
└ 1

julia> @info query(String, orm, 'g', "people/name")
┌ Info: 
└ emma

julia> @info query(String, orm, 'g', "people/age")
┌ Info: 
└ 25

There are a plethora of other commands, though not all of them are fully implemented or tested yet. Fortunately, testing for a server like this will be relatively straightforward. When it comes to data-bases, security is naturally one of the biggest concerns, so there are a number of security features going on under the hood. The most obvious of which is AES256 cryptography, but expanding on that there are also transaction IDs. For this server, I designed a two byte header that contains two flag sections of 4 bits each and a command byte. The 4th–8th bits that are returned from the server on each response must be the same as the ones that were previously given. This is in addition to the client already being in a connection loop, so it is pretty unlikely at this point that someone could spoof being that someone they are not.

There are also some other work-in-progress additions to the ORM . In the examples above, I provide String as the first argument to query :

julia> @info query(String, orm, 'i', "people/name", "emma")
┌ Info: 
└ 1

julia> @info query(String, orm, 'g', "people/name")
┌ Info: 
└ emma

julia> @info query(String, orm, 'g', "people/age")
┌ Info: 
└ 25

Here, we are providing the type we want to parse the return into. For example, we could perform a query for a Vector of integers, or a single integer.

julia> query(Int64, orm, 'i', "people/name", "emma")

julia> query(Vector{Int64}, orm, 'g', "people/age")

I would already consider these to be light ORM features, but the plan is to eventually implement more ORM features, including ORM indexing. For example,

ORM["people/name", 1:3]

Would be equivalent to…

query(Vector{Int64}, orm, 'g', "people/age", "1:3")

This ORM framework is primarily created for my personal use-case, which will involve a ChiDB server and the ff Feature File file format. However, the project is called ToolipsORM — not ChiDBORM . My goal for this package is to eventually provide support for a lot more query drivers. Considering this, my Minimum Viable Product for this project will be a full working implementation of the :ff feature file. After release, more planned drivers will eventually be implemented.

Regarding the database itself, there are some more improvements I want to make. Though I am sure it is not entirely necessary, I am considering an end-to-end encryption system where an encryption key is sent to the connector on the initial connection. This would complicate everything, but would give an extra layer of protection that might be appreciated. My use-case for this is solely through local traffic — if ever the data is needed outside of my local network, I will be using an API. Considering this, it might be overkill but it still might be something I choose to do.

Another thing I would like to add for security is salting. Salting is a simple concept, we add random elements to each password before encrypting it, usually in segments of 32 bytes or 256 bits. Despite its simplicity, actually implementing it will be somewhat complicated — we need to keep track of what we add to each string and where, and that salt needs to be loaded again when we start the data-base again. This would add substantially to the security within the database, and would instantly bring this project a lot closer to production-ready.

My security situation is ideal; all connections are sent to my Local Area Network (LAN) via a proxy and all of the database traffic will be LAN traffic. ToolipsORM also includes ‘API functionality’ that makes it super simple to deploy an ORM-based API, and all of the actual data serving will be taking place through that API. ToolipsORM also plans to support querying to APIs with ORM, which will make creating an API alongside my database server extremely straightforward and possibly almost entirely automated.

Though I am going to make a try catch for actually performing the command, but the goal is for every error in any command to return either an error message or a confirmation via a blank String and 0 code. Needless to say, once it becomes a large number of commands this becomes quite difficult to do.

Eventually, there will also be more store-able data-types — I built an entire parametric dispatch system to make implementing these easy and fast in the future. This will include types like compressed strings and ‘crypt’ strings. I am nearly done with the commands, but there is still a long way to go before this is fully operational. I am probably about half or 2/3rds of the way through this project.

Despite the long road ahead, it is easy to see how much ground I have already covered. At this rate, it won’t be long before I have a basic functional MVP. I am thankful for how much I have learned from this project. Going into ‘ building a database server,’ I thought I would be using UDP and I thought my approach to this was going to be entirely different. One of the first things I learned is that TCP is more common, which makes sense when you consider its reliability and the type of communication we want from a data-base server. Of course, we want our queries to be reliably sent and responded to, and TCP is the tool for that job. It was also really nice to get a view from the database server’s perspective, rather than the querying perspective we are used to — especially with all the modern ORM, driver, and other obfuscation layers that developers often use to get away from low-level networking protocol design.

impending olive

The last thing I will touch on in this small update is the incoming breaking version of Olive . Olive will finally be getting the 0.2 treatment — becoming a largely improved step from 0.1 wherein a good portion of the base will be refactored. This breaking update is primarily seeking to deprecate the OliveModifier in favor of a regular ComponentModifier and replace Olive keys with ToolipsSession Session keys. All in all, this will help to reduce load and memory usage in several instances and overall will make some pretty massive improvements to Olive itself. This may also come with some new types of OliveExtension -based extensions, for example startup extensions or client data extensions — extensions that don’t need to run every time the page loads like a load extension would. Perhaps even extensions that change the response body before writing. As of right now, extensions are exclusively loaded on response.

Though this might sound all-encompassing, 0.2 is a relatively small update that is simply bringing Olive inline with the latest version of the Toolips ecosystem. Specifically, this will be for the (currently unreleased) Session 0.5 , a new breaking version of ToolipsSession that also only makes some slight adjustments. The main goal with both of these updates is to further reduce memory usage and also improve performance by removing any and all redundancies.

Fortunately, after having ChifiDocs — a pretty memory-heavy application that explicitly preloads all of the markdown documentation for each chifi module into Components — deployed for a few months now I am really happy with Julia’s garbage collection. I can definitely be quoted somewhere saying “ Though Julia takes more memory and is implicit in its garbage collection, that implicit garbage collection is actually really good — especially when used in tandem with GC.gc .“ and having this server in production has been a successful test for that claim. The server has yet to use over a gigabyte of memory, which is far less than I would typically allocate for a publicly available web-server. I planned to use, or ‘set aside’ 5GB of Clara’s 32GB of RAM and the server uses less than 1GB; so it is working quite well.

final steps

At this point, we are right on the cusp of, at long last, bringing my overly-ambitious collaborative notebook-editing platform into fruition. As a reminder, here is a short video demo of collaboration cells and the types of multi-user experiences I am planning to bring into fruition:

[embed]

Everything just needs a little bit more work to get us there. Once I am finished with this database server project, I plan to move onto three things…

• ToolipsSession update
• ChifiDocs update
• and the Olive update,

… before finally focusing exclusively on OliveCreator ; finally becoming a normal web-developer for a little bit instead of building the entire world to do that web-development. It is a lot, but the road we have walked is a lot longer than the one ahead… For this project at least; I will definitely continue doing open-source stuff beyond maintaining these projects as always. Art is just too much fun to create! I am super excited to start sharing invitation links when the time comes.

Again, I apologize for the delay in content on my Medium. I have not forgotten about it! I have definitely been prioritizing work on these packages and my other projects over this, which I think is still the right choice for me right now. After working on the project and all of its complicated dependencies every month for the past three years, I am trying to finally see its release. Hopefully it is easier to understand my priorities right now, but I will always continue writing.

As always, thank you all for sticking with the project and please have a wonderful day ❤. Soon we shall reap the reward!