System libraries refactor, enhanced documentation, and advanced query capabilities

In the past few weeks, Graffe has undergone significant changes. Below is an overview of the latest features and improvements. To give an idea: 1465 files changed, 29400 lines of code-change.

Highlights

We've implemented a major overhaul of the entire library system. This involved consolidating multiple repositories, each with extensive dependencies and intricate release procedures, into a streamlined, single repository structure. Now, we have a system repository and a library repository that encompasses all streamlets, which have been refactored for improved type cleanliness and structural consistency.

Another key update is the simplication in building streamlets. We've standardized this across all streamlets, ensuring full implementation throughout the platform.

We've also established documentation for Graffe (insiders) to facilitate the creation and deployment of streamlets. This comprehensive guidance covers the command-line interface, documentation practices for streamlets, and includes basic code examples. To support this, we added template streamlets to @graffe/example for reference.

The metadata system for streamlets has been enhanced across the Graffe ecosystem. It now boasts a simplified structure, preserving all previous features but allowing for metadata to be incorporated using a single markdown statement. This approach retains advanced metadata capabilities while streamlining extraction from the margin, paving the way for comprehensive data lineage and provenance documentation and enabling section extraction essential for Graffe's functionality.

We have simplified the usage of inline Graffe data streams within SQL statements, streamlining previously complex definitions. A new query interface within the web application allows for straightforward experimentation with querying Graffe's dataset. For a practical illustration, please refer to the accompanying video.

Further expanding our platform support, we've released a Linux ARM build, thus extending our compatibility to all Mac and Linux environments.

Also a new graffe release command was created to simplify and streamline the process of releasing cells and updates to cells. This reduces many complex actions to a single command.

Beyond these key highlights, we have made numerous additional enhancements and bugfixes aimed at improving the development experience, code optimization, workflow efficiency, and library robustness through shared code practices. Another battery of tests was added. The full extent of these changes is substantial and extends beyond the scope of this brief update.

Screencast

Quick walkthrough of some of the changes (with sound).

Getting started

Become an insider (opens in a new tab). This grants you early access and thanks to your sponsoring you help making Graffe available to all of humanity!

Furthermore you can join our community (opens in a new tab) for updates.