Crate cargo_crev_reviews

Expand description

cargo_crev_reviews

Write cargo-crev reviews in GUI with a cross-platform app written in full-stack Rust
repository; version: 2022.122.1621 date: 2022-01-22 authors: Luciano Bestia

Try it

Install cargo_crev_reviews:

cargo install cargo_crev_reviews

Go to a Rust project directory where the Cargo.toml file is, and start the program:

cd ~/rustprojects/your-project-name
cargo_crev_reviews

And follow the simple instructions…

Backend CLI in Linux terminal:
screen_5
Frontend GUI in browser:
screen_6

cargo tree

Cargo-tree is a Rust utility that shows all the ramification of dependencies in your Rust project. It is included inside the cargo utility.

Your personal reviews are the most important. Ideally, you want to personally review every crate, rate it and write something about it for your own use. You want to know that the dependencies your program is using are not malicious or unsound. If you have a boss, he will sooner or later ask you if you reviewed all the dependencies. With cargo_crev_reviews you have a basic tool to do that.

Don’t panic!

To write a review you need to see the exact source code and other metadata about the crate. Click on the crate name in the list and it will open:

VSCode in the directory with a copy of the original dependency source code
crev.dev for other people reviews
crates.io for basic information
lib.rs for extended information
a list of all versions of that crate with your reviews added

Be warned that modern browsers block pop-ups and you have to allow that explicitly for this site 127.0.0.1.
If you don’t have VSCode, you can change the code_editor in the Config menu.

Reputation vs. code review

Personally, I think that the reputation of the author is important. For some highly visible and respected members of the Rust community I don’t review the code. The reputation of the author is enough to make me feel safe. This method is not perfect, because there can be identity theft or a faulty version. But I still think that it is an efficient and effective method for me.
Crates.io made a confusion with authors and owners. Lately they introduced the published_by field for every published version. For me this is the main person responsible for any eventual issue of this specific published version. Crates.io guarantees that published versions cannot be modified later. They are read-only, so you know that you see exactly the right code.

When you are satisfied with your own reviews, you then publish them for other developers to read. In the same way, reviews from other developers help you when analyzing the dependencies. More people that write reviews, better the information we get for our decisions.
One crate can have many versions. Every version is separately reviewed. Often, the review will be just the same for many versions, but that is good. When a developer wants to see the reviews for a specific version, he does not need to watch the reviews of all other versions of the same crate.

TL;DR

For developers that want to understand, tweak the code and contribute to the project, below is a long description of the important aspects of the project. If you are only a user of cargo_crev_reviews, you don’t need to read all of this.

Motivation

I think cargo-crev is a great tool to express trustworthiness in the open-source community, especially for the Rust programming language. I fear so much of supply chain attacks using dependencies from crates.io. For the smallest project you can get 100 dependencies easily. How to trust them all? To review them all manually? It is just crazy.
But if enough people write reviews, it will be so much easier to trust the code. It is the same principle as booking.com or air-bnb. Guests of a hotel write a review about their actual experience in the hotel. And you can read a hopefully truthful review and can understand if the hotel is good or bad. Sometimes can happen to find a fake review, but if there is enough people, most of them will be sincere.
Sadly, writing reviews in cargo-crev is hard. Let’s make a GUI wrapper around cargo-crev to make write reviews easier.
We will see walking the path what obstacles we must overcome.

Technical decisions

Rust does not have a true GUI story. It is mostly for CLI and libraries. Because GUI is mostly non cross-platform. But Rust is the best language for Wasm/Webassembly. So let’s combine this.
I will make a Rust workspace (multi-projects):

CLI for a web server (local micro-server)
Wasm for the browser (chrome and similar)

The web server CLI will access files, commands, libraries and the network. This will work only in Linux, but today Win10 has integrated Linux with WSL2. It will work just fine on all the operating systems sensible for Rust development.
Wasm in browser will just access the local micro web server. This is gonna be the GUI and because the browser works in every OS, this is cross-platform development.

I want the simplest web server ever. It will be used exclusively locally from one super simple web-application, so don’t need to care much about security. I choose simple server from the Rust book. I don’t care about multi-threading or async , because it will be used by only one browser. The example from the book evolved into the github repository of the author of the book github.com/steveklabnik. I cloned it and update the dependencies and consequently fixed some broken code. I published it as dev_bestia_simple_server on crates.io.

For the browser I will create a simple web app. All the code will be in Rust, I will avoid javascript. The GUI will be in HTML5 and CSS3. This is all supported by all modern browsers.

Development

I will use cargo-auto to automate the tasks needed to build the project.
The sub-directory automation_tasks_rs is the Rust project for cargo-auto.
My project dev_bestia_cargo_completion helps with bash auto-completion.

The Rust workspace is made of members:

backend CLI (this will be the main and only project to be published on crates.io)
GUI frontend

The sub-directory web_server_folder contains all the files and folder structure for a working development web_server.
But this files are not used directly. Because of the way the publish to crates.io works, I will embed them inside the Rust code as strings (base64 encoded if needed). I will make an automation task for that.

There is a file auto_generated_mod.rs where automation tasks generates boilerplate code. I prefer generated code to procedural macros because it is easy to write(generate), read and debug the code. Also the tools for auto-completion can have problems with procedural macros.

backend - cargo_crev_reviews

This is a micro-web-server intended for local use with only one local browser connected to it.
It is the backend of the application cargo_crev_reviews. I had to use the same name here.
Together the backend and the frontend form a complete application that is cross-platform.
They share some structs for communication that are defined in the common_structs_mod module. One automation task copies the content from backend to frontend projects to keep them in sync.
The only URL the server operates is: http://127.0.0.1:8182/cargo_crev_reviews

The web server CLI will access files, commands, libraries and the network. For the signing process of crev reviews it need the crev passphrase. This can be entered interactively or with the env variable _export CREV_PASSPHRASE=your_passphrase.
Add a space before the export command to avoid the secret to be saved in the bash history.

If I want to publish this on crates.io it must all be inside one binary executable file.
It means that all the static files: css, html, icons, images, … must be inside the Rust code.
For developing it is practical to have all this files as files.
But before release an automation task converts this files to strings and put them into the Rust code.

The micro-server will accept mostly POST with json similar to json-rpc. But sure I had to modify it to something more adequate for my use-case. I think in the future I will change that even more to something even more adequate.

Syntax:

--> data sent to Server - request
<-- data sent to Client - response

rpc call with named parameters:

--> {
"request_method": "subtract_calculate",
"request_data": {
    "subtrahend": 23,
    "minuend": 42,
    }
}

<-- {
"response_method": "subtract_show",
"response_data": {
    "subtracted": 19,     
    },
"response_html":"..."    
}

An example how to test a POST request with curl:

curl -d '{"request_method": "subtract", "request_data": {"subtrahend": 23, "minuend": 42}}' -H 'Content-Type: application/json' http://127.0.0.1:8182/cargo_crev_reviews

There are also a small number of GET requests for static files mostly to start the communication between the browser and the server.
TODO: one day I will add also a websocket communication, so the client can show a progress bar for actions processing on the server.

GUI frontend - cargo_crev_reviews_wasm

This simple web app is the GUI frontend of the application cargo_crev_reviews.
It is strictly designed for use on desktops as it is a tool for programers. No need to do a mobile version. The response from the rpc server contains:

response method name
html template
data

The method name is used to match and call the appropriate function.
The html template must be microXml compatible. The wasm code reads element by element and when finds a marker, inserts the data. I wanted the html template to be complete with some sample texts. So the markers are added in front of the element or attribute they are meant to replace.

common structs - common_structs_mod.rs

Common structures between backend and frontend. It is kind of a contract for communication.
All in 100% Rust language. One automation task keeps in sync the backend and frontend module.
TODO: Using structs is not generic enough. Most of the time I need then name of a field. Because with the name I can bind in different scenarios. Using structs I don’t have the name of the field at runtime. I think I will ditch most of the structs to have just a plain old flat text with QVS21. Inside that text every field will have a slice and a name. And I can then use that in runtime for bindings.

cargo-crev integration

The cargo-crev project contains many crates. The crates crev-lib and crev-data are libraries for integration. All the code working with crev is encapsulated in the crev_mod.rs module.

cargo registry

The cargo application is essential for work with the Rust language. Cargo maintains a local cargo registry in the directory ~/.cargo/registry/. It has 3 basic sub-directories: index, cache and src.
The registry index is the database with crates metadata and is git fetched from github. Path to an index file:
~/.cargo/registry/index/github.com-1ecc6299db9ec823/cache/re/ad/reader_for_microxml

The content of this file looks like this:

bc688d353fc7c7a2f3f1f5fed9a27fc1773fc710
1.0.0 {
    "name": "reader_for_microxml",
    "vers": "1.0.0",
    "deps": [],
    "cksum": "623616f68a6441e2f61aa01c9bbcf76f4c9989328e0e10ab747e936718791912",
    "features": {},
    "yanked": true,
    "links": null
}
1.1.11 {
    "name": "reader_for_microxml",
    "vers": "1.1.11",
    "deps": [],
    "cksum": "fd50abb1f0d11a59ebe6d3f31446e4af8d0f8a7df668034b6c9b94453fa30c42",
    "features": {},
    "yanked": true,
    "links": null
}

Cargo downloads from crates.io the complete source code for every dependency it needs for your project.
First it downloads the tar gz file ending with .crate into the cache directory:
~/.cargo/registry/cache/github.com-1ecc6299db9ec823/cargo_auto_lib-0.7.23.crate
Probably the cksum field in the registry index is calculated from this .crate file.
Then this is unzipped into the src folder as the complete source code directories and files:
~/.cargo/registry/src/github.com-1ecc6299db9ec823/.
Crates.io guarantees the .crate file for a crate+version cannot be altered or deleted and are always available for download from crates.io (even when yanked).
We can review exactly this local code with confidence, because we know it will never change.
This local files should not be altered in any way. But it can happen unknowingly and unwillingly, if we open a code editor with intellisense in this folder. It will create the target folder and Cargo.lock file. It can happen also that Go to definition opens this files in the editor and maybe we alter some comment or code. This is no good.
In the config and utils it is possible to check that these files are not tempered. If something is modified it is easy to just remove that folder from src. Then we can download and unpack the missing crates.

crates.io API

Some data are not available locally in the cargo registry and need to be obtained from https://crates.io//api/v1/crates/{}/{}/. Then I store them in ~/.config/crev/cargo_crev_reviews_data/db. Data from crates.io are immutable all, except yanked. I will find in the local registry the data for yanked and new versions. That will trigger the download of data from crates.io.

RustSec cargo audit

I added the result of cargo audit --json into the cargo tree list.

trusted publishers

There is a confusion on crates.io who is the owner, author or group that is responsible for a crate version. Lately they added a published_by field for a crate_version. That sounds more accurate. The one that published is responsible to check there is no malware inside.
You can edit your list of Trusted publishers in the app.

Code-flow

Everything is compiled into one single executable binary for Linux: cargo_crev_reviews.
First it opens the default browser with xdg-open on http://127.0.0.1:8182/cargo_crev_reviews/index.html.
I received a comment that xdg-open is not preinstalled on every Linux distro. I use Debian 10 and have it.
On other distros it is possible to install it with xdg-utils.
You can also change it with the env variable export CREV_BROWSER_PATH=/usr/bin/xdg-open for the command that exists on your system. Later you can change this command in the Config menu.
If your WSL2 does not have yet a default browser run this:

ln -s "/mnt/c/Program Files/Mozilla Firefox/firefox.exe" /usr/bin/browser_in_win
export BROWSER='/usr/bin/browser_in_win'

The command ln -sf is permanent and persistent. It makes a symbolic link file that stays there forever.
But export BROWSER= is NOT persistent. You need to add this command to ~/.bashrc that runs it on every start of terminal.

In the next millisecond the web server starts listening to 127.0.0.1 port 8182.
The first set of requests are GET and response is “static” files embedded in auto_generated_files_mod.rs

browser request for /cargo_crev_reviews/index.html is GET, the response is html text file embedded in auto_generated_files_mod.rs in the function: index_html()
This html is just an empty shell that gets the css and wasm code. There is no real content inside. This concept is Single-page application SPA.
index.html requests: 3 css files, pkg/cargo_crev_reviews.js, pkg/cargo_crev_reviews_bg.wasm, “favicon” icons/icon-032.png. All these requests are GET and responses come from auto_generated_files_mod.rs functions, some are text files and others are base64 files.
the browser imports the wasm module and starts the init function that requests request_cargo_tree_list. This responds with: response_method_name, response_html and response_data.
wasm (inside the browser) is Rust code. First it matches method_name and calls the appropriate function. It processes the html with the data and inserts it into index.html (the empty shell).
the browser renders our first page. Hooray!
the user clicks on some button.
the macro on_click! or row_on_click! hides the ugly Rust code behind the definition of an event handler in web_sys and calls a function
wasm creates a rpc request and sends/POST to the server
the request is POST, the server first matches the method_name and calls the appropriate function. The function processes the call and prepares some data. It loads the html template.
The response contains the html to be rendered and data to be inserted in this html before rendering.

How to add a page with data

Define the data structs

In common_structs_mod.rs add the structs like:

#[derive(Serialize, Deserialize, Debug, Default)]
pub struct VersionItemData {
    pub crate_name: String,
    pub crate_version: String,
}

#[derive(Serialize, Deserialize, Debug, Default)]
pub struct VersionListData {
    pub list_of_version: Vec<VersionItemData>,
}

Create a “standard” html page

The html page has to be MicroXml compatible, basically XHtml. Copy for example web_server_folder/review_list.html to a new html file. Open this file with a browser to preview it. I use the VSCode extension vscode-open-wsl and right-click on the file and Open with default application. In WSL2 I use my project wsl_open_browser to open the default browser from wsl2 to Windows. Now edit the html file to your liking and reload the browser with F5 to see the changes. Use some sample text to make it look as close to what you want. These texts will be later programmatically replaced, but they are invaluable while designing a nice layout.

Add markers

Inside the html you want to replace the sample texts with the data from the server. Before the text add the (invisible) marker for example . You can replace also an attribute if you insert an attribute before it like this data-wt_variable_name="next_attribute_name".
Now run the automation task cargo auto build that will copy/embed this file into auto_generated_files_mod.rs.

Write a server functions

In srv_methods_mod.rs add a function like this:

#[named]
pub fn srv_function_name(request_data: serde_json::Value) -> anyhow::Result<String> {
    log::info!(function_name!());
    let filter: ReviewFilterData = unwrap!(serde_json::from_value(request_data));
    let response_data  = get_some_data(filter)?;
    let response_html = crate::auto_generated_files_mod::review_edit_html();
    // cln_methods::cln_review_edit(response_data, response_html)
}

Now run the automation task cargo auto build that generates auto_generated_mod.rs.

client module

If I use a different struct for data, I must have different client modules and functions.
With a generic data-struct all of this could be generic.
TODO: use QVS21

In cln_version_mod.rs add a client function like this:

pub fn cln_review_edit(srv_response: RpcResponse) {
    let html = extract_html(&srv_response);
    store_to_review_item_data(srv_response);

    // the mutex is locked inside a scope. When this structure falls out of scope, the lock will be unlocked.
    let html_after_process = {
        let data = REVIEW_ITEM_DATA.lock().unwrap().deref();
        process_html(data, &html)
    };

    inject_into_html(&html_after_process);

    on_click!("button_review_save", request_review_save);
    on_click!("button_review_list", request_review_list);
}

sled database

I don’t want to repeatedly use crates.io api for the same data. I need a disk persistent storage for this data.
I will have a try with the sled database. A lightweight pure-Rust high-performance transactional embedded database. This is a key-value database. The value can be any struct. There can be multiple separate trees/keyspaces: crates, versions, reviews, yanked,….

plantUml

Write your diagrams in code with plantUml. The language syntax is pretty easy. I have a text file images/server_plantuml_v3.txt. The I use the online service plantuml.com/plantuml/proxy to create a png image. Because of caching I have the version of the file in the file name. When the version changes the old cache is not in use any more.
The server modules:

Bad surprise on WSL2

My git repository got corrupted. It looks that this happens to many people on WSL2.
The cure is:

find .git/objects/ -type f -empty | xargs rm
git fetch -p
git fsck --full

Some other info

Cargo-crev knows also about Issues and Advisories. But for this project I focused only on Reviews. You can rate the version as Negative if there is something really broken with it. I also limited the crates source to only crates.io. Other sources are not publicly interesting.

cargo crev reviews and advisory

We live in times of danger with supply chain attacks.
It is recommended to always use cargo-crev
to verify the trustworthiness of each of your dependencies.
Please, spread this info.
You can also read reviews quickly on the web:
https://web.crev.dev/rust-reviews/crates/

open-source and free as a beer

My open-source projects are free as a beer (MIT license).
I just love programming.
But I need also to drink. If you find my projects and tutorials helpful,
please buy me a beer donating on my paypal.
You know the price of a beer in your local bar ;-)
So I can drink a free beer for your health :-)
Na zdravje! Alla salute! Prost! Nazdravlje! 🍻

Structs

ansi clear all

ansi clear line

ansi color

ansi color

ansi reset color

SERVER_FIRST_SUBDIRECTORY

127.0.0.1

8182

ansi unhide cursor

ansi color

Enums

Status

Functions

browser_not_installed

warning browser from config not installed

cargo_audit_not_installed

warning cargo-audit not installed

cargo_crev_not_installed

warning cargo-crev not installed

db_sled_migration_update

if the data structure changes then it must be updated by a migration

get_config

host_port_is_busy

check that this is the only instance of this server return None if the host+port is free if the server host+port is not free, returns the String for the error message.

not_started_inside_rust_project

warning not started in Rust project

one_instance_of_the_program_already_running

warning that one instance is already running

parse_get_uri_and_response_file

GET is used only to request files Files are stored in functions in the auto_generated_files_mod.rs module there is an automation task to copy files from web_server_folder to the module

parse_post_data_and_match_method

response_404_not_found

response_err_message

response_modal_message

sync_in_background_reviews

working with crev data looks slow. I will sync in background with fast sled database and work from there.

sync_in_background_verify

working with crev verify looks slow. I will sync in background with fast sled database and work from there.

sync_in_background_yanked

from cargo registry read all versions that are in db_version only the yanked store in db_yanked

unlock_crev_id_interactively

unlock crev_id interactively

welcome_print

print the welcome text