trek-rs 0.2.0

A web content extraction library that removes clutter from web pages
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148

# Trek

![Trek Banner](https://github.com/officialunofficial/trek/blob/main/banner.png)

[![Crates.io](https://img.shields.io/crates/v/trek-rs.svg)](https://crates.io/crates/trek-rs) [![npm](https://img.shields.io/npm/v/@officialunofficial/trek.svg)](https://www.npmjs.com/package/@officialunofficial/trek)

A modern web content extraction library written in Rust, compiled to WebAssembly.

Trek removes clutter from web pages and extracts clean, readable content. It's designed as a modern alternative to Mozilla Readability with enhanced features like mobile-aware extraction and consistent HTML standardization.

## Features

- 🦀 Written in Rust for performance and safety
- 🌐 Compiles to WebAssembly for browser usage
- 📱 Mobile-aware content extraction
- 🎯 Site-specific extractors for popular platforms
- 🔧 Configurable extraction options
- 📊 Content scoring algorithm
- 🏷️ Metadata extraction (title, author, date, etc.)

## Installation

### As a Rust library

```toml
[dependencies]
trek-rs = "0.1"
```

### As a WASM/JavaScript module

```bash
npm install @officialunofficial/trek
```

Or with other package managers:

```bash
# Yarn
yarn add @officialunofficial/trek

# pnpm
pnpm add @officialunofficial/trek

# Bun
bun add @officialunofficial/trek
```

## Usage

### Rust

```rust
use trek_rs::{Trek, TrekOptions};

let options = TrekOptions {
    debug: false,
    url: Some("https://example.com".to_string()),
    ..Default::default()
};

let trek = Trek::new(options);
let result = trek.parse(html_content)?;

println!("Title: {}", result.metadata.title);
println!("Content: {}", result.content);
```

### Web Playground

Trek includes an interactive web playground for testing content extraction:

```bash
# Build WASM and start the playground server
make playground

# Open http://localhost:8000/playground/ in your browser
```

The playground provides:
- **Live Extraction**: Paste HTML and see extracted content instantly
- **Multiple Views**: Switch between content, metadata, raw JSON, and debug tabs
- **Extraction Options**: Toggle clutter removal and metadata inclusion
- **Example Content**: Pre-loaded example to demonstrate Trek's capabilities

#### Playground Features

- **Content Tab**: Shows the extracted article content with proper formatting
- **Metadata Tab**: Displays title, author, word count, and other metadata
- **Raw JSON Tab**: View the complete extraction response
- **Debug Tab**: See extraction details and performance metrics

### JavaScript/TypeScript

```javascript
import init, { TrekWasm } from '@officialunofficial/trek';

// Initialize the WASM module
await init();

const trek = new TrekWasm({
    debug: false,
    url: 'https://example.com'
});

const result = await trek.parse(htmlContent);

console.log('Title:', result.title);
console.log('Content:', result.content);
```

## Building

### Native library

```bash
cargo build --release
```

### WebAssembly

```bash
wasm-pack build --target web --out-dir pkg
```

## Development

```bash
# Run tests
cargo test

# Run clippy
cargo clippy --all-targets --all-features

# Format code
cargo fmt

# Generate changelog
git cliff -o CHANGELOG.md
```

## Credits

Trek is a fork of [Defuddle](https://github.com/kepano/defuddle) by [@kepano](https://github.com/kepano), refactored into Rust, adding WebAssembly support, site-specific extractors, and additional features.

## License

MIT