scrapelect 0.3.2

Interpreter for scrapelect, a CSS-inspired web scraping DSL
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131

# Filters

Now we've seen how to identify parts of a page to turn into data,
let's look at how to manipulate that data.  `scrapelect` does this
using **filters**.  We've seen a couple already, like `take` and `text`,
but let's look at them more closely.

Every filter takes in a value, and a list of *named arguments* (which can
be empty), and returns the result of applying that filter.

We call a filter with `value | filter_name(arg: value, ...)`. The simplest filter is the `id` filter, which takes the value and
returns that same value: if we let `a: 5 | id();`, we get
`{ "a": 5 }`.

Another useful filter is the `dbg` filter, which, like `id`,
returns the original value, but it prints the value to the
terminal as well.  `dbg` has an *optional argument* msg, a
String, which specifies a message to include, if set.  If not
provided to the filter, it prints `debug message: ...`.

```scrp
a: 1 | dbg();
b: 2 | dbg(message: "from b");
```

will output to the console:

```
debug message: 1
from b: 2
```

and return `{ "a": 1, "b": 2 }`.

## Modifying filters

Often, though, it is useful to use a filter that modifies the passed
value in some way.  One useful filter is `strip`, which trims
leading and trailing whitespace from a string, which is often found
inside HTML elements.

```scrp
trimmed: "    hellooooo   " | strip();
```

outputs

```json
{
  "trimmed": "hellooooo"
}
```

Note that this doesn't *mutate* the original value passed in;
that is, if you read another binding as the input to a filter,
applying the filter will not change the value in the original binding:

```scrp
bind: "5";
new: $bind | int();
```

results (where `int` converts the value into the integer type):

```json
{
  "bind": "5",
  "new": 5
}
```

where `bind` is still a string but `new` is an int.

## Filter documentation

Documentation for all of the built-in filters is available at
[docs.rs](https://docs.rs/scrapelect/latest/scrapelect/interpreter/filter/builtin/),
which lists filter signatures, descriptions, and examples.

## Chaining filters

Filters are designed to be executed in a pipeline, passing the
output of one filter to the input of another:

```scrp
is-not-five: "5" | int() | eq(to: 5) | not();
```

outputs

```json
{
  "is-not-five": false
}
```

where `eq` returns whether the value is equal to `to`, and `not`
returns the boolean NOT (opposite) of the input value.

## Qualifiers

Similar to element blocks, filters can have qualifiers at the end.
The same qualifiers (none for one, `?` for optional, and `*` for list)
can be placed at the end of a filter call to modify its behavior:

- `value | filter(...)?` applies the filter to `value` if it is not `null`,
  or returns `null` if it is.
- `value | filter(...)*`, when `value` is a list, returns the list
  of every element in the list with the filter applied to it
  (similar to the map operation in other languages).  It is an
  error to use the `*` qualifier if the input value is not a list.

### Example

```scrp
floats: "1 2.3 4.5" | split() | float()*;
optional: "3.4" | float()?;
optional2: null | float()?;
```

returns, where `split` turns a string into an array of strings
split on a delimiter (by default, whitespace), and `float` turns
a value into the float type:

```json
{
  "floats": [1.0, 2.3, 4.5],
  "optional": 3.4,
  "optional2": null
}
```