mocra 0.1.6

A distributed, event-driven crawling and data collection framework
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

# mocra


A distributed, event-driven crawling and data collection framework for Rust.

## Documentation

- version 0.1.5

- Config precedence fix (v0.1.5):
	- Effective config now consistently follows: ORM/module config > config.toml > hardcoded default.
	- Fixed fallback behavior for `module_locker` and `wss_timeout` when ORM value is missing.
	- Fixed boolean fallback in download config loading to inherit from config.toml when module value is absent.

- Middleware interface update (v0.1.2):
	- `DownloadMiddleware` / `DataMiddleware` / `DataStoreMiddleware` methods now use `&mut self`.
	- Added store lifecycle hooks in `DataStoreMiddleware`:
		- `before_store(&mut self, _config: &Option<ModuleConfig>) -> Result<()>`
		- `after_store(&mut self, _config: &Option<ModuleConfig>) -> Result<()>`
	- Store execution order is now unified as:
		1. `before_store`
		2. `store_data`
		3. `after_store`
	- If `before_store` fails, `store_data` and `after_store` are skipped and the middleware enters existing error/retry flow.

- ParserData update (v0.1.1): `ParserData.parser_task` changed from `Option<ParserTaskModel>` to `Vec<ParserTaskModel>`.
	- Parser can now return multiple next parser tasks in a single round.
	- Existing `with_task(...)` remains available (now appends to the vector).
	- Recommended check for "has next task": use `!parser_task.is_empty()`.
    
- Project docs: `docs/README.md`
- API docs (after publish): <https://docs.rs/mocra>

## License


Licensed under either of:

- MIT license
- Apache License, Version 2.0