Expand description
rs-ingest
This crate provides primitives for building custom ingestion engines on Futurenet using Rust.
It’s inspired by the go/ingest package from the Stellar repository.
Developers often require ingestion features that are outside the scope of Horizon’s capabilities or need higher availability for data processing. This crate aims to cater to such needs by providing the means to build custom ingestion mechanisms tailored to specific requirements.
Note: This crate is still a work in progress. Its current capabilities are limited.
Note: Only POSIX systems are currently supported.
§Features
The crate offers the following features:
§Running Offline
Running offline involves replaying historical data through a catchup process. This crate supports two offline modes:
- Single-thread mode: Waits for the core subprocess to finish catching up and allows retrieval of ledger metadata.
- Multi-thread mode: Returns a
Receiver<MetaResult>that receives newly decoded ledger metadata as it becomes available.
§Running Online
Running online involves syncing with Futurenet and receiving ledger close metadata in real-time. This mode is useful for tasks requiring real-time data ingestion, such as event streaming.
Note: Currently, online mode only supports multi-threaded execution.
To learn more about the crate and check out a couple of examples see the README
Structs§
- Bounded
Range - Represents a bounded range
- Buffered
Ledger Meta Reader - Struct for reading buffered ledger metadata.
- Captive
Core - Represents a captive instance of the Stellar Core.
- Context
Path - Context path object.
- Ingestion
Config - Configuration settings
- Ledger
Close Meta Reader - Ledger
Close Meta Wrapper - Wrapper struct to hold the
LedgerCloseMetadata. - Meta
Result - Represents the result of processing ledger metadata.
- Meta
Result Reader - Stellar
Core Runner - Core runner object.
Enums§
- BufReader
Error - Enum to represent different types of errors related to
BufReaderoperations. - Buffered
Ledger Meta Reader Mode - Enum to indicate the mode of operation for
BufferedLedgerMetaReader. - Error
- Enum to represent different types of errors related to the CaptiveCore.
- Range
- Ranges supported. Currently unbounded ranges are not supported.
- Reader
Error - Runner
Error - Represents the potential errors that can occur during runner operations.
- Runner
Status - Represents the status of a core runner.
- Supported
Network - Enum describing the network to run stellar-core on.
Constants§
- DEFAULT_
XDR_ RW_ DEPTH_ LIMIT - prevents stack overflow
Traits§
- Multi
Thread Buffered Ledger Meta Reader - Trait for reading ledger metadata in multi-thread mode from a buffered source.
- Single
Thread Buffered Ledger Meta Reader - Trait for reading ledger metadata in single-thread mode from a buffered source.
- Stellar
Core Runner Public - Public interface for interacting with the Stellar Core runner. These actions should not be directory called by the implementor.