Crate ingest

rs-ingest

This crate provides primitives for building custom ingestion engines on Futurenet using Rust. It’s inspired by the go/ingest package from the Stellar repository.

Developers often require ingestion features that are outside the scope of Horizon’s capabilities or need higher availability for data processing. This crate aims to cater to such needs by providing the means to build custom ingestion mechanisms tailored to specific requirements.

Note: This crate is still a work in progress. Its current capabilities are limited.

Note: Only POSIX systems are currently supported.

Features

The crate offers the following features:

Running Offline

Running offline involves replaying historical data through a catchup process. This crate supports two offline modes:

• Single-thread mode: Waits for the core subprocess to finish catching up and allows retrieval of ledger metadata.
• Multi-thread mode: Returns a Receiver<MetaResult> that receives newly decoded ledger metadata as it becomes available.

Running Online

Running online involves syncing with Futurenet and receiving ledger close metadata in real-time. This mode is useful for tasks requiring real-time data ingestion, such as event streaming.

Note: Currently, online mode only supports multi-threaded execution.

To learn more about the crate and check out a couple of examples see the README

Structs

• BoundedRange
  Represents a bounded range
• BufferedLedgerMetaReader
  Struct for reading buffered ledger metadata.
• CaptiveCore
  Represents a captive instance of the Stellar Core.
• ContextPath
  Context path object.
• IngestionConfig
  Configuration settings
• LedgerCloseMetaReader
• LedgerCloseMetaWrapper
  Wrapper struct to hold the LedgerCloseMeta data.
• MetaResult
  Represents the result of processing ledger metadata.
• MetaResultReader
• StellarCoreRunner
  Core runner object.

Enums

• BufReaderError
  Enum to represent different types of errors related to BufReader operations.
• BufferedLedgerMetaReaderMode
  Enum to indicate the mode of operation for BufferedLedgerMetaReader.
• Error
  Enum to represent different types of errors related to the CaptiveCore.
• Range
  Ranges supported. Currently unbounded ranges are not supported.
• ReaderError
• RunnerError
  Represents the potential errors that can occur during runner operations.
• RunnerStatus
  Represents the status of a core runner.
• SupportedNetwork
  Enum describing the network to run stellar-core on.

Traits

• MultiThreadBufferedLedgerMetaReader
  Trait for reading ledger metadata in multi-thread mode from a buffered source.
• SingleThreadBufferedLedgerMetaReader
  Trait for reading ledger metadata in single-thread mode from a buffered source.
• StellarCoreRunnerPublic
  Public interface for interacting with the Stellar Core runner. These actions should not be directory called by the implementor.