pub struct DevTweaks {Show 27 fields
pub buffer_cache_strategy: Option<BufferCacheStrategy>,
pub buffer_max_buckets: Option<usize>,
pub buffer_cache_allocation_strategy: Option<BufferCacheAllocationStrategy>,
pub fbuf_slab_bytes_per_class: Option<usize>,
pub fetch_join: Option<bool>,
pub fetch_distinct: Option<bool>,
pub merger: Option<MergerType>,
pub storage_mb_max: Option<u64>,
pub stack_overflow_backtrace: Option<bool>,
pub splitter_chunk_size_records: Option<u64>,
pub adaptive_joins: Option<bool>,
pub eager_evict: Option<bool>,
pub balancer_min_relative_improvement_threshold: Option<f64>,
pub balancer_min_absolute_improvement_threshold: Option<u64>,
pub balancer_balance_tax: Option<f64>,
pub balancer_key_distribution_refresh_threshold: Option<f64>,
pub bloom_false_positive_rate: Option<f64>,
pub enable_roaring: Option<bool>,
pub max_level0_batch_size_records: Option<u16>,
pub merger_threads: Option<u16>,
pub negative_weight_multiplier: Option<u16>,
pub disable_auto_transaction: Option<bool>,
pub now_offset: Option<DateTime<Utc>>,
pub now_http_driven: Option<bool>,
pub streaming_exchange: Option<bool>,
pub optimize_input_during_commit: Option<bool>,
pub other_options: BTreeMap<String, Value>,
}Expand description
Optional settings for tweaking Feldera internals.
These settings reflect experiments that may come and go and change from version to version. Users should not consider them to be stable.
Fields§
§buffer_cache_strategy: Option<BufferCacheStrategy>Buffer-cache implementation to use for storage reads.
The default is s3_fifo.
buffer_max_buckets: Option<usize>Override the number of buckets/shards used by sharded buffer caches.
This only applies when buffer_cache_strategy = "s3_fifo". Values are
rounded up to the next power of two because the current implementation
shards by hash(key) & (n - 1).
buffer_cache_allocation_strategy: Option<BufferCacheAllocationStrategy>How S3-FIFO caches are assigned to foreground/background workers.
This only applies when buffer_cache_strategy = "s3_fifo". The
default is shared_per_worker_pair; LRU always uses per_thread.
fbuf_slab_bytes_per_class: Option<usize>Target number of cached bytes retained in each FBuf slab size class.
The default is 16 MiB.
fetch_join: Option<bool>Whether to asynchronously fetch keys needed for the join operator from storage. Asynchronous fetching should be faster for high-latency storage, such as object storage, but it could use excessive amounts of memory if the number of keys fetched is very large.
fetch_distinct: Option<bool>Whether to asynchronously fetch keys needed for the distinct operator from storage. Asynchronous fetching should be faster for high-latency storage, such as object storage, but it could use excessive amounts of memory if the number of keys fetched is very large.
merger: Option<MergerType>Which merger to use.
storage_mb_max: Option<u64>If set, the maximum amount of storage, in MiB, for the POSIX backend to
allow to be in use before failing all writes with StorageFull. This
is useful for testing on top of storage that does not implement its own
quota mechanism.
stack_overflow_backtrace: Option<bool>Attempt to print a stack trace on stack overflow.
To be used for debugging only; do not enable in production.
splitter_chunk_size_records: Option<u64>Controls the maximal number of records output by splitter operators (joins, distinct, aggregation, rolling window and group operators) at each step.
The default value is 10,000 records.
adaptive_joins: Option<bool>Enable adaptive joins.
Adaptive joins dynamically change their partitioning policy to avoid skew.
Adaptive joins are disabled by default.
eager_evict: Option<bool>Evict eagerly from buffer caches as files get deleted.
This is an optimization that drops files from the cache as soon as they are deleted.
It has unknown (no?) performance benefits from what I can tell.
Historically it made sense to do this for two reasons: a) we know with 100% guarantee that the file won’t ever be read again. b) we could do this in O(logn) time with the LRU cache. This is no longer true for s3-fifo where it is O(n).
If the eviction is expensive, (many small objects in the cache) this can cause a regression.
New default disables this behavior by making it false.
If this doesn’t cause regression we will remove this option in the future.
balancer_min_relative_improvement_threshold: Option<f64>The minimum relative improvement threshold for the join balancer.
The join balancer is a component that dynamically chooses an optimal partitioning policy for adaptive join operators. This parameter prevents the join balancer from making changes to the partitioning policy if the improvement is not significant, since the overhead of such rebalancing, especially when performed frequently, can exceed the benefits.
A rebalancing is considered significant if the relative estimated improvement for the cluster of joins where the rebalancing is applied is at least this threshold.
A rebalancing is applied if both this threshold and
balancer_min_absolute_improvement_threshold are met.
The default value is 1.2.
balancer_min_absolute_improvement_threshold: Option<u64>The minimum absolute improvement threshold for the balancer.
The join balancer is a component that dynamically chooses an optimal partitioning policy for adaptive join operators. This parameter prevents the join balancer from making changes to the partitioning policy if the improvement is not significant, since the overhead of such rebalancing, especially when performed frequently, can exceed the benefits.
A rebalancing is considered significant if the absolute estimated improvement for the cluster of joins where the rebalancing is applied is at least this threshold. The cost model used by the balancer is based on the number of records in the largest partition of a collection.
A rebalancing is applied if both this threshold and
balancer_min_relative_improvement_threshold are met.
The default value is 10,000.
balancer_balance_tax: Option<f64>Factor that discourages the use of the Balance policy in a perfectly balanced collection.
Assuming a perfectly balanced key distribution, the Balance policy is slightly less efficient than Shard,
since it requires computing the hash of the entire key/value pair. This factor discourages the use of this policy
if the skew is <balancer_balance_tax.
The default value is 1.1.
balancer_key_distribution_refresh_threshold: Option<f64>The balancer threshold for checking for an improved partitioning policy for a stream.
Finding a good partitioning policy for a circuit involves solving an optimization problem, which can be relatively expensive. Instead of doing this on every step, the balancer only checks for an improved partitioning policy if the key distribution of a stream has changed significantly since the current solution was computed. Specifically, it only kicks in when the size of at least one shard of at least one stream in the cluster has changed by more than this threshold.
The default value is 0.1.
bloom_false_positive_rate: Option<f64>False-positive rate for Bloom filters on batches on storage, as a fraction f, where 0 < f < 1.
The false-positive rate trades off between the amount of memory used by Bloom filters and how frequently storage needs to be searched for keys that are not actually present. Typical false-positive rates and their corresponding memory costs are:
- 0.1: 4.8 bits per key
- 0.01: 9.6 bits per key
- 0.001: 14.4 bits per key
- 0.0001: 19.2 bits per key (default)
Values outside the valid range, such as 0.0, disable Bloom filters.
enable_roaring: Option<bool>Whether file-backed batches may use roaring membership filters when the key type supports them.
max_level0_batch_size_records: Option<u16>Maximum batch size in records for level 0 merges.
merger_threads: Option<u16>The number of merger threads.
The default value is equal to the number of worker threads.
negative_weight_multiplier: Option<u16>Additional bias the merger assigns to records with negative weights (retractions) to promote them to higher levels of the LSM tree sooner.
Reasonable values for this parameter are in the range [0, 10].
The default value is 0, which means that retractions are not given any additional bias.
disable_auto_transaction: Option<bool>Don’t automatically start a transaction for every step.
now_offset: Option<DateTime<Utc>>Override the timestamp returned by SQL NOW() at pipeline start.
When set, the clock connector anchors NOW() to this RFC 3339
timestamp the first time the pipeline starts and advances at
wall-clock cadence from there:
NOW() = now_offset + (wall_clock - wall_clock_at_start).
Any RFC 3339 timestamp parseable by chrono::DateTime<Utc> is
accepted (years 0001 through 9999), in the past or future
relative to wall clock.
This is a testing knob for queries that depend on NOW().
On resume the clock continues from the last journaled NOW();
now_offset’s value is honored only on a fresh start:
| Initial run | Resume from checkpoint | Post-replay NOW() |
|---|---|---|
| no offset | no offset | wall clock (unchanged) |
| offset | offset | wall-clock pace from the last journaled value; the new offset value is ignored |
| offset | no offset | jumps to wall clock (explicit opt-out of the anchor) |
| no offset | offset | wall-clock pace from the last journaled value; the new offset value is ignored |
now_http_driven: Option<bool>Drive NOW() from an external HTTP endpoint instead of wall clock.
When true, the clock connector emits one initial tick (using
now_offset if set, otherwise wall clock) and then holds that
value. Subsequent calls to POST /clock/advance move NOW()
forward by the requested delta. Negative deltas are rejected;
the clock is forward-only.
streaming_exchange: Option<bool>Enable streaming exchange.
optimize_input_during_commit: Option<bool>Optimize input operators during transaction commit.
other_options: BTreeMap<String, Value>Options not understood by this particular version.
This allows the pipeline manager to take options that a custom or old runtime version accepts.
Implementations§
Source§impl DevTweaks
impl DevTweaks
pub fn buffer_cache_strategy(&self) -> BufferCacheStrategy
pub fn buffer_cache_allocation_strategy(&self) -> BufferCacheAllocationStrategy
pub fn effective_buffer_cache_allocation_strategy( &self, ) -> BufferCacheAllocationStrategy
pub fn fetch_join(&self) -> bool
pub fn fetch_distinct(&self) -> bool
pub fn merger(&self) -> MergerType
pub fn stack_overflow_backtrace(&self) -> bool
pub fn splitter_chunk_size_records(&self) -> u64
pub fn adaptive_joins(&self) -> bool
pub fn balancer_min_relative_improvement_threshold(&self) -> f64
pub fn balancer_min_absolute_improvement_threshold(&self) -> u64
pub fn balancer_balance_tax(&self) -> f64
pub fn balancer_key_distribution_refresh_threshold(&self) -> f64
pub fn bloom_false_positive_rate(&self) -> f64
pub fn enable_roaring(&self) -> bool
pub fn negative_weight_multiplier(&self) -> u16
pub fn disable_auto_transaction(&self) -> bool
Sourcepub fn now_offset_ms(&self) -> Option<i64>
pub fn now_offset_ms(&self) -> Option<i64>
Configured now_offset as milliseconds since the Unix epoch,
or None if no override is set.