[−][src]Trait dbcrossbarlib::Locator
Specify the the location of data or a schema.
Required methods
fn as_any(&self) -> &dyn Any
Provide a mechanism for casting a dyn Locator
back to the underlying,
concrete locator type using Rust's Any
type.
See this StackOverflow question for a discussion of the technical
details, and why we need a Locator::as_any
method to use Any
.
This is a bit of a sketchy feature to provide, but we provide it for use
with supports_write_remote_data
and write_remote_data
, which are
used for certain locator pairs (i.e., Google Cloud Storage and BigQuery)
to bypass our normal local_data
and write_local_data
transfers and
use an external, optimized transfer method (such as direct loads from
Google Cloud Storage into BigQuery).
This should always be implemented as follows:
impl Locator for MyLocator {
fn as_any(&self) -> &dyn Any {
self
}
}
Provided methods
fn schema(&self, _ctx: Context) -> BoxFuture<Option<Table>>
Return a table schema, if available.
fn write_schema(
&self,
_ctx: Context,
_schema: Table,
_if_exists: IfExists
) -> BoxFuture<()>
&self,
_ctx: Context,
_schema: Table,
_if_exists: IfExists
) -> BoxFuture<()>
Write a table schema to this locator, if that's the sort of thing that we can do.
fn count(
&self,
_ctx: Context,
_shared_args: SharedArguments<Unverified>,
_source_args: SourceArguments<Unverified>
) -> BoxFuture<usize>
&self,
_ctx: Context,
_shared_args: SharedArguments<Unverified>,
_source_args: SourceArguments<Unverified>
) -> BoxFuture<usize>
Count the records specified by this locator.
fn local_data(
&self,
_ctx: Context,
_shared_args: SharedArguments<Unverified>,
_source_args: SourceArguments<Unverified>
) -> BoxFuture<Option<BoxStream<CsvStream>>>
&self,
_ctx: Context,
_shared_args: SharedArguments<Unverified>,
_source_args: SourceArguments<Unverified>
) -> BoxFuture<Option<BoxStream<CsvStream>>>
If this locator can be used as a local data source, return a stream of CSV streams. This function type is bit hairy:
- The outermost
BoxFuture
is essentially an asyncResult
, returning either a value or an error. It's boxed because we don't know what concrete type it will actually be, just that it will implementFuture
. - The
Option
will beNone
if we have no local data, orSome
if we can provide one or more CSV streams. - The
BoxStream
returns a "stream of streams". This could be aVec<CsvStream>
, but that would force us to, say, open up hundreds of CSV files or S3 objects at once, causing us to run out of file descriptors. By returning a stream, we allow our caller to open up files or start downloads only when needed. - The innermost
CsvStream
is a stream of raw CSV data plus some other information, like the original filename.
fn display_output_locators(&self) -> DisplayOutputLocators
Should we display the individual output locations?
fn write_local_data(
&self,
_ctx: Context,
_data: BoxStream<CsvStream>,
_shared_args: SharedArguments<Unverified>,
_dest_args: DestinationArguments<Unverified>
) -> BoxFuture<BoxStream<BoxFuture<BoxLocator>>>
&self,
_ctx: Context,
_data: BoxStream<CsvStream>,
_shared_args: SharedArguments<Unverified>,
_dest_args: DestinationArguments<Unverified>
) -> BoxFuture<BoxStream<BoxFuture<BoxLocator>>>
If this locator can be used as a local data sink, write data to it.
This function takes a stream data
as input, the elements of which are
individual CsvStream
values. An implementation should normally use
map
or and_then
to write those CSV streams to storage associated
with the locator, and return a stream of BoxFuture<()>
values:
# Pseudo code for parallel output.
data.map(async |csv_stream| {
write(csv_stream).await?;
Ok(())
})
For cases where output must be serialized, it's OK to consume the entire
data
stream, and return a single-item stream containing ()
.
The caller of write_local_data
will pull several items at a time from
the returned BoxStream<BoxFuture<()>>
and evaluate them in parallel.
fn supports_write_remote_data(&self, _source: &dyn Locator) -> bool
Can we access the data at source
directly using write_remote_data
?
fn write_remote_data(
&self,
_ctx: Context,
source: BoxLocator,
_shared_args: SharedArguments<Unverified>,
_source_args: SourceArguments<Unverified>,
_dest_args: DestinationArguments<Unverified>
) -> BoxFuture<Vec<BoxLocator>>
&self,
_ctx: Context,
source: BoxLocator,
_shared_args: SharedArguments<Unverified>,
_source_args: SourceArguments<Unverified>,
_dest_args: DestinationArguments<Unverified>
) -> BoxFuture<Vec<BoxLocator>>
Take the data at source
, and write to this locator directly, without
passing it through the local system.
This is used to bypass source.local_data
and dest.write_local_data
when we don't need them.
Implementors
impl Locator for BigQueryLocator
[src]
fn as_any(&self) -> &dyn Any
[src]
fn schema(&self, ctx: Context) -> BoxFuture<Option<Table>>
[src]
fn count(
&self,
ctx: Context,
shared_args: SharedArguments<Unverified>,
source_args: SourceArguments<Unverified>
) -> BoxFuture<usize>
[src]
&self,
ctx: Context,
shared_args: SharedArguments<Unverified>,
source_args: SourceArguments<Unverified>
) -> BoxFuture<usize>
fn local_data(
&self,
ctx: Context,
shared_args: SharedArguments<Unverified>,
source_args: SourceArguments<Unverified>
) -> BoxFuture<Option<BoxStream<CsvStream>>>
[src]
&self,
ctx: Context,
shared_args: SharedArguments<Unverified>,
source_args: SourceArguments<Unverified>
) -> BoxFuture<Option<BoxStream<CsvStream>>>
fn write_local_data(
&self,
ctx: Context,
data: BoxStream<CsvStream>,
shared_args: SharedArguments<Unverified>,
dest_args: DestinationArguments<Unverified>
) -> BoxFuture<BoxStream<BoxFuture<BoxLocator>>>
[src]
&self,
ctx: Context,
data: BoxStream<CsvStream>,
shared_args: SharedArguments<Unverified>,
dest_args: DestinationArguments<Unverified>
) -> BoxFuture<BoxStream<BoxFuture<BoxLocator>>>
fn supports_write_remote_data(&self, source: &dyn Locator) -> bool
[src]
fn write_remote_data(
&self,
ctx: Context,
source: BoxLocator,
shared_args: SharedArguments<Unverified>,
source_args: SourceArguments<Unverified>,
dest_args: DestinationArguments<Unverified>
) -> BoxFuture<Vec<BoxLocator>>
[src]
&self,
ctx: Context,
source: BoxLocator,
shared_args: SharedArguments<Unverified>,
source_args: SourceArguments<Unverified>,
dest_args: DestinationArguments<Unverified>
) -> BoxFuture<Vec<BoxLocator>>
impl Locator for BigQuerySchemaLocator
[src]
fn as_any(&self) -> &dyn Any
[src]
fn schema(&self, ctx: Context) -> BoxFuture<Option<Table>>
[src]
fn write_schema(
&self,
ctx: Context,
table: Table,
if_exists: IfExists
) -> BoxFuture<()>
[src]
&self,
ctx: Context,
table: Table,
if_exists: IfExists
) -> BoxFuture<()>
impl Locator for DbcrossbarSchemaLocator
[src]
fn as_any(&self) -> &dyn Any
[src]
fn schema(&self, ctx: Context) -> BoxFuture<Option<Table>>
[src]
fn write_schema(
&self,
ctx: Context,
table: Table,
if_exists: IfExists
) -> BoxFuture<()>
[src]
&self,
ctx: Context,
table: Table,
if_exists: IfExists
) -> BoxFuture<()>
impl Locator for DbcrossbarTsLocator
[src]
impl Locator for PostgresLocator
[src]
fn as_any(&self) -> &dyn Any
[src]
fn schema(&self, _ctx: Context) -> BoxFuture<Option<Table>>
[src]
fn count(
&self,
ctx: Context,
shared_args: SharedArguments<Unverified>,
source_args: SourceArguments<Unverified>
) -> BoxFuture<usize>
[src]
&self,
ctx: Context,
shared_args: SharedArguments<Unverified>,
source_args: SourceArguments<Unverified>
) -> BoxFuture<usize>
fn local_data(
&self,
ctx: Context,
shared_args: SharedArguments<Unverified>,
source_args: SourceArguments<Unverified>
) -> BoxFuture<Option<BoxStream<CsvStream>>>
[src]
&self,
ctx: Context,
shared_args: SharedArguments<Unverified>,
source_args: SourceArguments<Unverified>
) -> BoxFuture<Option<BoxStream<CsvStream>>>
fn write_local_data(
&self,
ctx: Context,
data: BoxStream<CsvStream>,
shared_args: SharedArguments<Unverified>,
dest_args: DestinationArguments<Unverified>
) -> BoxFuture<BoxStream<BoxFuture<BoxLocator>>>
[src]
&self,
ctx: Context,
data: BoxStream<CsvStream>,
shared_args: SharedArguments<Unverified>,
dest_args: DestinationArguments<Unverified>
) -> BoxFuture<BoxStream<BoxFuture<BoxLocator>>>
impl Locator for PostgresSqlLocator
[src]
fn as_any(&self) -> &dyn Any
[src]
fn schema(&self, ctx: Context) -> BoxFuture<Option<Table>>
[src]
fn write_schema(
&self,
ctx: Context,
table: Table,
if_exists: IfExists
) -> BoxFuture<()>
[src]
&self,
ctx: Context,
table: Table,
if_exists: IfExists
) -> BoxFuture<()>
impl Locator for RedshiftLocator
[src]
fn as_any(&self) -> &dyn Any
[src]
fn schema(&self, ctx: Context) -> BoxFuture<Option<Table>>
[src]
fn local_data(
&self,
ctx: Context,
shared_args: SharedArguments<Unverified>,
source_args: SourceArguments<Unverified>
) -> BoxFuture<Option<BoxStream<CsvStream>>>
[src]
&self,
ctx: Context,
shared_args: SharedArguments<Unverified>,
source_args: SourceArguments<Unverified>
) -> BoxFuture<Option<BoxStream<CsvStream>>>
fn write_local_data(
&self,
ctx: Context,
data: BoxStream<CsvStream>,
shared_args: SharedArguments<Unverified>,
dest_args: DestinationArguments<Unverified>
) -> BoxFuture<BoxStream<BoxFuture<BoxLocator>>>
[src]
&self,
ctx: Context,
data: BoxStream<CsvStream>,
shared_args: SharedArguments<Unverified>,
dest_args: DestinationArguments<Unverified>
) -> BoxFuture<BoxStream<BoxFuture<BoxLocator>>>
fn supports_write_remote_data(&self, source: &dyn Locator) -> bool
[src]
fn write_remote_data(
&self,
ctx: Context,
source: BoxLocator,
shared_args: SharedArguments<Unverified>,
source_args: SourceArguments<Unverified>,
dest_args: DestinationArguments<Unverified>
) -> BoxFuture<Vec<BoxLocator>>
[src]
&self,
ctx: Context,
source: BoxLocator,
shared_args: SharedArguments<Unverified>,
source_args: SourceArguments<Unverified>,
dest_args: DestinationArguments<Unverified>
) -> BoxFuture<Vec<BoxLocator>>