Struct pachyderm::pfs::PutFileRequest

pub struct PutFileRequest {
    pub file: Option<File>,
    pub value: Vec<u8>,
    pub url: String,
    pub recursive: bool,
    pub delimiter: i32,
    pub target_file_datums: i64,
    pub target_file_bytes: i64,
    pub header_records: i64,
    pub overwrite_index: Option<OverwriteIndex>,
    pub delete: bool,
}

Fields

file: Option<File>

value: Vec<u8>

url: String

recursive: bool

applies only to URLs that can be recursively walked, for example s3:// URLs

delimiter: i32

Delimiter causes data to be broken up into separate files with File.Path as a prefix.

target_file_datums: i64

TargetFileDatums specifies the target number of datums in each written file it may be lower if data does not split evenly, but will never be higher, unless the value is 0.

target_file_bytes: i64

TargetFileBytes specifies the target number of bytes in each written file, files may have more or fewer bytes than the target.

header_records: i64

header_records is an option for splitting data when 'delimiter' is not NONE (or SQL). It specifies the number of records that are converted to a header and applied to all file shards.

This is particularly useful for CSV files, where the first row often contains column titles; if 'header_records' is set to one in that case, the first row will be associated with the directory that contains the rest of the split-up csv rows as files, and if any data is retrieved from that directory by GetFile, it will appear to begin with that first row of column labels (including in pipeline workers).

Note that SQL files have their own logic for determining headers (their header is not a number of records, but a collection of SQL commands that create the relevant tables and such). This way, SQL files retrieved by GetFile can be passed to psql, and they will set up the appropriate tables before inserting the records in the files that were retrieved.

overwrite_index: Option<OverwriteIndex>

overwrite_index is the object index where the write starts from. All existing objects starting from the index are deleted.

delete: bool

delete indicates that the file should be deleted, this is redundant with DeleteFile, but is necessary because it allows you to send file deletes atomically with other PutFile operations.

Implementations

impl PutFileRequest

pub fn delimiter(&self) -> Delimiter

Returns the enum value of delimiter, or the default if the field is set to an invalid enum value.

pub fn set_delimiter(&mut self, value: Delimiter)

Sets delimiter to the provided enum value.

Trait Implementations

impl Clone for PutFileRequest

fn clone(&self) -> PutFileRequest

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for PutFileRequest

fn fmt(&self, f: &mut Formatter) -> Result

Formats the value using the given formatter.

impl Default for PutFileRequest

fn default() -> PutFileRequest

Returns the "default value" for a type.

impl Message for PutFileRequest

fn encode_raw<B>(&self, buf: &mut B) where
    B: BufMut, 

fn merge_field<B>(
    &mut self,
    tag: u32,
    wire_type: WireType,
    buf: &mut B,
    ctx: DecodeContext
) -> Result<(), DecodeError> where
    B: Buf, 

fn encoded_len(&self) -> usize

Returns the encoded length of the message without a length delimiter.

fn clear(&mut self)

Clears the message, resetting all fields to their default.

fn encode<B>(&self, buf: &mut B) -> Result<(), EncodeError> where
    B: BufMut, 

Encodes the message to a buffer.

fn encode_length_delimited<B>(&self, buf: &mut B) -> Result<(), EncodeError> where
    B: BufMut, 

Encodes the message with a length-delimiter to a buffer.

fn decode<B>(buf: B) -> Result<Self, DecodeError> where
    B: Buf,
    Self: Default, 

Decodes an instance of the message from a buffer.

fn decode_length_delimited<B>(buf: B) -> Result<Self, DecodeError> where
    B: Buf,
    Self: Default, 

Decodes a length-delimited instance of the message from the buffer.

fn merge<B>(&mut self, buf: B) -> Result<(), DecodeError> where
    B: Buf, 

Decodes an instance of the message from a buffer, and merges it into self.

fn merge_length_delimited<B>(&mut self, buf: B) -> Result<(), DecodeError> where
    B: Buf, 

Decodes a length-delimited instance of the message from buffer, and merges it into self.

impl PartialEq<PutFileRequest> for PutFileRequest

fn eq(&self, other: &PutFileRequest) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &PutFileRequest) -> bool

This method tests for !=.

impl StructuralPartialEq for PutFileRequest