Enum Modality

pub enum Modality {
    Text {
        data: String,
        controls: Option<Vec<TextControl>>,
    },
    Image {
        data: String,
        x: Option<i32>,
        y: Option<i32>,
        size: Option<i32>,
        controls: Option<Vec<ImageControl>>,
    },
    TokenIds {
        data: Vec<u32>,
        controls: Option<Vec<TokenControl>>,
    },
}

The prompt for models can be a combination of different modalities (Text and Image). The type of modalities which are supported depend on the Model in question.

Variants

Text

The only type of prompt which can be used with pure language models

Fields

data: String

controls: Option<Vec<TextControl>>

Image

An image input into the model. See Modality::from_image_path.

Fields

data: String

An image send as part of a prompt to a model. The image is represented as base64.

Note: The models operate on square images. All non-square images are center-cropped before going to the model, so portions of the image may not be visible.

You can supply specific cropping parameters if you like, to choose a different area of the image than a center-crop. Or, you can always transform the image yourself to a square before sending it.

x: Option<i32>

x-coordinate of top left corner of cropping box in pixels

y: Option<i32>

y-coordinate of top left corner of cropping box in pixels

size: Option<i32>

Size of the cropping square in pixels

controls: Option<Vec<ImageControl>>

TokenIds

Fields

data: Vec<u32>

controls: Option<Vec<TokenControl>>

Implementations

impl Modality

pub fn from_text( text: impl Into<String>, controls: Option<Vec<TextControl>>, ) -> Self

Instantiates a text prompt

pub fn from_token_ids( ids: Vec<u32>, controls: Option<Vec<TokenControl>>, ) -> Self

Instantiates a token_ids prompt

pub fn from_image_path(path: impl AsRef<Path>) -> Result<Self, LoadImageError>

pub fn from_image(image: &DynamicImage) -> Result<Self, LoadImageError>

Image input for model

The model can only see squared pictures. Images are centercropped. You may want to use this method instead of Self::from_image_path in case you have the image in memory already and do not want to load it from a file again.

Trait Implementations

impl Clone for Modality

fn clone(&self) -> Modality

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Modality

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Serialize for Modality

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.