Enum Modality

pub enum Modality<'a> {
    Text {
        data: Cow<'a, str>,
    },
    Image {
        data: Cow<'a, str>,
    },
}

The prompt for models can be a combination of different modalities (Text and Image). The type of modalities which are supported depend on the Model in question.

Variants

Text

The only type of prompt which can be used with pure language models

Fields

data: Cow<'a, str>

Image

An image input into the model. See Modality::from_image_path.

Fields

data: Cow<'a, str>

Implementations

impl<'a> Modality<'a>

pub fn from_text(text: impl Into<Cow<'a, str>>) -> Self

Instantiates a text prompt

pub fn from_image_path(path: impl AsRef<Path>) -> Result<Self, LoadImageError>

Image input for model, from file path.

The model can only see squared pictures. Images are centercropped.

use aleph_alpha_client::{Client, How, Modality, Prompt, Sampling, Stopping, TaskCompletion,
    Task, Logprobs};
use dotenvy::dotenv;
use std::path::PathBuf;

#[tokio::main(flavor = "current_thread")]
async fn main() {
    // Create client
    let client = Client::from_env().unwrap();
    // Define task
    let task = TaskCompletion {
        prompt: Prompt::from_vec(vec![
            Modality::from_image_path("cat.png").unwrap(),
            Modality::from_text("A picture of "),
        ]),
        stopping: Stopping::from_maximum_tokens(10),
        sampling: Sampling::MOST_LIKELY,
        special_tokens: false,
        logprobs: Logprobs::No,
        echo: false,
    };
    // Execute
    let model = "luminous-base";
    let job = task.with_model(model);
    let response = client.output_of(&job, &How::default()).await.unwrap();
    // Show result
    println!("{}", response.completion);
}

pub fn from_image(image: &DynamicImage) -> Result<Self, LoadImageError>

Image input for model

The model can only see squared pictures. Images are centercropped. You may want to use this method instead of Self::from_image_path in case you have the image in memory already and do not want to load it from a file again.

pub fn borrow(&self) -> Modality<'_>

Create a semantically idetical entry of modality which borrows the contents of this one.

It is very practical to allow Modality of e.g. Text to take both ownership of the string it contains as well as borrow a slice. However then we are creating a body from the user input we want to avoid copying everything and needing to allocate for that modality again. This is there this borrow function really shines.

Trait Implementations

impl<'a> Clone for Modality<'a>

fn clone(&self) -> Modality<'a>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'a> Debug for Modality<'a>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'a> PartialEq for Modality<'a>

fn eq(&self, other: &Modality<'a>) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl<'a> Serialize for Modality<'a>

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl<'a> Eq for Modality<'a>

impl<'a> StructuralPartialEq for Modality<'a>