Crate page_turner

A generic abstraction of paginated APIs

Imagine, you need to use the following API to find the most upvoted comment under a blog post.

struct GetCommentsRequest {
    blog_post_id: BlogPostId,
    page_number: u32,
}

struct GetCommentsResponse {
    comments: Vec<Comment>,
    more_comments_available: bool,
}

In order to do that you will need to write a hairy loop that checks the more_comments_available flag, increments page_number, and updates a variable that stores the resulting value. This crate helps to abstract away any paginated API and allows you to work with such APIs uniformly with the help of async streams. Both cursor and non-cursor pagniations are supported. All you need to do is to implement the PageTurner trait for the client that sends GetCommentsRequest.

In PageTurner you specify what data you fetch and what errors may occur for a particular request, then you implement the turn_page method where you describe how to query a single page and how to prepare a request for the next page.

use page_turner::prelude::*;

impl PageTurner<GetCommentsRequest> for BlogClient {
    type PageItems = Vec<Comment>;
    type PageError = BlogClientError;

    async fn turn_page(&self, mut request: GetCommentsRequest) -> TurnedPageResult<Self, GetCommentsRequest> {
        let response = self.get_comments(request.clone()).await?;

        if response.more_comments_available {
            request.page_number += 1;
            Ok(TurnedPage::next(response.comments, request))
        } else {
            Ok(TurnedPage::last(response.comments))
        }
    }
}

PageTurner then provides default implementations for PageTurner::pages and PageTurner::into_pages methods that you can use to get a stream of pages and, optionally, to turn it into a stream of page items if you need. Now we can use our client to find the most upvoted comment like that:

let client = BlogClient::new();

let most_upvoted_comment = client
    .pages(GetCommentsRequest { blog_post_id, page_number: 1 })
    .items()
    .try_fold(None::<Comment>, |most_upvoted, next_comment| async move {
        match most_upvoted {
            Some(comment) if next_comment.upvotes > comment.upvotes => Ok(Some(next_comment)),
            current @ Some(_) => Ok(current),
            None => Ok(Some(next_comment)),
        }
    })
    .await?
    .unwrap();

assert_eq!(most_upvoted_comment.text, "Yeet");
assert_eq!(most_upvoted_comment.upvotes, 5);

// Or we can process the whole pages if needed

let mut comment_pages = std::pin::pin!(client.pages(GetCommentsRequest { blog_post_id, page_number: 1 }));

while let Some(comment_page) = comment_pages.try_next().await? {
    detect_spam(comment_page);
}

Notice, that with this kind of the API we don’t require any data from response to construct the next valid request. We can take an advantage on such APIs by implementing the RequestAhead trait on a request type. For requests that implement RequestAhead PageTurner provides additional methods - PageTurner::pages_ahead and PageTurner::pages_ahead_unordered. These methods allow to query multiple pages concurrently using an optimal sliding window request scheduling.

impl RequestAhead for GetCommentsRequest {
    fn next_request(&self) -> Self {
        Self {
            blog_post_id: self.blog_post_id,
            page_number: self.page_number + 1,
        }
    }
}

let client = BlogClient::new();

// Now instead of querying pages one by one we make 4 concurrent requests
// for multiple pages under the hood but besides using a different PageTurner
// method nothing changes in the user code.
let most_upvoted_comment = client
    .pages_ahead(4, Limit::None, GetCommentsRequest { blog_post_id, page_number: 1 })
    .items()
    .try_fold(None::<Comment>, |most_upvoted, next_comment| async move {
        match most_upvoted {
            Some(comment) if next_comment.upvotes > comment.upvotes => Ok(Some(next_comment)),
            current @ Some(_) => Ok(current),
            None => Ok(Some(next_comment)),
        }
    })
    .await?
    .unwrap();

assert_eq!(most_upvoted_comment.text, "Yeet");
assert_eq!(most_upvoted_comment.upvotes, 5);

// In the example above the order of pages being returned corresponds to the order
// of requests which means the stream is blocked until the first page is ready
// even if the second and the third pages are already received. For this use case
// we don't really care about the order of the comments so we can use
// pages_ahead_unordered to unblock the stream as soon as we receive a response to
// any of the concurrent requests.
let most_upvoted_comment = client
    .pages_ahead_unordered(4, Limit::None, GetCommentsRequest { blog_post_id, page_number: 1 })
    .items()
    .try_fold(None::<Comment>, |most_upvoted, next_comment| async move {
        match most_upvoted {
            Some(comment) if next_comment.upvotes > comment.upvotes => Ok(Some(next_comment)),
            current @ Some(_) => Ok(current),
            None => Ok(Some(next_comment)),
        }
    })
    .await?
    .unwrap();

assert_eq!(most_upvoted_comment.text, "Yeet");
assert_eq!(most_upvoted_comment.upvotes, 5);

Page turner flavors and crate features

There are multiple falvors of page turners suitable for different contexts and each flavor is available behind a feature flag. By default the mt feature is enabled which provides you crate::mt::PageTurner that works with multithreaded executors. The crate root prelude just reexports the crate::mt::prelude::*, and types at the crate root are reexported from crate::mt. Overall there are the following page turner flavors and corresponding features available:

• local: Provides a less constrained crate::local::PageTurner suitable for singlethreaded executors. Use page_turner::local::prelude::* to work with it.
• mutable: A crate::mutable::PageTurner is like local but even allows your client to mutate during the request execution. Use page_turner::mutable::prelude::* to work with it.
• mt: A crate::mt::PageTurner for multithreaded executors. It’s reexported by default but if you enable additional features in the same project it is recommended to use page_turner::mt::prelude::* to distinguish between different flavors of page turners.
• dynamic: An object safe crate::dynamic::PageTurner that requires async_trait to be implemented and can be used as an object with dynamic dispatch.

Re-exports

• pub use local::mutable;
  mutable
• pub use mt::dynamic;
  dynamic
• pub use crate::mt::*;

Modules

• locallocal
  A page turner suitable for singlethreaded executors. See mutable for a version that allows to use &mut self in methods
• mt
  A page turner suitable for multithreaded executors. This is what you need in most cases. See dynamic if you also need dyn PageTurner objects for some reason.
• prelude
  Minimal reexports you need to work with a page turner.

Structs

• TurnedPage
  A struct that combines items queried for the current page and an optional request to query the next page. If next_request is None PageTurner stops querying pages.

Enums

• Limit
  If you use pages_ahead or pages_ahead_unordered families of methods and you know in advance how many pages you need to query, specify Limit::Pages to prevent redundant querying past the last existing page from being executed.

Traits

• RequestAhead
  If a request for the next page doesn’t require any data from the response and can be made out of the request for the current page implement this trait to enable pages_ahead, pages_ahead_unordered families of methods that query pages concurrently.