# solana-shardmap Documentation
## Table of Contents
1. [Core Concepts](#core-concepts)
2. [API Reference](#api-reference)
3. [Architecture Patterns](#architecture-patterns)
4. [Performance Guide](#performance-guide)
5. [Common Patterns](#common-patterns)
6. [Error Handling](#error-handling)
7. [Best Practices](#best-practices)
## Core Concepts
### What is Sharding?
Sharding is a database partitioning technique where data is split across multiple storage units. In Solana's context, this means distributing your key-value pairs across multiple accounts to overcome the ~10KB account size limit.
### Why Shard on Solana?
```rust
// ❌ This won't scale beyond ~10KB
#[account]
pub struct LargeMap {
pub data: Vec<(Pubkey, UserData)>, // Limited by account size
}
// ✅ This scales horizontally across multiple accounts
#[account]
pub struct ShardedMap {
pub shard: MappingShard<Pubkey, UserData>, // Each shard ~10KB
}
```
### Shard Distribution Strategy
```rust
// Hash-based distribution ensures even spread
fn get_shard_for_user(user: &Pubkey, total_shards: u8) -> u8 {
let hash = hash_pubkey(user);
(hash % total_shards as u64) as u8
}
```
## API Reference
### Core Types
#### `MappingShard<K, V>`
The main shard structure that stores key-value pairs with bounded capacity.
```rust
pub struct MappingShard<K: ShardKey, V: ShardValue> {
pub shard_id: u8, // Identifier for this shard
pub items: Vec<(K, V)>, // The actual key-value pairs
pub item_count: u16, // Cached count for efficiency
pub max_items: u16, // Maximum capacity
}
```
#### `ShardedMap<K, V>` Trait
Core operations that every shard must implement:
```rust
pub trait ShardedMap<K, V> {
// Single operations
fn insert(&mut self, key: K, value: V) -> Result<()>;
fn get(&self, key: &K) -> Option<V>;
fn remove(&mut self, key: &K) -> Result<()>;
fn len(&self) -> usize;
fn is_empty(&self) -> bool;
fn max_capacity(&self) -> usize;
// Batch operations
fn insert_batch(&mut self, items: Vec<(K, V)>) -> Result<Vec<Result<()>>>;
fn get_batch(&self, keys: &[K]) -> Vec<Option<V>>;
fn remove_batch(&mut self, keys: &[K]) -> Result<Vec<Result<()>>>;
}
```
### Capacity Management Methods
#### Basic Capacity Info
```rust
let shard = MappingShard::<Pubkey, u64>::new(0, 100);
// Basic capacity information
assert_eq!(shard.remaining_capacity(), 100);
assert_eq!(shard.utilization_percentage(), 0.0);
assert_eq!(shard.load_factor(), 0.0);
assert!(shard.is_empty());
assert!(!shard.is_full());
```
#### Advanced Capacity Management
```rust
// Check if near capacity threshold
if shard.is_near_capacity(80.0) {
println!("Shard is 80% full - consider scaling");
}
// Estimate space for new items
let new_keys = [key1, key2, key3];
let available_space = shard.space_for_new_items(&new_keys);
// Resize capacity (useful for account resizing)
shard.resize_capacity(200)?;
// Memory optimization
shard.shrink_to_fit(); // Reduce memory usage
shard.reserve(50); // Pre-allocate for performance
shard.clear(); // Remove all items but keep capacity
```
#### Capacity Statistics
```rust
let stats = shard.capacity_stats();
println!("Shard Health Report:");
println!(" Items: {}/{}", stats.current_items, stats.max_capacity);
println!(" Utilization: {:.1}%", stats.utilization_percentage);
println!(" Load Factor: {:.2}", stats.load_factor);
println!(" Vec Capacity: {}", stats.vec_capacity);
println!(" Status: {}", if stats.is_full { "FULL" } else { "OK" });
```
### Batch Operations
#### Batch Insert
```rust
// Prepare batch data
let batch_items = vec![
(user1, profile1),
(user2, profile2),
(user3, profile3),
];
// Method 1: Insert with detailed results
let results = shard.insert_batch(batch_items.clone())?;
for (i, result) in results.iter().enumerate() {
match result {
Ok(_) => println!("Item {} inserted successfully", i),
Err(e) => println!("Item {} failed: {}", i, e),
}
}
// Method 2: All-or-nothing approach
if shard.can_insert_batch(&batch_items) {
let count = shard.try_insert_batch(batch_items)?;
println!("Inserted {} items", count);
} else {
println!("Batch would exceed capacity");
}
```
#### Batch Retrieval
```rust
let keys = [user1, user2, user3, nonexistent_user];
let values = shard.get_batch(&keys);
// Process results
for (key, value) in keys.iter().zip(values.iter()) {
match value {
Some(profile) => println!("Found profile for {}: {:?}", key, profile),
None => println!("No profile found for {}", key),
}
}
```
#### Batch Removal
```rust
let users_to_remove = [inactive_user1, inactive_user2];
let results = shard.remove_batch(&users_to_remove)?;
let removed_count = results.iter().filter(|r| r.is_ok()).count();
println!("Removed {} inactive users", removed_count);
```
## Architecture Patterns
### Pattern 1: Simple Fixed Sharding
```rust
// Configuration
const TOTAL_SHARDS: u8 = 10;
const ITEMS_PER_SHARD: u16 = 100;
// Shard selection
fn get_shard_index(user: &Pubkey) -> u8 {
let hash = hash_pubkey(user);
(hash % TOTAL_SHARDS as u64) as u8
}
// PDA derivation
fn get_shard_pda(program_id: &Pubkey, shard_index: u8) -> (Pubkey, u8) {
derive_shard_pda(program_id, shard_index)
}
```
### Pattern 2: Dynamic Shard Management
```rust
pub struct ShardManager {
pub total_shards: u8,
pub items_per_shard: u16,
pub program_id: Pubkey,
}
impl ShardManager {
pub fn new(program_id: Pubkey) -> Self {
Self {
total_shards: 1,
items_per_shard: 100,
program_id,
}
}
pub fn should_add_shard(&self, current_loads: &[usize]) -> bool {
let max_load = current_loads.iter().max().unwrap_or(&0);
*max_load >= (self.items_per_shard as usize * 80 / 100) // 80% threshold
}
pub fn add_shard(&mut self) -> Result<()> {
if self.total_shards == 255 {
return Err(/* max shards reached */);
}
self.total_shards += 1;
Ok(())
}
}
```
### Pattern 3: Multi-Level Sharding
```rust
// For very large datasets, use hierarchical sharding
fn get_hierarchical_shard(user: &Pubkey) -> (u8, u8) {
let hash = hash_pubkey(user);
let level1 = (hash % 10) as u8; // 10 top-level shards
let level2 = ((hash / 10) % 10) as u8; // 10 sub-shards each
(level1, level2)
}
fn derive_hierarchical_pda(program_id: &Pubkey, l1: u8, l2: u8) -> (Pubkey, u8) {
Pubkey::find_program_address(
&[b"shard", &[l1], &[l2]],
program_id
)
}
```
## Performance Guide
### Benchmarking Results
| Insert | 50μs | 800μs | 37.5x faster |
| Retrieval | 30μs | 400μs | 75x faster |
| Removal | 40μs | 600μs | 66.7x faster |
### Optimization Strategies
#### 1. Batch Everything
```rust
// ❌ Inefficient: Multiple transactions
for user in users {
program.methods.updateProfile(user.profile)
.accounts({...})
.rpc();
}
// ✅ Efficient: Single batch transaction
program.methods.batchUpdateProfiles(user_profiles)
.accounts({...})
.rpc();
```
#### 2. Pre-allocate Memory
```rust
// Before large operations
shard.reserve(expected_items);
// Process operations
// ...
// Cleanup after
shard.shrink_to_fit();
```
#### 3. Monitor and Rebalance
```rust
async fn monitor_shard_health(shards: Vec<ShardAccount>) {
for shard in shards {
if shard.utilization_percentage() > 90.0 {
// Trigger rebalancing or new shard creation
create_additional_shard().await?;
}
}
}
```
## Common Patterns
### User Profile Management
```rust
#[derive(AnchorSerialize, AnchorDeserialize, Clone)]
pub struct UserProfile {
pub username: String,
pub level: u32,
pub xp: u64,
pub last_login: i64,
pub achievements: Vec<u8>,
}
impl UserProfile {
pub fn new(username: String) -> Self {
Self {
username,
level: 1,
xp: 0,
last_login: Clock::get().unwrap().unix_timestamp,
achievements: vec![],
}
}
pub fn add_xp(&mut self, amount: u64) {
self.xp += amount;
self.level = (self.xp / 1000) as u32 + 1;
}
}
```
### Leaderboard System
```rust
#[derive(AnchorSerialize, AnchorDeserialize, Clone, PartialEq, Eq)]
pub struct LeaderboardEntry {
pub player: Pubkey,
pub score: u64,
pub timestamp: i64,
}
impl PartialOrd for LeaderboardEntry {
fn partial_cmp(&self, other: &Self) -> Option<std::cmp::Ordering> {
Some(self.cmp(other))
}
}
impl Ord for LeaderboardEntry {
fn cmp(&self, other: &Self) -> std::cmp::Ordering {
self.score.cmp(&other.score).reverse() // Higher scores first
}
}
```
### Inventory System
```rust
#[derive(AnchorSerialize, AnchorDeserialize, Clone)]
pub struct InventoryItem {
pub item_id: u32,
pub quantity: u32,
pub metadata: Vec<u8>,
}
#[derive(AnchorSerialize, AnchorDeserialize, Clone)]
pub struct PlayerInventory {
pub items: Vec<InventoryItem>,
pub capacity: u32,
}
impl PlayerInventory {
pub fn add_item(&mut self, item_id: u32, quantity: u32) -> Result<()> {
if let Some(existing) = self.items.iter_mut().find(|i| i.item_id == item_id) {
existing.quantity += quantity;
} else {
self.items.push(InventoryItem {
item_id,
quantity,
metadata: vec![],
});
}
Ok(())
}
}
```
## Error Handling
### Built-in Errors
```rust
#[error_code]
pub enum ShardError {
#[msg("Shard is full.")]
ShardFull,
#[msg("Key not found.")]
KeyNotFound,
#[msg("Invalid shard index or PDA mismatch.")]
InvalidShard,
#[msg("Invalid capacity: new capacity cannot be smaller than current item count")]
InvalidCapacity,
}
```
### Error Handling Patterns
```rust
// Pattern 1: Graceful degradation
match shard.insert(key, value) {
Ok(_) => msg!("Profile updated successfully"),
Err(ShardError::ShardFull) => {
msg!("Shard full - creating new shard");
// Trigger new shard creation
}
Err(e) => return Err(e.into()),
}
// Pattern 2: Batch error analysis
let results = shard.insert_batch(items)?;
let (successes, failures): (Vec<_>, Vec<_>) = results
.into_iter()
.enumerate()
.partition(|(_, result)| result.is_ok());
msg!("Batch insert: {} successes, {} failures",
successes.len(), failures.len());
```
## Best Practices
### 1. Shard Sizing
```rust
// Optimal shard sizes for different use cases
const USER_PROFILES_PER_SHARD: u16 = 100; // ~8KB per shard
const GAME_SCORES_PER_SHARD: u16 = 200; // ~6KB per shard
const INVENTORY_ITEMS_PER_SHARD: u16 = 50; // ~9KB per shard
```
### 2. Monitoring and Alerting
```rust
pub fn check_shard_health(shard: &MappingShard<K, V>) -> HealthStatus {
let utilization = shard.utilization_percentage();
match utilization {
x if x >= 95.0 => HealthStatus::Critical,
x if x >= 85.0 => HealthStatus::Warning,
x if x >= 70.0 => HealthStatus::Caution,
_ => HealthStatus::Healthy,
}
}
```
### 3. Account Space Estimation
```rust
use solana_shardmap::estimate_shard_account_size;
// Calculate required space accurately
let key_size = 32; // Pubkey
let value_size = std::mem::size_of::<UserProfile>();
let max_items = 100;
let required_space = estimate_shard_account_size(key_size, value_size, max_items);
println!("Required account space: {} bytes", required_space);
```
### 4. Migration Strategies
```rust
// Plan for data migration when resharding
pub async fn migrate_shard_data(
old_shard: &MappingShard<K, V>,
new_shards: &mut [MappingShard<K, V>],
shard_selector: impl Fn(&K) -> usize,
) -> Result<()> {
for (key, value) in old_shard.items.iter() {
let target_shard = shard_selector(key);
new_shards[target_shard].insert(key.clone(), value.clone())?;
}
Ok(())
}
```
### 5. Testing Strategies
```rust
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_shard_capacity_limits() {
let mut shard = MappingShard::<u8, u8>::new(0, 2);
// Fill to capacity
assert!(shard.insert(1, 10).is_ok());
assert!(shard.insert(2, 20).is_ok());
// Verify capacity enforcement
assert!(shard.insert(3, 30).is_err());
assert_eq!(shard.len(), 2);
assert!(shard.is_full());
}
#[test]
fn test_batch_operations_performance() {
let mut shard = MappingShard::<u32, String>::new(0, 1000);
// Prepare large batch
let items: Vec<(u32, String)> = (0..500)
.map(|i| (i, format!("value_{}", i)))
.collect();
// Batch insert should succeed
let results = shard.insert_batch(items).unwrap();
assert!(results.iter().all(|r| r.is_ok()));
assert_eq!(shard.len(), 500);
}
}
```
---
For more examples and patterns, check the [test suite](src/tests.rs) and [examples](examples/) directory.