CW-Storage-Plus: Enhanced/experimental storage engines for CosmWasm
The ideas in here are based on the cosmwasm-storage
crate. However,
after much usage, we decided a complete rewrite could allow us to add
more powerful and easy to use interfaces. Here are those interfaces.
Status: beta
This has been heavily used in many production-quality contracts and heavily refined. There is one planned API break (dealing with auto-deserializing keys in range queries), but the code has demonstrated itself to be stable and powerful. Please feel free to use in your contracts.
Usage Overview
We introduce two main classes to provide a productive abstraction
on top of cosmwasm_std::Storage
. They are Item
, which is
a typed wrapper around one database key, providing some helper functions
for interacting with it without dealing with raw bytes. And Map
,
which allows you to store multiple unique typed objects under a prefix,
indexed by a simple (&[u8]
) or compound (eg. (&[u8], &[u8])
) primary key.
These correspond to the concepts represented in cosmwasm_storage
by
Singleton
and Bucket
, but with a re-designed API and implementation
to require less typing for developers and less gas usage in the contracts.
Item
The usage of an Item
is pretty straight-forward.
You must simply provide the proper type, as well as a database key not
used by any other item. Then it will provide you with a nice interface
to interact with such data.
If you are coming from using Singleton
, the biggest change is that
we no longer store Storage
inside, meaning we don't need read and write
variants of the object, just one type. Furthermore, we use const fn
to create the Item
, allowing it to be defined as a global compile-time
constant rather than a function that must be constructed each time,
which saves gas as well as typing.
Example Usage:
// note const constructor rather than 2 functions with Singleton
const CONFIG: = new;
Map
The usage of an Map
is a little more complex, but
is still pretty straight-forward. You can imagine it as a storage-backed
BTreeMap
, allowing key-value lookups with typed values. In addition,
we support not only simple binary keys (&[u8]
), but tuples, which are
combined. This allows us to store allowances as composite keys
eg. (owner, spender)
to look up the balance.
Beyond direct lookups, we have a super power not found in Ethereum -
iteration. That's right, you can list all items in a Map
, or only
part of them. We can efficiently allow pagination over these items as
well, starting at the point the last query ended, with low gas costs.
This requires the iterator
feature to be enabled in cw-storage-plus
(which automatically enables it in cosmwasm-std
as well, and which is
enabled by default).
If you are coming from using Bucket
, the biggest change is that
we no longer store Storage
inside, meaning we don't need read and write
variants of the object, just one type. Furthermore, we use const fn
to create the Bucket
, allowing it to be defined as a global compile-time
constant rather than a function that must be constructed each time,
which saves gas as well as typing. In addition, the composite indexes
(tuples) is more ergonomic and expressive of intention, and the range
interface has been improved.
Here is an example with normal (simple) keys:
const PEOPLE: = new;
Key types
A Map
key can be anything that implements the PrimaryKey
trait. There are a series of implementations of
PrimaryKey
already provided (see packages/storage-plus/src/keys.rs
):
impl<'a> PrimaryKey<'a> for &'a [u8]
impl<'a> PrimaryKey<'a> for &'a str
impl<'a> PrimaryKey<'a> for Vec<u8>
impl<'a> PrimaryKey<'a> for String
impl<'a> PrimaryKey<'a> for Addr
impl<'a> PrimaryKey<'a> for &'a Addr
impl<'a, T: PrimaryKey<'a> + Prefixer<'a>, U: PrimaryKey<'a>> PrimaryKey<'a> for (T, U)
impl<'a, T: PrimaryKey<'a> + Prefixer<'a>, U: PrimaryKey<'a> + Prefixer<'a>, V: PrimaryKey<'a>> PrimaryKey<'a> for (T, U, V)
impl<'a, T: Endian + Clone> PrimaryKey<'a> for IntKey<T>
That means that byte and string slices, byte vectors, and strings, can be conveniently used as keys. Moreover, some other types can be used as well, like addresses and addresses references, pairs and triples, and integer types.
If the key represents and address, we suggest using &Addr
for keys in storage, instead of String
or string slices. This implies doing address validation
through addr_validate
on any address passed in via a message, to ensure it's a legitimate address, and not random text
which will fail later.
Thus, pub fn addr_validate(&self, &str) -> Addr
in deps.api
can be used for address validation, and the returned
Addr
can be conveniently used as key in a Map
or similar structure.
Composite Keys
There are times when we want to use multiple items as a key, for example, when storing allowances based on account owner and spender. We could try to manually concatenate them before calling, but that can lead to overlap, and is a bit low-level for us. Also, by explicitly separating the keys, we can easily provide helpers to do range queries over a prefix, such as "show me all allowances for one owner" (first part of the composite key). Just like you'd expect from your favorite database.
Here how we use it with composite keys. Just define a tuple as a key and use that everywhere you used a byte slice above.
// Note the tuple for primary key. We support one slice, or a 2 or 3-tuple
// adding longer tuples is quite easy but unlikely to be needed.
const ALLOWANCE: = new;
Path
Under the scenes, we create a Path
from the Map
when accessing a key.
PEOPLE.load(&store, b"jack") == PEOPLE.key(b"jack").load()
.
Map.key()
returns a Path
, which has the same interface as Item
,
reusing the calculated path to this key.
For simple keys, this is just a bit less typing and a bit less gas if you
use the same key for many calls. However, for composite keys, like
(b"owner", b"spender")
it is much less typing. And highly recommended anywhere
you will use the a composite key even twice:
const PEOPLE: = new;
const ALLOWANCE: = new;
Prefix
In addition to getting one particular item out of a map, we can iterate over the map
(or a subset of the map). This let us answer questions like "show me all tokens",
and we provide some nice Bound
s helpers to easily allow pagination or custom ranges.
The general format is to get a Prefix
by calling map.prefix(k)
, where k
is exactly
one less item than the normal key (If map.key()
took (&[u8], &[u8])
, then map.prefix()
takes &[u8]
.
If map.key()
took &[u8]
, map.prefix()
takes ()
). Once we have a prefix space, we can iterate
over all items with range(store, min, max, order)
. It supports Order::Ascending
or Order::Descending
.
min
is the lower bound and max
is the higher bound.
If the min
and max
bounds, it will return all items under this prefix. You can use .take(n)
to
limit the results to n
items and start doing pagination. You can also set the min
bound to
eg. Bound::Exclusive(last_value)
to start iterating over all items after the last value. Combined with
take
, we easily have pagination support. You can also use Bound::Inclusive(x)
when you want to include any
perfect matches. To better understand the API, please read the following example:
const PEOPLE: = new;
const ALLOWANCE: = new;
IndexedMap
In cw-plus, there's currently one example of IndexedMap
usage, in the cw721-base
contract.
Let's use it to illustrate IndexedMap
definition and usage.
Definition
Let's discuss this piece by piece:
These are the index definitions. Here there's only one index, called owner
. There could be more, as public
members of the TokenIndexes
struct.
We see that the owner
index is a MultiIndex
. A multi-index can have repeated values as keys. That's why
the primary key is being added as the last element of the multi-index key.
Like the name implies, this is an index over tokens, by owner. Given that an owner can have multiple tokens,
we need a MultiIndex
to be able to list / iterate over all the tokens a given owner has.
So, to recap, the TokenInfo
data will originally be stored by token_id
(which is a string value).
You can see this in the token creation code:
tokens.update?;
(Incidentally, this is using update
instead of save
, to avoid overwriting an already existing token).
Then, it will be indexed by token owner
(which is an Addr
), so that we can list all the tokens an owner has.
That's why the owner
index key is (Addr, Vec<u8>)
. The first owned element is the owner
data
, whereas the second one is the token_id
(converted internally to Vec<u8>
).
The important thing here is that the key (and its components, in the case of a combined key) must implement
the PrimaryKey
trait. You can see that the 2-tuple (_, _)
, Addr
, and Vec<u8>
do implement PrimaryKey
:
We can now see how it all works, taking a look at the remaining code:
This implements the IndexList
trait for TokenIndexes
.
Note: this code is more or less boiler-plate, and needed for the internals. Do not try to customize this;
just return a list of all indexes.
Implementing this trait serves two purposes (which are really one, and the same): it allows the indexes
to be queried through get_indexes
, and, it allows TokenIndexes
to be treated as an IndexList
. So that
it can be passed as a parameter during IndexedMap
construction, below:
Here tokens()
is just a helper function, that simplifies the IndexedMap
construction for us. First the
index (es) is (are) created, and then, the IndexedMap
is created (using IndexedMap::new
), and returned.
During index creation, we must supply an index function per index
owner: new
, which is the one that will take the value, and the primary key (which is always in Vec<u8>
form) of the
original map, and create the index key from them. Of course, this requires that the elements required
for the index key are present in the value (which makes sense).
Besides the index function, we must also supply the namespace of the pk, and the one for the new index.
After that, we just create and return the IndexedMap
:
new
Here of course, the namespace of the pk must match the one used during index(es) creation. And, we pass our
TokenIndexes
(as an IndexList
-type parameter) as second argument. Connecting in this way the underlying Map
for the pk, with the defined indexes.
So, IndexedMap
(and the other Indexed*
types) is just a wrapper / extension around Map
, that provides
a number of index functions and namespaces to create indexes over the original Map
data. It also implements
calling these index functions during value storage / update / removal, so that you can forget about it,
and just use the indexed data.
Usage
An example of use, where owner
is a String
value passed as a parameter, and start_after
and limit
optionally
define the pagination range:
Notice this uses prefix()
, explained above in the Map
section.
let limit = limit.unwrap_or.min as usize;
let start = start_after.map;
let owner_addr = deps.api.addr_validate?;
let res: = tokens
.idx
.owner
.prefix
.range
.take
.collect;
let tokens = res?;
Now tokens
contains (token_id, TokenInfo)
pairs for the given owner
.
The pk values are Vec<u8>
, as this is a limitation of the current implementation.
Another example that is similar, but returning only the token_id
s, using the keys()
method:
let pks: = tokens
.idx
.owner
.prefix
.keys
.take
.collect;
Now pks
contains token_id
values (as Vec<u8>
s) for the given owner
.