osrscache/
lib.rs

1//! A read-only, high-level, virtual file API for the RuneScape cache.
2//!
3//! This crate provides high performant data reads into the [Oldschool RuneScape] and [RuneScape 3] cache file systems.
4//! It can read the necessary data to synchronize the client's cache with the server. There are also some
5//! [loaders](#loaders) that give access to definitions from the cache such as items or npcs.
6//!
7//! For read-heavy workloads, a writer can be used to prevent continuous buffer allocations.
8//! By default every read will allocate a writer with the correct capacity.
9//!
10//! RuneScape's chat system uses huffman coding to compress messages. In order to decompress them this library has
11//! a [`Huffman`] implementation.
12//!
13//! When a RuneScape client sends game packets the id's are encoded and can be decoded with the [`IsaacRand`]
14//! implementation. These id's are encoded by the client in a predictable random order which can be reversed if
15//! the server has its own `IsaacRand` with the same encoder/decoder keys. These keys are sent by the client
16//! on login and are user specific. It will only send encoded packet id's if the packets are game packets.
17//!
18//! Note that this crate is still evolving; both OSRS & RS3 are not fully supported/implemented and
19//! will probably contain bugs or miss core features. If you require features or find bugs consider [opening
20//! an issue].
21//!
22//! # Safety
23//!
24//! In order to read bytes in a high performant way the cache uses [memmap2]. This can be unsafe because of its potential for
25//! _Undefined Behaviour_ when the underlying file is subsequently modified, in or out of process.
26//! Using `Mmap` here is safe because the RuneScape cache is a read-only binary file system. The map will remain valid even
27//! after the `File` is dropped, it's completely independent of the `File` used to create it. Therefore, the use of unsafe is
28//! not propagated outwards. When the `Cache` is dropped memory will be subsequently unmapped.
29//!
30//! # Features
31//!
32//! The cache's protocol defaults to OSRS. In order to use the RS3 protocol you can enable the `rs3` feature flag.
33//! A lot of types derive [serde]'s `Serialize` and `Deserialize`. The `serde-derive` feature flag can be used to
34//! enable (de)serialization on any compatible types.
35//!
36//! # Quick Start
37//!
38//! ```
39//! use osrscache::Cache;
40//!
41//! # fn main() -> Result<(), osrscache::Error> {
42//! let cache = Cache::new("./data/osrs_cache")?;
43//!
44//! let index_id = 2; // Config index.
45//! let archive_id = 10; // Archive containing item definitions.
46//!
47//! let buffer = cache.read(index_id, archive_id)?;
48//! # Ok(())
49//! # }
50//! ```
51//!
52//! # Loaders
53//!
54//! In order to get [definitions](crate::definition) you can look at the [loaders](crate::loader) this library provides.
55//! The loaders use the cache as a dependency to parse in their data and cache the relevant definitions internally.
56//! The loader module also tells you how to make a loader if this crate doesn't (yet) provide it.
57//!
58//! Note: Some loaders cache these definitions lazily because of either the size of the data or the
59//! performance. The map loader for example is both slow and large so caching is by default lazy.
60//! Lazy loaders require mutability.
61//!
62//! [Oldschool RuneScape]: https://oldschool.runescape.com/
63//! [RuneScape 3]: https://www.runescape.com/
64//! [opening an issue]: https://github.com/jimvdl/rs-cache/issues/new
65//! [serde]: https://crates.io/crates/serde
66//! [memmap2]: https://crates.io/crates/memmap2
67//! [`Huffman`]: crate::util::Huffman
68//! [`IsaacRand`]: crate::util::IsaacRand
69#![cfg_attr(docsrs, feature(doc_cfg))]
70#![deny(
71    clippy::all,
72    clippy::correctness,
73    clippy::suspicious,
74    clippy::style,
75    clippy::complexity,
76    clippy::perf
77)]
78
79#[macro_use]
80pub mod util;
81pub mod checksum;
82pub mod definition;
83pub mod error;
84pub mod extension;
85pub mod loader;
86
87#[doc(inline)]
88pub use error::Error;
89use error::Result;
90
91use checksum::Checksum;
92#[cfg(feature = "rs3")]
93use checksum::{RsaChecksum, RsaKeys};
94use runefs::codec::{Buffer, Decoded, Encoded};
95use runefs::error::{Error as RuneFsError, ReadError};
96use runefs::{ArchiveRef, Dat2, Indices, MAIN_DATA};
97use std::{io::Write, path::Path};
98
99/// A complete virtual representation of the RuneScape cache file system.
100#[derive(Debug)]
101pub struct Cache {
102    data: Dat2,
103    pub(crate) indices: Indices,
104}
105
106impl Cache {
107    /// Creates a high level virtual memory map over the cache directory.
108    /// 
109    /// All files are isolated on allocation by keeping them as in-memory files.
110    ///
111    /// # Errors
112    ///
113    /// The bulk of the errors which might occur are mostely I/O related due to acquiring 
114    /// file handles.
115    /// 
116    /// Other errors might include protocol changes in newer caches.
117    /// Any error unrelated to I/O at this stage should be considered a bug.
118    pub fn new<P: AsRef<Path>>(path: P) -> crate::Result<Self> {
119        Ok(Self {
120            data: Dat2::new(path.as_ref().join(MAIN_DATA))?,
121            indices: Indices::new(path)?,
122        })
123    }
124
125    /// Generate a checksum based on the current cache.
126    /// 
127    /// The `Checksum` acts as a validator for individual cache files.
128    /// Any RuneScape client will request a list of crc's to check the validity
129    /// of all of the file data that was transferred.
130    pub fn checksum(&self) -> crate::Result<Checksum> {
131        Checksum::new(self)
132    }
133
134    /// Generate a checksum based on the current cache with RSA encryption.
135    /// 
136    /// `RsaChecksum` wraps a regular `Checksum` with the added benefit of encrypting
137    /// the whirlpool hash into the checksum buffer.
138    #[cfg(feature = "rs3")]
139    #[cfg_attr(docsrs, doc(cfg(feature = "rs3")))]
140    pub fn checksum_with<'a>(&self, keys: RsaKeys<'a>) -> crate::Result<RsaChecksum<'a>> {
141        RsaChecksum::with_keys(self, keys)
142    }
143
144    /// Retrieves and constructs data corresponding to the given index and archive.
145    ///
146    /// # Errors
147    ///
148    /// When trying to retrieve data from an index or an archive that does not exist 
149    /// the `IndexNotFound` or `ArchiveNotFound` errors are returned, respectively.
150    /// 
151    /// Any other errors such as sector validation failures or failed parsers should
152    /// be considered a bug.
153    pub fn read(&self, index_id: u8, archive_id: u32) -> crate::Result<Buffer<Encoded>> {
154        let index = self
155            .indices
156            .get(&index_id)
157            .ok_or(RuneFsError::Read(ReadError::IndexNotFound(index_id)))?;
158
159        let archive = index
160            .archive_refs
161            .get(&archive_id)
162            .ok_or(RuneFsError::Read(ReadError::ArchiveNotFound {
163                idx: index_id,
164                arc: archive_id,
165            }))?;
166
167        let buffer = self.data.read(archive)?;
168
169        assert_eq!(buffer.len(), archive.length);
170
171        Ok(buffer)
172    }
173
174    pub(crate) fn read_archive(&self, archive: &ArchiveRef) -> crate::Result<Buffer<Encoded>> {
175        self.read(archive.index_id, archive.id)
176    }
177
178    /// Retrieves and writes data corresponding to the given index and archive into `W`.
179    ///
180    /// # Errors
181    ///
182    /// See the error section on [`read`](Cache::read) for more details.
183    pub fn read_into_writer<W: Write>(
184        &self,
185        index_id: u8,
186        archive_id: u32,
187        writer: &mut W,
188    ) -> crate::Result<()> {
189        let index = self
190            .indices
191            .get(&index_id)
192            .ok_or(RuneFsError::Read(ReadError::IndexNotFound(index_id)))?;
193
194        let archive = index
195            .archive_refs
196            .get(&archive_id)
197            .ok_or(RuneFsError::Read(ReadError::ArchiveNotFound {
198                idx: index_id,
199                arc: archive_id,
200            }))?;
201        Ok(self.data.read_into_writer(archive, writer)?)
202    }
203
204    /// Retrieves the huffman table.
205    ///
206    /// Required when decompressing chat messages, see [`Huffman`](crate::util::Huffman).
207    pub fn huffman_table(&self) -> crate::Result<Buffer<Decoded>> {
208        let index_id = 10;
209
210        let archive = self.archive_by_name(index_id, "huffman")?;
211        let buffer = self.read_archive(archive)?;
212
213        assert_eq!(buffer.len(), archive.length);
214
215        Ok(buffer.decode()?)
216    }
217
218    pub(crate) fn archive_by_name<T: AsRef<str>>(
219        &self,
220        index_id: u8,
221        name: T,
222    ) -> crate::Result<&ArchiveRef> {
223        let index = self
224            .indices
225            .get(&index_id)
226            .ok_or(RuneFsError::Read(ReadError::IndexNotFound(index_id)))?;
227        let hash = util::djd2::hash(&name);
228
229        let archive = index
230            .metadata
231            .iter()
232            .find(|archive| archive.name_hash == hash)
233            .ok_or_else(|| crate::error::NameHashMismatch {
234                hash,
235                name: name.as_ref().into(),
236                idx: index_id,
237            })?;
238
239        let archive_ref = index
240            .archive_refs
241            .get(&archive.id)
242            .ok_or(RuneFsError::Read(ReadError::ArchiveNotFound {
243                idx: index_id,
244                arc: archive.id,
245            }))?;
246
247        Ok(archive_ref)
248    }
249}
250
251#[cfg(test)]
252mod test_util {
253    use super::Cache;
254    use sha1::Sha1;
255
256    fn is_normal<T: Send + Sync + Sized + Unpin>() {}
257
258    #[test]
259    fn normal_types() {
260        is_normal::<super::Cache>();
261    }
262
263    pub fn osrs_cache() -> crate::Result<Cache> {
264        Cache::new("./data/osrs_cache")
265    }
266
267    #[cfg(all(test, feature = "rs3"))]
268    pub fn rs3_cache() -> crate::Result<Cache> {
269        Cache::new("./data/rs3_cache")
270    }
271
272    pub fn hash(buffer: &[u8]) -> String {
273        let mut m = Sha1::new();
274
275        m.update(buffer);
276        m.digest().to_string()
277    }
278}
279
280#[cfg(test)]
281mod read {
282    mod osrs {
283        use crate::test_util;
284
285        #[test]
286        fn ref_table() -> crate::Result<()> {
287            let cache = test_util::osrs_cache()?;
288            let buffer = cache.read(255, 10)?;
289            let hash = test_util::hash(&buffer);
290            assert_eq!(&hash, "9a9a50a0bd25d6246ad5fe3ebc5b14e9a5df7227");
291            assert_eq!(buffer.len(), 77);
292            Ok(())
293        }
294        #[test]
295        fn random_read() -> crate::Result<()> {
296            let cache = test_util::osrs_cache()?;
297            let buffer = cache.read(0, 191)?;
298            let hash = test_util::hash(&buffer);
299            assert_eq!(&hash, "cd459f6ccfbd81c1e3bfadf899624f2519e207a9");
300            assert_eq!(buffer.len(), 2055);
301            Ok(())
302        }
303        #[test]
304        fn large_read() -> crate::Result<()> {
305            let cache = test_util::osrs_cache()?;
306            let buffer = cache.read(2, 10)?;
307            let hash = test_util::hash(&buffer);
308            assert_eq!(&hash, "6d3396a9d5a7729b3b8069ac32e384e5da58096b");
309            assert_eq!(buffer.len(), 282396);
310            Ok(())
311        }
312        #[test]
313        fn deep_archive() -> crate::Result<()> {
314            let cache = test_util::osrs_cache()?;
315            let buffer = cache.read(7, 24918)?;
316            let hash = test_util::hash(&buffer);
317            assert_eq!(&hash, "fe91e9e9170a5a05ed2684c1db1169aa7ef4906e");
318            assert_eq!(buffer.len(), 803);
319            Ok(())
320        }
321
322        #[test]
323        fn single_data_len() -> crate::Result<()> {
324            let cache = test_util::osrs_cache()?;
325            let buffer = cache.read(3, 278)?;
326
327            let hash = test_util::hash(&buffer);
328            assert_eq!(&hash, "036abb64d3f1734d892f69b1253a87639b7bcb44");
329            assert_eq!(buffer.len(), 512);
330            Ok(())
331        }
332
333        #[test]
334        fn double_data_len() -> crate::Result<()> {
335            let cache = test_util::osrs_cache()?;
336            let buffer = cache.read(0, 1077)?;
337
338            let hash = test_util::hash(&buffer);
339            assert_eq!(&hash, "fbe9d365cf0c3efa94e0d4a2c5e607b28a1279b9");
340            assert_eq!(buffer.len(), 1024);
341            Ok(())
342        }
343
344        #[test]
345        fn fails() -> crate::Result<()> {
346            let cache = test_util::osrs_cache()?;
347            assert!(cache.read(2, 25_000).is_err());
348            Ok(())
349        }
350    }
351
352    #[cfg(feature = "rs3")]
353    mod rs3 {
354        use crate::test_util;
355
356        #[test]
357        fn random_0_read() -> crate::Result<()> {
358            let cache = test_util::rs3_cache()?;
359            let buffer = cache.read(0, 25)?;
360            let hash = test_util::hash(&buffer);
361            assert_eq!(&hash, "81e455fc58fe5ac98fee4df5b78600bbf43e83f7");
362            assert_eq!(buffer.len(), 1576);
363            Ok(())
364        }
365
366        #[test]
367        fn between_single_double() -> crate::Result<()> {
368            let cache = test_util::rs3_cache()?;
369            let buffer = cache.read(7, 0)?;
370
371            let hash = test_util::hash(&buffer);
372            assert_eq!(&hash, "b33919c6e4677abc6ec1c0bdd9557f820a163559");
373            assert_eq!(buffer.len(), 529);
374            Ok(())
375        }
376
377        #[test]
378        fn fails() -> crate::Result<()> {
379            let cache = test_util::rs3_cache()?;
380            assert!(cache.read(2, 25_000).is_err());
381            Ok(())
382        }
383    }
384}
385
386#[cfg(test)]
387mod osrs {
388    use super::test_util;
389    use super::Cache;
390
391    #[test]
392    fn new() {
393        assert!(Cache::new("./data/osrs_cache").is_ok());
394    }
395
396    #[test]
397    fn new_wrong_path() {
398        assert!(Cache::new("./wrong/path").is_err());
399    }
400
401    #[test]
402    fn huffman_table() -> crate::Result<()> {
403        let cache = test_util::osrs_cache()?;
404        let buffer = cache.huffman_table()?;
405
406        let hash = test_util::hash(&buffer);
407        assert_eq!(&hash, "664e89cf25a0af7da138dd0f3904ca79cd1fe767");
408        assert_eq!(buffer.len(), 256);
409
410        Ok(())
411    }
412}
413
414#[cfg(all(test, feature = "rs3"))]
415mod rs3 {
416    use super::test_util;
417    use super::Cache;
418
419    #[test]
420    fn new() {
421        assert!(Cache::new("./data/rs3_cache").is_ok());
422    }
423
424    #[test]
425    fn new_wrong_path() {
426        assert!(Cache::new("./wrong/path").is_err());
427    }
428
429    #[test]
430    fn huffman_table() -> crate::Result<()> {
431        let cache = test_util::rs3_cache()?;
432        let buffer = cache.huffman_table()?;
433
434        let hash = test_util::hash(&buffer);
435        assert_eq!(&hash, "664e89cf25a0af7da138dd0f3904ca79cd1fe767");
436        assert_eq!(buffer.len(), 256);
437
438        Ok(())
439    }
440}
osrscache/lib.rs

osrscache/
lib.rs
