[−][src]Module xnde::fields

fields

Introduction

NDE (Nullsoft Database Engine) field types. I've only implemented the fields needed to read my muisc library. Contact me if you're using this & have fields in your library that aren't handled here.

Discussion

As mentioned in the xnde docs, an NDE table consists of a collection of records serialized very compactly. An index is a list of offsets into the data (the .dat file on disk) at each each row begins, in the order corresponding to the index. E.g. in my database, index 255 is the primary index and just gives the record offsets in sequential order, while index 0 lists the exact same set of records, but in lexicographical order of the "filename" field.

In my database, the first record contains the columns for the table, and the second the indicies.

Record Formats

Each record consists of a sequence of fields. Each field has the form:

    +----+------+----------+------+------+------+
    | ID | type | max_size | prev | next | data |
    +----+------+----------+------+------+------+

ID is a 8-bit unsigned int; this appears to be unique to each column, and serves to map a field to a column in other records. Types as an 8-bit unsigned int describing the field type. In the NDE implementation, these map to a set of contants:

#define FIELD_UNKNOWN   255
enum
{
    FIELD_COLUMN     =  0,
    FIELD_INDEX      =  1,
    FIELD_REDIRECTOR =  2,
    FIELD_STRING     =  3,
    FIELD_INTEGER    =  4,
    FIELD_BOOLEAN    =  5,
    FIELD_BINARY     =  6, // max size 65536
    FIELD_GUID       =  7,
    FIELD_PRIVATE    =  8,
    FIELD_BITMAP     =  6,
    FIELD_FLOAT      =  9,
    FIELD_DATETIME   = 10,
    FIELD_LENGTH     = 11,
    FIELD_FILENAME   = 12,
    FIELD_INT64      = 13,
    FIELD_BINARY32   = 14, // binary field, but 32bit sizes instead of 16bit
    FIELD_INT128     = 15, // mainly for storing MD5 hashes
};

In this crate, they are represented by the enum FieldType.

max_size is a 32-bit little-endian unsigned int containing the size of the data field (i.e. the field-specific blob after the common header). If fields were guaranteed to be sequential within a record, this quantity could be computed from next, so I can only assume that is not the case generally, even though it was for my database.

Also, note that this is the serialized size on disk; I have seen fields where the field- specific data took up less than this. I suspect, but haven't verified, that this allows the write implementation to update a field whose new serialized representation happens to be smaller by just writing it in place and not updating the rest of the file to "pack" it more tightly.

prev is a 32-bit little-endian unsigned int containing the offset of the previous field in this record, and next is the same giving the offset of the next. Not all columns need appear in each record, and the record length is nowhere written down; the reader must simply read one field after another until encountering one whose next field is zero. As an aside, this strongly suggests a buffered read implementation, which the NDE uses.

Field Formats

The following diagrams display field layouts after the common field header.

Column

    +----------+------------------+--------------+--------------------+
    | type: u8 | unique index: i8 | name len: u8 | name: ASCII string |
    +----------+------------------+--------------+--------------------+

The column name is a length-prefixed ASCII string (i.e. no trailing nil).

Filename & String

    +----+------+
    | cb | text |
    +----+------+

cb is a sixteen-bit, little-endian unsigned integer containing the number of bytes in the filename or string. The text may be UTF-16 encoded; in that case we expect a BOM. Else the reference implementation simply copies the bytes; this implementation assumes UTF-8. Note that the string is not null-terminated.

Index

    +-----+------+----+------+
    | pos | type | cb | name |
    +-----+------+----+------+

pos is a 32-bit, little endian signed int. I never figured out what it does. type' is also 32-bit, LE, signed int, and refers to the type of field found in this column. cbis a 32-bit, LE unsigned int describing the length of thename` field, which is the ASCII text of the filter name.

Int64

    +-----+
    | val |
    +-----+

A 64-bit, little-endian, signed integer.

Datetime, Integer, Length

    +-------+
    | value |
    +-------+

value is a 32-bit little-endian integer. I still haven't unravelled how to interpret it in all cases, but it is a signed integer (i.e. not a simple Unix-style "seconds-since-epoch" value for time, or seconds for length).

Structs

ColumnField	NDE FIELD_COLUMN
DatetimeField	NDE FIELD_DATETIME
Error
FilenameField	NDE FIELD_FILENAME
IndexField	NDE FIELD_INDEX
Int64Field	NDE FIELD_INT64
IntegerField	NDE FIELD_INTEGER
LengthField	NDE FIELD_LENGTH
NdeFieldBase	Common NDE Field attributes: id, next-field, prev-field
StringField	NDE FIELD_STRING
UnsupportedNdeField	An NDE Field which we do not know how to parse

Enums

Cause
FieldType	NDE field types, maintaining the associated C numeric constants
FieldValue