Any Data object has necessary a Type.
The least specific Type is ZL_Type_serial,
which means it’s just a blob of bytes.
Codecs can only accept and produce specified data Types.
In contrast, Selectors & Graphs may optionally accept multiple data Types,
using bitmap masking (example: ZL_Type_struct | ZL_Type_numeric).
@brief Pass @p compressor as a ZL_Compressor* object to the compression
state. Compression will start with the default Starting GraphID of @p
compressor, using its default parameters provided at registration time.
@note Only one compressor can be referenced at a time.
Referencing a new compressor deletes previous reference.
@note If a custom GraphID and parameters were previously set,
invoking this method will reset them to default.
@pre @p compressor must remain valid for the duration of its usage.
@pre @p compressor must be already validated.
@brief Set the starting Graph of next compression,
as @p graphID referenced in the provided @p compressor,
optionally providing it with some runtime parameters.
@pre @p compressor must remain valid for the duration of its usage.
@pre @p compressor must be already validated.
Sets the Arena for Data* objects in the CCtx.
This frees the previous Data Arena and creates a new one.
This choice remains sticky, until set again.
The default Data Arena is HeapArena.
Read the serialized compressor from @p serialized and find all of the nodes
and graphs that the serialized compressor refers to but that aren’t present
in the serialized compressor. I.e., lists all the components that must be
present on a destination compressor in order for this serialized compressor
to correctly materialize onto that destination compressor.
Equivalent @ref ZL_CompressorSerializer_serialize, but produces a human-
readable output for debugging. This output format cannot currently be
consumed by OpenZL.
@returns If the provided @p graphid was created by modifying another
existing graph, i.e., its type is ZL_GraphType_parameterized,
the ZL_GraphID of that other graph. Otherwise, it returns
ZL_GRAPH_ILLEGAL.
@returns The custom graphs of @p graphid, which is only non-empty for
selector, dynamic, and multi-input graphs.
@note The array is valid for the lifetime of the @p compressor.
@returns The custom nodes of @p graphid, which is only non-empty for
dynamic and multi-input graphs.
@note The array is valid for the lifetime of the @p compressor.
@returns The successor of the head node in the graph if @p graphid is a
static graph, otherwise returns an empty list.
@note The array is valid for the lifetime of the @p compressor.
Inserts a placeholder for the Automated Compressor Explorer (ACE) to
replace with an automatically generated graph. It accepts a single input of
any type.
Build a new graph out of pre-existing components. The new graph passes its
data to @p headNode, and then each output of @p headNode is set to the
corresponding @p successorGraph.
@brief Simplified variant of @ref ZL_Compressor_registerParameterizedNode().
Clone an existing @ref ZL_NodeID from an already registered
@p nodeid but employs new parameters, set via @p localParams.
Calls @p callback on every graph registered in the @p compressor.
If @p callback returns an error, short-circuit and return that error.
@returns Success if all callbacks succeed, or the first error.
@note The callback will not be called on standard graphs, since they
aren’t registered in the @p compressor.
Calls @p callback on every node registered in the @p compressor.
If @p callback returns an error, short-circuit and return that error.
@returns Success if all callbacks succeed, or the first error.
@note The callback will not be called on standard nodes, since they
aren’t registered in the @p compressor.
Calls @p callback on every param set in the @p compressor.
If @p callback returns an error, short-circuit and return that error.
@returns Success if all callbacks succeed, or the first error.
@note The callback will not be called on params that are equal to
the default value (0).
@returns A verbose error string containing context about the error that
occurred. This is useful for debugging, and for submitting bug reports to
OpenZL developers.
@note This string is stored within the @p compressor and is only valid for
the lifetime of the @p compressor.
@param[out] graphID If the function returns true, filled with
the starting GraphID, otherwise filled with ZL_GRAPH_ILLEGAL.
@returns True iff the compressor has the starting GraphID set.
@returns The array of warnings that were encountered during the creation
of the compressor.
@note The array’s and the errors’ lifetimes are valid until the next non-
const call on the compressor.
@brief Initialize a @p compressor object with a ZL_GraphFn Graph function
@p f. It will register a few custom graphs and custom nodes, and set the
starting Graph ID. This is a convenience function, which is equivalent to
calling the Graph function, then ZL_Compressor_selectStartingGraphID()
followed by ZL_Compressor_validate()
@returns Success or an error which can be checked with ZL_isError().
@param f The function used to build the ZL_GraphID.
Parameterized brute force selector that selects the best successor from a
user-provided list of candidates.
@param successors the list of successors to select from. Each successor must
be equipped to handle the input stream type.
Tokenize with a custom tokenization function. This is useful if you want to
define a custom order for your alphabet that is neither insertion nor sorted
order.
@param nbOutputs - the number of output streams to be generated. Passed as a
local param to the transform.
@param dispatchIndices - the array of indices to be used for dispatching.
Will be passed as a local param to the transform. The lifetime of the array
is to be managed by the caller and should outlive the transform execution.
Creates a graph for ZL_NODE_MERGE_SORTED that first detects whether
the input has <= 64 sorted runs. If it does it selects the node.
Otherwise it selects the backupGraph.
@brief Create a new GraphID by the one from @p gid,
just replacing the @p localParams by the provided ones. Used to create
custom variants of Standard Graphs for example.
Register a custom pipe encoder, to be employed in upcoming Graph.
This registration is specialized for simple pipe transforms.
Counterpart: ZL_DCtx_registerPipeDecoder().
This action creates a NodeID, provided as @return value.
Graphs using custom encoders are only decodable if,
at decompression time, a custom decoder of same CTid is registered.
Split-by-struct
This operation splits a serialized input
defined as an array of structures of fixed size,
by grouping same fields into their own stream.
All fields are considered concatenated back-to-back (no alignment).
For this transform to work, input must be an exact multiple of struct_size,
with struct_size = sum(field_sizes).
Each output stream is then assigned a successor Graph.
Split-by-param
This operation splits a serialized input
into segments, defined by array @segmentSizes[].
The nb of segments and their size is static,
except for the last segment size, which can receive a size value 0,
meaning “whatever is left in the stream”.
Each segment is then into its own output,
and then sent to the next processing stage defined by @successors[].
@brief Set global parameters via @p compressor. In this construction, global
parameters are attached to a Compressor object. Global Parameters set at
Compressor level can be overridden later at CCtx level.
@brief Decompresses typed content from frames with a single typed output
into a pre-allocated buffer @p dst .
Information about output type is written into @p outputInfo .
Register a custom decoder transform, should it be needed during
decompression of a zstrong frame.
A decoder transform is expected to reverse the encoding transform of same
@CTid.
This function is specialized for the registration of simple pipe decoder
transforms. Counterpart to ZL_Compressor_registerPipeEncoder().
Register a custom split decoder transform.
This is supposed to be the reverse of the encoding transform of same @CTid.
Counterpart to ZS2_registerSplitTransform().
@return element width in nb of bytes
This is only valid for fixed size elements,
such as ZL_Type_struct or ZL_Type_numeric.
If Type is ZL_Type_string, it returns 0 instead.
@note invoking ZL_Data_numElts() is only valid for committed Data.
If the Data object was received as an input, it’s necessarily valid.
So the issue can only happen for outputs,
between allocation and commit.
Querying ZL_Data_numElts() is not expected to be useful for output Data.
@note ZL_Type_serial doesn’t really have a concept of “elt”.
In this case, it returns Data size in bytes.
These methods provide direct access to internal buffer.
Warning : users must pay attention to buffer boundaries.
@return pointer to the beginning of buffer.
@note for ZL_Type_string, returns a pointer to the buffer containing the
concatenated strings.
This method is only valid for ZL_Type_string Data.
@return a pointer to the array of string lengths.
The size of this array is == ZL_Data_numElts(data).
@return NULL if incorrect data type, or StringLens not allocated yet.
This method is only valid for ZL_Type_string Data.
It reserves memory space for StringLens array, and returns a pointer to it.
The buffer is owned by @p data and has the same lifetime.
The returned pointer can be used to write into the array.
After writing into the array, the nb of String Lengths provided must be
signaled using @ref ZL_Data_commit().
This method will fail if StringLens is already allocated.
@return NULL if incorrect data type, or allocation error.
This method is only valid for ZL_Type_string Data.
It requires write access into StringLens array.
Only valid if StringLens array has already been allocated
and not yet written into.
@return NULL when any of the above conditions is violated.
@returns The opaque pointer provided in the transform description.
@warning Usage of the opaque pointer must be thread-safe, and must
not not modify the state in any way that impacts decoding!
Provides an opaque pointer that can be useful to provide state to the parser.
For example, it can be used by language bindings to allow parsers written in
languages other than C.
Attempts to add more information to the error represented by @p error.
Narrowly, this means trying to append a stack frame to the stacktrace that
rich errors accumulate. In service of that, it also tries to up-convert the
error to a rich error if it isn’t already. @p fmt and optional additional
following args can also be used to append an arbitrary formatted string of
information into the error.
Actual implementation function which accepts all of the explicit arguments
that are set up for you by the macros elsewhere. Prefer to use those macros
rather than this function directly.
@returns The opaque pointer provided in the transform description.
@warning Usage of the opaque pointer must be thread-safe, and must
not not modify the state in any way that impacts encoding!
@returns Returns a state, as generated by the Transform’s State Manager.
The state’s lifetime is managed by the host CCtx, it will be free
automatically at end of CCtx lifetime (and can’t be forcefully free
manually). The state may be cached from a previous run with a compatible
transform (same state signature).
Registration might fail if the Descriptor is incorrectly filled,
Any further operation attempted with such a Graph will also fail.
Such an outcome can be tested with ZL_GraphID_isValid().
Note: this is mostly for debugging purposes,
once a Descriptor is valid, registration can be assumed to be successful.
@returns a NodeID that implements transpose split for the given @p eltWidth
that will work with any Zstrong format version. If no node exists, then
returns ZL_NODE_ILLEGAL. This can happen for format version <= 10 when
@p eltWidth != 2,4,8.
Determines whether @nodeid is supported given the applied global parameters
for the compression.
Notably the ZL_CParam_formatVersion parameter can determine if a node is
valid for the given encoding version.
Registration might fail, for example if the Descriptor is incorrectly filled.
In which case, the returned nodeid is ZL_NODE_ILLEGAL.
Any further operation attempted with such a Node will also fail.
Such an outcome can be tested with ZL_NodeID_isValid().
Note: this is mostly for debugging,
once a Descriptor is valid, registration can be assumed to remain successful.
@returns The capacity of the buffer reserved for @p output in bytes. If
@p output has not been reserved, it returns an error. For string types, this
is the sum of the lengths of each string that can be written into the buffer.
@returns The content size in bytes that has been committed to @p output.
For non-string types, this is the eltWidth * numElts. For string types, this
is the sum of the lengths of each stream. If @p output has not been commited,
it returns an error.
@returns The element width of the @p output if it has a buffer reserved,
otherwise it returns NULL. Within a custom codec, this function always
succeeds, because the output always has a buffer reserved. If the type
of the output is String, it returns 0 instead.
@returns The capacity of the buffer reserved for @p output in number of
elements. If @p output has not been reserved, it returns an error.
For string types, this is the number of strings that can be written into the
buffer.
@returns The number of elements committed in @p output if it has been
committed. Otherwise, returns an error if the write to @p output has not been
committed.
@returns The dctx that will be used for decompression so the user
can set parameters and register custom transforms if the helpers
provided aren’t sufficient.
@returns The total number of streams in the decoder graph. This includes
streams stored in the frame, input streams to the frame, and intermediate
streams.
@pre ZL_ReflectionCtx_setCompressedFrame() returned success.
Utility function to get the input types supported by @gid as an input mask.
Throws an error if @gid is not valid or does not have exactly one input.
See ZL_Compressor_Graph_getInput0Mask() for more details.
Provides an opaque pointer that can be useful to provide state to the parser.
For example, it can be used by language bindings to allow parsers written in
languages other than C.
Creates a ZL_TypedRef that references an array of numeric values,
employing the local host’s endianness.
Supported widths are 1, 2, 4, and 8 and the input array must be properly
aligned (in local ABI).
Creates a ZL_TypedRef referencing a “flat-strings” representation. All
“strings” are concatenated into @p strBuffer and their lengths are stored in
a @p strLens array.
@brief Gets the size of the compressed frame.
This method could be useful when the compressed frame only represents
a first portion of a larger buffer.
A custom tokenization function to tokenize the input. The output of this
function is not checked in production builds, and it is UB to tokenize
incorrectly.
Internally, there are two kinds of error info objects: dynamic and static,
which are, like they sound, respectively dynamically allocated (and which
must be freed) or statically allocated, but which therefore can’t contain
any runtime information.
@brief While it’s possible to add elements (graphs, selectors or nodes) to a
Compressor one by one, and then finalize it by selecting a starting graph ID,
it’s generally common for all these steps to be regrouped into a single
initialization function.
The following signature corresponds such a function.
It returns a GraphID which, by convention, is the starting GraphID.
Internally, there are two kinds of error info objects: dynamic and static,
which are, like they sound, respectively dynamically allocated (and which
must be freed) or statically allocated, but which therefore can’t contain
any runtime information.