QMX compression improved (smaller and faster to decode) More...
#include <compress_integer_qmx_improved.h>
Public Member Functions | |
| compress_integer_qmx_improved () | |
| Constructor. | |
| virtual | ~compress_integer_qmx_improved () |
| Destructor. | |
| virtual size_t | encode (void *encoded, size_t encoded_buffer_length, const integer *source, size_t source_integers) |
| Encode a sequence of integers returning the number of bytes used for the encoding, or 0 if the encoded sequence doesn't fit in the buffer. More... | |
| virtual void | decode (integer *decoded, size_t integers_to_decode, const void *source, size_t source_length) |
| Decode a sequence of integers encoded with this codex. More... | |
 Public Member Functions inherited from JASS::compress_integer | |
| compress_integer () | |
| Constructor. | |
| virtual | ~compress_integer () |
| Destructor. | |
Static Public Member Functions | |
| static void | unittest_one (const std::vector< uint32_t > &sequence) |
| Test one sequence to make sure it encodes and decodes to the same thing. Assert if not. More... | |
| static void | unittest (void) |
| Unit test this class. | |
| Static Public Member Functions inherited from JASS::compress_integer | |
| static size_t | d1_encode (integer *encoded, const integer *source, size_t source_integers) |
| Convert an array of integers into an array of D1 (delta, d-gap) encoded integers. More... | |
| static size_t | d1_decode (integer *decoded, const integer *source, size_t source_integers) |
| Convert a D1 encoded array of integers into an array of integers. More... | |
| static size_t | dn_encode (integer *encoded, const integer *source, size_t source_integers, size_t n=1) |
| Convert an array of integers into an array of Dn (delta, d-gap) encoded integers with a gap of n. More... | |
| static size_t | dn_decode (integer *decoded, const integer *source, size_t source_integers, size_t n=1) |
| Convert a Dn encoded array of integers into an array of integers. More... | |
| static void | unittest_one (compress_integer &encoder, const std::vector< uint32_t > &sequence) |
| Test one sequence to make sure it encodes and decodes to the same thing. Assert if not. More... | |
| static void | unittest (compress_integer &compressor, uint32_t staring_from=0) |
| Unit test this class, assert on failure. More... | |
Private Member Functions | |
| void | write_out (uint8_t **buffer, uint32_t *source, uint32_t raw_count, uint32_t size_in_bits, uint8_t **length_buffer) |
| Encode and write out the sequence into the buffer. More... | |
Private Attributes | |
| uint8_t * | length_buffer |
| Stores the number of bits needed to compress each integer. | |
| uint64_t | length_buffer_length |
| The length of length_buffer. | |
| uint32_t * | full_length_buffer |
| If the run_length is too short then 0-pad into this buffer. | |
Additional Inherited Members | |
| Public Types inherited from JASS::compress_integer | |
| typedef uint32_t | integer |
| This class and descendants will work on integers of this size. Do not change without also changing JASS_COMPRESS_INTEGER_BITS_PER_INTEGER. | |
Detailed Description
QMX compression improved (smaller and faster to decode)
Trotman & Lin describe several improvements to the QMX codex in:
A. Trotman, J. Lin (2016), In Vacuo and In Situ Evaluation of SIMD Codecs, Proceedings of The 21st Australasian Document Computing Symposium (ADCS 2016
including removal of the vbyte encoded length from the end of the encoded sequence. This version of QMX is the original QMX with that improvement added, but none of the other imprivements suggested by Trotman & Lin. This makes the encoded sequence smaller, and faster to decode, than any of the other alrernatives suggested. It does not include the code to prevent read and write overruns from the encoded string and into the decode buffer. To account for overwrites make sure the decode-into buffer is at least 256 integers larger than required. To prevent over-reads from the encoded string make sure that that string is at least 16 bytes longer than needed.
At the request of Matthias Petri (University of Melbourne), the code no longer requires SIMD-word alignment to decode (the read and write instructions have been changed from aligned to unaligned since Intel made them faster).
For details on the original QMX encoding see:
A. Trotman (2014), Compression, SIMD, and Postings Lists, Proceedings of the 19th Australasian Document Computing Symposium (ADCS 2014)
Member Function Documentation
◆ decode()
|
virtual |
Decode a sequence of integers encoded with this codex.
- Parameters
-
decoded [out] The sequence of decoded integers. integers_to_decode [in] The minimum number of integers to decode (it may decode more). source [in] The encoded integers. source_length [in] The length (in bytes) of the source buffer.
Implements JASS::compress_integer.
◆ encode()
|
virtual |
Encode a sequence of integers returning the number of bytes used for the encoding, or 0 if the encoded sequence doesn't fit in the buffer.
- Parameters
-
encoded [out] The sequence of bytes that is the encoded sequence. encoded_buffer_length [in] The length (in bytes) of the output buffer, encoded. source [in] The sequence of integers to encode. source_integers [in] The length (in integers) of the source buffer.
- Returns
- The number of bytes used to encode the integer sequence, or 0 on error (i.e. overflow).
Implements JASS::compress_integer.
◆ unittest_one()
|
static |
Test one sequence to make sure it encodes and decodes to the same thing. Assert if not.
- Parameters
-
sequence [in] the sequernce to encode.
◆ write_out()
|
private |
Encode and write out the sequence into the buffer.
- Parameters
-
buffer [in] where to write the encoded sequence source [in] the integer sequence to encode raw_count [in] the numnber of integers to encode size_in_bits [in] the size, in bits, of the largest integer length_buffer [in] the length of buffer, in bytes
The documentation for this class was generated from the following files:
- source/compress_integer_qmx_improved.h
- source/compress_integer_qmx_improved.cpp
