Property store

Work in progress

TODO: TypedValue vs PropertyValue vs PropertyStore

Idea behind property store

Properties (specifically PropertyValues) get copied very often - we can compress them to optimize memory usage and performance.

Implementation

The values themselves are stored encoded in a flat buffer. On an insertion the underlying storage buffer is resized if necessary and on removal the underlying storage buffer is shrinked if the new data can fit into a buffer that is 1/3 smaller than the current buffer. If it can't fit into a smaller buffer, the current buffer is used. All mappings are encoded independently of each other. TODO: elaborate more

Metadata

Each mapping starts with an encoded metadata field that is used for several purposes:

to determine the encoded type
to determine the encoded property ID size
to determine the encoded payload size

The metadata field is always a single byte and its bits are used as follows:

bits 0-3 →type (4 bits)
bits 4-5 →size of property ID (2 bits)
bits 6-7 →size of payload OR size of payload size indicator (2 bits) When encoding integers (int64_t and uint32_t) they are compressed so that they are stored into 1, 2, 4 or 8 bytes depending on their value

The size of the metadata field is very important because it is encoded with each and every ID to value mapping. That is why every possible bit is used to store some useful information. Increasing the size of the metadata field will increase memory usage for every stored ID to value mapping. Bits 0-5 are always used in the same way but bits 6-7 can be used for various memory optimizations.

Property store/compression

This section displays how each data type is stored, first row indicates header

Null
- Bits 6-7 not used
Bool
- Bits 6-7 used for encoded value
- Encoded property ID
Int
- Bits 6-7 are used to indicate whether the value is encoded as int8_t , int16_t , int32_t or int64_t
- Encoded property id
- Encoded property value
Double
- Bits 6-7 not used
- Encoded property id
- Encoded property value