Involved Source Filesbitreader.gobitwriter.gocompress.godecompress.go This file contains the specialisation of Decoder.Decompress4X
and Decoder.Decompress1X that use an asm implementation of thir main loops. Package huff0 provides fast huffman encoding as used in zstd.
See README.md at https://github.com/klauspost/compress/tree/master/huff0 for details.decompress_amd64.s
Package-Level Type Names (total 13, in which 3 are exported)
/* sort exporteds by: | */
Decoder provides stateless decoding.actualTableLoguint8bufs*sync.PooldtdTable Decompress1X will decompress a 1X encoded stream.
The cap of the output buffer will be the maximum decompressed size.
The length of the supplied input must match the end of a block exactly. Decompress4X will decompress a 4X encoded stream.
The length of the supplied input must match the end of a block exactly.
The *capacity* of the dst slice must match the destination size of
the uncompressed data exactly.(*Decoder) buffer() *[4][256]byte decompress1X8Bit will decompress a 1X encoded stream with tablelog <= 8.
The cap of the output buffer will be the maximum decompressed size.
The length of the supplied input must match the end of a block exactly. decompress1X8Bit will decompress a 1X encoded stream with tablelog <= 8.
The cap of the output buffer will be the maximum decompressed size.
The length of the supplied input must match the end of a block exactly. Decompress4X will decompress a 4X encoded stream.
The length of the supplied input must match the end of a block exactly.
The *capacity* of the dst slice must match the destination size of
the uncompressed data exactly. Decompress4X will decompress a 4X encoded stream.
The length of the supplied input must match the end of a block exactly.
The *capacity* of the dst slice must match the destination size of
the uncompressed data exactly.
func (*Scratch).Decoder() *Decoder
MaxDecodedSize will set the maximum allowed output size.
This value will automatically be set to BlockSizeMax if not set.
Decoders will return ErrMaxDecodedSizeExceeded is this limit is exceeded. MaxSymbolValue will override the maximum symbol value of the next block. Out is output buffer.
If the scratch is re-used before the caller is done processing the output,
set this field to nil.
Otherwise the output buffer will be re-used for next Compression/Decompression step
and allocation will be avoided. OutData will contain the compressed data.
Slice of the returned data. OutTable will contain the table data only, if a new table has been generated.
Slice of the returned data. Reuse will specify the reuse policy TableLog will attempt to override the tablelog for the next block.
Must be <= 11 and >= 5. WantLogLess allows to specify a log 2 reduction that should at least be achieved,
otherwise the block will be returned as incompressible.
The reduction should then at least be (input size >> WantLogLess)
If WantLogLess == 0 any improvement will do. // Selected tablelog. // compression table // clear countcount[256]uint32 // *[4][256]byte buffers. // decompression tablefse*fse.ScratchhuffWeight[256]byte // count of the most probable symbolnodes[]nodeElt // Table used for previous compression. // Tablelog for previous tablesrcLenint // Length of active part of the symbol table.tmpOut[4][]byte Decoder will return a stateless decoder that can be used by multiple
decompressors concurrently.
Before this is called, the table must be initialized with ReadTable.
The Decoder is still linked to the scratch buffer so that cannot be reused.
However, it is safe to discard the scratch. Decompress1X will decompress a 1X encoded stream.
The length of the supplied input must match the end of a block exactly.
Before this is called, the table must be initialized with ReadTable unless
the encoder re-used the table.
deprecated: Use the stateless Decoder() to get a concurrent version. Decompress4X will decompress a 4X encoded stream.
Before this is called, the table must be initialized with ReadTable unless
the encoder re-used the table.
The length of the supplied input must match the end of a block exactly.
The destination size of the uncompressed data must be known and provided.
deprecated: Use the stateless Decoder() to get a concurrent version. TransferCTable will transfer the previously used compression table.(*Scratch) buildCTable() error(*Scratch) canUseTable(c cTable) bool(*Scratch) compress1X(src []byte) ([]byte, error)(*Scratch) compress1xDo(dst, src []byte) []byte(*Scratch) compress4X(src []byte) ([]byte, error) compress4Xp will compress 4 streams using separate goroutines. countSimple will create a simple histogram in s.count.
Returns the biggest count.
Does not update s.clearCount. huffSort will sort symbols, decreasing order. matches will compare a decoding table to a coding table.
Errors are written to the writer.
Nothing will be written if table is ok. minSize returns the minimum possible size considering the shannon limit. minTableLog provides the minimum logSize to safely represent a distribution. optimalTableLog calculates and sets the optimal tableLog in s.actualTableLog(*Scratch) prepare(in []byte) (*Scratch, error)(*Scratch) setMaxHeight(lastNonNull int) uint8(*Scratch) validateTable(c cTable) bool
func ReadTable(in []byte, s *Scratch) (s2 *Scratch, remain []byte, err error)
func (*Scratch).prepare(in []byte) (*Scratch, error)
func Compress1X(in []byte, s *Scratch) (out []byte, reUsed bool, err error)
func Compress4X(in []byte, s *Scratch) (out []byte, reUsed bool, err error)
func EstimateSizes(in []byte, s *Scratch) (tableSz, dataSz, reuseSz int, err error)
func ReadTable(in []byte, s *Scratch) (s2 *Scratch, remain []byte, err error)
func (*Scratch).TransferCTable(src *Scratch)
func compress(in []byte, s *Scratch, compressor func(src []byte) ([]byte, error)) (out []byte, reUsed bool, err error)
bitReader reads a bitstream in reverse.
The last set bit indicates the start of the stream and is used
for aligning the input.bitsReaduint8in[]byte // next byte to read is at in[off - 1]valueuint64(*bitReaderBytes) advance(n uint8) close the bitstream and returns an error if out-of-buffer reads occurred. fill() will make sure at least 32 bits are available. fillFast() will make sure at least 32 bits are available.
There must be at least 4 bytes available. fillFastStart() assumes the bitReaderBytes is empty and there is at least 8 bytes to read. finished returns true if all bits have been read from the bit stream. init initializes and resets the bit reader. peekByteFast requires that at least one byte is requested every time.
There are no checks if the buffer is filled.(*bitReaderBytes) remaining() uint
bitReaderShifted reads a bitstream in reverse.
The last set bit indicates the start of the stream and is used
for aligning the input.bitsReaduint8in[]byte // next byte to read is at in[off - 1]valueuint64(*bitReaderShifted) advance(n uint8) close the bitstream and returns an error if out-of-buffer reads occurred. fill() will make sure at least 32 bits are available. fillFast() will make sure at least 32 bits are available.
There must be at least 4 bytes available. fillFastStart() assumes the bitReaderShifted is empty and there is at least 8 bytes to read. init initializes and resets the bit reader. peekBitsFast requires that at least one bit is requested every time.
There are no checks if the buffer is filled.(*bitReaderShifted) remaining() uint
bitWriter will write bits.
First bit will be LSB of the first byte of output.bitContaineruint64nBitsuint8out[]byte addBits16Clean will add up to 16 bits. value may not contain more set bits than indicated.
It will not check if there is space for them, so the caller must ensure that it has flushed recently. close will write the alignment bit and write the final byte(s)
to the output. encFourSymbols adds up to 32 bits from four symbols.
It will not check if there is space for them,
so the caller must ensure that b has been flushed recently. encSymbol will add up to 16 bits. value may not contain more set bits than indicated.
It will not check if there is space for them, so the caller must ensure that it has flushed recently. encTwoSymbols will add up to 32 bits. value may not contain more set bits than indicated.
It will not check if there is space for them, so the caller must ensure that it has flushed recently. flush32 will flush out, so there are at least 32 bits available for writing. flushAlign will flush remaining full bytes and align to next byte boundary.
A nodeElt is the fields
count uint32
parent uint16
symbol byte
nbBits uint8
in some order, all squashed into an integer so that the compiler
always loads and stores entire nodeElts instead of separate fields.(*nodeElt) count() uint32(*nodeElt) nbBits() uint8(*nodeElt) parent() uint16(*nodeElt) setCount(c uint32)(*nodeElt) setNbBits(n uint8)(*nodeElt) setParent(p int16)(*nodeElt) symbol() byte
func makeNodeElt(count uint32, symbol byte) nodeElt
Package-Level Functions (total 11, in which 4 are exported)
Compress1X will compress the input.
The output can be decoded using Decompress1X.
Supply a Scratch object. The scratch object contains state about re-use,
So when sharing across independent encodes, be sure to set the re-use policy.
Compress4X will compress the input. The input is split into 4 independent blocks
and compressed similar to Compress1X.
The output can be decoded using Decompress4X.
Supply a Scratch object. The scratch object contains state about re-use,
So when sharing across independent encodes, be sure to set the re-use policy.
EstimateSizes will estimate the data sizes
ReadTable will read a table from the input.
The size of the input may be larger than the table definition.
Any content remaining after the table definition will be returned.
If no Scratch is provided a new one is allocated.
The returned Scratch can be used for encoding or decoding input using this table.
Package-Level Constants (total 14, in which 5 are exported)
BlockSizeMax is maximum input size for a single block uncompressed.
ReusePolicyAllow will allow reuse if it produces smaller output.
ReusePolicyMust must allow reuse and produce smaller output.
ReusePolicyNone will disable re-use of tables.
This is slightly faster than ReusePolicyAllow but may produce larger output.
ReusePolicyPrefer will re-use aggressively if possible.
This will not check if a new table will produce smaller output,
except if the current table is impossible to use or
compressed output is bigger than input.
Error reported by asm implementations
fallback8BitSize is the size where using Go version is faster.
zstandard limits tablelog to 11, see:
https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md#huffman-tree-description
Uses special code for all tables that are < 8 bits.
The pages are generated with Goldsv0.8.4. (GOOS=linux GOARCH=amd64)
Golds is a Go 101 project developed by Tapir Liu.
PR and bug reports are welcome and can be submitted to the issue list.
Please follow @zigo_101 (reachable from the left QR code) to get the latest news of Golds.