Involved Source Filesbitreader.gobitwriter.gobytereader.gocompress.godecompress.go Package fse provides Finite State Entropy encoding and decoding.
Finite State Entropy encoding provides a fast near-optimal symbol encoding/decoding
for byte blocks as implemented in zstd.
See https://github.com/klauspost/compress/tree/master/fse for more information.
Package-Level Type Names (total 9, in which 1 is exported)
/* sort exporteds by: | */
Scratch provides temporary storage for compression and decompression. DecompressLimit limits the maximum decoded size acceptable.
If > 0 decompression will stop when approximately this many bytes
has been decoded.
If 0, maximum size will be 2GB. MaxSymbolValue will override the maximum symbol value of the next block. Out is output buffer.
If the scratch is re-used before the caller is done processing the output,
set this field to nil.
Otherwise the output buffer will be re-used for next Compression/Decompression step
and allocation will be avoided. TableLog will attempt to override the tablelog for the next block. // Selected tablelog.bitsbitReaderbrbyteReaderbwbitWriter // clear count Private // Compression tables. // Decompression table. // count of the most probable symbolnorm[256]int16 // Length of active part of the symbol table. // no bits has prob > 50%. Histogram allows to populate the histogram and skip that step in the compression,
It otherwise allows to inspect the histogram when compression is done.
To indicate that you have populated the histogram call HistogramFinished
with the value of the highest populated symbol, as well as the number of entries
in the most populated entry. These are accepted at face value.
The returned slice will always be length 256. HistogramFinished can be called to indicate that the histogram has been populated.
maxSymbol is the index of the highest set symbol of the next data segment.
maxCount is the number of entries in the most populated entry.
These are accepted at face value. allocCtable will allocate tables needed for compression.
If existing tables a re big enough, they are simply re-used. allocDtable will allocate decoding tables if they are not big enough. buildCTable will populate the compression table so it is ready to be used. buildDtable will build the decoding table. compress is the main compression loop that will encode the input from the last byte to the first. countSimple will create a simple histogram in s.count.
Returns the biggest count.
Does not update s.clearCount. decompress will decompress the bitstream.
If the buffer is over-read an error is returned. minTableLog provides the minimum logSize to safely represent a distribution. normalizeCount will normalize the count of the symbols so
the total is equal to the table size. Secondary normalization method.
To be used when primary method fails. optimalTableLog calculates and sets the optimal tableLog in s.actualTableLog prepare will prepare and allocate scratch tables used for both compression and decompression. readNCount will read the symbol distribution so decoding tables can be constructed. validateNorm validates the normalized histogram table. writeCount will write the normalized histogram count to header.
This is read back by readNCount.
func (*Scratch).prepare(in []byte) (*Scratch, error)
func Compress(in []byte, s *Scratch) ([]byte, error)
func Decompress(b []byte, s *Scratch) ([]byte, error)
bitReader reads a bitstream in reverse.
The last set bit indicates the start of the stream and is used
for aligning the input.bitsReaduint8in[]byte // next byte to read is at in[off - 1]valueuint64 close the bitstream and returns an error if out-of-buffer reads occurred. fill() will make sure at least 32 bits are available. fillFast() will make sure at least 32 bits are available.
There must be at least 4 bytes available. fillFastStart() assumes the bitreader is empty and there is at least 8 bytes to read. finished returns true if all bits have been read from the bit stream. getBits will return n bits. n can be 0. getBitsFast requires that at least one bit is requested every time.
There are no checks if the buffer is filled. init initializes and resets the bit reader.
bitWriter will write bits.
First bit will be LSB of the first byte of output.bitContaineruint64nBitsuint8out[]byte addBits16Clean will add up to 16 bits. value may not contain more set bits than indicated.
It will not check if there is space for them, so the caller must ensure that it has flushed recently. addBits16NC will add up to 16 bits.
It will not check if there is space for them,
so the caller must ensure that it has flushed recently. addBits16ZeroNC will add up to 16 bits.
It will not check if there is space for them,
so the caller must ensure that it has flushed recently.
This is fastest if bits can be zero. close will write the alignment bit and write the final byte(s)
to the output. flush will flush all pending full bytes.
There will be at least 56 bits available for writing when this has been called.
Using flush32 is faster, but leaves less space for writing. flush32 will flush out, so there are at least 32 bits available for writing. flushAlign will flush remaining full bytes and align to next byte boundary. reset and continue writing by appending to out.
byteReader provides a byte reader that reads
little endian values from a byte stream.
The input stream is manually advanced.
The reader performs no bounds checks.b[]byteoffint Uint32 returns a little endian uint32 starting at current offset. advance the stream b n bytes. init will initialize the reader and set the input. remain will return the number of bytes remaining. unread returns the unread portion of the input.
cState contains the compression state of a stream.bw*bitWriterstateuint16stateTable[]uint16 encode the output symbol provided and write it to the bitstream. encode the output symbol provided and write it to the bitstream. flush will write the tablelog to the output and flush the remaining full bytes. init will initialize the compression state to the first symbol of the stream.
decoder keeps track of the current state and updates it from the bitstream.br*bitReaderdt[]decSymbolstateuint16 final returns the current state symbol without decoding the next. finished returns true if all bits have been read from the bitstream
and the next state would require reading bits from the input. init will initialize the decoder and read the first state from the stream. next returns the next symbol and sets the next state.
At least tablelog bits must be available in the bit reader. nextFast returns the next symbol and sets the next state.
This can only be used if no symbols are 0 bits.
At least tablelog bits must be available in the bit reader.
decSymbol contains information about a state entry,
Including the state offset base, the output symbol and
the number of bits to read for the low part of the destination state.nbBitsuint8newStateuint16symboluint8
symbolTransform contains the state transform for a symbol.deltaFindStateint32deltaNbBitsuint32 String prints values as a human readable string.
symbolTransform : fmt.Stringer
symbolTransform : context.stringer
symbolTransform : runtime.stringer
Package-Level Functions (total 4, in which 2 are exported)
Compress the input bytes. Input must be < 2GB.
Provide a Scratch buffer to avoid memory allocations.
Note that the output is also kept in the scratch buffer.
If input is too hard to compress, ErrIncompressible is returned.
If input is a single byte value repeated ErrUseRLE is returned.
Decompress a block of data.
You can provide a scratch buffer to avoid allocations.
If nil is provided a temporary one will be allocated.
It is possible, but by no way guaranteed that corrupt data will
return an error.
It is up to the caller to verify integrity of the returned data.
Use a predefined Scratch to set maximum acceptable output size.
!MEMORY_USAGE :
* Memory usage formula : N->2^N Bytes (examples : 10 -> 1KB; 12 -> 4KB ; 16 -> 64KB; 20 -> 1MB; etc.)
* Increasing memory usage improves compression ratio
* Reduced memory usage can improve speed, due to cache effect
* Recommended max value is 14, for 16KB, which nicely fits into Intel x86 L1 cache
The pages are generated with Goldsv0.8.4. (GOOS=linux GOARCH=amd64)
Golds is a Go 101 project developed by Tapir Liu.
PR and bug reports are welcome and can be submitted to the issue list.
Please follow @zigo_101 (reachable from the left QR code) to get the latest news of Golds.
