ston1th/docstore
1
1
Fork 0
Code Issues 3 Releases Activity

updated dependencies

Browse source
This commit is contained in:
ston1th 2023-01-15 16:27:22 +01:00
commit 89ed7a4100
364 changed files with 28950 additions and 17035 deletions
Download patch file Download diff file Expand all files Collapse all files

1
vendor/github.com/RoaringBitmap/roaring/.gitignore generated vendored
View file

@ -3,4 +3,3 @@ roaring-fuzz.zip
workdir
coverage.out
testdata/all3.classic
testdata/all3.msgp.snappy

32
vendor/github.com/RoaringBitmap/roaring/.travis.yml generated vendored
View file

@ -1,32 +0,0 @@
language: go
sudo: false
install:
- go get -t github.com/RoaringBitmap/roaring
- go get -t golang.org/x/tools/cmd/cover
- go get -t github.com/mattn/goveralls
- go get -t github.com/mschoch/smat
notifications:
email: false
go:
- "1.13.x"
- "1.14.x"
- tip
# whitelist
branches:
only:
- master
script:
- goveralls -v -service travis-ci -ignore arraycontainer_gen.go,bitmapcontainer_gen.go,rle16_gen.go,rle_gen.go,roaringarray_gen.go,rle.go || go test
- go test -race -run TestConcurrent*
- go build -tags appengine
- go test -tags appengine
- GOARCH=arm64 go build
- GOARCH=386 go build
- GOARCH=386 go test
- GOARCH=arm go build
- GOARCH=arm64 go build
matrix:
allow_failures:
- go: tip

6
vendor/github.com/RoaringBitmap/roaring/Makefile generated vendored
View file

@ -64,7 +64,7 @@ qa: fmtcheck test vet lint
# Get the dependencies
deps:
GOPATH=$(GOPATH) go get github.com/stretchr/testify
GOPATH=$(GOPATH) go get github.com/willf/bitset
GOPATH=$(GOPATH) go get github.com/bits-and-blooms/bitset
GOPATH=$(GOPATH) go get github.com/golang/lint/golint
GOPATH=$(GOPATH) go get github.com/mschoch/smat
GOPATH=$(GOPATH) go get github.com/dvyukov/go-fuzz/go-fuzz
@ -97,10 +97,6 @@ nuke:
rm -rf ./target
GOPATH=$(GOPATH) go clean -i ./...
ser:
go generate
cover:
go test -coverprofile=coverage.out
go tool cover -html=coverage.out

19
vendor/github.com/RoaringBitmap/roaring/README.md generated vendored
View file

@ -1,4 +1,4 @@
roaring [![Build Status](https://travis-ci.org/RoaringBitmap/roaring.png)](https://travis-ci.org/RoaringBitmap/roaring) [![GoDoc](https://godoc.org/github.com/RoaringBitmap/roaring?status.svg)](https://godoc.org/github.com/RoaringBitmap/roaring) [![GoDoc](https://godoc.org/github.com/RoaringBitmap/roaring/roaring64?status.svg)](https://godoc.org/github.com/RoaringBitmap/roaring/roaring64) [![Go Report Card](https://goreportcard.com/badge/RoaringBitmap/roaring)](https://goreportcard.com/report/github.com/RoaringBitmap/roaring)
roaring [![GoDoc](https://godoc.org/github.com/RoaringBitmap/roaring/roaring64?status.svg)](https://godoc.org/github.com/RoaringBitmap/roaring/roaring64) [![Go Report Card](https://goreportcard.com/badge/RoaringBitmap/roaring)](https://goreportcard.com/report/github.com/RoaringBitmap/roaring)
[![Build Status](https://cloud.drone.io/api/badges/RoaringBitmap/roaring/status.svg)](https://cloud.drone.io/RoaringBitmap/roaring)
![Go-CI](https://github.com/RoaringBitmap/roaring/workflows/Go-CI/badge.svg)
![Go-ARM-CI](https://github.com/RoaringBitmap/roaring/workflows/Go-ARM-CI/badge.svg)
@ -7,10 +7,8 @@ roaring [![Build Status](https://travis-ci.org/RoaringBitmap/roaring.png)](https
This is a go version of the Roaring bitmap data structure.
Roaring bitmaps are used by several major systems such as [Apache Lucene][lucene] and derivative systems such as [Solr][solr] and
[Elasticsearch][elasticsearch], [Apache Druid (Incubating)][druid], [LinkedIn Pinot][pinot], [Netflix Atlas][atlas], [Apache Spark][spark], [OpenSearchServer][opensearchserver], [Cloud Torrent][cloudtorrent], [Whoosh][whoosh], [Pilosa][pilosa], [Microsoft Visual Studio Team Services (VSTS)][vsts], and eBay's [Apache Kylin][kylin]. The YouTube SQL Engine, [Google Procella](https://research.google/pubs/pub48388/), uses Roaring bitmaps for indexing.
[Elasticsearch][elasticsearch], [Apache Druid (Incubating)][druid], [LinkedIn Pinot][pinot], [Netflix Atlas][atlas], [Apache Spark][spark], [OpenSearchServer][opensearchserver], [anacrolix/torrent][anacrolix/torrent], [Whoosh][whoosh], [Pilosa][pilosa], [Microsoft Visual Studio Team Services (VSTS)][vsts], and eBay's [Apache Kylin][kylin]. The YouTube SQL Engine, [Google Procella](https://research.google/pubs/pub48388/), uses Roaring bitmaps for indexing.
[lucene]: https://lucene.apache.org/
[solr]: https://lucene.apache.org/solr/
@ -18,7 +16,7 @@ Roaring bitmaps are used by several major systems such as [Apache Lucene][lucene
[druid]: https://druid.apache.org/
[spark]: https://spark.apache.org/
[opensearchserver]: http://www.opensearchserver.com
[cloudtorrent]: https://github.com/jpillora/cloud-torrent
[anacrolix/torrent]: https://github.com/anacrolix/torrent
[whoosh]: https://bitbucket.org/mchaput/whoosh/wiki/Home
[pilosa]: https://www.pilosa.com/
[kylin]: http://kylin.apache.org/
@ -32,7 +30,7 @@ Roaring bitmaps are found to work well in many important applications:
The ``roaring`` Go library is used by
* [Cloud Torrent](https://github.com/jpillora/cloud-torrent)
* [anacrolix/torrent]
* [runv](https://github.com/hyperhq/runv)
* [InfluxDB](https://www.influxdata.com)
* [Pilosa](https://www.pilosa.com/)
@ -42,6 +40,7 @@ The ``roaring`` Go library is used by
* [SourceGraph](https://github.com/sourcegraph/sourcegraph)
* [M3](https://github.com/m3db/m3)
* [trident](https://github.com/NetApp/trident)
* [Husky](https://www.datadoghq.com/blog/engineering/introducing-husky/)
This library is used in production in several systems, it is part of the [Awesome Go collection](https://awesome-go.com).
@ -84,7 +83,7 @@ When the bitset approach is applicable, it can be orders of
magnitude faster than other possible implementation of a set (e.g., as a hash set)
while using several times less memory.
However, a bitset, even a compressed one is not always applicable. For example, if the
However, a bitset, even a compressed one is not always applicable. For example, if
you have 1000 random-looking integers, then a simple array might be the best representation.
We refer to this case as the "sparse" scenario.
@ -158,7 +157,7 @@ http://arxiv.org/abs/1402.6407 This paper used data from http://lemire.me/data/r
Dependencies are fetched automatically by giving the `-t` flag to `go get`.
they include
- github.com/willf/bitset
- github.com/bits-and-blooms/bitset
- github.com/mschoch/smat
- github.com/glycerine/go-unsnap-stream
- github.com/philhofer/fwd
@ -384,12 +383,14 @@ You can help us test further the library with fuzzy testing:
go get github.com/dvyukov/go-fuzz/go-fuzz-build
go test -tags=gofuzz -run=TestGenerateSmatCorpus
go-fuzz-build github.com/RoaringBitmap/roaring
go-fuzz -bin=./roaring-fuzz.zip -workdir=workdir/ -timeout=200
go-fuzz -bin=./roaring-fuzz.zip -workdir=workdir/ -timeout=200 -func FuzzSmat
Let it run, and if the # of crashers is > 0, check out the reports in
the workdir where you should be able to find the panic goroutine stack
traces.
You may also replace `-func FuzzSmat` by `-func FuzzSerializationBuffer` or `-func FuzzSerializationStream`.
### Alternative in Go
There is a Go version wrapping the C/C++ implementation https://github.com/RoaringBitmap/gocroaring

69
vendor/github.com/RoaringBitmap/roaring/arraycontainer.go generated vendored
View file

@ -4,8 +4,6 @@ import (
"fmt"
)
//go:generate msgp -unexported
type arrayContainer struct {
content []uint16
}
@ -18,10 +16,11 @@ func (ac *arrayContainer) String() string {
return s + "}"
}
func (ac *arrayContainer) fillLeastSignificant16bits(x []uint32, i int, mask uint32) {
func (ac *arrayContainer) fillLeastSignificant16bits(x []uint32, i int, mask uint32) int {
for k := 0; k < len(ac.content); k++ {
x[k+i] = uint32(ac.content[k]) | mask
}
return i + len(ac.content)
}
func (ac *arrayContainer) iterate(cb func(x uint16) bool) bool {
@ -396,11 +395,19 @@ func (ac *arrayContainer) iorBitmap(bc2 *bitmapContainer) container {
}
func (ac *arrayContainer) iorRun16(rc *runContainer16) container {
bc1 := ac.toBitmapContainer()
bc2 := rc.toBitmapContainer()
bc1.iorBitmap(bc2)
*ac = *newArrayContainerFromBitmap(bc1)
return ac
runCardinality := rc.getCardinality()
// heuristic for if the container should maybe be an
// array container.
if runCardinality < ac.getCardinality() &&
runCardinality+ac.getCardinality() < arrayDefaultMaxSize {
var result container
result = ac
for _, run := range rc.iv {
result = result.iaddRange(int(run.start), int(run.start)+int(run.length)+1)
}
return result
}
return rc.orArray(ac)
}
func (ac *arrayContainer) lazyIOR(a container) container {
@ -485,7 +492,7 @@ func (ac *arrayContainer) orArrayCardinality(value2 *arrayContainer) int {
func (ac *arrayContainer) lazyorArray(value2 *arrayContainer) container {
value1 := ac
maxPossibleCardinality := value1.getCardinality() + value2.getCardinality()
if maxPossibleCardinality > arrayLazyLowerBound { // it could be a bitmap!^M
if maxPossibleCardinality > arrayLazyLowerBound { // it could be a bitmap!
bc := newBitmapContainer()
for k := 0; k < len(value2.content); k++ {
v := value2.content[k]
@ -845,6 +852,10 @@ func (ac *arrayContainer) getCardinality() int {
return len(ac.content)
}
func (ac *arrayContainer) isEmpty() bool {
return len(ac.content) == 0
}
func (ac *arrayContainer) rank(x uint16) int {
answer := binarySearch(ac.content, x)
if answer >= 0 {
@ -884,7 +895,7 @@ func (ac *arrayContainer) resetTo(a container) {
x.fillArray(ac.content)
case *runContainer16:
card := int(x.cardinality())
card := int(x.getCardinality())
ac.realloc(card)
cur := 0
for _, r := range x.iv {
@ -958,10 +969,10 @@ func (ac *arrayContainer) numberOfRuns() (nr int) {
runlen++
} else {
if cur < prev {
panic("then fundamental arrayContainer assumption of sorted ac.content was broken")
panic("the fundamental arrayContainer assumption of sorted ac.content was broken")
}
if cur == prev {
panic("then fundamental arrayContainer assumption of deduplicated content was broken")
panic("the fundamental arrayContainer assumption of deduplicated content was broken")
} else {
nr++
runlen = 0
@ -996,16 +1007,42 @@ func (ac *arrayContainer) containerType() contype {
return arrayContype
}
func (ac *arrayContainer) addOffset(x uint16) []container {
low := &arrayContainer{}
high := &arrayContainer{}
func (ac *arrayContainer) addOffset(x uint16) (container, container) {
var low, high *arrayContainer
if len(ac.content) == 0 {
return nil, nil
}
if y := uint32(ac.content[0]) + uint32(x); highbits(y) == 0 {
// Some elements will fall into low part, allocate a container.
// Checking the first one is enough because they are ordered.
low = &arrayContainer{}
}
if y := uint32(ac.content[len(ac.content)-1]) + uint32(x); highbits(y) > 0 {
// Some elements will fall into high part, allocate a container.
// Checking the last one is enough because they are ordered.
high = &arrayContainer{}
}
for _, val := range ac.content {
y := uint32(val) + uint32(x)
if highbits(y) > 0 {
// OK, if high == nil then highbits(y) == 0 for all y.
high.content = append(high.content, lowbits(y))
} else {
// OK, if low == nil then highbits(y) > 0 for all y.
low.content = append(low.content, lowbits(y))
}
}
return []container{low, high}
// Ensure proper nil interface.
if low == nil {
return nil, high
}
if high == nil {
return low, nil
}
return low, high
}

134
vendor/github.com/RoaringBitmap/roaring/arraycontainer_gen.go generated vendored
View file

@ -1,134 +0,0 @@
package roaring
// NOTE: THIS FILE WAS PRODUCED BY THE
// MSGP CODE GENERATION TOOL (github.com/tinylib/msgp)
// DO NOT EDIT
import "github.com/tinylib/msgp/msgp"
// Deprecated: DecodeMsg implements msgp.Decodable
func (z *arrayContainer) DecodeMsg(dc *msgp.Reader) (err error) {
var field []byte
_ = field
var zbzg uint32
zbzg, err = dc.ReadMapHeader()
if err != nil {
return
}
for zbzg > 0 {
zbzg--
field, err = dc.ReadMapKeyPtr()
if err != nil {
return
}
switch msgp.UnsafeString(field) {
case "content":
var zbai uint32
zbai, err = dc.ReadArrayHeader()
if err != nil {
return
}
if cap(z.content) >= int(zbai) {
z.content = (z.content)[:zbai]
} else {
z.content = make([]uint16, zbai)
}
for zxvk := range z.content {
z.content[zxvk], err = dc.ReadUint16()
if err != nil {
return
}
}
default:
err = dc.Skip()
if err != nil {
return
}
}
}
return
}
// Deprecated: EncodeMsg implements msgp.Encodable
func (z *arrayContainer) EncodeMsg(en *msgp.Writer) (err error) {
// map header, size 1
// write "content"
err = en.Append(0x81, 0xa7, 0x63, 0x6f, 0x6e, 0x74, 0x65, 0x6e, 0x74)
if err != nil {
return err
}
err = en.WriteArrayHeader(uint32(len(z.content)))
if err != nil {
return
}
for zxvk := range z.content {
err = en.WriteUint16(z.content[zxvk])
if err != nil {
return
}
}
return
}
// Deprecated: MarshalMsg implements msgp.Marshaler
func (z *arrayContainer) MarshalMsg(b []byte) (o []byte, err error) {
o = msgp.Require(b, z.Msgsize())
// map header, size 1
// string "content"
o = append(o, 0x81, 0xa7, 0x63, 0x6f, 0x6e, 0x74, 0x65, 0x6e, 0x74)
o = msgp.AppendArrayHeader(o, uint32(len(z.content)))
for zxvk := range z.content {
o = msgp.AppendUint16(o, z.content[zxvk])
}
return
}
// Deprecated: UnmarshalMsg implements msgp.Unmarshaler
func (z *arrayContainer) UnmarshalMsg(bts []byte) (o []byte, err error) {
var field []byte
_ = field
var zcmr uint32
zcmr, bts, err = msgp.ReadMapHeaderBytes(bts)
if err != nil {
return
}
for zcmr > 0 {
zcmr--
field, bts, err = msgp.ReadMapKeyZC(bts)
if err != nil {
return
}
switch msgp.UnsafeString(field) {
case "content":
var zajw uint32
zajw, bts, err = msgp.ReadArrayHeaderBytes(bts)
if err != nil {
return
}
if cap(z.content) >= int(zajw) {
z.content = (z.content)[:zajw]
} else {
z.content = make([]uint16, zajw)
}
for zxvk := range z.content {
z.content[zxvk], bts, err = msgp.ReadUint16Bytes(bts)
if err != nil {
return
}
}
default:
bts, err = msgp.Skip(bts)
if err != nil {
return
}
}
}
o = bts
return
}
// Deprecated: Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message
func (z *arrayContainer) Msgsize() (s int) {
s = 1 + 8 + msgp.ArrayHeaderSize + (len(z.content) * (msgp.Uint16Size))
return
}

78
vendor/github.com/RoaringBitmap/roaring/bitmapcontainer.go generated vendored
View file

@ -5,8 +5,6 @@ import (
"unsafe"
)
//go:generate msgp -unexported
type bitmapContainer struct {
cardinality int
bitmap []uint64
@ -115,7 +113,7 @@ type bitmapContainerShortIterator struct {
func (bcsi *bitmapContainerShortIterator) next() uint16 {
j := bcsi.i
bcsi.i = bcsi.ptr.NextSetBit(bcsi.i + 1)
bcsi.i = bcsi.ptr.NextSetBit(uint(bcsi.i) + 1)
return uint16(j)
}
func (bcsi *bitmapContainerShortIterator) hasNext() bool {
@ -128,7 +126,7 @@ func (bcsi *bitmapContainerShortIterator) peekNext() uint16 {
func (bcsi *bitmapContainerShortIterator) advanceIfNeeded(minval uint16) {
if bcsi.hasNext() && bcsi.peekNext() < minval {
bcsi.i = bcsi.ptr.NextSetBit(int(minval))
bcsi.i = bcsi.ptr.NextSetBit(uint(minval))
}
}
@ -266,7 +264,7 @@ func bitmapEquals(a, b []uint64) bool {
return true
}
func (bc *bitmapContainer) fillLeastSignificant16bits(x []uint32, i int, mask uint32) {
func (bc *bitmapContainer) fillLeastSignificant16bits(x []uint32, i int, mask uint32) int {
// TODO: should be written as optimized assembly
pos := i
base := mask
@ -280,6 +278,7 @@ func (bc *bitmapContainer) fillLeastSignificant16bits(x []uint32, i int, mask ui
}
base += 64
}
return pos
}
func (bc *bitmapContainer) equals(o container) bool {
@ -351,6 +350,10 @@ func (bc *bitmapContainer) getCardinality() int {
return bc.cardinality
}
func (bc *bitmapContainer) isEmpty() bool {
return bc.cardinality == 0
}
func (bc *bitmapContainer) clone() container {
ptr := bitmapContainer{bc.cardinality, make([]uint64, len(bc.bitmap))}
copy(ptr.bitmap, bc.bitmap[:])
@ -1009,20 +1012,23 @@ func (bc *bitmapContainer) fillArray(container []uint16) {
}
}
func (bc *bitmapContainer) NextSetBit(i int) int {
x := i / 64
if x >= len(bc.bitmap) {
func (bc *bitmapContainer) NextSetBit(i uint) int {
var (
x = i / 64
length = uint(len(bc.bitmap))
)
if x >= length {
return -1
}
w := bc.bitmap[x]
w = w >> uint(i%64)
if w != 0 {
return i + countTrailingZeros(w)
return int(i) + countTrailingZeros(w)
}
x++
for ; x < len(bc.bitmap); x++ {
for ; x < length; x++ {
if bc.bitmap[x] != 0 {
return (x * 64) + countTrailingZeros(bc.bitmap[x])
return int(x*64) + countTrailingZeros(bc.bitmap[x])
}
}
return -1
@ -1118,34 +1124,60 @@ func (bc *bitmapContainer) containerType() contype {
return bitmapContype
}
func (bc *bitmapContainer) addOffset(x uint16) []container {
low := newBitmapContainer()
high := newBitmapContainer()
func (bc *bitmapContainer) addOffset(x uint16) (container, container) {
var low, high *bitmapContainer
if bc.cardinality == 0 {
return nil, nil
}
b := uint32(x) >> 6
i := uint32(x) % 64
end := uint32(1024) - b
low = newBitmapContainer()
if i == 0 {
copy(low.bitmap[b:], bc.bitmap[:end])
copy(high.bitmap[:b], bc.bitmap[end:])
} else {
low.bitmap[b] = bc.bitmap[0] << i
for k := uint32(1); k < end; k++ {
newval := bc.bitmap[k] << i
if newval == 0 {
newval = bc.bitmap[k-1] >> (64 - i)
}
newval |= bc.bitmap[k-1] >> (64 - i)
low.bitmap[b+k] = newval
}
}
low.computeCardinality()
if low.cardinality == bc.cardinality {
// All elements from bc ended up in low, meaning high will be empty.
return low, nil
}
if low.cardinality == 0 {
// low is empty, let's reuse the container for high.
high = low
low = nil
} else {
// None of the containers will be empty, so allocate both.
high = newBitmapContainer()
}
if i == 0 {
copy(high.bitmap[:b], bc.bitmap[end:])
} else {
for k := end; k < 1024; k++ {
newval := bc.bitmap[k] << i
if newval == 0 {
newval = bc.bitmap[k-1] >> (64 - i)
}
newval |= bc.bitmap[k-1] >> (64 - i)
high.bitmap[k-end] = newval
}
high.bitmap[b] = bc.bitmap[1023] >> (64 - i)
}
low.computeCardinality()
high.computeCardinality()
return []container{low, high}
// Ensure proper nil interface.
if low == nil {
return nil, high
}
return low, high
}

415
vendor/github.com/RoaringBitmap/roaring/bitmapcontainer_gen.go generated vendored
View file

@ -1,415 +0,0 @@
package roaring
// NOTE: THIS FILE WAS PRODUCED BY THE
// MSGP CODE GENERATION TOOL (github.com/tinylib/msgp)
// DO NOT EDIT
import "github.com/tinylib/msgp/msgp"
// Deprecated: DecodeMsg implements msgp.Decodable
func (z *bitmapContainer) DecodeMsg(dc *msgp.Reader) (err error) {
var field []byte
_ = field
var zbzg uint32
zbzg, err = dc.ReadMapHeader()
if err != nil {
return
}
for zbzg > 0 {
zbzg--
field, err = dc.ReadMapKeyPtr()
if err != nil {
return
}
switch msgp.UnsafeString(field) {
case "cardinality":
z.cardinality, err = dc.ReadInt()
if err != nil {
return
}
case "bitmap":
var zbai uint32
zbai, err = dc.ReadArrayHeader()
if err != nil {
return
}
if cap(z.bitmap) >= int(zbai) {
z.bitmap = (z.bitmap)[:zbai]
} else {
z.bitmap = make([]uint64, zbai)
}
for zxvk := range z.bitmap {
z.bitmap[zxvk], err = dc.ReadUint64()
if err != nil {
return
}
}
default:
err = dc.Skip()
if err != nil {
return
}
}
}
return
}
// Deprecated: EncodeMsg implements msgp.Encodable
func (z *bitmapContainer) EncodeMsg(en *msgp.Writer) (err error) {
// map header, size 2
// write "cardinality"
err = en.Append(0x82, 0xab, 0x63, 0x61, 0x72, 0x64, 0x69, 0x6e, 0x61, 0x6c, 0x69, 0x74, 0x79)
if err != nil {
return err
}
err = en.WriteInt(z.cardinality)
if err != nil {
return
}
// write "bitmap"
err = en.Append(0xa6, 0x62, 0x69, 0x74, 0x6d, 0x61, 0x70)
if err != nil {
return err
}
err = en.WriteArrayHeader(uint32(len(z.bitmap)))
if err != nil {
return
}
for zxvk := range z.bitmap {
err = en.WriteUint64(z.bitmap[zxvk])
if err != nil {
return
}
}
return
}
// Deprecated: MarshalMsg implements msgp.Marshaler
func (z *bitmapContainer) MarshalMsg(b []byte) (o []byte, err error) {
o = msgp.Require(b, z.Msgsize())
// map header, size 2
// string "cardinality"
o = append(o, 0x82, 0xab, 0x63, 0x61, 0x72, 0x64, 0x69, 0x6e, 0x61, 0x6c, 0x69, 0x74, 0x79)
o = msgp.AppendInt(o, z.cardinality)
// string "bitmap"
o = append(o, 0xa6, 0x62, 0x69, 0x74, 0x6d, 0x61, 0x70)
o = msgp.AppendArrayHeader(o, uint32(len(z.bitmap)))
for zxvk := range z.bitmap {
o = msgp.AppendUint64(o, z.bitmap[zxvk])
}
return
}
// Deprecated: UnmarshalMsg implements msgp.Unmarshaler
func (z *bitmapContainer) UnmarshalMsg(bts []byte) (o []byte, err error) {
var field []byte
_ = field
var zcmr uint32
zcmr, bts, err = msgp.ReadMapHeaderBytes(bts)
if err != nil {
return
}
for zcmr > 0 {
zcmr--
field, bts, err = msgp.ReadMapKeyZC(bts)
if err != nil {
return
}
switch msgp.UnsafeString(field) {
case "cardinality":
z.cardinality, bts, err = msgp.ReadIntBytes(bts)
if err != nil {
return
}
case "bitmap":
var zajw uint32
zajw, bts, err = msgp.ReadArrayHeaderBytes(bts)
if err != nil {
return
}
if cap(z.bitmap) >= int(zajw) {
z.bitmap = (z.bitmap)[:zajw]
} else {
z.bitmap = make([]uint64, zajw)
}
for zxvk := range z.bitmap {
z.bitmap[zxvk], bts, err = msgp.ReadUint64Bytes(bts)
if err != nil {
return
}
}
default:
bts, err = msgp.Skip(bts)
if err != nil {
return
}
}
}
o = bts
return
}
// Deprecated: Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message
func (z *bitmapContainer) Msgsize() (s int) {
s = 1 + 12 + msgp.IntSize + 7 + msgp.ArrayHeaderSize + (len(z.bitmap) * (msgp.Uint64Size))
return
}
// Deprecated: DecodeMsg implements msgp.Decodable
func (z *bitmapContainerShortIterator) DecodeMsg(dc *msgp.Reader) (err error) {
var field []byte
_ = field
var zhct uint32
zhct, err = dc.ReadMapHeader()
if err != nil {
return
}
for zhct > 0 {
zhct--
field, err = dc.ReadMapKeyPtr()
if err != nil {
return
}
switch msgp.UnsafeString(field) {
case "ptr":
if dc.IsNil() {
err = dc.ReadNil()
if err != nil {
return
}
z.ptr = nil
} else {
if z.ptr == nil {
z.ptr = new(bitmapContainer)
}
var zcua uint32
zcua, err = dc.ReadMapHeader()
if err != nil {
return
}
for zcua > 0 {
zcua--
field, err = dc.ReadMapKeyPtr()
if err != nil {
return
}
switch msgp.UnsafeString(field) {
case "cardinality":
z.ptr.cardinality, err = dc.ReadInt()
if err != nil {
return
}
case "bitmap":
var zxhx uint32
zxhx, err = dc.ReadArrayHeader()
if err != nil {
return
}
if cap(z.ptr.bitmap) >= int(zxhx) {
z.ptr.bitmap = (z.ptr.bitmap)[:zxhx]
} else {
z.ptr.bitmap = make([]uint64, zxhx)
}
for zwht := range z.ptr.bitmap {
z.ptr.bitmap[zwht], err = dc.ReadUint64()
if err != nil {
return
}
}
default:
err = dc.Skip()
if err != nil {
return
}
}
}
}
case "i":
z.i, err = dc.ReadInt()
if err != nil {
return
}
default:
err = dc.Skip()
if err != nil {
return
}
}
}
return
}
// Deprecated: EncodeMsg implements msgp.Encodable
func (z *bitmapContainerShortIterator) EncodeMsg(en *msgp.Writer) (err error) {
// map header, size 2
// write "ptr"
err = en.Append(0x82, 0xa3, 0x70, 0x74, 0x72)
if err != nil {
return err
}
if z.ptr == nil {
err = en.WriteNil()
if err != nil {
return
}
} else {
// map header, size 2
// write "cardinality"
err = en.Append(0x82, 0xab, 0x63, 0x61, 0x72, 0x64, 0x69, 0x6e, 0x61, 0x6c, 0x69, 0x74, 0x79)
if err != nil {
return err
}
err = en.WriteInt(z.ptr.cardinality)
if err != nil {
return
}
// write "bitmap"
err = en.Append(0xa6, 0x62, 0x69, 0x74, 0x6d, 0x61, 0x70)
if err != nil {
return err
}
err = en.WriteArrayHeader(uint32(len(z.ptr.bitmap)))
if err != nil {
return
}
for zwht := range z.ptr.bitmap {
err = en.WriteUint64(z.ptr.bitmap[zwht])
if err != nil {
return
}
}
}
// write "i"
err = en.Append(0xa1, 0x69)
if err != nil {
return err
}
err = en.WriteInt(z.i)
if err != nil {
return
}
return
}
// Deprecated: MarshalMsg implements msgp.Marshaler
func (z *bitmapContainerShortIterator) MarshalMsg(b []byte) (o []byte, err error) {
o = msgp.Require(b, z.Msgsize())
// map header, size 2
// string "ptr"
o = append(o, 0x82, 0xa3, 0x70, 0x74, 0x72)
if z.ptr == nil {
o = msgp.AppendNil(o)
} else {
// map header, size 2
// string "cardinality"
o = append(o, 0x82, 0xab, 0x63, 0x61, 0x72, 0x64, 0x69, 0x6e, 0x61, 0x6c, 0x69, 0x74, 0x79)
o = msgp.AppendInt(o, z.ptr.cardinality)
// string "bitmap"
o = append(o, 0xa6, 0x62, 0x69, 0x74, 0x6d, 0x61, 0x70)
o = msgp.AppendArrayHeader(o, uint32(len(z.ptr.bitmap)))
for zwht := range z.ptr.bitmap {
o = msgp.AppendUint64(o, z.ptr.bitmap[zwht])
}
}
// string "i"
o = append(o, 0xa1, 0x69)
o = msgp.AppendInt(o, z.i)
return
}
// Deprecated: UnmarshalMsg implements msgp.Unmarshaler
func (z *bitmapContainerShortIterator) UnmarshalMsg(bts []byte) (o []byte, err error) {
var field []byte
_ = field
var zlqf uint32
zlqf, bts, err = msgp.ReadMapHeaderBytes(bts)
if err != nil {
return
}
for zlqf > 0 {
zlqf--
field, bts, err = msgp.ReadMapKeyZC(bts)
if err != nil {
return
}
switch msgp.UnsafeString(field) {
case "ptr":
if msgp.IsNil(bts) {
bts, err = msgp.ReadNilBytes(bts)
if err != nil {
return
}
z.ptr = nil
} else {
if z.ptr == nil {
z.ptr = new(bitmapContainer)
}
var zdaf uint32
zdaf, bts, err = msgp.ReadMapHeaderBytes(bts)
if err != nil {
return
}
for zdaf > 0 {
zdaf--
field, bts, err = msgp.ReadMapKeyZC(bts)
if err != nil {
return
}
switch msgp.UnsafeString(field) {
case "cardinality":
z.ptr.cardinality, bts, err = msgp.ReadIntBytes(bts)
if err != nil {
return
}
case "bitmap":
var zpks uint32
zpks, bts, err = msgp.ReadArrayHeaderBytes(bts)
if err != nil {
return
}
if cap(z.ptr.bitmap) >= int(zpks) {
z.ptr.bitmap = (z.ptr.bitmap)[:zpks]
} else {
z.ptr.bitmap = make([]uint64, zpks)
}
for zwht := range z.ptr.bitmap {
z.ptr.bitmap[zwht], bts, err = msgp.ReadUint64Bytes(bts)
if err != nil {
return
}
}
default:
bts, err = msgp.Skip(bts)
if err != nil {
return
}
}
}
}
case "i":
z.i, bts, err = msgp.ReadIntBytes(bts)
if err != nil {
return
}
default:
bts, err = msgp.Skip(bts)
if err != nil {
return
}
}
}
o = bts
return
}
// Deprecated: Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message
func (z *bitmapContainerShortIterator) Msgsize() (s int) {
s = 1 + 4
if z.ptr == nil {
s += msgp.NilSize
} else {
s += 1 + 12 + msgp.IntSize + 7 + msgp.ArrayHeaderSize + (len(z.ptr.bitmap) * (msgp.Uint64Size))
}
s += 2 + msgp.IntSize
return
}

161
vendor/github.com/RoaringBitmap/roaring/byte_input.go generated vendored
View file

@ -1,161 +0,0 @@
package roaring
import (
"encoding/binary"
"io"
)
type byteInput interface {
// next returns a slice containing the next n bytes from the buffer,
// advancing the buffer as if the bytes had been returned by Read.
next(n int) ([]byte, error)
// readUInt32 reads uint32 with LittleEndian order
readUInt32() (uint32, error)
// readUInt16 reads uint16 with LittleEndian order
readUInt16() (uint16, error)
// getReadBytes returns read bytes
getReadBytes() int64
// skipBytes skips exactly n bytes
skipBytes(n int) error
}
func newByteInputFromReader(reader io.Reader) byteInput {
return &byteInputAdapter{
r: reader,
readBytes: 0,
}
}
func newByteInput(buf []byte) byteInput {
return &byteBuffer{
buf: buf,
off: 0,
}
}
type byteBuffer struct {
buf []byte
off int
}
// next returns a slice containing the next n bytes from the reader
// If there are fewer bytes than the given n, io.ErrUnexpectedEOF will be returned
func (b *byteBuffer) next(n int) ([]byte, error) {
m := len(b.buf) - b.off
if n > m {
return nil, io.ErrUnexpectedEOF
}
data := b.buf[b.off : b.off+n]
b.off += n
return data, nil
}
// readUInt32 reads uint32 with LittleEndian order
func (b *byteBuffer) readUInt32() (uint32, error) {
if len(b.buf)-b.off < 4 {
return 0, io.ErrUnexpectedEOF
}
v := binary.LittleEndian.Uint32(b.buf[b.off:])
b.off += 4
return v, nil
}
// readUInt16 reads uint16 with LittleEndian order
func (b *byteBuffer) readUInt16() (uint16, error) {
if len(b.buf)-b.off < 2 {
return 0, io.ErrUnexpectedEOF
}
v := binary.LittleEndian.Uint16(b.buf[b.off:])
b.off += 2
return v, nil
}
// getReadBytes returns read bytes
func (b *byteBuffer) getReadBytes() int64 {
return int64(b.off)
}
// skipBytes skips exactly n bytes
func (b *byteBuffer) skipBytes(n int) error {
m := len(b.buf) - b.off
if n > m {
return io.ErrUnexpectedEOF
}
b.off += n
return nil
}
// reset resets the given buffer with a new byte slice
func (b *byteBuffer) reset(buf []byte) {
b.buf = buf
b.off = 0
}
type byteInputAdapter struct {
r io.Reader
readBytes int
}
// next returns a slice containing the next n bytes from the buffer,
// advancing the buffer as if the bytes had been returned by Read.
func (b *byteInputAdapter) next(n int) ([]byte, error) {
buf := make([]byte, n)
m, err := io.ReadAtLeast(b.r, buf, n)
b.readBytes += m
if err != nil {
return nil, err
}
return buf, nil
}
// readUInt32 reads uint32 with LittleEndian order
func (b *byteInputAdapter) readUInt32() (uint32, error) {
buf, err := b.next(4)
if err != nil {
return 0, err
}
return binary.LittleEndian.Uint32(buf), nil
}
// readUInt16 reads uint16 with LittleEndian order
func (b *byteInputAdapter) readUInt16() (uint16, error) {
buf, err := b.next(2)
if err != nil {
return 0, err
}
return binary.LittleEndian.Uint16(buf), nil
}
// getReadBytes returns read bytes
func (b *byteInputAdapter) getReadBytes() int64 {
return int64(b.readBytes)
}
// skipBytes skips exactly n bytes
func (b *byteInputAdapter) skipBytes(n int) error {
_, err := b.next(n)
return err
}
// reset resets the given buffer with a new stream
func (b *byteInputAdapter) reset(stream io.Reader) {
b.r = stream
b.readBytes = 0
}

2
vendor/github.com/RoaringBitmap/roaring/clz.go generated vendored
View file

@ -1,4 +1,6 @@
//go:build go1.9
// +build go1.9
// "go1.9", from Go version 1.9 onward
// See https://golang.org/pkg/go/build/#hdr-Build_Constraints

1
vendor/github.com/RoaringBitmap/roaring/clz_compat.go generated vendored
View file

@ -1,3 +1,4 @@
//go:build !go1.9
// +build !go1.9
package roaring

2
vendor/github.com/RoaringBitmap/roaring/ctz.go generated vendored
View file

@ -1,4 +1,6 @@
//go:build go1.9
// +build go1.9
// "go1.9", from Go version 1.9 onward
// See https://golang.org/pkg/go/build/#hdr-Build_Constraints

1
vendor/github.com/RoaringBitmap/roaring/ctz_compat.go generated vendored
View file

@ -1,3 +1,4 @@
//go:build !go1.9
// +build !go1.9
package roaring

33
vendor/github.com/RoaringBitmap/roaring/fastaggregation.go generated vendored
View file

@ -33,15 +33,6 @@ main:
s2 = x2.highlowcontainer.getKeyAtIndex(pos2)
} else {
c1 := x1.highlowcontainer.getContainerAtIndex(pos1)
switch t := c1.(type) {
case *arrayContainer:
c1 = t.toBitmapContainer()
case *runContainer16:
if !t.isFull() {
c1 = t.toBitmapContainer()
}
}
answer.highlowcontainer.appendContainer(s1, c1.lazyOR(x2.highlowcontainer.getContainerAtIndex(pos2)), false)
pos1++
pos2++
@ -89,18 +80,7 @@ main:
}
s2 = x2.highlowcontainer.getKeyAtIndex(pos2)
} else {
c1 := x1.highlowcontainer.getContainerAtIndex(pos1)
switch t := c1.(type) {
case *arrayContainer:
c1 = t.toBitmapContainer()
case *runContainer16:
if !t.isFull() {
c1 = t.toBitmapContainer()
}
case *bitmapContainer:
c1 = x1.highlowcontainer.getWritableContainerAtIndex(pos1)
}
c1 := x1.highlowcontainer.getWritableContainerAtIndex(pos1)
x1.highlowcontainer.containers[pos1] = c1.lazyIOR(x2.highlowcontainer.getContainerAtIndex(pos2))
x1.highlowcontainer.needCopyOnWrite[pos1] = false
pos1++
@ -301,9 +281,6 @@ func (x1 *Bitmap) AndAny(bitmaps ...*Bitmap) {
tmpBitmap = newBitmapContainer()
}
tmpBitmap.resetTo(keyContainers[0])
for _, c := range keyContainers[1:] {
tmpBitmap.ior(c)
}
ored = tmpBitmap
} else {
if tmpArray == nil {
@ -311,15 +288,15 @@ func (x1 *Bitmap) AndAny(bitmaps ...*Bitmap) {
}
tmpArray.realloc(maxPossibleOr)
tmpArray.resetTo(keyContainers[0])
for _, c := range keyContainers[1:] {
tmpArray.ior(c)
}
ored = tmpArray
}
for _, c := range keyContainers[1:] {
ored = ored.ior(c)
}
}
result := x1.highlowcontainer.getWritableContainerAtIndex(basePos).iand(ored)
if result.getCardinality() > 0 {
if !result.isEmpty() {
x1.highlowcontainer.replaceKeyAndContainerAtIndex(intersections, baseKey, result, false)
intersections++
}

166
vendor/github.com/RoaringBitmap/roaring/internal/byte_input.go generated vendored Normal file
View file

@ -0,0 +1,166 @@
package internal
import (
"encoding/binary"
"io"
)
// ByteInput typed interface around io.Reader or raw bytes
type ByteInput interface {
// Next returns a slice containing the next n bytes from the buffer,
// advancing the buffer as if the bytes had been returned by Read.
Next(n int) ([]byte, error)
// ReadUInt32 reads uint32 with LittleEndian order
ReadUInt32() (uint32, error)
// ReadUInt16 reads uint16 with LittleEndian order
ReadUInt16() (uint16, error)
// GetReadBytes returns read bytes
GetReadBytes() int64
// SkipBytes skips exactly n bytes
SkipBytes(n int) error
}
// NewByteInputFromReader creates reader wrapper
func NewByteInputFromReader(reader io.Reader) ByteInput {
return &ByteInputAdapter{
r: reader,
readBytes: 0,
}
}
// NewByteInput creates raw bytes wrapper
func NewByteInput(buf []byte) ByteInput {
return &ByteBuffer{
buf: buf,
off: 0,
}
}
// ByteBuffer raw bytes wrapper
type ByteBuffer struct {
buf []byte
off int
}
// Next returns a slice containing the next n bytes from the reader
// If there are fewer bytes than the given n, io.ErrUnexpectedEOF will be returned
func (b *ByteBuffer) Next(n int) ([]byte, error) {
m := len(b.buf) - b.off
if n > m {
return nil, io.ErrUnexpectedEOF
}
data := b.buf[b.off : b.off+n]
b.off += n
return data, nil
}
// ReadUInt32 reads uint32 with LittleEndian order
func (b *ByteBuffer) ReadUInt32() (uint32, error) {
if len(b.buf)-b.off < 4 {
return 0, io.ErrUnexpectedEOF
}
v := binary.LittleEndian.Uint32(b.buf[b.off:])
b.off += 4
return v, nil
}
// ReadUInt16 reads uint16 with LittleEndian order
func (b *ByteBuffer) ReadUInt16() (uint16, error) {
if len(b.buf)-b.off < 2 {
return 0, io.ErrUnexpectedEOF
}
v := binary.LittleEndian.Uint16(b.buf[b.off:])
b.off += 2
return v, nil
}
// GetReadBytes returns read bytes
func (b *ByteBuffer) GetReadBytes() int64 {
return int64(b.off)
}
// SkipBytes skips exactly n bytes
func (b *ByteBuffer) SkipBytes(n int) error {
m := len(b.buf) - b.off
if n > m {
return io.ErrUnexpectedEOF
}
b.off += n
return nil
}
// Reset resets the given buffer with a new byte slice
func (b *ByteBuffer) Reset(buf []byte) {
b.buf = buf
b.off = 0
}
// ByteInputAdapter reader wrapper
type ByteInputAdapter struct {
r io.Reader
readBytes int
}
// Next returns a slice containing the next n bytes from the buffer,
// advancing the buffer as if the bytes had been returned by Read.
func (b *ByteInputAdapter) Next(n int) ([]byte, error) {
buf := make([]byte, n)
m, err := io.ReadAtLeast(b.r, buf, n)
b.readBytes += m
if err != nil {
return nil, err
}
return buf, nil
}
// ReadUInt32 reads uint32 with LittleEndian order
func (b *ByteInputAdapter) ReadUInt32() (uint32, error) {
buf, err := b.Next(4)
if err != nil {
return 0, err
}
return binary.LittleEndian.Uint32(buf), nil
}
// ReadUInt16 reads uint16 with LittleEndian order
func (b *ByteInputAdapter) ReadUInt16() (uint16, error) {
buf, err := b.Next(2)
if err != nil {
return 0, err
}
return binary.LittleEndian.Uint16(buf), nil
}
// GetReadBytes returns read bytes
func (b *ByteInputAdapter) GetReadBytes() int64 {
return int64(b.readBytes)
}
// SkipBytes skips exactly n bytes
func (b *ByteInputAdapter) SkipBytes(n int) error {
_, err := b.Next(n)
return err
}
// Reset resets the given buffer with a new stream
func (b *ByteInputAdapter) Reset(stream io.Reader) {
b.r = stream
b.readBytes = 0
}

21
vendor/github.com/RoaringBitmap/roaring/internal/pools.go generated vendored Normal file
View file

@ -0,0 +1,21 @@
package internal
import (
"sync"
)
var (
// ByteInputAdapterPool shared pool
ByteInputAdapterPool = sync.Pool{
New: func() interface{} {
return &ByteInputAdapter{}
},
}
// ByteBufferPool shared pool
ByteBufferPool = sync.Pool{
New: func() interface{} {
return &ByteBuffer{}
},
}
)

9
vendor/github.com/RoaringBitmap/roaring/parallel.go generated vendored
View file

@ -166,7 +166,6 @@ func appenderRoutine(bitmapChan chan<- *Bitmap, resultChan <-chan keyedContainer
make([]container, 0, expectedKeys),
make([]bool, 0, expectedKeys),
false,
nil,
},
}
for i := range keys {
@ -286,14 +285,14 @@ func ParAnd(parallelism int, bitmaps ...*Bitmap) *Bitmap {
for input := range inputChan {
c := input.containers[0].and(input.containers[1])
for _, next := range input.containers[2:] {
if c.getCardinality() == 0 {
if c.isEmpty() {
break
}
c = c.iand(next)
}
// Send a nil explicitly if the result of the intersection is an empty container
if c.getCardinality() == 0 {
if c.isEmpty() {
c = nil
}
@ -355,10 +354,10 @@ func ParOr(parallelism int, bitmaps ...*Bitmap) *Bitmap {
if lKey == MaxUint16 && hKey == 0 {
return New()
} else if len(bitmaps) == 1 {
return bitmaps[0]
return bitmaps[0].Clone()
}
keyRange := hKey - lKey + 1
keyRange := int(hKey) - int(lKey) + 1
if keyRange == 1 {
// revert to FastOr. Since the key range is 0
// no container-level aggregation parallelism is achievable

2
vendor/github.com/RoaringBitmap/roaring/popcnt.go generated vendored
View file

@ -1,4 +1,6 @@
//go:build go1.9
// +build go1.9
// "go1.9", from Go version 1.9 onward
// See https://golang.org/pkg/go/build/#hdr-Build_Constraints

1
vendor/github.com/RoaringBitmap/roaring/popcnt_asm.go generated vendored
View file

@ -1,3 +1,4 @@
//go:build amd64 && !appengine && !go1.9
// +build amd64,!appengine,!go1.9
package roaring

1
vendor/github.com/RoaringBitmap/roaring/popcnt_compat.go generated vendored
View file

@ -1,3 +1,4 @@
//go:build !go1.9
// +build !go1.9
package roaring

1
vendor/github.com/RoaringBitmap/roaring/popcnt_generic.go generated vendored
View file

@ -1,3 +1,4 @@
//go:build !amd64 || appengine || go1.9
// +build !amd64 appengine go1.9
package roaring

338
vendor/github.com/RoaringBitmap/roaring/roaring.go generated vendored
View file

@ -11,7 +11,8 @@ import (
"fmt"
"io"
"strconv"
"sync"
"github.com/RoaringBitmap/roaring/internal"
)
// Bitmap represents a compressed bitmap where you can add integers.
@ -52,27 +53,72 @@ func (rb *Bitmap) ToBytes() ([]byte, error) {
return rb.highlowcontainer.toBytes()
}
// Deprecated: WriteToMsgpack writes a msgpack2/snappy-streaming compressed serialized
// version of this bitmap to stream. The format is not
// compatible with the WriteTo() format, and is
// experimental: it may produce smaller on disk
// footprint and/or be faster to read, depending
// on your content. Currently only the Go roaring
// implementation supports this format.
func (rb *Bitmap) WriteToMsgpack(stream io.Writer) (int64, error) {
return 0, rb.highlowcontainer.writeToMsgpack(stream)
// Checksum computes a hash (currently FNV-1a) for a bitmap that is suitable for
// using bitmaps as elements in hash sets or as keys in hash maps, as well as
// generally quicker comparisons.
// The implementation is biased towards efficiency in little endian machines, so
// expect some extra CPU cycles and memory to be used if your machine is big endian.
// Likewise, don't use this to verify integrity unless you're certain you'll load
// the bitmap on a machine with the same endianess used to create it.
func (rb *Bitmap) Checksum() uint64 {
const (
offset = 14695981039346656037
prime = 1099511628211
)
var bytes []byte
hash := uint64(offset)
bytes = uint16SliceAsByteSlice(rb.highlowcontainer.keys)
for _, b := range bytes {
hash ^= uint64(b)
hash *= prime
}
for _, c := range rb.highlowcontainer.containers {
// 0 separator
hash ^= 0
hash *= prime
switch c := c.(type) {
case *bitmapContainer:
bytes = uint64SliceAsByteSlice(c.bitmap)
case *arrayContainer:
bytes = uint16SliceAsByteSlice(c.content)
case *runContainer16:
bytes = interval16SliceAsByteSlice(c.iv)
default:
panic("invalid container type")
}
if len(bytes) == 0 {
panic("empty containers are not supported")
}
for _, b := range bytes {
hash ^= uint64(b)
hash *= prime
}
}
return hash
}
// ReadFrom reads a serialized version of this bitmap from stream.
// The format is compatible with other RoaringBitmap
// implementations (Java, C) and is documented here:
// https://github.com/RoaringBitmap/RoaringFormatSpec
func (rb *Bitmap) ReadFrom(reader io.Reader) (p int64, err error) {
stream := byteInputAdapterPool.Get().(*byteInputAdapter)
stream.reset(reader)
// Since io.Reader is regarded as a stream and cannot be read twice.
// So add cookieHeader to accept the 4-byte data that has been read in roaring64.ReadFrom.
// It is not necessary to pass cookieHeader when call roaring.ReadFrom to read the roaring32 data directly.
func (rb *Bitmap) ReadFrom(reader io.Reader, cookieHeader ...byte) (p int64, err error) {
stream := internal.ByteInputAdapterPool.Get().(*internal.ByteInputAdapter)
stream.Reset(reader)
p, err = rb.highlowcontainer.readFrom(stream)
byteInputAdapterPool.Put(stream)
p, err = rb.highlowcontainer.readFrom(stream, cookieHeader...)
internal.ByteInputAdapterPool.Put(stream)
return
}
@ -100,29 +146,15 @@ func (rb *Bitmap) ReadFrom(reader io.Reader) (p int64, err error) {
// call CloneCopyOnWriteContainers on all such bitmaps.
//
func (rb *Bitmap) FromBuffer(buf []byte) (p int64, err error) {
stream := byteBufferPool.Get().(*byteBuffer)
stream.reset(buf)
stream := internal.ByteBufferPool.Get().(*internal.ByteBuffer)
stream.Reset(buf)
p, err = rb.highlowcontainer.readFrom(stream)
byteBufferPool.Put(stream)
internal.ByteBufferPool.Put(stream)
return
}
var (
byteBufferPool = sync.Pool{
New: func() interface{} {
return &byteBuffer{}
},
}
byteInputAdapterPool = sync.Pool{
New: func() interface{} {
return &byteInputAdapter{}
},
}
)
// RunOptimize attempts to further compress the runs of consecutive values found in the bitmap
func (rb *Bitmap) RunOptimize() {
rb.highlowcontainer.runOptimize()
@ -133,14 +165,6 @@ func (rb *Bitmap) HasRunCompression() bool {
return rb.highlowcontainer.hasRunCompression()
}
// Deprecated: ReadFromMsgpack reads a msgpack2/snappy-streaming serialized
// version of this bitmap from stream. The format is
// expected is that written by the WriteToMsgpack()
// call; see additional notes there.
func (rb *Bitmap) ReadFromMsgpack(stream io.Reader) (int64, error) {
return 0, rb.highlowcontainer.readFromMsgpack(stream)
}
// MarshalBinary implements the encoding.BinaryMarshaler interface for the bitmap
// (same as ToBytes)
func (rb *Bitmap) MarshalBinary() ([]byte, error) {
@ -180,8 +204,7 @@ func (rb *Bitmap) ToArray() []uint32 {
hs := uint32(rb.highlowcontainer.getKeyAtIndex(pos)) << 16
c := rb.highlowcontainer.getContainerAtIndex(pos)
pos++
c.fillLeastSignificant16bits(array, pos2, hs)
pos2 += c.getCardinality()
pos2 = c.fillLeastSignificant16bits(array, pos2, hs)
}
return array
}
@ -248,6 +271,14 @@ type intIterator struct {
hs uint32
iter shortPeekable
highlowcontainer *roaringArray
// These embedded iterators per container type help reduce load in the GC.
// This way, instead of making up-to 64k allocations per full iteration
// we get a single allocation and simply reinitialize the appropriate
// iterator and point to it in the generic `iter` member on each key bound.
shortIter shortIterator
runIter runIterator16
bitmapIter bitmapContainerShortIterator
}
// HasNext returns true if there are more integers to iterate over
@ -257,8 +288,19 @@ func (ii *intIterator) HasNext() bool {
func (ii *intIterator) init() {
if ii.highlowcontainer.size() > ii.pos {
ii.iter = ii.highlowcontainer.getContainerAtIndex(ii.pos).getShortIterator()
ii.hs = uint32(ii.highlowcontainer.getKeyAtIndex(ii.pos)) << 16
c := ii.highlowcontainer.getContainerAtIndex(ii.pos)
switch t := c.(type) {
case *arrayContainer:
ii.shortIter = shortIterator{t.content, 0}
ii.iter = &ii.shortIter
case *runContainer16:
ii.runIter = runIterator16{rc: t, curIndex: 0, curPosInIndex: 0}
ii.iter = &ii.runIter
case *bitmapContainer:
ii.bitmapIter = bitmapContainerShortIterator{t, t.NextSetBit(0)}
ii.iter = &ii.bitmapIter
}
}
}
@ -279,14 +321,14 @@ func (ii *intIterator) PeekNext() uint32 {
// AdvanceIfNeeded advances as long as the next value is smaller than minval
func (ii *intIterator) AdvanceIfNeeded(minval uint32) {
to := minval >> 16
to := minval & 0xffff0000
for ii.HasNext() && (ii.hs>>16) < to {
for ii.HasNext() && ii.hs < to {
ii.pos++
ii.init()
}
if ii.HasNext() && (ii.hs>>16) == to {
if ii.HasNext() && ii.hs == to {
ii.iter.advanceIfNeeded(lowbits(minval))
if !ii.iter.hasNext() {
@ -296,12 +338,17 @@ func (ii *intIterator) AdvanceIfNeeded(minval uint32) {
}
}
func newIntIterator(a *Bitmap) *intIterator {
p := new(intIterator)
// IntIterator is meant to allow you to iterate through the values of a bitmap, see Initialize(a *Bitmap)
type IntIterator = intIterator
// Initialize configures the existing iterator so that it can iterate through the values of
// the provided bitmap.
// The iteration results are undefined if the bitmap is modified (e.g., with Add or Remove).
func (p *intIterator) Initialize(a *Bitmap) {
p.pos = 0
p.highlowcontainer = &a.highlowcontainer
p.init()
return p
}
type intReverseIterator struct {
@ -309,6 +356,10 @@ type intReverseIterator struct {
hs uint32
iter shortIterable
highlowcontainer *roaringArray
shortIter reverseIterator
runIter runReverseIterator16
bitmapIter reverseBitmapContainerShortIterator
}
// HasNext returns true if there are more integers to iterate over
@ -318,8 +369,30 @@ func (ii *intReverseIterator) HasNext() bool {
func (ii *intReverseIterator) init() {
if ii.pos >= 0 {
ii.iter = ii.highlowcontainer.getContainerAtIndex(ii.pos).getReverseIterator()
ii.hs = uint32(ii.highlowcontainer.getKeyAtIndex(ii.pos)) << 16
c := ii.highlowcontainer.getContainerAtIndex(ii.pos)
switch t := c.(type) {
case *arrayContainer:
ii.shortIter = reverseIterator{t.content, len(t.content) - 1}
ii.iter = &ii.shortIter
case *runContainer16:
index := int(len(t.iv)) - 1
pos := uint16(0)
if index >= 0 {
pos = t.iv[index].length
}
ii.runIter = runReverseIterator16{rc: t, curIndex: index, curPosInIndex: pos}
ii.iter = &ii.runIter
case *bitmapContainer:
pos := -1
if t.cardinality > 0 {
pos = int(t.maximum())
}
ii.bitmapIter = reverseBitmapContainerShortIterator{t, pos}
ii.iter = &ii.bitmapIter
}
} else {
ii.iter = nil
}
@ -335,12 +408,16 @@ func (ii *intReverseIterator) Next() uint32 {
return x
}
func newIntReverseIterator(a *Bitmap) *intReverseIterator {
p := new(intReverseIterator)
// IntReverseIterator is meant to allow you to iterate through the values of a bitmap, see Initialize(a *Bitmap)
type IntReverseIterator = intReverseIterator
// Initialize configures the existing iterator so that it can iterate through the values of
// the provided bitmap.
// The iteration results are undefined if the bitmap is modified (e.g., with Add or Remove).
func (p *intReverseIterator) Initialize(a *Bitmap) {
p.highlowcontainer = &a.highlowcontainer
p.pos = a.highlowcontainer.size() - 1
p.init()
return p
}
// ManyIntIterable allows you to iterate over the values in a Bitmap
@ -356,12 +433,27 @@ type manyIntIterator struct {
hs uint32
iter manyIterable
highlowcontainer *roaringArray
shortIter shortIterator
runIter runIterator16
bitmapIter bitmapContainerManyIterator
}
func (ii *manyIntIterator) init() {
if ii.highlowcontainer.size() > ii.pos {
ii.iter = ii.highlowcontainer.getContainerAtIndex(ii.pos).getManyIterator()
ii.hs = uint32(ii.highlowcontainer.getKeyAtIndex(ii.pos)) << 16
c := ii.highlowcontainer.getContainerAtIndex(ii.pos)
switch t := c.(type) {
case *arrayContainer:
ii.shortIter = shortIterator{t.content, 0}
ii.iter = &ii.shortIter
case *runContainer16:
ii.runIter = runIterator16{rc: t, curIndex: 0, curPosInIndex: 0}
ii.iter = &ii.runIter
case *bitmapContainer:
ii.bitmapIter = bitmapContainerManyIterator{t, -1, 0}
ii.iter = &ii.bitmapIter
}
} else {
ii.iter = nil
}
@ -403,12 +495,17 @@ func (ii *manyIntIterator) NextMany64(hs64 uint64, buf []uint64) int {
return n
}
func newManyIntIterator(a *Bitmap) *manyIntIterator {
p := new(manyIntIterator)
// ManyIntIterator is meant to allow you to iterate through the values of a bitmap, see Initialize(a *Bitmap)
type ManyIntIterator = manyIntIterator
// Initialize configures the existing iterator so that it can iterate through the values of
// the provided bitmap.
// The iteration results are undefined if the bitmap is modified (e.g., with Add or Remove).
func (p *manyIntIterator) Initialize(a *Bitmap) {
p.pos = 0
p.highlowcontainer = &a.highlowcontainer
p.init()
return p
}
// String creates a string representation of the Bitmap
@ -440,7 +537,7 @@ func (rb *Bitmap) String() string {
// Iterate iterates over the bitmap, calling the given callback with each value in the bitmap. If the callback returns
// false, the iteration is halted.
// The iteration results are undefined if the bitmap is modified (e.g., with Add or Remove).
// There is no guarantee as to what order the values will be iterated
// There is no guarantee as to what order the values will be iterated.
func (rb *Bitmap) Iterate(cb func(x uint32) bool) {
for i := 0; i < rb.highlowcontainer.size(); i++ {
hs := uint32(rb.highlowcontainer.getKeyAtIndex(i)) << 16
@ -472,19 +569,25 @@ func (rb *Bitmap) Iterate(cb func(x uint32) bool) {
// Iterator creates a new IntPeekable to iterate over the integers contained in the bitmap, in sorted order;
// the iterator becomes invalid if the bitmap is modified (e.g., with Add or Remove).
func (rb *Bitmap) Iterator() IntPeekable {
return newIntIterator(rb)
p := new(intIterator)
p.Initialize(rb)
return p
}
// ReverseIterator creates a new IntIterable to iterate over the integers contained in the bitmap, in sorted order;
// the iterator becomes invalid if the bitmap is modified (e.g., with Add or Remove).
func (rb *Bitmap) ReverseIterator() IntIterable {
return newIntReverseIterator(rb)
p := new(intReverseIterator)
p.Initialize(rb)
return p
}
// ManyIterator creates a new ManyIntIterable to iterate over the integers contained in the bitmap, in sorted order;
// the iterator becomes invalid if the bitmap is modified (e.g., with Add or Remove).
func (rb *Bitmap) ManyIterator() ManyIntIterable {
return newManyIntIterator(rb)
p := new(manyIntIterator)
p.Initialize(rb)
return p
}
// Clone creates a copy of the Bitmap
@ -496,11 +599,17 @@ func (rb *Bitmap) Clone() *Bitmap {
// Minimum get the smallest value stored in this roaring bitmap, assumes that it is not empty
func (rb *Bitmap) Minimum() uint32 {
if len(rb.highlowcontainer.containers) == 0 {
panic("Empty bitmap")
}
return uint32(rb.highlowcontainer.containers[0].minimum()) | (uint32(rb.highlowcontainer.keys[0]) << 16)
}
// Maximum get the largest value stored in this roaring bitmap, assumes that it is not empty
func (rb *Bitmap) Maximum() uint32 {
if len(rb.highlowcontainer.containers) == 0 {
panic("Empty bitmap")
}
lastindex := len(rb.highlowcontainer.containers) - 1
return uint32(rb.highlowcontainer.containers[lastindex].maximum()) | (uint32(rb.highlowcontainer.keys[lastindex]) << 16)
}
@ -544,34 +653,38 @@ func AddOffset64(x *Bitmap, offset int64) (answer *Bitmap) {
containerOffset64 = offset >> 16
}
if containerOffset64 >= (1<<16) || containerOffset64 <= -(1<<16) {
return New()
answer = New()
if containerOffset64 >= (1<<16) || containerOffset64 < -(1<<16) {
return answer
}
containerOffset := int32(containerOffset64)
inOffset := (uint16)(offset - containerOffset64*(1<<16))
if inOffset == 0 {
answer = x.Clone()
for pos := 0; pos < answer.highlowcontainer.size(); pos++ {
key := int32(answer.highlowcontainer.getKeyAtIndex(pos))
key += containerOffset
if key >= 0 && key <= MaxUint16 {
answer.highlowcontainer.keys[pos] = uint16(key)
}
}
} else {
answer = New()
for pos := 0; pos < x.highlowcontainer.size(); pos++ {
key := int32(x.highlowcontainer.getKeyAtIndex(pos))
key += containerOffset
c := x.highlowcontainer.getContainerAtIndex(pos)
offsetted := c.addOffset(inOffset)
if key >= 0 && key <= MaxUint16 {
c := x.highlowcontainer.getContainerAtIndex(pos).clone()
answer.highlowcontainer.appendContainer(uint16(key), c, false)
}
}
} else {
for pos := 0; pos < x.highlowcontainer.size(); pos++ {
key := int32(x.highlowcontainer.getKeyAtIndex(pos))
key += containerOffset
if offsetted[0].getCardinality() > 0 && (key >= 0 && key <= MaxUint16) {
if key+1 < 0 || key > MaxUint16 {
continue
}
c := x.highlowcontainer.getContainerAtIndex(pos)
lo, hi := c.addOffset(inOffset)
if lo != nil && key >= 0 {
curSize := answer.highlowcontainer.size()
lastkey := int32(0)
@ -581,15 +694,15 @@ func AddOffset64(x *Bitmap, offset int64) (answer *Bitmap) {
if curSize > 0 && lastkey == key {
prev := answer.highlowcontainer.getContainerAtIndex(curSize - 1)
orrseult := prev.ior(offsetted[0])
answer.highlowcontainer.setContainerAtIndex(curSize-1, orrseult)
orresult := prev.ior(lo)
answer.highlowcontainer.setContainerAtIndex(curSize-1, orresult)
} else {
answer.highlowcontainer.appendContainer(uint16(key), offsetted[0], false)
answer.highlowcontainer.appendContainer(uint16(key), lo, false)
}
}
if offsetted[1].getCardinality() > 0 && ((key+1) >= 0 && (key+1) <= MaxUint16) {
answer.highlowcontainer.appendContainer(uint16(key+1), offsetted[1], false)
if hi != nil && key+1 <= MaxUint16 {
answer.highlowcontainer.appendContainer(uint16(key+1), hi, false)
}
}
}
@ -659,13 +772,13 @@ func (rb *Bitmap) Remove(x uint32) {
if i >= 0 {
c := rb.highlowcontainer.getWritableContainerAtIndex(i).iremoveReturnMinimized(lowbits(x))
rb.highlowcontainer.setContainerAtIndex(i, c)
if rb.highlowcontainer.getContainerAtIndex(i).getCardinality() == 0 {
if rb.highlowcontainer.getContainerAtIndex(i).isEmpty() {
rb.highlowcontainer.removeAtIndex(i)
}
}
}
// CheckedRemove removes the integer x from the bitmap and return true if the integer was effectively remove (and false if the integer was not present)
// CheckedRemove removes the integer x from the bitmap and return true if the integer was effectively removed (and false if the integer was not present)
func (rb *Bitmap) CheckedRemove(x uint32) bool {
// TODO: add unit tests for this method
hb := highbits(x)
@ -675,7 +788,7 @@ func (rb *Bitmap) CheckedRemove(x uint32) bool {
oldcard := C.getCardinality()
C = C.iremoveReturnMinimized(lowbits(x))
rb.highlowcontainer.setContainerAtIndex(i, C)
if rb.highlowcontainer.getContainerAtIndex(i).getCardinality() == 0 {
if rb.highlowcontainer.getContainerAtIndex(i).isEmpty() {
rb.highlowcontainer.removeAtIndex(i)
return true
}
@ -723,15 +836,12 @@ func (rb *Bitmap) Rank(x uint32) uint64 {
// the smallest element. Note that this function differs in convention from
// the Rank function which returns 1 on the smallest value.
func (rb *Bitmap) Select(x uint32) (uint32, error) {
if rb.GetCardinality() <= uint64(x) {
return 0, fmt.Errorf("can't find %dth integer in a bitmap with only %d items", x, rb.GetCardinality())
}
remaining := x
for i := 0; i < rb.highlowcontainer.size(); i++ {
c := rb.highlowcontainer.getContainerAtIndex(i)
if remaining >= uint32(c.getCardinality()) {
remaining -= uint32(c.getCardinality())
card := uint32(c.getCardinality())
if remaining >= card {
remaining -= card
} else {
key := rb.highlowcontainer.getKeyAtIndex(i)
return uint32(key)<<16 + uint32(c.selectInt(uint16(remaining))), nil
@ -758,7 +868,7 @@ main:
c1 := rb.highlowcontainer.getWritableContainerAtIndex(pos1)
c2 := x2.highlowcontainer.getContainerAtIndex(pos2)
diff := c1.iand(c2)
if diff.getCardinality() > 0 {
if !diff.isEmpty() {
rb.highlowcontainer.replaceKeyAndContainerAtIndex(intersectionsize, s1, diff, false)
intersectionsize++
}
@ -889,6 +999,28 @@ main:
return answer
}
// IntersectsWithInterval checks whether a bitmap 'rb' and an open interval '[x,y)' intersect.
func (rb *Bitmap) IntersectsWithInterval(x, y uint64) bool {
if x >= y {
return false
}
if x > MaxUint32 {
return false
}
it := intIterator{}
it.Initialize(rb)
it.AdvanceIfNeeded(uint32(x))
if !it.HasNext() {
return false
}
if uint64(it.Next()) >= y {
return false
}
return true
}
// Intersects checks whether two bitmap intersects, bitmaps are not modified
func (rb *Bitmap) Intersects(x2 *Bitmap) bool {
pos1 := 0
@ -960,7 +1092,7 @@ func (rb *Bitmap) Xor(x2 *Bitmap) {
} else {
// TODO: couple be computed in-place for reduced memory usage
c := rb.highlowcontainer.getContainerAtIndex(pos1).xor(x2.highlowcontainer.getContainerAtIndex(pos2))
if c.getCardinality() > 0 {
if !c.isEmpty() {
rb.highlowcontainer.setContainerAtIndex(pos1, c)
pos1++
} else {
@ -1040,7 +1172,7 @@ main:
c1 := rb.highlowcontainer.getWritableContainerAtIndex(pos1)
c2 := x2.highlowcontainer.getContainerAtIndex(pos2)
diff := c1.iandNot(c2)
if diff.getCardinality() > 0 {
if !diff.isEmpty() {
rb.highlowcontainer.replaceKeyAndContainerAtIndex(intersectionsize, s1, diff, false)
intersectionsize++
}
@ -1149,7 +1281,7 @@ main:
C := x1.highlowcontainer.getContainerAtIndex(pos1)
C = C.and(x2.highlowcontainer.getContainerAtIndex(pos2))
if C.getCardinality() > 0 {
if !C.isEmpty() {
answer.highlowcontainer.appendContainer(s1, C, false)
}
pos1++
@ -1196,7 +1328,7 @@ func Xor(x1, x2 *Bitmap) *Bitmap {
pos2++
} else {
c := x1.highlowcontainer.getContainerAtIndex(pos1).xor(x2.highlowcontainer.getContainerAtIndex(pos2))
if c.getCardinality() > 0 {
if !c.isEmpty() {
answer.highlowcontainer.appendContainer(s1, c, false)
}
pos1++
@ -1239,7 +1371,7 @@ main:
c1 := x1.highlowcontainer.getContainerAtIndex(pos1)
c2 := x2.highlowcontainer.getContainerAtIndex(pos2)
diff := c1.andNot(c2)
if diff.getCardinality() > 0 {
if !diff.isEmpty() {
answer.highlowcontainer.appendContainer(s1, diff, false)
}
pos1++
@ -1329,7 +1461,7 @@ func (rb *Bitmap) Flip(rangeStart, rangeEnd uint64) {
if i >= 0 {
c := rb.highlowcontainer.getWritableContainerAtIndex(i).inot(int(containerStart), int(containerLast)+1)
if c.getCardinality() > 0 {
if !c.isEmpty() {
rb.highlowcontainer.setContainerAtIndex(i, c)
} else {
rb.highlowcontainer.removeAtIndex(i)
@ -1410,7 +1542,7 @@ func (rb *Bitmap) RemoveRange(rangeStart, rangeEnd uint64) {
return
}
c := rb.highlowcontainer.getWritableContainerAtIndex(i).iremoveRange(int(lbStart), int(lbLast+1))
if c.getCardinality() > 0 {
if !c.isEmpty() {
rb.highlowcontainer.setContainerAtIndex(i, c)
} else {
rb.highlowcontainer.removeAtIndex(i)
@ -1423,7 +1555,7 @@ func (rb *Bitmap) RemoveRange(rangeStart, rangeEnd uint64) {
if ifirst >= 0 {
if lbStart != 0 {
c := rb.highlowcontainer.getWritableContainerAtIndex(ifirst).iremoveRange(int(lbStart), int(max+1))
if c.getCardinality() > 0 {
if !c.isEmpty() {
rb.highlowcontainer.setContainerAtIndex(ifirst, c)
ifirst++
}
@ -1434,7 +1566,7 @@ func (rb *Bitmap) RemoveRange(rangeStart, rangeEnd uint64) {
if ilast >= 0 {
if lbLast != max {
c := rb.highlowcontainer.getWritableContainerAtIndex(ilast).iremoveRange(int(0), int(lbLast+1))
if c.getCardinality() > 0 {
if !c.isEmpty() {
rb.highlowcontainer.setContainerAtIndex(ilast, c)
} else {
ilast++
@ -1490,7 +1622,7 @@ func Flip(bm *Bitmap, rangeStart, rangeEnd uint64) *Bitmap {
if i >= 0 {
c := bm.highlowcontainer.getContainerAtIndex(i).not(int(containerStart), int(containerLast)+1)
if c.getCardinality() > 0 {
if !c.isEmpty() {
answer.highlowcontainer.insertNewKeyValueAt(-j-1, uint16(hb), c)
}

165
vendor/github.com/RoaringBitmap/roaring/roaringarray.go generated vendored
View file

@ -4,16 +4,15 @@ import (
"bytes"
"encoding/binary"
"fmt"
"github.com/RoaringBitmap/roaring/internal"
"io"
snappy "github.com/glycerine/go-unsnap-stream"
"github.com/tinylib/msgp/msgp"
)
//go:generate msgp -unexported
type container interface {
addOffset(uint16) []container
// addOffset returns the (low, high) parts of the shifted container.
// Whenever one of them would be empty, nil will be returned instead to
// avoid unnecessary allocations.
addOffset(uint16) (container, container)
clone() container
and(container) container
@ -21,6 +20,7 @@ type container interface {
iand(container) container // i stands for inplace
andNot(container) container
iandNot(container) container // i stands for inplace
isEmpty() bool
getCardinality() int
// rank returns the number of integers that are
// smaller or equal to x. rank(infinity) would be getCardinality().
@ -51,7 +51,7 @@ type container interface {
// any of the implementations.
equals(r container) bool
fillLeastSignificant16bits(array []uint32, i int, mask uint32)
fillLeastSignificant16bits(array []uint32, i int, mask uint32) int
or(r container) container
orCardinality(r container) int
isFull() bool
@ -103,18 +103,6 @@ type roaringArray struct {
containers []container `msg:"-"` // don't try to serialize directly.
needCopyOnWrite []bool
copyOnWrite bool
// conserz is used at serialization time
// to serialize containers. Otherwise empty.
conserz []containerSerz
}
// containerSerz facilitates serializing container (tricky to
// serialize because it is an interface) by providing a
// light wrapper with a type identifier.
type containerSerz struct {
t contype `msg:"t"` // type
r msgp.Raw `msg:"r"` // Raw msgpack of the actual container type
}
func newRoaringArray() *roaringArray {
@ -246,7 +234,6 @@ func (ra *roaringArray) resize(newsize int) {
func (ra *roaringArray) clear() {
ra.resize(0)
ra.copyOnWrite = false
ra.conserz = nil
}
func (ra *roaringArray) clone() *roaringArray {
@ -566,51 +553,58 @@ func (ra *roaringArray) toBytes() ([]byte, error) {
return buf.Bytes(), err
}
func (ra *roaringArray) readFrom(stream byteInput) (int64, error) {
cookie, err := stream.readUInt32()
if err != nil {
return stream.getReadBytes(), fmt.Errorf("error in roaringArray.readFrom: could not read initial cookie: %s", err)
func (ra *roaringArray) readFrom(stream internal.ByteInput, cookieHeader ...byte) (int64, error) {
var cookie uint32
var err error
if len(cookieHeader) > 0 && len(cookieHeader) != 4 {
return int64(len(cookieHeader)), fmt.Errorf("error in roaringArray.readFrom: could not read initial cookie: incorrect size of cookie header")
}
if len(cookieHeader) == 4 {
cookie = binary.LittleEndian.Uint32(cookieHeader)
} else {
cookie, err = stream.ReadUInt32()
if err != nil {
return stream.GetReadBytes(), fmt.Errorf("error in roaringArray.readFrom: could not read initial cookie: %s", err)
}
}
var size uint32
var isRunBitmap []byte
if cookie&0x0000FFFF == serialCookie {
size = uint32(uint16(cookie>>16) + 1)
size = uint32(cookie>>16 + 1)
// create is-run-container bitmap
isRunBitmapSize := (int(size) + 7) / 8
isRunBitmap, err = stream.next(isRunBitmapSize)
isRunBitmap, err = stream.Next(isRunBitmapSize)
if err != nil {
return stream.getReadBytes(), fmt.Errorf("malformed bitmap, failed to read is-run bitmap, got: %s", err)
return stream.GetReadBytes(), fmt.Errorf("malformed bitmap, failed to read is-run bitmap, got: %s", err)
}
} else if cookie == serialCookieNoRunContainer {
size, err = stream.readUInt32()
size, err = stream.ReadUInt32()
if err != nil {
return stream.getReadBytes(), fmt.Errorf("malformed bitmap, failed to read a bitmap size: %s", err)
return stream.GetReadBytes(), fmt.Errorf("malformed bitmap, failed to read a bitmap size: %s", err)
}
} else {
return stream.getReadBytes(), fmt.Errorf("error in roaringArray.readFrom: did not find expected serialCookie in header")
return stream.GetReadBytes(), fmt.Errorf("error in roaringArray.readFrom: did not find expected serialCookie in header")
}
if size > (1 << 16) {
return stream.getReadBytes(), fmt.Errorf("it is logically impossible to have more than (1<<16) containers")
return stream.GetReadBytes(), fmt.Errorf("it is logically impossible to have more than (1<<16) containers")
}
// descriptive header
buf, err := stream.next(2 * 2 * int(size))
buf, err := stream.Next(2 * 2 * int(size))
if err != nil {
return stream.getReadBytes(), fmt.Errorf("failed to read descriptive header: %s", err)
return stream.GetReadBytes(), fmt.Errorf("failed to read descriptive header: %s", err)
}
keycard := byteSliceAsUint16Slice(buf)
if isRunBitmap == nil || size >= noOffsetThreshold {
if err := stream.skipBytes(int(size) * 4); err != nil {
return stream.getReadBytes(), fmt.Errorf("failed to skip bytes: %s", err)
if err := stream.SkipBytes(int(size) * 4); err != nil {
return stream.GetReadBytes(), fmt.Errorf("failed to skip bytes: %s", err)
}
}
@ -641,30 +635,29 @@ func (ra *roaringArray) readFrom(stream byteInput) (int64, error) {
if isRunBitmap != nil && isRunBitmap[i/8]&(1<<(i%8)) != 0 {
// run container
nr, err := stream.readUInt16()
nr, err := stream.ReadUInt16()
if err != nil {
return 0, fmt.Errorf("failed to read runtime container size: %s", err)
}
buf, err := stream.next(int(nr) * 4)
buf, err := stream.Next(int(nr) * 4)
if err != nil {
return stream.getReadBytes(), fmt.Errorf("failed to read runtime container content: %s", err)
return stream.GetReadBytes(), fmt.Errorf("failed to read runtime container content: %s", err)
}
nb := runContainer16{
iv: byteSliceAsInterval16Slice(buf),
card: int64(card),
iv: byteSliceAsInterval16Slice(buf),
}
ra.containers[i] = &nb
} else if card > arrayDefaultMaxSize {
// bitmap container
buf, err := stream.next(arrayDefaultMaxSize * 2)
buf, err := stream.Next(arrayDefaultMaxSize * 2)
if err != nil {
return stream.getReadBytes(), fmt.Errorf("failed to read bitmap container: %s", err)
return stream.GetReadBytes(), fmt.Errorf("failed to read bitmap container: %s", err)
}
nb := bitmapContainer{
@ -675,10 +668,10 @@ func (ra *roaringArray) readFrom(stream byteInput) (int64, error) {
ra.containers[i] = &nb
} else {
// array container
buf, err := stream.next(card * 2)
buf, err := stream.Next(card * 2)
if err != nil {
return stream.getReadBytes(), fmt.Errorf("failed to read array container: %s", err)
return stream.GetReadBytes(), fmt.Errorf("failed to read array container: %s", err)
}
nb := arrayContainer{
@ -689,7 +682,7 @@ func (ra *roaringArray) readFrom(stream byteInput) (int64, error) {
}
}
return stream.getReadBytes(), nil
return stream.GetReadBytes(), nil
}
func (ra *roaringArray) hasRunCompression() bool {
@ -702,84 +695,6 @@ func (ra *roaringArray) hasRunCompression() bool {
return false
}
func (ra *roaringArray) writeToMsgpack(stream io.Writer) error {
ra.conserz = make([]containerSerz, len(ra.containers))
for i, v := range ra.containers {
switch cn := v.(type) {
case *bitmapContainer:
bts, err := cn.MarshalMsg(nil)
if err != nil {
return err
}
ra.conserz[i].t = bitmapContype
ra.conserz[i].r = bts
case *arrayContainer:
bts, err := cn.MarshalMsg(nil)
if err != nil {
return err
}
ra.conserz[i].t = arrayContype
ra.conserz[i].r = bts
case *runContainer16:
bts, err := cn.MarshalMsg(nil)
if err != nil {
return err
}
ra.conserz[i].t = run16Contype
ra.conserz[i].r = bts
default:
panic(fmt.Errorf("Unrecognized container implementation: %T", cn))
}
}
w := snappy.NewWriter(stream)
err := msgp.Encode(w, ra)
ra.conserz = nil
return err
}
func (ra *roaringArray) readFromMsgpack(stream io.Reader) error {
r := snappy.NewReader(stream)
err := msgp.Decode(r, ra)
if err != nil {
return err
}
if len(ra.containers) != len(ra.keys) {
ra.containers = make([]container, len(ra.keys))
}
for i, v := range ra.conserz {
switch v.t {
case bitmapContype:
c := &bitmapContainer{}
_, err = c.UnmarshalMsg(v.r)
if err != nil {
return err
}
ra.containers[i] = c
case arrayContype:
c := &arrayContainer{}
_, err = c.UnmarshalMsg(v.r)
if err != nil {
return err
}
ra.containers[i] = c
case run16Contype:
c := &runContainer16{}
_, err = c.UnmarshalMsg(v.r)
if err != nil {
return err
}
ra.containers[i] = c
default:
return fmt.Errorf("unrecognized contype serialization code: '%v'", v.t)
}
}
ra.conserz = nil
return nil
}
func (ra *roaringArray) advanceUntil(min uint16, pos int) int {
lower := pos + 1

529
vendor/github.com/RoaringBitmap/roaring/roaringarray_gen.go generated vendored
View file

@ -1,529 +0,0 @@
package roaring
// NOTE: THIS FILE WAS PRODUCED BY THE
// MSGP CODE GENERATION TOOL (github.com/tinylib/msgp)
// DO NOT EDIT
import (
"github.com/tinylib/msgp/msgp"
)
// Deprecated: DecodeMsg implements msgp.Decodable
func (z *containerSerz) DecodeMsg(dc *msgp.Reader) (err error) {
var field []byte
_ = field
var zxvk uint32
zxvk, err = dc.ReadMapHeader()
if err != nil {
return
}
for zxvk > 0 {
zxvk--
field, err = dc.ReadMapKeyPtr()
if err != nil {
return
}
switch msgp.UnsafeString(field) {
case "t":
{
var zbzg uint8
zbzg, err = dc.ReadUint8()
z.t = contype(zbzg)
}
if err != nil {
return
}
case "r":
err = z.r.DecodeMsg(dc)
if err != nil {
return
}
default:
err = dc.Skip()
if err != nil {
return
}
}
}
return
}
// Deprecated: EncodeMsg implements msgp.Encodable
func (z *containerSerz) EncodeMsg(en *msgp.Writer) (err error) {
// map header, size 2
// write "t"
err = en.Append(0x82, 0xa1, 0x74)
if err != nil {
return err
}
err = en.WriteUint8(uint8(z.t))
if err != nil {
return
}
// write "r"
err = en.Append(0xa1, 0x72)
if err != nil {
return err
}
err = z.r.EncodeMsg(en)
if err != nil {
return
}
return
}
// Deprecated: MarshalMsg implements msgp.Marshaler
func (z *containerSerz) MarshalMsg(b []byte) (o []byte, err error) {
o = msgp.Require(b, z.Msgsize())
// map header, size 2
// string "t"
o = append(o, 0x82, 0xa1, 0x74)
o = msgp.AppendUint8(o, uint8(z.t))
// string "r"
o = append(o, 0xa1, 0x72)
o, err = z.r.MarshalMsg(o)
if err != nil {
return
}
return
}
// Deprecated: UnmarshalMsg implements msgp.Unmarshaler
func (z *containerSerz) UnmarshalMsg(bts []byte) (o []byte, err error) {
var field []byte
_ = field
var zbai uint32
zbai, bts, err = msgp.ReadMapHeaderBytes(bts)
if err != nil {
return
}
for zbai > 0 {
zbai--
field, bts, err = msgp.ReadMapKeyZC(bts)
if err != nil {
return
}
switch msgp.UnsafeString(field) {
case "t":
{
var zcmr uint8
zcmr, bts, err = msgp.ReadUint8Bytes(bts)
z.t = contype(zcmr)
}
if err != nil {
return
}
case "r":
bts, err = z.r.UnmarshalMsg(bts)
if err != nil {
return
}
default:
bts, err = msgp.Skip(bts)
if err != nil {
return
}
}
}
o = bts
return
}
// Deprecated: Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message
func (z *containerSerz) Msgsize() (s int) {
s = 1 + 2 + msgp.Uint8Size + 2 + z.r.Msgsize()
return
}
// Deprecated: DecodeMsg implements msgp.Decodable
func (z *contype) DecodeMsg(dc *msgp.Reader) (err error) {
{
var zajw uint8
zajw, err = dc.ReadUint8()
(*z) = contype(zajw)
}
if err != nil {
return
}
return
}
// Deprecated: EncodeMsg implements msgp.Encodable
func (z contype) EncodeMsg(en *msgp.Writer) (err error) {
err = en.WriteUint8(uint8(z))
if err != nil {
return
}
return
}
// Deprecated: MarshalMsg implements msgp.Marshaler
func (z contype) MarshalMsg(b []byte) (o []byte, err error) {
o = msgp.Require(b, z.Msgsize())
o = msgp.AppendUint8(o, uint8(z))
return
}
// Deprecated: UnmarshalMsg implements msgp.Unmarshaler
func (z *contype) UnmarshalMsg(bts []byte) (o []byte, err error) {
{
var zwht uint8
zwht, bts, err = msgp.ReadUint8Bytes(bts)
(*z) = contype(zwht)
}
if err != nil {
return
}
o = bts
return
}
// Deprecated: Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message
func (z contype) Msgsize() (s int) {
s = msgp.Uint8Size
return
}
// Deprecated: DecodeMsg implements msgp.Decodable
func (z *roaringArray) DecodeMsg(dc *msgp.Reader) (err error) {
var field []byte
_ = field
var zlqf uint32
zlqf, err = dc.ReadMapHeader()
if err != nil {
return
}
for zlqf > 0 {
zlqf--
field, err = dc.ReadMapKeyPtr()
if err != nil {
return
}
switch msgp.UnsafeString(field) {
case "keys":
var zdaf uint32
zdaf, err = dc.ReadArrayHeader()
if err != nil {
return
}
if cap(z.keys) >= int(zdaf) {
z.keys = (z.keys)[:zdaf]
} else {
z.keys = make([]uint16, zdaf)
}
for zhct := range z.keys {
z.keys[zhct], err = dc.ReadUint16()
if err != nil {
return
}
}
case "needCopyOnWrite":
var zpks uint32
zpks, err = dc.ReadArrayHeader()
if err != nil {
return
}
if cap(z.needCopyOnWrite) >= int(zpks) {
z.needCopyOnWrite = (z.needCopyOnWrite)[:zpks]
} else {
z.needCopyOnWrite = make([]bool, zpks)
}
for zcua := range z.needCopyOnWrite {
z.needCopyOnWrite[zcua], err = dc.ReadBool()
if err != nil {
return
}
}
case "copyOnWrite":
z.copyOnWrite, err = dc.ReadBool()
if err != nil {
return
}
case "conserz":
var zjfb uint32
zjfb, err = dc.ReadArrayHeader()
if err != nil {
return
}
if cap(z.conserz) >= int(zjfb) {
z.conserz = (z.conserz)[:zjfb]
} else {
z.conserz = make([]containerSerz, zjfb)
}
for zxhx := range z.conserz {
var zcxo uint32
zcxo, err = dc.ReadMapHeader()
if err != nil {
return
}
for zcxo > 0 {
zcxo--
field, err = dc.ReadMapKeyPtr()
if err != nil {
return
}
switch msgp.UnsafeString(field) {
case "t":
{
var zeff uint8
zeff, err = dc.ReadUint8()
z.conserz[zxhx].t = contype(zeff)
}
if err != nil {
return
}
case "r":
err = z.conserz[zxhx].r.DecodeMsg(dc)
if err != nil {
return
}
default:
err = dc.Skip()
if err != nil {
return
}
}
}
}
default:
err = dc.Skip()
if err != nil {
return
}
}
}
return
}
// Deprecated: EncodeMsg implements msgp.Encodable
func (z *roaringArray) EncodeMsg(en *msgp.Writer) (err error) {
// map header, size 4
// write "keys"
err = en.Append(0x84, 0xa4, 0x6b, 0x65, 0x79, 0x73)
if err != nil {
return err
}
err = en.WriteArrayHeader(uint32(len(z.keys)))
if err != nil {
return
}
for zhct := range z.keys {
err = en.WriteUint16(z.keys[zhct])
if err != nil {
return
}
}
// write "needCopyOnWrite"
err = en.Append(0xaf, 0x6e, 0x65, 0x65, 0x64, 0x43, 0x6f, 0x70, 0x79, 0x4f, 0x6e, 0x57, 0x72, 0x69, 0x74, 0x65)
if err != nil {
return err
}
err = en.WriteArrayHeader(uint32(len(z.needCopyOnWrite)))
if err != nil {
return
}
for zcua := range z.needCopyOnWrite {
err = en.WriteBool(z.needCopyOnWrite[zcua])
if err != nil {
return
}
}
// write "copyOnWrite"
err = en.Append(0xab, 0x63, 0x6f, 0x70, 0x79, 0x4f, 0x6e, 0x57, 0x72, 0x69, 0x74, 0x65)
if err != nil {
return err
}
err = en.WriteBool(z.copyOnWrite)
if err != nil {
return
}
// write "conserz"
err = en.Append(0xa7, 0x63, 0x6f, 0x6e, 0x73, 0x65, 0x72, 0x7a)
if err != nil {
return err
}
err = en.WriteArrayHeader(uint32(len(z.conserz)))
if err != nil {
return
}
for zxhx := range z.conserz {
// map header, size 2
// write "t"
err = en.Append(0x82, 0xa1, 0x74)
if err != nil {
return err
}
err = en.WriteUint8(uint8(z.conserz[zxhx].t))
if err != nil {
return
}
// write "r"
err = en.Append(0xa1, 0x72)
if err != nil {
return err
}
err = z.conserz[zxhx].r.EncodeMsg(en)
if err != nil {
return
}
}
return
}
// Deprecated: MarshalMsg implements msgp.Marshaler
func (z *roaringArray) MarshalMsg(b []byte) (o []byte, err error) {
o = msgp.Require(b, z.Msgsize())
// map header, size 4
// string "keys"
o = append(o, 0x84, 0xa4, 0x6b, 0x65, 0x79, 0x73)
o = msgp.AppendArrayHeader(o, uint32(len(z.keys)))
for zhct := range z.keys {
o = msgp.AppendUint16(o, z.keys[zhct])
}
// string "needCopyOnWrite"
o = append(o, 0xaf, 0x6e, 0x65, 0x65, 0x64, 0x43, 0x6f, 0x70, 0x79, 0x4f, 0x6e, 0x57, 0x72, 0x69, 0x74, 0x65)
o = msgp.AppendArrayHeader(o, uint32(len(z.needCopyOnWrite)))
for zcua := range z.needCopyOnWrite {
o = msgp.AppendBool(o, z.needCopyOnWrite[zcua])
}
// string "copyOnWrite"
o = append(o, 0xab, 0x63, 0x6f, 0x70, 0x79, 0x4f, 0x6e, 0x57, 0x72, 0x69, 0x74, 0x65)
o = msgp.AppendBool(o, z.copyOnWrite)
// string "conserz"
o = append(o, 0xa7, 0x63, 0x6f, 0x6e, 0x73, 0x65, 0x72, 0x7a)
o = msgp.AppendArrayHeader(o, uint32(len(z.conserz)))
for zxhx := range z.conserz {
// map header, size 2
// string "t"
o = append(o, 0x82, 0xa1, 0x74)
o = msgp.AppendUint8(o, uint8(z.conserz[zxhx].t))
// string "r"
o = append(o, 0xa1, 0x72)
o, err = z.conserz[zxhx].r.MarshalMsg(o)
if err != nil {
return
}
}
return
}
// Deprecated: UnmarshalMsg implements msgp.Unmarshaler
func (z *roaringArray) UnmarshalMsg(bts []byte) (o []byte, err error) {
var field []byte
_ = field
var zrsw uint32
zrsw, bts, err = msgp.ReadMapHeaderBytes(bts)
if err != nil {
return
}
for zrsw > 0 {
zrsw--
field, bts, err = msgp.ReadMapKeyZC(bts)
if err != nil {
return
}
switch msgp.UnsafeString(field) {
case "keys":
var zxpk uint32
zxpk, bts, err = msgp.ReadArrayHeaderBytes(bts)
if err != nil {
return
}
if cap(z.keys) >= int(zxpk) {
z.keys = (z.keys)[:zxpk]
} else {
z.keys = make([]uint16, zxpk)
}
for zhct := range z.keys {
z.keys[zhct], bts, err = msgp.ReadUint16Bytes(bts)
if err != nil {
return
}
}
case "needCopyOnWrite":
var zdnj uint32
zdnj, bts, err = msgp.ReadArrayHeaderBytes(bts)
if err != nil {
return
}
if cap(z.needCopyOnWrite) >= int(zdnj) {
z.needCopyOnWrite = (z.needCopyOnWrite)[:zdnj]
} else {
z.needCopyOnWrite = make([]bool, zdnj)
}
for zcua := range z.needCopyOnWrite {
z.needCopyOnWrite[zcua], bts, err = msgp.ReadBoolBytes(bts)
if err != nil {
return
}
}
case "copyOnWrite":
z.copyOnWrite, bts, err = msgp.ReadBoolBytes(bts)
if err != nil {
return
}
case "conserz":
var zobc uint32
zobc, bts, err = msgp.ReadArrayHeaderBytes(bts)
if err != nil {
return
}
if cap(z.conserz) >= int(zobc) {
z.conserz = (z.conserz)[:zobc]
} else {
z.conserz = make([]containerSerz, zobc)
}
for zxhx := range z.conserz {
var zsnv uint32
zsnv, bts, err = msgp.ReadMapHeaderBytes(bts)
if err != nil {
return
}
for zsnv > 0 {
zsnv--
field, bts, err = msgp.ReadMapKeyZC(bts)
if err != nil {
return
}
switch msgp.UnsafeString(field) {
case "t":
{
var zkgt uint8
zkgt, bts, err = msgp.ReadUint8Bytes(bts)
z.conserz[zxhx].t = contype(zkgt)
}
if err != nil {
return
}
case "r":
bts, err = z.conserz[zxhx].r.UnmarshalMsg(bts)
if err != nil {
return
}
default:
bts, err = msgp.Skip(bts)
if err != nil {
return
}
}
}
}
default:
bts, err = msgp.Skip(bts)
if err != nil {
return
}
}
}
o = bts
return
}
// Deprecated: Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message
func (z *roaringArray) Msgsize() (s int) {
s = 1 + 5 + msgp.ArrayHeaderSize + (len(z.keys) * (msgp.Uint16Size)) + 16 + msgp.ArrayHeaderSize + (len(z.needCopyOnWrite) * (msgp.BoolSize)) + 12 + msgp.BoolSize + 8 + msgp.ArrayHeaderSize
for zxhx := range z.conserz {
s += 1 + 2 + msgp.Uint8Size + 2 + z.conserz[zxhx].r.Msgsize()
}
return
}

732
vendor/github.com/RoaringBitmap/roaring/runcontainer.go generated vendored
View file

File diff suppressed because it is too large Load diff

1104
vendor/github.com/RoaringBitmap/roaring/runcontainer_gen.go generated vendored
View file

File diff suppressed because it is too large Load diff

15
vendor/github.com/RoaringBitmap/roaring/serialization.go generated vendored
View file

@ -3,8 +3,6 @@ package roaring
import (
"encoding/binary"
"io"
"github.com/tinylib/msgp/msgp"
)
// writeTo for runContainer16 follows this
@ -19,16 +17,3 @@ func (b *runContainer16) writeTo(stream io.Writer) (int, error) {
}
return stream.Write(buf)
}
func (b *runContainer16) writeToMsgpack(stream io.Writer) (int, error) {
bts, err := b.MarshalMsg(nil)
if err != nil {
return 0, err
}
return stream.Write(bts)
}
func (b *runContainer16) readFromMsgpack(stream io.Reader) (int, error) {
err := msgp.Decode(stream, b)
return 0, err
}

12
vendor/github.com/RoaringBitmap/roaring/serialization_generic.go generated vendored
View file

@ -1,3 +1,4 @@
//go:build (!amd64 && !386 && !arm && !arm64 && !ppc64le && !mipsle && !mips64le && !mips64p32le && !wasm) || appengine
// +build !amd64,!386,!arm,!arm64,!ppc64le,!mipsle,!mips64le,!mips64p32le,!wasm appengine
package roaring
@ -84,6 +85,17 @@ func uint16SliceAsByteSlice(slice []uint16) []byte {
return by
}
func interval16SliceAsByteSlice(slice []interval16) []byte {
by := make([]byte, len(slice)*4)
for i, v := range slice {
binary.LittleEndian.PutUint16(by[i*2:], v.start)
binary.LittleEndian.PutUint16(by[i*2+2:], v.length)
}
return by
}
func byteSliceAsUint16Slice(slice []byte) []uint16 {
if len(slice)%2 != 0 {
panic("Slice size should be divisible by 2")

518
vendor/github.com/RoaringBitmap/roaring/serialization_littleendian.go generated vendored
View file

@ -1,8 +1,10 @@
//go:build (386 && !appengine) || (amd64 && !appengine) || (arm && !appengine) || (arm64 && !appengine) || (ppc64le && !appengine) || (mipsle && !appengine) || (mips64le && !appengine) || (mips64p32le && !appengine) || (wasm && !appengine)
// +build 386,!appengine amd64,!appengine arm,!appengine arm64,!appengine ppc64le,!appengine mipsle,!appengine mips64le,!appengine mips64p32le,!appengine wasm,!appengine
package roaring
import (
"encoding/binary"
"errors"
"io"
"reflect"
@ -55,6 +57,22 @@ func uint16SliceAsByteSlice(slice []uint16) []byte {
return result
}
func interval16SliceAsByteSlice(slice []interval16) []byte {
// make a new slice header
header := *(*reflect.SliceHeader)(unsafe.Pointer(&slice))
// update its capacity and length
header.Len *= 4
header.Cap *= 4
// instantiate result and use KeepAlive so data isn't unmapped.
result := *(*[]byte)(unsafe.Pointer(&header))
runtime.KeepAlive(&slice)
// return it
return result
}
func (bc *bitmapContainer) asLittleEndianByteSlice() []byte {
return uint64SliceAsByteSlice(bc.bitmap)
}
@ -132,3 +150,503 @@ func byteSliceAsInterval16Slice(slice []byte) (result []interval16) {
// return result
return
}
func byteSliceAsContainerSlice(slice []byte) (result []container) {
var c container
containerSize := int(unsafe.Sizeof(c))
if len(slice)%containerSize != 0 {
panic("Slice size should be divisible by unsafe.Sizeof(container)")
}
// reference: https://go101.org/article/unsafe.html
// make a new slice header
bHeader := (*reflect.SliceHeader)(unsafe.Pointer(&slice))
rHeader := (*reflect.SliceHeader)(unsafe.Pointer(&result))
// transfer the data from the given slice to a new variable (our result)
rHeader.Data = bHeader.Data
rHeader.Len = bHeader.Len / containerSize
rHeader.Cap = bHeader.Cap / containerSize
// instantiate result and use KeepAlive so data isn't unmapped.
runtime.KeepAlive(&slice) // it is still crucial, GC can free it)
// return result
return
}
func byteSliceAsBitsetSlice(slice []byte) (result []bitmapContainer) {
bitsetSize := int(unsafe.Sizeof(bitmapContainer{}))
if len(slice)%bitsetSize != 0 {
panic("Slice size should be divisible by unsafe.Sizeof(bitmapContainer)")
}
// reference: https://go101.org/article/unsafe.html
// make a new slice header
bHeader := (*reflect.SliceHeader)(unsafe.Pointer(&slice))
rHeader := (*reflect.SliceHeader)(unsafe.Pointer(&result))
// transfer the data from the given slice to a new variable (our result)
rHeader.Data = bHeader.Data
rHeader.Len = bHeader.Len / bitsetSize
rHeader.Cap = bHeader.Cap / bitsetSize
// instantiate result and use KeepAlive so data isn't unmapped.
runtime.KeepAlive(&slice) // it is still crucial, GC can free it)
// return result
return
}
func byteSliceAsArraySlice(slice []byte) (result []arrayContainer) {
arraySize := int(unsafe.Sizeof(arrayContainer{}))
if len(slice)%arraySize != 0 {
panic("Slice size should be divisible by unsafe.Sizeof(arrayContainer)")
}
// reference: https://go101.org/article/unsafe.html
// make a new slice header
bHeader := (*reflect.SliceHeader)(unsafe.Pointer(&slice))
rHeader := (*reflect.SliceHeader)(unsafe.Pointer(&result))
// transfer the data from the given slice to a new variable (our result)
rHeader.Data = bHeader.Data
rHeader.Len = bHeader.Len / arraySize
rHeader.Cap = bHeader.Cap / arraySize
// instantiate result and use KeepAlive so data isn't unmapped.
runtime.KeepAlive(&slice) // it is still crucial, GC can free it)
// return result
return
}
func byteSliceAsRun16Slice(slice []byte) (result []runContainer16) {
run16Size := int(unsafe.Sizeof(runContainer16{}))
if len(slice)%run16Size != 0 {
panic("Slice size should be divisible by unsafe.Sizeof(runContainer16)")
}
// reference: https://go101.org/article/unsafe.html
// make a new slice header
bHeader := (*reflect.SliceHeader)(unsafe.Pointer(&slice))
rHeader := (*reflect.SliceHeader)(unsafe.Pointer(&result))
// transfer the data from the given slice to a new variable (our result)
rHeader.Data = bHeader.Data
rHeader.Len = bHeader.Len / run16Size
rHeader.Cap = bHeader.Cap / run16Size
// instantiate result and use KeepAlive so data isn't unmapped.
runtime.KeepAlive(&slice) // it is still crucial, GC can free it)
// return result
return
}
func byteSliceAsBoolSlice(slice []byte) (result []bool) {
boolSize := int(unsafe.Sizeof(true))
if len(slice)%boolSize != 0 {
panic("Slice size should be divisible by unsafe.Sizeof(bool)")
}
// reference: https://go101.org/article/unsafe.html
// make a new slice header
bHeader := (*reflect.SliceHeader)(unsafe.Pointer(&slice))
rHeader := (*reflect.SliceHeader)(unsafe.Pointer(&result))
// transfer the data from the given slice to a new variable (our result)
rHeader.Data = bHeader.Data
rHeader.Len = bHeader.Len / boolSize
rHeader.Cap = bHeader.Cap / boolSize
// instantiate result and use KeepAlive so data isn't unmapped.
runtime.KeepAlive(&slice) // it is still crucial, GC can free it)
// return result
return
}
// FrozenView creates a static view of a serialized bitmap stored in buf.
// It uses CRoaring's frozen bitmap format.
//
// The format specification is available here:
// https://github.com/RoaringBitmap/CRoaring/blob/2c867e9f9c9e2a3a7032791f94c4c7ae3013f6e0/src/roaring.c#L2756-L2783
//
// The provided byte array (buf) is expected to be a constant.
// The function makes the best effort attempt not to copy data.
// Only little endian is supported. The function will err if it detects a big
// endian serialized file.
// You should take care not to modify buff as it will likely result in
// unexpected program behavior.
// If said buffer comes from a memory map, it's advisable to give it read
// only permissions, either at creation or by calling Mprotect from the
// golang.org/x/sys/unix package.
//
// Resulting bitmaps are effectively immutable in the following sense:
// a copy-on-write marker is used so that when you modify the resulting
// bitmap, copies of selected data (containers) are made.
// You should *not* change the copy-on-write status of the resulting
// bitmaps (SetCopyOnWrite).
//
// If buf becomes unavailable, then a bitmap created with
// FromBuffer would be effectively broken. Furthermore, any
// bitmap derived from this bitmap (e.g., via Or, And) might
// also be broken. Thus, before making buf unavailable, you should
// call CloneCopyOnWriteContainers on all such bitmaps.
//
func (rb *Bitmap) FrozenView(buf []byte) error {
return rb.highlowcontainer.frozenView(buf)
}
/* Verbatim specification from CRoaring.
*
* FROZEN SERIALIZATION FORMAT DESCRIPTION
*
* -- (beginning must be aligned by 32 bytes) --
* <bitset_data> uint64_t[BITSET_CONTAINER_SIZE_IN_WORDS * num_bitset_containers]
* <run_data> rle16_t[total number of rle elements in all run containers]
* <array_data> uint16_t[total number of array elements in all array containers]
* <keys> uint16_t[num_containers]
* <counts> uint16_t[num_containers]
* <typecodes> uint8_t[num_containers]
* <header> uint32_t
*
* <header> is a 4-byte value which is a bit union of FROZEN_COOKIE (15 bits)
* and the number of containers (17 bits).
*
* <counts> stores number of elements for every container.
* Its meaning depends on container type.
* For array and bitset containers, this value is the container cardinality minus one.
* For run container, it is the number of rle_t elements (n_runs).
*
* <bitset_data>,<array_data>,<run_data> are flat arrays of elements of
* all containers of respective type.
*
* <*_data> and <keys> are kept close together because they are not accessed
* during deserilization. This may reduce IO in case of large mmaped bitmaps.
* All members have their native alignments during deserilization except <header>,
* which is not guaranteed to be aligned by 4 bytes.
*/
const FROZEN_COOKIE = 13766
var (
FrozenBitmapInvalidCookie = errors.New("header does not contain the FROZEN_COOKIE")
FrozenBitmapBigEndian = errors.New("loading big endian frozen bitmaps is not supported")
FrozenBitmapIncomplete = errors.New("input buffer too small to contain a frozen bitmap")
FrozenBitmapOverpopulated = errors.New("too many containers")
FrozenBitmapUnexpectedData = errors.New("spurious data in input")
FrozenBitmapInvalidTypecode = errors.New("unrecognized typecode")
FrozenBitmapBufferTooSmall = errors.New("buffer too small")
)
func (ra *roaringArray) frozenView(buf []byte) error {
if len(buf) < 4 {
return FrozenBitmapIncomplete
}
headerBE := binary.BigEndian.Uint32(buf[len(buf)-4:])
if headerBE&0x7fff == FROZEN_COOKIE {
return FrozenBitmapBigEndian
}
header := binary.LittleEndian.Uint32(buf[len(buf)-4:])
buf = buf[:len(buf)-4]
if header&0x7fff != FROZEN_COOKIE {
return FrozenBitmapInvalidCookie
}
nCont := int(header >> 15)
if nCont > (1 << 16) {
return FrozenBitmapOverpopulated
}
// 1 byte per type, 2 bytes per key, 2 bytes per count.
if len(buf) < 5*nCont {
return FrozenBitmapIncomplete
}
types := buf[len(buf)-nCont:]
buf = buf[:len(buf)-nCont]
counts := byteSliceAsUint16Slice(buf[len(buf)-2*nCont:])
buf = buf[:len(buf)-2*nCont]
keys := byteSliceAsUint16Slice(buf[len(buf)-2*nCont:])
buf = buf[:len(buf)-2*nCont]
nBitmap, nArray, nRun := 0, 0, 0
nArrayEl, nRunEl := 0, 0
for i, t := range types {
switch t {
case 1:
nBitmap++
case 2:
nArray++
nArrayEl += int(counts[i]) + 1
case 3:
nRun++
nRunEl += int(counts[i])
default:
return FrozenBitmapInvalidTypecode
}
}
if len(buf) < (1<<13)*nBitmap+4*nRunEl+2*nArrayEl {
return FrozenBitmapIncomplete
}
bitsetsArena := byteSliceAsUint64Slice(buf[:(1<<13)*nBitmap])
buf = buf[(1<<13)*nBitmap:]
runsArena := byteSliceAsInterval16Slice(buf[:4*nRunEl])
buf = buf[4*nRunEl:]
arraysArena := byteSliceAsUint16Slice(buf[:2*nArrayEl])
buf = buf[2*nArrayEl:]
if len(buf) != 0 {
return FrozenBitmapUnexpectedData
}
var c container
containersSz := int(unsafe.Sizeof(c))*nCont
bitsetsSz := int(unsafe.Sizeof(bitmapContainer{}))*nBitmap
arraysSz := int(unsafe.Sizeof(arrayContainer{}))*nArray
runsSz := int(unsafe.Sizeof(runContainer16{}))*nRun
needCOWSz := int(unsafe.Sizeof(true))*nCont
bitmapArenaSz := containersSz + bitsetsSz + arraysSz + runsSz + needCOWSz
bitmapArena := make([]byte, bitmapArenaSz)
containers := byteSliceAsContainerSlice(bitmapArena[:containersSz])
bitmapArena = bitmapArena[containersSz:]
bitsets := byteSliceAsBitsetSlice(bitmapArena[:bitsetsSz])
bitmapArena = bitmapArena[bitsetsSz:]
arrays := byteSliceAsArraySlice(bitmapArena[:arraysSz])
bitmapArena = bitmapArena[arraysSz:]
runs := byteSliceAsRun16Slice(bitmapArena[:runsSz])
bitmapArena = bitmapArena[runsSz:]
needCOW := byteSliceAsBoolSlice(bitmapArena)
iBitset, iArray, iRun := 0, 0, 0
for i, t := range types {
needCOW[i] = true
switch t {
case 1:
containers[i] = &bitsets[iBitset]
bitsets[iBitset].cardinality = int(counts[i]) + 1
bitsets[iBitset].bitmap = bitsetsArena[:1024]
bitsetsArena = bitsetsArena[1024:]
iBitset++
case 2:
containers[i] = &arrays[iArray]
sz := int(counts[i]) + 1
arrays[iArray].content = arraysArena[:sz]
arraysArena = arraysArena[sz:]
iArray++
case 3:
containers[i] = &runs[iRun]
runs[iRun].iv = runsArena[:counts[i]]
runsArena = runsArena[counts[i]:]
iRun++
}
}
// Not consuming the full input is a bug.
if iBitset != nBitmap || len(bitsetsArena) != 0 ||
iArray != nArray || len(arraysArena) != 0 ||
iRun != nRun || len(runsArena) != 0 {
panic("we missed something")
}
ra.keys = keys
ra.containers = containers
ra.needCopyOnWrite = needCOW
ra.copyOnWrite = true
return nil
}
func (bm *Bitmap) GetFrozenSizeInBytes() uint64 {
nBits, nArrayEl, nRunEl := uint64(0), uint64(0), uint64(0)
for _, c := range bm.highlowcontainer.containers {
switch v := c.(type) {
case *bitmapContainer:
nBits++
case *arrayContainer:
nArrayEl += uint64(len(v.content))
case *runContainer16:
nRunEl += uint64(len(v.iv))
}
}
return 4 + 5*uint64(len(bm.highlowcontainer.containers)) +
(nBits << 13) + 2*nArrayEl + 4*nRunEl
}
func (bm *Bitmap) Freeze() ([]byte, error) {
sz := bm.GetFrozenSizeInBytes()
buf := make([]byte, sz)
_, err := bm.FreezeTo(buf)
return buf, err
}
func (bm *Bitmap) FreezeTo(buf []byte) (int, error) {
containers := bm.highlowcontainer.containers
nCont := len(containers)
nBits, nArrayEl, nRunEl := 0, 0, 0
for _, c := range containers {
switch v := c.(type) {
case *bitmapContainer:
nBits++
case *arrayContainer:
nArrayEl += len(v.content)
case *runContainer16:
nRunEl += len(v.iv)
}
}
serialSize := 4 + 5*nCont + (1<<13)*nBits + 4*nRunEl + 2*nArrayEl
if len(buf) < serialSize {
return 0, FrozenBitmapBufferTooSmall
}
bitsArena := byteSliceAsUint64Slice(buf[:(1<<13)*nBits])
buf = buf[(1<<13)*nBits:]
runsArena := byteSliceAsInterval16Slice(buf[:4*nRunEl])
buf = buf[4*nRunEl:]
arraysArena := byteSliceAsUint16Slice(buf[:2*nArrayEl])
buf = buf[2*nArrayEl:]
keys := byteSliceAsUint16Slice(buf[:2*nCont])
buf = buf[2*nCont:]
counts := byteSliceAsUint16Slice(buf[:2*nCont])
buf = buf[2*nCont:]
types := buf[:nCont]
buf = buf[nCont:]
header := uint32(FROZEN_COOKIE | (nCont << 15))
binary.LittleEndian.PutUint32(buf[:4], header)
copy(keys, bm.highlowcontainer.keys[:])
for i, c := range containers {
switch v := c.(type) {
case *bitmapContainer:
copy(bitsArena, v.bitmap)
bitsArena = bitsArena[1024:]
counts[i] = uint16(v.cardinality - 1)
types[i] = 1
case *arrayContainer:
copy(arraysArena, v.content)
arraysArena = arraysArena[len(v.content):]
elems := len(v.content)
counts[i] = uint16(elems - 1)
types[i] = 2
case *runContainer16:
copy(runsArena, v.iv)
runs := len(v.iv)
runsArena = runsArena[runs:]
counts[i] = uint16(runs)
types[i] = 3
}
}
return serialSize, nil
}
func (bm *Bitmap) WriteFrozenTo(wr io.Writer) (int, error) {
// FIXME: this is a naive version that iterates 4 times through the
// containers and allocates 3*len(containers) bytes; it's quite likely
// it can be done more efficiently.
containers := bm.highlowcontainer.containers
written := 0
for _, c := range containers {
c, ok := c.(*bitmapContainer)
if !ok {
continue
}
n, err := wr.Write(uint64SliceAsByteSlice(c.bitmap))
written += n
if err != nil {
return written, err
}
}
for _, c := range containers {
c, ok := c.(*runContainer16)
if !ok {
continue
}
n, err := wr.Write(interval16SliceAsByteSlice(c.iv))
written += n
if err != nil {
return written, err
}
}
for _, c := range containers {
c, ok := c.(*arrayContainer)
if !ok {
continue
}
n, err := wr.Write(uint16SliceAsByteSlice(c.content))
written += n
if err != nil {
return written, err
}
}
n, err := wr.Write(uint16SliceAsByteSlice(bm.highlowcontainer.keys))
written += n
if err != nil {
return written, err
}
countTypeBuf := make([]byte, 3*len(containers))
counts := byteSliceAsUint16Slice(countTypeBuf[:2*len(containers)])
types := countTypeBuf[2*len(containers):]
for i, c := range containers {
switch c := c.(type) {
case *bitmapContainer:
counts[i] = uint16(c.cardinality - 1)
types[i] = 1
case *arrayContainer:
elems := len(c.content)
counts[i] = uint16(elems - 1)
types[i] = 2
case *runContainer16:
runs := len(c.iv)
counts[i] = uint16(runs)
types[i] = 3
}
}
n, err = wr.Write(countTypeBuf)
written += n
if err != nil {
return written, err
}
header := uint32(FROZEN_COOKIE | (len(containers) << 15))
if err := binary.Write(wr, binary.LittleEndian, header); err != nil {
return written, err
}
written += 4
return written, nil
}

1
vendor/github.com/RoaringBitmap/roaring/serializationfuzz.go generated vendored
View file

@ -1,3 +1,4 @@
//go:build gofuzz
// +build gofuzz
package roaring

1
vendor/github.com/RoaringBitmap/roaring/setutil_arm64.go generated vendored
View file

@ -1,3 +1,4 @@
//go:build arm64 && !gccgo && !appengine
// +build arm64,!gccgo,!appengine
package roaring

1
vendor/github.com/RoaringBitmap/roaring/setutil_generic.go generated vendored
View file

@ -1,3 +1,4 @@
//go:build !arm64 || gccgo || appengine
// +build !arm64 gccgo appengine
package roaring

3
vendor/github.com/RoaringBitmap/roaring/smat.go generated vendored
View file

@ -1,3 +1,4 @@
//go:build gofuzz
// +build gofuzz
/*
@ -62,8 +63,8 @@ import (
"fmt"
"sort"
"github.com/bits-and-blooms/bitset"
"github.com/mschoch/smat"
"github.com/willf/bitset"
)
// fuzz test using state machine driven by byte stream.