Scorch! a New Index for Bleve

Scorch! a New Index for Bleve

Bleve, an open-source full-text search library for Go, has moved beyond the general-purpose key/value store and now implements its own custom binary index format named Scorch. Learn about the data-structures and Go libraries we've chosen to build this solution.

16cdfb0c4af5297e261cb36e30fa5c20?s=128

Marty Schoch

August 03, 2018
Tweet

Transcript

  1. 2.
  2. 3.
  3. 5.
  4. 7.
  5. 8.
  6. 9.
  7. 14.
  8. 34.
  9. 35.
  10. 36.
  11. 37.
  12. 38.
  13. 39.
  14. 42.
  15. 46.
  16. 47.
  17. 51.

    Roaring Bitmap backed Postings List •Uncompressed Bitsets •Slice of Integers

    •Run-length encoding •Chunked Encoding, combining each
  18. 53.
  19. 55.

    $

  20. 69.

    Index Snapshot - epoch 27 Introducer Application Goroutine introduction channel

    read read/write f Persister - lastEpoch 26 Persister - lastEpoch 26
  21. 70.

    Index Snapshot - epoch 27 Introducer Application Goroutine introduction channel

    read read/write f Persister - lastEpoch 26 Persister - lastEpoch 26
  22. 71.

    Index Snapshot - epoch 27 Introducer Application Goroutine introduction channel

    read read/write f Persister - lastEpoch 26 Persister - lastEpoch 26
  23. 72.

    Index Snapshot - epoch 27 Introducer Application Goroutine introduction channel

    read read/write f Persister - lastEpoch 26 Persister - lastEpoch 26
  24. 73.

    Index Snapshot - epoch 27 Introducer Application Goroutine introduction channel

    read read/write f Persister - lastEpoch 26 Persister - lastEpoch 26 f.bolt
  25. 74.

    Index Snapshot - epoch 27 Introducer Application Goroutine introduction channel

    read read/write f Persister - lastEpoch 26 Persister - lastEpoch 26 f.bolt root.bolt 27 a, b, c, d, e, f
  26. 75.

    Index Snapshot - epoch 27 Introducer Application Goroutine introduction channel

    read read/write f Persister - lastEpoch 26 Persister - lastEpoch 26 f.bolt root.bolt 27 a, b, c, d, e, f Persister - lastEpoch 27
  27. 76.

    Index Snapshot - epoch 27 Introducer Application Goroutine introduction channel

    read read/write f Persister - lastEpoch 26 Persister - lastEpoch 26 f.bolt root.bolt 27 a, b, c, d, e, f Persister - lastEpoch 27
  28. 77.

    Index Snapshot - epoch 27 Introducer Application Goroutine introduction channel

    read read/write f Persister - lastEpoch 26 Persister - lastEpoch 26 f.bolt root.bolt 27 a, b, c, d, e, f Persister - lastEpoch 27
  29. 78.

    Index Snapshot - epoch 27 Introducer Application Goroutine introduction channel

    read read/write f Persister - lastEpoch 26 Persister - lastEpoch 26 f.bolt root.bolt 27 a, b, c, d, e, f Persister - lastEpoch 27
  30. 79.

    Index Snapshot - epoch 27 Introducer Application Goroutine introduction channel

    read read/write f Persister - lastEpoch 26 Persister - lastEpoch 26 f.bolt root.bolt 27 a, b, c, d, e, f Persister - lastEpoch 27
  31. 80.
  32. 81.

    Index Snapshot - epoch 27 Introducer Application Goroutine introduction channel

    read read/write f Persister - lastEpoch 26 Persister - lastEpoch 26 f.zap root.bolt 27 a, b, c, d, e, f Persister - lastEpoch 27
  33. 83.
  34. 84.
  35. 85.
  36. 87.

    Index Snapshot - epoch 27 Introducer Application Goroutine introduction channel

    read read/write Persister f Merger - lastPlanned 26
  37. 88.

    Index Snapshot - epoch 27 Introducer Application Goroutine introduction channel

    read read/write Persister f Merger - lastPlanned 26
  38. 89.

    Index Snapshot - epoch 27 Introducer Application Goroutine introduction channel

    read read/write Persister f Merger - lastPlanned 26
  39. 90.

    Index Snapshot - epoch 27 Introducer Application Goroutine introduction channel

    read read/write Persister f Merger - lastPlanned 26
  40. 91.

    Index Snapshot - epoch 27 Introducer Application Goroutine introduction channel

    read read/write Persister f Merger - lastPlanned 26 Merge Introduction g.zap
  41. 92.

    Index Snapshot - epoch 27 Introducer Application Goroutine introduction channel

    read read/write Persister f Merger - lastPlanned 26
  42. 93.

    Index Snapshot - epoch 27 Introducer Application Goroutine introduction channel

    read read/write Persister f Merger - lastPlanned 26 Merge Introduction g.zap
  43. 94.

    Index Snapshot - epoch 27 Introducer Application Goroutine introduction channel

    read read/write Persister f Merger - lastPlanned 26 Merge Introduction g.zap Merger - lastPlanned 27
  44. 95.

    Index Snapshot - epoch 27 Introducer Application Goroutine introduction channel

    read read/write Persister f Merger - lastPlanned 26 Merge Introduction g.zap Merger - lastPlanned 27
  45. 96.

    Index Snapshot - epoch 27 Introducer Application Goroutine introduction channel

    read read/write Persister Merger - lastPlanned 26 Merge Introduction g.zap Merger - lastPlanned 27
  46. 97.

    Index Snapshot - epoch 27 Introducer Application Goroutine introduction channel

    read read/write Persister Merger - lastPlanned 26 Merge Introduction g.zap Merger - lastPlanned 27 g
  47. 98.

    Index Snapshot - epoch 27 Introducer Application Goroutine introduction channel

    read read/write Persister Merger - lastPlanned 26 Merge Introduction g.zap Merger - lastPlanned 27 g Index Snapshot - epoch 28
  48. 99.

    Index Snapshot - epoch 27 Introducer Application Goroutine introduction channel

    read read/write Persister Merger - lastPlanned 26 Merge Introduction g.zap Merger - lastPlanned 27 g Index Snapshot - epoch 28
  49. 100.

    Index Snapshot - epoch 27 Introducer Application Goroutine introduction channel

    read read/write Persister Merger - lastPlanned 26 Merger - lastPlanned 27 g Index Snapshot - epoch 28
  50. 103.
  51. 104.
  52. 105.
  53. 106.
  54. 107.
  55. 108.
  56. 109.
  57. 110.
  58. 111.
  59. 113.
  60. 114.
  61. 115.
  62. 118.
  63. 119.
  64. 120.
  65. 127.
  66. 129.
  67. 130.
  68. 131.
  69. 132.
  70. 133.