* Think of a Block as an array of values for a sequence of lucene documents. That's * almost true! For the purposes of implementing {@link BlockLoader}, it's close enough. * The compute engine operates on arrays because the good folks that build CPUs have * spent the past 40 years making them really really good at running tight loops over * arrays of data. So we play along with the CPU and make arrays. *
** There are a lot of interesting choices hiding in here to make getting those arrays * out of lucene work well: *
** Generally reads are faster from {@code doc_values}, slower from {@code stored} fields, * and even slower from {@code _source}. If we get to chose, we pick {@code doc_values}. * But we work with what's on disk and that's a product of the field type and what the user's * configured. Picking the optimal choice given what's on disk is the responsibility of each * field's {@link MappedFieldType#blockLoader} method. The more configurable the field's * storage strategies the more {@link BlockLoader}s you have to implement to integrate it * with ESQL. It can get to be a lot. Sorry. *
** For a field to be supported by ESQL fully it has to be loadable if it was configured to be * stored in any way. It's possible to turn off storage entirely by turning off * {@code doc_values} and {@code _source} and {@code stored} fields. In that case, it's * acceptable to return {@link ConstantNull}. User turned the field off, best we can do * is {@code null}. *
** We also sometimes want to "push" executing some ESQL functions into the block loader itself. * Usually we do this when it's a ton faster. See the docs for {@code BlockLoaderExpression} * for why and how we do this. *
** For example, {@code long} fields implement these block loaders: *
** NOTE: We can't read from {@code long}s from {@code stored} fields which is a * bug, but maybe not * a terrible one because it's very uncommon to configure {@code long} to be {@code stored} * but to disable {@code _source} and {@code doc_values}. Nothing's perfect. Especially * code. *
** When we described how to read from {@code doc_values} we said we prefer * to use {@link ColumnAtATimeReader}. But some callers don't support reading column-at-a-time * and need to read row-by-row. So we also need an implementation of {@link RowStrideReader} * that reads from {@code doc_values}. Usually it's most convenient to implement both of those * in the same {@code class}. {@link AllReader} is an interface for those sorts of classes, and * you'll see it in the {@code doc_values} code frequently. *
** When decompressing {@code stored} fields lucene can skip stored field that aren't used. They * still have to be decompressed, but they aren't turned into java objects which saves a fair bit * of work. If you don't need any stored fields return {@link StoredFieldsSpec#NO_REQUIREMENTS}. * Otherwise, return what you need. *
** Instances of this class must be immutable and thread safe. Instances of * {@link ColumnAtATimeReader} and {@link RowStrideReader} are all mutable and can only * be accessed by one thread at a time but may be passed between threads. * See implementations {@link Reader#canReuse} for how that's handled. "Normal" java objects * don't need to do anything special to be kicked from thread to thread - the transfer itself * establishes a {@code happens-before} relationship that makes everything you need visible. * But Lucene's readers aren't "normal" java objects and sometimes need to be rebuilt if we * shift threads. *
*/ public interface BlockLoader { /** * @deprecated remove me once serverless migrates */ @Deprecated static BlockLoader constantBytes(BytesRef value) { return new ConstantBytes(value); } interface Reader { /** * Checks if the reader can be used to read a range documents starting with the given docID by the current thread. */ boolean canReuse(int startingDocID); } interface ColumnAtATimeReader extends Reader { /** * Reads the values of all documents in {@code docs}. * * @param nullsFiltered if {@code true}, then target docs are guaranteed to have a value for the field; * otherwise, the guarantee is unknown. This enables optimizations for block loaders, * treating the field as dense (every document has value) even if it is sparse in * the index. For example, "FROM index | WHERE x != null | STATS sum(x)", after filtering out * documents without value for field x, all target documents returned from the source operator * will have a value for field x whether x is dense or sparse in the index. */ BlockLoader.Block read(BlockFactory factory, Docs docs, int offset, boolean nullsFiltered) throws IOException; } /** * An interface for readers that attempt to load all document values in a column-at-a-time fashion. *
* Unlike {@link ColumnAtATimeReader}, implementations may return {@code null} if they are unable
* to load the requested values, for example due to unsupported underlying data.
* This allows callers to optimistically try optimized loading strategies first, and fall back if necessary.
*/
interface OptionalColumnAtATimeReader {
/**
* Attempts to read the values of all documents in {@code docs}
* Returns {@code null} if unable to load the values.
*
* @param nullsFiltered if {@code true}, then target docs are guaranteed to have a value for the field.
* see {@link ColumnAtATimeReader#read(BlockFactory, Docs, int, boolean)}
* @param toDouble a function to convert long values to double, or null if no conversion is needed/supported
* @param toInt whether to convert to int in case int block / vector is needed
* @param binaryMultiValuedFormat whether the multi-valued binary format is used (CustomBinaryDocValuesField).
*/
@Nullable
BlockLoader.Block tryRead(
BlockFactory factory,
Docs docs,
int offset,
boolean nullsFiltered,
BlockDocValuesReader.ToDouble toDouble,
boolean toInt,
boolean binaryMultiValuedFormat
) throws IOException;
}
interface RowStrideReader extends Reader {
/**
* Reads the values of the given document into the builder.
*/
void read(int docId, StoredFields storedFields, Builder builder) throws IOException;
}
interface AllReader extends ColumnAtATimeReader, RowStrideReader {}
interface StoredFields {
/**
* The {@code _source} of the document.
*/
Source source() throws IOException;
/**
* @return the ID for the current document
*/
String id() throws IOException;
/**
* @return the routing path for the current document
*/
String routing() throws IOException;
/**
* @return stored fields for the current document
*/
Map
* For example, a text field with a synthetic source can be loaded from its child keyword field when possible,
* which is faster than loading from stored fields:
*
* This is just a marker interface for these results. The compute engine
* has fleshed out implementations.
*
* {
* "parent_text_field": {
* "type": "text",
* "fields": {
* "child_keyword_field": {
* "type": "keyword",
* "ignore_above": 256
* }
* }
* }
* }
*
* If no values in a segment exceed the 256-character limit, then we can safely use the block loader from the
* keyword field for the entire segment. Alternatively, on a per-document basis, if doc-1 has the value "a"
* (under the limit) and doc-2 has the value "bcd..." (exceeds the limit), we can load doc-1 from the doc_values
* of keyword field and doc-2 from the slower stored fields.
*/
abstract class ConditionalBlockLoader implements BlockLoader {
private final BlockLoader preferLoader;
private final BlockLoader fallbackLoader;
protected ConditionalBlockLoader(BlockLoader preferLoader, BlockLoader fallbackLoader) {
this.preferLoader = preferLoader;
this.fallbackLoader = fallbackLoader;
}
@Override
public Builder builder(BlockFactory factory, int expectedCount) {
return fallbackLoader.builder(factory, expectedCount);
}
/**
* Determines whether the preferred loader can be used for all documents in the given leaf context.
*/
protected abstract boolean canUsePreferLoaderForLeaf(LeafReaderContext context) throws IOException;
/**
* Whether we can use the prefer loader for the given doc ID. If {@code true}, then the preferred loader is used
* to avoid loading stored fields or source.
*/
protected abstract boolean canUsePreferLoaderForDoc(int docId) throws IOException;
@Override
public ColumnAtATimeReader columnAtATimeReader(LeafReaderContext context) throws IOException {
if (canUsePreferLoaderForLeaf(context)) {
return preferLoader.columnAtATimeReader(context);
} else {
return fallbackLoader.columnAtATimeReader(context);
}
}
@Override
public RowStrideReader rowStrideReader(LeafReaderContext context) throws IOException {
if (preferLoader.rowStrideStoredFieldSpec().noRequirements() == false) {
return fallbackLoader.rowStrideReader(context);
}
RowStrideReader preferReader = preferLoader.rowStrideReader(context);
if (canUsePreferLoaderForLeaf(context)) {
return preferReader;
}
RowStrideReader fallbackReader = fallbackLoader.rowStrideReader(context);
return new RowStrideReader() {
@Override
public void read(int docId, StoredFields storedFields, Builder builder) throws IOException {
if (storedFields.loaded() == false && canUsePreferLoaderForDoc(docId)) {
preferReader.read(docId, storedFields, builder);
} else {
fallbackReader.read(docId, storedFields, builder);
}
}
@Override
public boolean canReuse(int startingDocID) {
return fallbackReader.canReuse(startingDocID) && preferReader.canReuse(startingDocID);
}
};
}
@Override
public StoredFieldsSpec rowStrideStoredFieldSpec() {
return fallbackLoader.rowStrideStoredFieldSpec();
}
@Override
public boolean supportsOrdinals() {
return false;
}
@Override
public SortedSetDocValues ordinals(LeafReaderContext context) throws IOException {
return null;
}
}
/**
* A list of documents to load. Documents are always in non-decreasing order.
*/
interface Docs {
int count();
int get(int i);
}
/**
* Builds block "builders" for loading data into blocks for the compute engine.
* It's important for performance that this only have one implementation in
* production code. That implementation sits in the "compute" project. The is
* also a test implementation, but there may be no more other implementations.
*/
interface BlockFactory {
/**
* Adjust the circuit breaker with the given delta, if the delta is negative, the breaker will
* be adjusted without tripping.
* @throws CircuitBreakingException if the breaker was put above its limit
*/
void adjustBreaker(long delta) throws CircuitBreakingException;
/**
* Build a builder to load booleans as loaded from doc values. Doc values
* load booleans in sorted order.
*/
BooleanBuilder booleansFromDocValues(int expectedCount);
/**
* Build a builder to load booleans without any loading constraints.
*/
BooleanBuilder booleans(int expectedCount);
/**
* Build a builder to load {@link BytesRef}s as loaded from doc values.
* Doc values load {@linkplain BytesRef}s deduplicated and in sorted order.
*/
BytesRefBuilder bytesRefsFromDocValues(int expectedCount);
/**
* Build a builder to load {@link BytesRef}s without any loading constraints.
*/
BytesRefBuilder bytesRefs(int expectedCount);
/**
* Build a specialized builder for singleton dense {@link BytesRef} fields with the following constraints:
*
*
*
* @param expectedCount The maximum number of values to be collected.
*/
SingletonBytesRefBuilder singletonBytesRefs(int expectedCount);
/**
* Build a builder to load doubles as loaded from doc values.
* Doc values load doubles in sorted order.
*/
DoubleBuilder doublesFromDocValues(int expectedCount);
/**
* Build a builder to load doubles without any loading constraints.
*/
DoubleBuilder doubles(int expectedCount);
/**
* Build a builder to load dense vectors without any loading constraints.
*/
FloatBuilder denseVectors(int expectedVectorsCount, int dimensions);
/**
* Build a builder to load ints as loaded from doc values.
* Doc values load ints in sorted order.
*/
IntBuilder intsFromDocValues(int expectedCount);
/**
* Build a builder to load ints without any loading constraints.
*/
IntBuilder ints(int expectedCount);
/**
* Build a builder to load longs as loaded from doc values.
* Doc values load longs in sorted order.
*/
LongBuilder longsFromDocValues(int expectedCount);
/**
* Build a builder to load longs without any loading constraints.
*/
LongBuilder longs(int expectedCount);
/**
* Build a specialized builder for singleton dense long based fields with the following constraints:
*
*
*
* @param expectedCount The maximum number of values to be collected.
*/
SingletonLongBuilder singletonLongs(int expectedCount);
/**
* Build a specialized builder for singleton dense int based fields with the following constraints:
*
*
*
* @param expectedCount The maximum number of values to be collected.
*/
SingletonIntBuilder singletonInts(int expectedCount);
/**
* Build a specialized builder for singleton dense double based fields with the following constraints:
*
*
*
* @param expectedCount The maximum number of values to be collected.
*/
SingletonDoubleBuilder singletonDoubles(int expectedCount);
/**
* Build a builder to load only {@code null}s.
*/
Builder nulls(int expectedCount);
/**
* Build a block that contains only {@code null}.
*/
Block constantNulls(int count);
/**
* Build a block that contains {@code value} repeated
* {@code size} times.
*/
Block constantBytes(BytesRef value, int count);
/**
* Build a block that contains {@code value} repeated
* {@code count} times.
*/
Block constantInt(int value, int count);
/**
* Build a reader for reading {@link SortedDocValues}
*/
SingletonOrdinalsBuilder singletonOrdinalsBuilder(SortedDocValues ordinals, int count, boolean isDense);
/**
* Build a reader for reading {@link SortedSetDocValues}
*/
SortedSetOrdinalsBuilder sortedSetOrdinalsBuilder(SortedSetDocValues ordinals, int count);
AggregateMetricDoubleBuilder aggregateMetricDoubleBuilder(int count);
LongRangeBuilder longRangeBuilder(int count);
Block buildAggregateMetricDoubleDirect(Block minBlock, Block maxBlock, Block sumBlock, Block countBlock);
ExponentialHistogramBuilder exponentialHistogramBlockBuilder(int count);
Block buildExponentialHistogramBlockDirect(
Block minima,
Block maxima,
Block sums,
Block valueCounts,
Block zeroThresholds,
Block encodedHistograms
);
Block buildTDigestBlockDirect(Block encodedDigests, Block minima, Block maxima, Block sums, Block valueCounts);
TDigestBuilder tdigestBlockBuilder(int count);
}
/**
* A columnar representation of homogenous data. It has a position (row) count, and
* various data retrieval methods for accessing the underlying data that is stored at a given
* position. In other words, a fancy wrapper over an array.
*