Nebula Graph 2.0 supports full-text indexing by using an external full-text search engine. To understand this new feature, let’s review the architecture and storage model of Nebula Graph 2.0.
As shown in the preceding figure, the Storage Service is composed of three layers. The bottom one is the Store Engine. It is a standalone local store engine, supporting get
, put
, scan
, and delete
operations on local data. The associated interfaces are in the kvstore/KVEngine.h file. Users can customize the local store plugins to meet their own needs. Currently, Nebula Graph provides a RocksDB-based store engine.
Above the local store engine, the consensus algorithm of multi-group Raft is implemented. With this implementation, each partition corresponds to one raft group, where a partition is a data shard in Nebula Graph. Hash-based sharding is used in Nebula Graph. For more information about how the hash functions work, see the 1.2.1 Data Storage in Nebula Graph. To create a graph space in Nebula Graph, the number of partitions is required and it cannot be changed after creation. The number of partitions must meet your needs of business expansion.
The top layer is the storage interfaces. A set of graph-related APIs is implemented in this layer. The API requests are translated into a set of KV operations on the corresponding partitions. This layer makes our storage service real graph storage. Without it, the Storage Service of Nebula Graph is just a KV storage solution. In Nebula Graph, the KV storage is not provided as a separate service. The main reason is that a lot of computations are required to execute a WHERE
clause and the schema of a graph is needed for the computations, but the schema is not implemented in the KV store layer. The design implemented in Nebula Graph makes computation pushdown easier.
In Nebula Graph 2.0, the storage structure containing vertices, edges, and indexes is improved. Now let’s review the storage structure of Nebula Graph 2.0, which could help you understand the implementation of data scanning and index scanning in Nebula Graph 2.0.
Nebula Graph stores vertices and edges based on the key-value storage model. In this section, the storage structure of the keys is introduced. The keys are composed of the following items:
Type
: One byte. It represents the key type, such as vertex, edge, index, or system.PartID
: Three bytes. It represents a partition. This field makes it easy to scan the entire partition data based on the prefix when the partition is re-balanced.VertexID
: n bytes. For an outgoing edge, it represents the ID of the source vertex. For an incoming edge, it represents the ID of the destination vertex.Edge Type
: Four bytes. It represents the type of edge. If it is greater than 0, the edge is outgoing. If it is less than 0, the edge is incoming.Rank
: Eight bytes. It is used to identify edges of the same edge type and with the same source and destination vertices. Users can use it to represent their own business attributes such as transaction time, transaction serial number, or a sorting weight.PlaceHolder
: One byte. It is invisible to users now. In the future, it will be used when we implement the distributed transaction.TagID
:Four bytes. It represents the type of tag.1.2.1.1 Vertex Key Format
TYPE (1 BYTE) | PARTID (3 BYTES) | VERTEXID (N BYTES) | TAGID (4 BYTES) |
---|
1.2.1.2 Edge Key Format
TYPE (1 BYTE) | PARTID (3 BYTES) | VERTEXID (N BYTES) | EDGETYPE (4 BYTES) | RANK (8 BYTES) | VERTEXID (N BYTES) | PLACEHOLDER (1 BYTE) |
---|
NULL
, 0xFF is used.1.2.2.1 Tag Index Key Format
TYPE (1 BYTE) | PARTID (3 BYTES) | INDEXID (4 BYTES) | PROPS BINARY (N BYTES) | NULLABLE BITSET (2 BYTES) | VERTEXID (N BYTES) |
---|
1.2.2.2 Edge Index Key Format
TYPE (1 BYTE) | PARTID (3 BYTES) | INDEXID (4 BYTES) | PROPS BINARY (N BYTES) | NULLABLE BITSET (2 BYTES) | VERTEXID (N BYTES) | RANK (8 BYTES) | VERTEXID (N BYTES) |
---|
From the preceding figure, you can see that if you want to perform a fuzzy query of a text on a property, a full table scan
or full index scan
statement is required and then the data is filtered row by row, which will compromise the query performance. If the amount of data is large, out-of-memory may occur before the scanning is done. Besides, inverted indexing is against the initial design principle of indexing in Nebula Graph, so it is not implemented for text search. After some research and discussion, to make the full-text search work greatly, we decided to introduce a full-text search engine from a third party. It can ensure query performance and reduce the development cost of the Nebula Graph kernel.
In Nebula Graph 2.0, only LOOKUP
supports text search. It means that when an external full-text search engine is available, users can run a LOOKUP
statement to perform a text search. For an external full-text search engine, only some basic functionalities, such as inserting data and querying data, are implemented. To implement some complex, plain text queries, Nebula Graph needs to be polished further. Any suggestions from the Nebula Graph community are welcome. The following are the text search expressions that are supported by Nebula Graph 2.0:
In this article, I will discuss data synchronization performance and query performance.
LOOKUP
supports text search, but the performance is inevitably lower than that of the native index scan of Nebula Graph, even sometimes the query performance of the external full-text search engine is low. To solve this problem, a timeliness mechanism, LIMIT
and TIMEOUT
, is needed to ensure the query performance. For more information, see the following sections.TERM | MEANING |
---|---|
Tag | Defines the property structure of vertices. Tags are identified by tagId . Multiple tags can be attached to one vertex. |
Edge | Defines the property structure of edges. Edge types are identified by edgetype . |
Property | Defines the properties of a tag or an edge type. Its data type is defined in a tag or an edge type. |
Partition | Represents the smallest logical store unit in Nebula Graph. A Storage Engine contains multiple partitions. The Leader or Follower role can be assigned to a partition. Raftex ensures data consistency between Leaders and Followers. |
Graph space | Each graph space is an independent business graph unit. Each graph space has its own independent tag and edge type set. A Nebula Graph cluster can have multiple graph spaces. |
Index | The referred index in the following sections represents the indexes on the properties of vertices and edges in Nebula Graph. Its data type is determined by the tag or edge type definition. |
TagIndex | Represents an index on a tag. A tag can have more than one index. Indexes across multiple tags have not been supported. |
EdgeIndex | Represents an index on an edge type. An edge type can have more than one index. Indexes across multiple edge types have not been supported. |
Scan Policy | Defines the index scan policy. Generally, a query statement can use multiple index scan policies, and Scan Policy decides which policy is used. |
Optimizer | Optimizes the query conditions to improve the query efficiency. For example, sorting, splitting, and merging sub-expression nodes on the expression tree of the WHERE clause. |
Elasticsearch is the external full-text search engine that is supported by Nebula Graph. In this section, I will introduce how Elasticsearch works with Nebula Graph 2.0.
PARTID(10 BYTES) | SCHEMAID(10 BYTES) | ENCODED_COLUMNNAME(32 BYTES) | ENCODED_VAL(MAX 344 BYTES) |
---|
partId
: Corresponds to the partition ID of Nebula Graph. Not available in Nebula Graph 2.0. It will be used for query pushdown and the routing feature of Elasticsearch in the future.schemaId
: Corresponds to the tagId
or edgetype
in Nebula Graph.encoded_columnName
: Corresponds to the property name of a tag or an edge type. The MD5 algorithm is used for encoding to avoid incompatible characters in Elasticsearch docID.encoded_val
: The maximum length is 344 bytes. To support some visible characters in the property values that are not supported by Elasticsearch docID, the Base64 algorithm is used to encode the property values, so the maximum length of encoded_val is 344 bytes. However, its actual size is up to 256 bytes only. Why is it 256 bytes? In the beginning, we just wanted to enable LOOKUP
to be used to perform a text search. Similar to MySQL, the length of an index in Nebula Graph is also limited and the recommended maximum length is 256 bytes. Therefore, the 256-byte length limit also is applied to the external search engine. So far, full-text search for long texts has not been supported.tagId
or edgetype
in Nebula Graph.In this section, I will introduce the details of synchronizing data asynchronously. Understanding Leader and Listener in Nebula Graph will help you understand the synchronization mechanism.
nebula-storage.conf
.nebula-storage-listener.conf
. As a listener, a Listener passively receives the WAL sent by the Leader, parses the WAL regularly, and calls the data insertion API of the external full-text search engine to synchronize the data with the external engine. Nebula Graph 2.0 supports the PUT
and BULK
interfaces of Elasticsearch.Now, let’s see how the data is synchronized:
INSERT
request is sent to the Leader of the related partitions via storageClient.INSERT
request and then synchronizes the WAL with the Listener.STRING
property values of the tags or edge types.PUT
or BULK
interface.In the preceding steps, if the Elasticsearch cluster or the Listener process crashes, the synchronization of the WAL will stop. When the system is restored, the data synchronization will continue with the last successful Log ID. We recommend that DBA should monitor the state of the Elasticsearch cluster in real-time by using an external monitoring tool. If the Elasticsearch cluster is inactive for a long time, a lot of logs will be generated to the Listener and the query cannot be performed normally.
From the preceding figure, we can see the key steps in text search as follows:
C1 == "A1" OR C1 == "A2"
is generated.LIMIT
and TIMEOUT
is adopted to interrupt the query on the Elasticsearch side in real-time.I assume that you are already familiar with the deployment of an Elasticsearch cluster, so I won’t describe it in detail. It should be noted that when the Elasticsearch cluster is successfully started, it is necessary to create a general template as follows.
{
"template": "nebula*",
"settings": {
"index": {
"number_of_shards": 3,
"number_of_replicas": 1
}
},
"mappings": {
"properties" : {
"tag_id" : { "type" : "long" },
"column_id" : { "type" : "text" },
"value" :{ "type" : "keyword"}
}
}
}
nebula-storaged-listener.conf
./bin/nebula-storaged --flagfile ${listener_config_path}/nebula-storaged-listener.conf
nebula> SIGN IN TEXT SERVICE (127.0.0.1:9200);
nebula> SHOW TEXT SEARCH CLIENTS;
+-------------+------+
| Host | Port |
+-------------+------+
| "127.0.0.1" | 9200 |
+-------------+------+
| "127.0.0.1" | 9200 |
+-------------+------+
| "127.0.0.1" | 9200 |
+-------------+------+
CREATE SPACE basketballplayer (partition_num=3,replica_factor=1, vid_type=fixed_string(30));
USE basketballplayer;
nebula> ADD LISTENER ELASTICSEARCH 192.168.8.5:46780,192.168.8.6:46780;
nebula> SHOW LISTENER;
+--------+-----------------+-----------------------+----------+
| PartId | Type | Host | Status |
+--------+-----------------+-----------------------+----------+
| 1 | "ELASTICSEARCH" | "[192.168.8.5:46780]" | "ONLINE" |
+--------+-----------------+-----------------------+----------+
| 2 | "ELASTICSEARCH" | "[192.168.8.5:46780]" | "ONLINE" |
+--------+-----------------+-----------------------+----------+
| 3 | "ELASTICSEARCH" | "[192.168.8.5:46780]" | "ONLINE" |
+--------+-----------------+-----------------------+----------+
The name
property should be shorter than 256 bytes. If the business permits, the name
property of the player tag should be the fixed_string
type and its length should be less than 256 bytes.
nebula> CREATE TAG player(name string, age int);
nebula> CREATE TAG INDEX name ON player(name(20));
nebula> INSERT VERTEX player(name, age) VALUES \
"Russell Westbrook": ("Russell Westbrook", 30), \
"Chris Paul": ("Chris Paul", 33),\
"Boris Diaw": ("Boris Diaw", 36),\
"David West": ("David West", 38),\
"Danny Green": ("Danny Green", 31),\
"Tim Duncan": ("Tim Duncan", 42),\
"James Harden": ("James Harden", 29),\
"Tony Parker": ("Tony Parker", 36),\
"Aron Baynes": ("Aron Baynes", 32),\
"Ben Simmons": ("Ben Simmons", 22),\
"Blake Griffin": ("Blake Griffin", 30);
nebula> LOOKUP ON player WHERE PREFIX(player.name, "B");
+-----------------+
| _vid |
+-----------------+
| "Boris Diaw" |
+-----------------+
| "Ben Simmons" |
+-----------------+
| "Blake Griffin" |
+-----------------+
In the process of setting up the system environment, errors in a step may make the functionalities unable to work normally. Based on user feedback, I summarized three possible error types. Here is how to analyze and solve these problems:
IP:Port
configuration of the Listeners does not conflict with that of the existing nebula-storaged process.IP:Port
configuration of Meta is consistent with that of the nebula-storaged process.listener_path
in the nebula-storaged-listener.conf
file.UPDATE CONFIGS storage:v=3
and make sure that the CURL command is executed successfully. If the execution fails, do a check of the Elasticsearch configuration or the compatibility between versions.UPDATE CONFIGS graph:v=3
and do a check of the graph logs to confirm the reasons for the CURL command failures.Department of Information Technologies: https://www.ibu.edu.ba/department-of-information-technologies/