The Evolution of Blockchain Data Indexing Technology: From Raw Nodes to AI-Driven Full-Chain Data Services

1. Introduction

Since the first batch of decentralized applications emerged in 2017, the blockchain ecosystem has developed a rich variety of financial, gaming, and social applications. When discussing these decentralized applications, have we ever considered the sources of the various types of data they use?

In 2024, artificial intelligence and Web3 have become hot topics. In the field of AI, data is like the source of life and is crucial for the growth and evolution of AI systems. Without data support, even the most sophisticated AI algorithms cannot exhibit their intended intelligence and effectiveness.

This article will analyze the evolution of data indexing in the development of the industry from the perspective of blockchain data accessibility, and compare data indexing protocols such as The Graph, Chainbase, and Space and Time, exploring the features of emerging protocols in data services and product architecture.

2. The Complexity and Simplicity of Data Indexing: From Blockchain Nodes to Full-Chain Database

2.1 Data Source: Blockchain Node

Blockchain nodes are the foundation of the entire network, responsible for recording, storing, and disseminating all on-chain transaction data. Each node keeps a complete copy of the blockchain data, ensuring the decentralized nature of the network. However, for ordinary users, building and maintaining a node is not an easy task, as it requires specialized skills and faces high hardware and bandwidth costs.

To solve this problem, RPC node providers have emerged. These providers are responsible for the management of nodes and provide data services through RPC endpoints. Users can easily access blockchain data without having to build their own nodes. Public RPC endpoints are free but have rate limits, while private RPC endpoints offer better performance but are less efficient and harder to scale. Nevertheless, the standardized API interfaces from node providers lower the barrier for users to access on-chain data, laying the foundation for subsequent data parsing and applications.

2.2 Data Parsing: From Prototype Data to Usable Data

The raw data provided by Blockchain nodes is often encrypted and encoded, increasing the difficulty of parsing. The data parsing process transforms complex prototype data into a format that is easy to understand and operate, making it a key link in the entire data indexing process.

Evolution of Data Indexers 2.3

As the amount of Blockchain data increases, the demand for data indexers is growing. Indexers organize on-chain data and send it to databases, making the data easy to query. They provide a unified query interface that allows developers to quickly retrieve the required information using standardized query languages.

Different types of indexers include:

1. Full Node Indexer
2. Lightweight Indexer
3. Dedicated Indexer
4. Aggregator Indexer

The emergence of indexers has greatly improved data indexing and query efficiency. Compared to traditional RPC endpoints, indexers support efficient indexing of large amounts of data and can perform complex queries. Some indexers also support aggregating data sources from multiple blockchains, avoiding the need for multi-chain applications to deploy multiple APIs.

2.4 Full Chain Database: Aligning to Stream Priority

As application demands become more complex, basic data indexers struggle to meet diverse query requirements. The "stream-first" approach in modern data pipeline architecture has become a solution to the limitations of traditional batch processing, enabling real-time data processing and analysis.

Blockchain data service providers are also moving towards building data streams. The Graph has launched Substreams, Chainbase, and SubSquid to develop real-time data lakes. These services are designed to address the need for real-time parsing of blockchain transactions and comprehensive querying capabilities.

3. AI + Database? In-depth comparison of The Graph, Chainbase, Space and Time

3.1 The Graph

The Graph provides multi-chain data indexing and query services through a decentralized node network. Its main product models include a data query execution market and a data indexing cache market. Subgraphs are the foundational data structure of The Graph network, defining the methods of data extraction and transformation.

The network consists of four key roles: Indexer, Curator, Delegator, and Developer. The Graph has transitioned to a fully decentralized subgraph hosting service, with economic incentives among participants to ensure the system operates.

The AutoAgora, Allocation Optimizer, and AgentC tools developed by Semiotic Labs utilize AI technology to optimize index pricing and user query experience, enhancing the system's intelligence and user-friendliness.

3.2 Chainbase

Chainbase is a full-chain data network that integrates all Blockchain data into one platform. Its unique features include:

• Real-time Data Lake
• Dual-chain architecture
• Innovative Data Format Standards
• Encrypted World Model

Chainbase's AI model Theia is based on NVIDIA's DORA model, combining on-chain and off-chain data analysis encryption patterns to provide users with intelligent data services.

3.3 Space and Time

Space and Time (SxT) aims to create a verifiable computation layer that expands zero-knowledge proofs on a decentralized data warehouse. Its innovative technology, Proof of SQL, ensures the tamper-proof and verifiability of SQL queries.

SxT collaborates with Microsoft's AI Lab to develop generative AI tools that enable users to process blockchain data through natural language.

Conclusion and Outlook

Blockchain data indexing technology has evolved from the initial node data source, through the development of data parsing and indexers, to AI-enabled full-chain data services, undergoing a process of gradual improvement. The advancements in these technologies have not only enhanced the efficiency and accuracy of data access but also brought about an intelligent experience.

In the future, with the development of new technologies such as AI and zero-knowledge proofs, Blockchain data services will become further intelligent and secure, continuously supporting industry innovation as infrastructure.