The Evolution and Innovation of Blockchain Data Indexing

Introduction

Since the first batch of decentralized applications emerged in 2017, the blockchain ecosystem has shown a flourishing trend. When discussing on-chain applications, have we ever considered the sources of the data used by these applications?

In 2024, artificial intelligence and Web3 have become hot topics. In the field of AI, data is like the source of life, driving systems to continuously learn and evolve. Without data support, even the most sophisticated AI algorithms cannot exert their intended intelligence and effectiveness.

This article will analyze the development of blockchain data indexing from the perspective of data accessibility, and compare the similarities and differences in technical features and product architecture between traditional indexing protocols and emerging data service protocols.

The Evolution of Data Indexing: From Blockchain Nodes to Full Chain Databases

Data Source: Blockchain Node

Blockchain is essentially a decentralized distributed ledger. Each node maintains a complete copy of the blockchain data, ensuring the decentralized nature of the network. However, for average users, setting up and maintaining a node requires specialized technical skills and high costs. Therefore, most users tend to rely on third-party services.

To solve this problem, RPC node providers have emerged. They are responsible for the management of nodes and provide data access services through RPC endpoints. Although public RPC endpoints are free to use, they have rate limits that can affect the application experience. Private RPC endpoints perform better, but have lower efficiency for complex queries, and their scalability and cross-network compatibility are limited.

Data Analysis: From Raw Data to Usable Data

The raw data provided by blockchain nodes is often encrypted and encoded, and using this data directly requires a significant amount of technical knowledge and computational resources. The data parsing process converts complex raw data into a format that is easy to understand and operate, which is a key step in the entire data indexing process.

The evolution of data indexers

As the amount of Blockchain data increases, the demand for data indexers is growing. Indexers simplify the data retrieval process by organizing on-chain data and providing a unified query interface. Different types of indexers include:

1. Full Node Indexer: Extract data directly from the full node to ensure data is complete and accurate.
2. Lightweight Indexer: Obtain specific data from full nodes as needed, reducing storage requirements.
3. Dedicated Indexer: Optimized retrieval for specific types of data or Blockchain.
4. Aggregator Indexer: Extracts data from multiple Blockchains and sources, suitable for multi-chain applications.

Compared to traditional RPC endpoints, the indexer supports complex queries and data filtering, improving data retrieval efficiency and reliability.

Full Chain Database: Aligning with Flow Priority

As application demands become more complex, basic data indexers struggle to meet diverse query needs. The "stream-first" approach in modern data pipeline architecture has become a solution for achieving real-time data processing and analysis. Blockchain data service providers are also moving towards building data streams, launching products such as real-time data lakes.

These services are designed to meet the needs of real-time parsing and comprehensive querying of blockchain transactions, supporting the development of more applications and assisting on-chain data analysis.

AI + Database: In-depth Comparison of The Graph, Chainbase, and Space and Time

The Graph

The Graph network provides multi-chain data indexing and querying services through a decentralized network of nodes. Its core products include the data query execution market and the data indexing cache market. Subgraphs are the fundamental data structures of The Graph, defining the methods of data extraction and transformation.

The network consists of four roles: indexers, curators, delegates, and developers, ensuring the system operates through economic incentives.

The Graph ecosystem is actively integrating AI technology. Tools developed by Semiotic Labs, such as AutoAgora, Allocation Optimizer, and AgentC, respectively enhance system performance in pricing mechanisms, resource allocation, and user experience.

Chainbase

Chainbase is a full-chain data network that integrates multi-chain data into one platform. Its key features include:

• Real-time Data Lake: Provides instant access to blockchain data streams.
• Dual-Chain Architecture: Built on Eigenlayer AVS to enhance cross-chain data processing capabilities.
• Innovative data format standard: Introduce "manuscripts" to optimize data structure.
• Cryptocurrency World Model: A model that combines AI technology to understand and predict Blockchain transactions.

The AI model Theia of Chainbase is based on NVIDIA's DORA model, deeply mining the value of on-chain data and providing intelligent data services.

Space and Time

Space and Time (SxT) is committed to building a verifiable computing layer that expands zero-knowledge proof technology. Its core innovation, Proof of SQL, realizes an efficient data verification method, providing a foundation for traditional industries that value data reliability to utilize Blockchain data.

SxT has partnered with Microsoft's AI Lab to develop generative AI tools that simplify user interactions with Blockchain data. Users can query using natural language, and the AI automatically converts it to SQL and executes it.

Conclusion and Outlook

The blockchain data indexing technology has undergone an evolution from node data sources to AI-powered full-chain data services. This development has not only improved data access efficiency and accuracy but also brought about an intelligent user experience.

In the future, with the development of technologies such as AI and zero-knowledge proofs, blockchain data services will become further intelligent and secure, continuing to serve as infrastructure to support industry innovation.