Performance & Reliability

Project Zero's modular ingestion pipeline is engineered to handle demanding performance requirements of high-throughput blockchain networks and large-scale historical data backfills. The architecture leverages parallelized processing and distributed systems to ingest, normalize, and transform data at scale, ensuring low latency for real-time applications.

Performance Architecture

Project Zero’s modular ingestion pipeline is engineered to handle demanding performance requirements. It efficiently streams blockchain data to various destinations, ensuring both real-time and historical data processing with minimal latency and high reliability. The architecture is optimized for high throughput, low-latency processing, and fault tolerance, making it suitable for mission-critical applications.

High-Throughput Processing

To support the high transaction volume of blockchain networks, Project Zero employs an optimized ingestion pipeline capable of processing thousands of transactions per second (TPS).

Key performance features include:

• Parallelized Data Processing:
  • Uses multi-threaded processing and asynchronous event-driven architecture to handle multiple blockchain transactions simultaneously.
  • Scales horizontally by adding more processing nodes when demand increases.
• Optimized Data Streaming:
  • Supports batch processing and real-time event-driven updates.
• Minimal Performance Overhead:
  • Efficient handling of blockchain RPC calls to avoid unnecessary requests.

Delivery Guarantees

Ensuring reliable and accurate data delivery is critical when working with blockchain data. Project Zero guarantees:

• 100% delivery assurance for all processed data
  • Acknowledges successful message delivery before committing data.
  • Implements retry mechanisms to ensure no data loss.
• Automatic failover and recovery mechanisms during outages
  • If a node fails, tasks are seamlessly reassigned to healthy nodes.
  • Persistent storage ensures no data is lost due to unexpected crashes.

A key challenge in blockchain data streaming is handling chain reorganizations, where multiple competing branches of the blockchain emerge due to network latency or mining conflicts. Project Zero is designed to detect, resolve, and seamlessly adapt to chain reorganizations in real-time, ensuring that data consumers always receive the most accurate and up-to-date blockchain state with low latency.

Chain Reorganization Handling

What is Blockchain Reorganization?

Blockchain reorgs happen when miners compete and find blocks at the same time. If two or more miners successfully mine valid blocks almost simultaneously, the blockchain splits into different branches. This split needs to be resolved to determine which branch holds the correct transaction history.

Importance of Consensus

Consensus is key for maintaining the blockchain’s integrity during reorganizations. When everyone in the network agrees on what’s right, it ensures that transactions are accurate and reliable. This helps to keep the blockchain trustworthy and ensures that recorded transactions are dependable.

Reorganization Process

Resolving blockchain reorganizations involves the following steps:

• Detecting the Conflict: Nodes within the network identify the presence of competing chains with conflicting transaction histories.
• Choosing the Winning Chain: Consensus mechanisms are employed to determine the most valid chain among the competing branches. Typically, the longest or most-work chain is selected as the winner, representing the definitive transaction history.
• Reorganizing the Chain: Once the winning chain is determined, nodes collaborate to revert transactions from the losing branch and apply those from the winning chain. This reorganization aligns the blockchain with the agreed-upon transaction history.
• Updating State: Transactions on the winning chain are confirmed, and the network’s state is updated accordingly. This ensures that the blockchain reflects the latest, consensus-approved transaction ledger. With the resolution of the reorganization, the network resumes its normal operational state.

This systematic approach to resolving blockchain reorganizations underscores the importance of consensus in maintaining the integrity and reliability of decentralized networks.

Handling Chain Reorganization

1. Detecting Chain Reorganization: While listening to a network node for new blocks, encountering a block with a number that has already been processed signals the occurrence of a chain reorganization.
2. Identifying Conflicting Blocks: Upon detecting a chain reorganization, the process involves determining conflicting blocks by tracing down the chain to their root block using their parent hash. It’s important to note that when a chain reorganization is detected, the depth of the reorganization is initially unknown.
3. Analyzing Conflicts: By examining the conflicting blocks, we can determine which chain holds the most valid transaction history. Typically, the longest chain is selected as the most valid representation.
4. Resolving Conflicts: Once the most valid chain is identified, conflicts must be resolved by selecting the correct branch and discarding blocks from the orphaned branch. Additionally, the pointer to the head block is adjusted accordingly.

By following these steps, network nodes can effectively manage chain reorganizations while polling, ensuring the integrity and continuity of the blockchain.