Skip to main content

EasyLayer EVM Crawler Documentation

EVM Crawler is a self-hosted application that enables monitoring of the blockchain state both historically and in real-time


Overview

EVM Crawler is a powerful self-hosted application designed for monitoring and analyzing the EVM blockchain. It provides developers with a flexible framework to track blockchain state both historically and in real-time, enabling them to build custom blockchain analytics and monitoring solutions.

The application is built on modern architectural patterns including CQRS (Command Query Responsibility Segregation) and Event Sourcing, ensuring reliable and consistent data processing. It offers multiple transport options (RPC, WebSocket, TCP, IPC) for accessing blockchain data and supports both SQLite and PostgreSQL for event storage.

Key Features

  • Self-Hosted Architecture: Full control over deployment and customization
  • Flexible Node Connectivity: Works with your own EVM node or providers like QuickNode
  • Real-time & Historical Processing: Process blockchain data from any block height with automatic reorganization support
  • Custom Model Definition: Define your own data models using TypeScript/JavaScript
  • Event-Based Processing: Create and handle custom events to track blockchain state changes
  • Multiple Transport Options: Access data through HTTP, WebSocket, TCP, or IPC protocols
  • Database Flexibility: Choose between SQLite (managed) or PostgreSQL (self-configured)

Performance (TODO)

EVM Crawler is engineered for high-speed operation, but actual performance is primarily influenced by two factors: network latency when fetching blocks from the blockchain and the efficiency of inserting large datasets into database, depending on your model structure.

Setup

Prerequisites

  • Node.js version 17 or higher
  • A EVM self node or QuickNode provider URL

Installation

Install the package using your preferred package manager:

# Using npm
npm install @easylayer/evm-crawler

# Using yarn
yarn add @easylayer/evm-crawler

Basic Usage

The @easylayer/evm-crawler package exports a bootstrap function that initializes the crawler. Here's a basic setup:

main.ts
import { bootstrap } from '@easylayer/evm-crawler';
import Model from './model';

bootstrap({
Models: [Model],
rpc: true,
});

Creating a Custom Model

Define your custom model by extending the base Model class:

model.ts
import { BasicEvent, EventBasePayload, Model, Block } from '@easylayer/evm-crawler';

// Define your custom event
export class YourCustomEvent<T extends EventBasePayload> extends BasicEvent<T> {};

// Create your model
export default class CustomModel extends Model {
address: string = '0x...';
balance: string = '0';

constructor() {
super('uniq-model-id'); // This ID will be used to fetch events and state
}

public async parseBlock({ block }: { block: Block }) {
// Implement this method to process blocks
// Create custom events using this.apply(new YourCustomEvent(data))
}

private onYourCustomEvent({ payload }: YourCustomEvent) {
// Handle your custom event
// Update model state based on the event payload
// See examples in the repository for detailed implementations
}
}

Bootstrap Configuration

The bootstrap function accepts the following configuration options:

interface BootstrapOptions {
Models: ModelType[]; // Array of your custom models
rpc?: boolean; // Enable RPC transport
ws?: boolean; // Enable WebSocket transport
tcp?: boolean; // Enable TCP transport
ipc?: boolean; // Enable IPC transport
}

You can enable multiple transports simultaneously and define multiple models for different business logic domains.

Transport API Reference

This document describes how clients can interact with the application via RPC, IPC, WS and TCP transports.


1. HTTP RPC (Queries Only)

The HTTP RPC transport allows clients to perform data retrieval queries using a standardized JSON-RPC-like protocol.

Connection Details

POST `https://localhost:3000/`
Content-Type: application/json

Available Queries

The application provides two main query types:

  1. GetModels - Retrieve model states at a specific block height
  2. FetchEvents - Retrieve events with pagination and filtering

GetModels Query

Retrieves the current state of one or more models at a specified block height.

Request Format

{
"requestId": "uuid-1001",
"action": "query",
"payload": {
"constructorName": "GetModels",
"dto": {
"modelIds": ["model1", "model2"],
"filter": {
"blockHeight": 100
}
}
}
}

Parameters

ParameterTypeRequiredDescription
modelIdsstring[]YesArray of model IDs to retrieve
filter.blockHeightnumberNoBlock height to get state at (defaults to latest)

Response Format

{
"requestId": "uuid-1001",
"action": "queryResponse",
"payload": [
{
"aggregateId": "model1",
"state": { /* model state */ }
},
{
"aggregateId": "model2",
"state": { /* model state */ }
}
]
}

FetchEvents Query

Retrieves events for one or more models with pagination and filtering options.

Request Format

{
"requestId": "uuid-1002",
"action": "query",
"payload": {
"constructorName": "FetchEvents",
"dto": {
"modelIds": ["model1"],
"filter": {
"blockHeight": 100
},
"paging": {
"limit": 10,
"offset": 0
}
}
}
}

Parameters

ParameterTypeRequiredDescription
modelIdsstring[]YesArray of model IDs to fetch events for
filter.blockHeightnumberNoFilter events by block height
filter.versionnumberNoFilter events by version
paging.limitnumberNoNumber of events to return (default: 10)
paging.offsetnumberNoNumber of events to skip (default: 0)

Response Format

{
"requestId": "uuid-1002",
"action": "queryResponse",
"payload": {
"events": [
{
"aggregateId": "model1",
"version": 5,
"blockHeight": 100,
"data": { /* event data */ }
}
],
"total": 100
}
}

Error Handling

Both queries return errors in the following format:

{
"requestId": "uuid-1003",
"action": "error",
"payload": {
"error": {
"message": "Error description"
}
}
}

2. Event Streaming (WS, TCP, IPC)

The application supports real-time event streaming through multiple transport protocols. All transports implement the same event communication patterns and use the same query interfaces as HTTP RPC.

Event Communication Patterns

1. Outgoing Events (Application → Client)

ActionDescriptionPayload
eventSingle event{ constructorName: string; dto: any }
batchMultiple eventsArray<{ constructorName: string; dto: any }>
pingConnection checkundefined
errorError notification{ message: string }
queryResponseResponse to querySame as HTTP RPC responses

2. Incoming Events (Client → Application)

ActionDescriptionPayload
pongResponse to pingundefined
queryQuery requestSame as HTTP RPC requests

Available Queries

All transports support the same queries as HTTP RPC:

  1. GetModels Query
{
"requestId": "uuid-1",
"action": "query",
"payload": {
"constructorName": "GetModels",
"dto": {
"modelIds": ["model1", "model2"],
"filter": {
"blockHeight": 100
}
}
}
}
  1. FetchEvents Query
{
"requestId": "uuid-2",
"action": "query",
"payload": {
"constructorName": "FetchEvents",
"dto": {
"modelIds": ["model1"],
"filter": {
"blockHeight": 100
},
"paging": {
"limit": 10,
"offset": 0
}
}
}
}

Connection Lifecycle

  1. Client establishes connection
  2. Application sends ping events periodically
  3. Client must respond with pong to maintain connection
  4. After successful pong, application starts streaming events

Message Interfaces

// Outgoing messages (Application → Client)
interface OutgoingMessage<A extends string = string, P = any> {
requestId?: string;
action: A;
payload?: P;
}

// Incoming messages (Client → Application)
interface IncomingMessage<A extends string = string, P = any> {
requestId: string;
action: A;
payload?: P;
}

Transport-Specific Details


2.1 WebSocket

Connection URL

ws://localhost:3000/events

2.2 IPC

Connection Details

IPC transport is only available when the application runs as a child process. The communication happens through Node.js child process IPC channel.

import { fork } from 'node:child_process';

// Start the application as a child process
const child = fork('./easylayer.js', [], {
stdio: ['inherit', 'inherit', 'inherit', 'ipc']
});

Configuration Reference

AppConfig

PropertyTypeDescriptionDefaultRequired
NODE_ENVstringNode environment"development"
HTTP_HOSTstringHttp Server host
HTTP_PORTnumberHttp Server port (0 or undefined to disable)
HTTP_SSL_ENABLEDbooleanEnable SSL for HTTP serverfalse
HTTP_SSL_KEY_PATHstringPath to SSL private key file for HTTP server
HTTP_SSL_CERT_PATHstringPath to SSL certificate file for HTTP server
HTTP_SSL_CA_PATHstringPath to SSL CA file for HTTP server
WS_HOSTstringWebSocket server host"0.0.0.0"
WS_PATHstringWebSocket Server path"/"
WS_PORTnumberWebSocket Server port (0 or undefined to disable)
HTTP_MAX_MESSAGE_SIZEnumberMaximum message size for HTTP transport in bytes1048576
WS_MAX_MESSAGE_SIZEnumberMaximum message size for WebSocket transport in bytes1048576
IPC_MAX_MESSAGE_SIZEnumberMaximum message size for IPC transport in bytes1048576
HEARTBEAT_TIMEOUTnumberHeartbeat timeout in milliseconds3000
CONNECTION_TIMEOUTnumberConnection timeout in milliseconds2000
WS_CORS_ORIGINstringCORS origin for WebSocket"*"
WS_CORS_CREDENTIALSbooleanCORS credentials for WebSocketfalse
WS_SSL_ENABLEDbooleanEnable SSL for WebSocketfalse
WS_SSL_KEY_PATHstringPath to SSL private key file for WebSocket
WS_SSL_CERT_PATHstringPath to SSL certificate file for WebSocket
WS_SSL_CA_PATHstringPath to SSL CA file for WebSocket
WS_TRANSPORTSarrayWebSocket transports (comma-separated: websocket,polling)"websocket,polling"

BlocksQueueConfig

PropertyTypeDescriptionDefaultRequired
EVM_CRAWLER_BLOCKS_QUEUE_LOADER_STRATEGY_NAMEstringLoader strategy name for the EVM blocks queue."subscribe"

BusinessConfig

PropertyTypeDescriptionDefaultRequired
EVM_CRAWLER_MAX_BLOCK_HEIGHTnumberMaximum block height to be processed. Defaults to infinity.9007199254740991
EVM_CRAWLER_START_BLOCK_HEIGHTnumberThe block height from which processing begins. If not set, only listen to new blocks.
EVM_CRAWLER_NETWORK_CHAIN_IDnumberChain ID of the EVM network
EVM_CRAWLER_NETWORK_NATIVE_CURRENCY_SYMBOLstringSymbol of the native currency
EVM_CRAWLER_NETWORK_NATIVE_CURRENCY_DECIMALSnumberDecimals of the native currency
EVM_CRAWLER_NETWORK_BLOCK_TIMEnumberAverage block time in seconds
EVM_CRAWLER_NETWORK_HAS_EIP1559booleanWhether the network supports EIP-1559
EVM_CRAWLER_NETWORK_HAS_WITHDRAWALSbooleanWhether the network supports withdrawals
EVM_CRAWLER_NETWORK_HAS_BLOB_TRANSACTIONSbooleanWhether the network supports blob transactions
EVM_CRAWLER_NETWORK_MAX_BLOCK_SIZEnumberMaximum execution block size in bytes (transactions only)
EVM_CRAWLER_NETWORK_MAX_BLOCK_WEIGHTnumberMaximum total block weight in bytes (including blob data)
EVM_CRAWLER_NETWORK_MAX_GAS_LIMITnumberMaximum gas limit per block
EVM_CRAWLER_NETWORK_MAX_TRANSACTION_SIZEnumberMaximum transaction size in bytes
EVM_CRAWLER_NETWORK_MIN_GAS_PRICEnumberMinimum gas price in wei
EVM_CRAWLER_NETWORK_MAX_BASE_FEE_PER_GASnumberMaximum base fee per gas in wei for EIP-1559 networks
EVM_CRAWLER_NETWORK_MAX_PRIORITY_FEE_PER_GASnumberMaximum priority fee per gas in wei for EIP-1559 networks
EVM_CRAWLER_NETWORK_MAX_BLOB_GAS_PER_BLOCKnumberMaximum blob gas per block for EIP-4844 networks
EVM_CRAWLER_NETWORK_TARGET_BLOB_GAS_PER_BLOCKnumberTarget blob gas per block for EIP-4844 networks
EVM_CRAWLER_NETWORK_MAX_CODE_SIZEnumberMaximum contract code size in bytes
EVM_CRAWLER_NETWORK_MAX_INIT_CODE_SIZEnumberMaximum init code size in bytes
EVM_CRAWLER_RATE_LIMIT_MAX_REQUESTS_PER_SECONDnumberMaximum requests per second
EVM_CRAWLER_RATE_LIMIT_MAX_CONCURRENT_REQUESTSnumberMaximum concurrent requests
EVM_CRAWLER_RATE_LIMIT_MAX_BATCH_SIZEnumberMaximum batch size for parallel requests
EVM_CRAWLER_RATE_LIMIT_BATCH_DELAY_MSnumberDelay between batches in milliseconds

EventStoreConfig

PropertyTypeDescriptionDefaultRequired
EVM_CRAWLER_EVENTSTORE_DB_NAMEstringFor SQLite: folder path where the database file will be created; For Postgres: name of the database to connect to."resolve(process.cwd(), 'eventstore"
EVM_CRAWLER_EVENTSTORE_DB_TYPEstringType of database for the eventstore."sqlite"
EVM_CRAWLER_EVENTSTORE_DB_SYNCHRONIZEbooleanAutomatic synchronization that creates or updates tables and columns. Use with caution.true
EVM_CRAWLER_EVENTSTORE_DB_HOSTstringHost for the eventstore database connection.
EVM_CRAWLER_EVENTSTORE_DB_PORTnumberPort for the eventstore database connection.
EVM_CRAWLER_EVENTSTORE_DB_USERNAMEstringUsername for the eventstore database connection.
EVM_CRAWLER_EVENTSTORE_DB_PASSWORDstringPassword for the eventstore database connection.

ProvidersConfig

PropertyTypeDescriptionDefaultRequired
EVM_CRAWLER_NETWORK_PROVIDER_NODE_HTTP_URLstringHTTP URL of the EVM-like network provider node
EVM_CRAWLER_NETWORK_PROVIDER_NODE_WS_URLstringWS URL of the EVM-like network provider node
EVM_CRAWLER_NETWORK_PROVIDER_TYPEstringType of the network provider
EVM_CRAWLER_NETWORK_PROVIDER_REQUEST_TIMEOUTnumberRequest timeout in milliseconds
EVM_CRAWLER_NETWORK_PROVIDER_RATE_LIMIT_MAX_CONCURRENT_REQUESTSnumberMaximum concurrent requests
EVM_CRAWLER_NETWORK_PROVIDER_RATE_LIMIT_MAX_BATCH_SIZEnumberMaximum batch size for parallel requests
EVM_CRAWLER_NETWORK_PROVIDER_RATE_LIMIT_REQUEST_DELAY_MSnumberDelay between batches in milliseconds