EasyLayer Bitcoin Crawler Documentation

Bitcoin Crawler is a self-hosted application that enables monitoring of the blockchain state both historically and in real-time

Overview

Bitcoin Crawler is a powerful self-hosted application designed for monitoring and analyzing the Bitcoin blockchain. It provides developers with a flexible framework to track blockchain state both historically and in real-time, enabling them to build custom blockchain analytics and monitoring solutions.

The application is built on modern architectural patterns including CQRS (Command Query Responsibility Segregation) and Event Sourcing, ensuring reliable and consistent data processing. It offers multiple transport options (RPC, WebSocket, TCP, IPC) for accessing blockchain data and supports both SQLite and PostgreSQL for event storage.

Key Features

Self-Hosted Architecture: Full control over deployment and customization
Flexible Node Connectivity: Works with your own Bitcoin node or providers like QuickNode
Real-time & Historical Processing: Process blockchain data from any block height with automatic reorganization support
Custom Model Definition: Define your own data models using TypeScript/JavaScript
Event-Based Processing: Create and handle custom events to track blockchain state changes
Multiple Transport Options: Access data through HTTP, WebSocket, TCP, or IPC protocols
Database Flexibility: Choose between SQLite (managed) or PostgreSQL (self-configured)

Performance (TODO)

Bitcoin Crawler is engineered for high-speed operation, but actual performance is primarily influenced by two factors: network latency when fetching blocks from the blockchain and the efficiency of inserting large datasets into database, depending on your model structure.

Setup

Prerequisites

Node.js version 17 or higher
A Bitcoin self node or QuickNode provider URL

Installation

Install the package using your preferred package manager:

# Using npm
npm install @easylayer/bitcoin-crawler

# Using yarn
yarn add @easylayer/bitcoin-crawler

Basic Usage

The @easylayer/bitcoin-crawler package exports a bootstrap function that initializes the crawler. Here's a basic setup:

main.ts
import { bootstrap } from '@easylayer/bitcoin-crawler';
import Model from './model';

bootstrap({
  Models: [Model],
  rpc: true,
});

Creating a Custom Model

Define your custom model by extending the base Model class:

model.ts
import { BasicEvent, EventBasePayload, Model, Block } from '@easylayer/bitcoin-crawler';

// Define your custom event
export class YourCustomEvent<T extends EventBasePayload> extends BasicEvent<T> {};

// Create your model
export default class CustomModel extends Model {
    address: string = '1A1zP1eP5QGefi2DMPTfTL5SLmv7DivfNa';
    balance: string = '0';

    constructor() {
      super('uniq-model-id'); // This ID will be used to fetch events and state
    }

    public async parseBlock({ block }: { block: Block }) {
      // Implement this method to process blocks
      // Create custom events using this.apply(new YourCustomEvent(data))
    }

    private onYourCustomEvent({ payload }: YourCustomEvent) {
      // Handle your custom event
      // Update model state based on the event payload
      // See examples in the repository for detailed implementations
    }
}

Bootstrap Configuration

The bootstrap function accepts the following configuration options:

interface BootstrapOptions {
  Models: ModelType[];      // Array of your custom models
  rpc?: boolean;           // Enable RPC transport
  ws?: boolean;            // Enable WebSocket transport
  tcp?: boolean;           // Enable TCP transport
  ipc?: boolean;           // Enable IPC transport
}

You can enable multiple transports simultaneously and define multiple models for different business logic domains.

Transport API Reference

This document describes how clients can interact with the application via RPC, IPC, WS and TCP transports.

1. HTTP RPC (Queries Only)

The HTTP RPC transport allows clients to perform data retrieval queries using a standardized JSON-RPC-like protocol.

Connection Details

POST `https://localhost:3000/`
Content-Type: application/json

Available Queries

The application provides two main query types:

GetModels - Retrieve model states at a specific block height
FetchEvents - Retrieve events with pagination and filtering

GetModels Query

Retrieves the current state of one or more models at a specified block height.

Request Format

{
  "requestId": "uuid-1001",
  "action": "query",
  "payload": {
    "constructorName": "GetModels",
    "dto": {
      "modelIds": ["model1", "model2"],
      "filter": {
        "blockHeight": 100
      }
    }
  }
}

Parameters

Parameter	Type	Required	Description
modelIds	string[]	Yes	Array of model IDs to retrieve
filter.blockHeight	number	No	Block height to get state at (defaults to latest)

Response Format

{
  "requestId": "uuid-1001",
  "action": "queryResponse",
  "payload": [
    {
      "aggregateId": "model1",
      "state": { /* model state */ }
    },
    {
      "aggregateId": "model2",
      "state": { /* model state */ }
    }
  ]
}

FetchEvents Query

Retrieves events for one or more models with pagination and filtering options.

Request Format

{
  "requestId": "uuid-1002",
  "action": "query",
  "payload": {
    "constructorName": "FetchEvents",
    "dto": {
      "modelIds": ["model1"],
      "filter": {
        "blockHeight": 100
      },
      "paging": {
        "limit": 10,
        "offset": 0
      }
    }
  }
}

Parameters

Parameter	Type	Required	Description
modelIds	string[]	Yes	Array of model IDs to fetch events for
filter.blockHeight	number	No	Filter events by block height
filter.version	number	No	Filter events by version
paging.limit	number	No	Number of events to return (default: 10)
paging.offset	number	No	Number of events to skip (default: 0)

Response Format

{
  "requestId": "uuid-1002",
  "action": "queryResponse",
  "payload": {
    "events": [
      {
        "aggregateId": "model1",
        "version": 5,
        "blockHeight": 100,
        "data": { /* event data */ }
      }
    ],
    "total": 100
  }
}

Error Handling

Both queries return errors in the following format:

{
  "requestId": "uuid-1003",
  "action": "error",
  "payload": {
    "error": {
      "message": "Error description"
    }
  }
}

2. Event Streaming (WS, TCP, IPC)

The application supports real-time event streaming through multiple transport protocols. All transports implement the same event communication patterns and use the same query interfaces as HTTP RPC.

Event Communication Patterns

1. Outgoing Events (Application → Client)

Action	Description	Payload
`event`	Single event	`{ constructorName: string; dto: any }`
`batch`	Multiple events	`Array<{ constructorName: string; dto: any }>`
`ping`	Connection check	`undefined`
`error`	Error notification	`{ message: string }`
`queryResponse`	Response to query	Same as HTTP RPC responses

2. Incoming Events (Client → Application)

Action	Description	Payload
`pong`	Response to ping	`undefined`
`query`	Query request	Same as HTTP RPC requests

Available Queries

All transports support the same queries as HTTP RPC:

GetModels Query

{
  "requestId": "uuid-1",
  "action": "query",
  "payload": {
    "constructorName": "GetModels",
    "dto": {
      "modelIds": ["model1", "model2"],
      "filter": {
        "blockHeight": 100
      }
    }
  }
}

FetchEvents Query

{
  "requestId": "uuid-2",
  "action": "query",
  "payload": {
    "constructorName": "FetchEvents",
    "dto": {
      "modelIds": ["model1"],
      "filter": {
        "blockHeight": 100
      },
      "paging": {
        "limit": 10,
        "offset": 0
      }
    }
  }
}

Connection Lifecycle

Client establishes connection
Application sends ping events periodically
Client must respond with pong to maintain connection
After successful pong, application starts streaming events

Message Interfaces

// Outgoing messages (Application → Client)
interface OutgoingMessage<A extends string = string, P = any> {
  requestId?: string;
  action: A;
  payload?: P;
}

// Incoming messages (Client → Application)
interface IncomingMessage<A extends string = string, P = any> {
  requestId: string;
  action: A;
  payload?: P;
}

Transport-Specific Details

2.1 WebSocket

Connection URL

ws://localhost:3000/events

2.2 TCP

Connection Details

Host: localhost
Port: 4000

2.3 IPC

Connection Details

IPC transport is only available when the application runs as a child process. The communication happens through Node.js child process IPC channel.

import { fork } from 'node:child_process';

// Start the application as a child process
const child = fork('./easylayer.js', [], {
  stdio: ['inherit', 'inherit', 'inherit', 'ipc']
});

Configuration Reference

AppConfig

Property	Type	Description	Default	Required
`NODE_ENV`	string	Node environment	`"development"`	✅
`HTTP_HOST`	string	Http Server host
`HTTP_PORT`	number	Http Server port	`3000`	✅
`TCP_HOST`	string	Tcp Server host
`TCP_PORT`	number	Tcp Server port	`4000`	✅

BlocksQueueConfig

Property	Type	Description	Default	Required
`BITCOIN_CRAWLER_BLOCKS_QUEUE_LOADER_STRATEGY_NAME`	string	Loader strategy name for the Bitcoin blocks queue.	`"pull-network-provider"`	✅
`BITCOIN_CRAWLER_BLOCKS_QUEUE_LOADER_CONCURRENCY_COUNT`	number	Concurrency сount of blocks download	`4`	✅

BusinessConfig

Property	Type	Description	Default	Required
`BITCOIN_CRAWLER_MAX_BLOCK_HEIGHT`	number	Maximum block height to be processed. Defaults to infinity.	`9007199254740991`	✅
`BITCOIN_CRAWLER_START_BLOCK_HEIGHT`	number	The block height from which processing begins.	`0`	✅
`BITCOIN_CRAWLER_ONE_BLOCK_SIZE`	number	The block size	`1048576`	✅

EventStoreConfig

Property	Type	Description	Default	Required
`BITCOIN_CRAWLER_EVENTSTORE_DB_NAME`	string	For SQLite: folder path where the database file will be created; For Postgres: name of the database to connect to.	`"resolve(process.cwd(), 'eventstore"`	✅
`BITCOIN_CRAWLER_EVENTSTORE_DB_TYPE`	string	Type of database for the eventstore.	`"sqlite"`	✅
`BITCOIN_CRAWLER_EVENTSTORE_DB_SYNCHRONIZE`	boolean	Automatic synchronization that creates or updates tables and columns. Use with caution.	`true`	✅
`BITCOIN_CRAWLER_EVENTSTORE_DB_HOST`	string	Host for the eventstore database connection.
`BITCOIN_CRAWLER_EVENTSTORE_DB_PORT`	number	Port for the eventstore database connection.
`BITCOIN_CRAWLER_EVENTSTORE_DB_USERNAME`	string	Username for the eventstore database connection.
`BITCOIN_CRAWLER_EVENTSTORE_DB_PASSWORD`	string	Password for the eventstore database connection.

ProvidersConfig

Property	Type	Description	Default	Required
`BITCOIN_CRAWLER_NETWORK_PROVIDER_SELF_NODE_URL`	string	URL of the user's own Bitcoin node. Format: `http://username:password@host:port`
`BITCOIN_CRAWLER_NETWORK_PROVIDER_QUICK_NODE_URLS`	array	Multiple QuickNode node URLs can be entered, separated by commas.

Overview​

Key Features​

Performance (TODO)​

Setup​

Prerequisites​

Installation​

Basic Usage​

Creating a Custom Model​

Bootstrap Configuration​

Transport API Reference

Connection Details​

Available Queries​

GetModels Query​

Request Format​

Parameters​

Response Format​

FetchEvents Query​

Request Format​

Parameters​

Response Format​

Error Handling​

Event Communication Patterns​

1. Outgoing Events (Application → Client)​

2. Incoming Events (Client → Application)​

Available Queries​

Connection Lifecycle​

Message Interfaces​

Transport-Specific Details​

Connection URL​

Connection Details​

Connection Details​

Configuration Reference​

AppConfig​

BlocksQueueConfig​

BusinessConfig​

EventStoreConfig​

ProvidersConfig​

Overview

Key Features

Performance (TODO)

Setup

Prerequisites

Installation

Basic Usage

Creating a Custom Model

Bootstrap Configuration

Connection Details

Available Queries

GetModels Query

Request Format

Parameters

Response Format

FetchEvents Query

Request Format

Parameters

Response Format

Error Handling

Event Communication Patterns

1. Outgoing Events (Application → Client)

2. Incoming Events (Client → Application)

Available Queries

Connection Lifecycle

Message Interfaces

Transport-Specific Details

Connection URL

Connection Details

Connection Details

Configuration Reference

AppConfig

BlocksQueueConfig

BusinessConfig

EventStoreConfig

ProvidersConfig