Choose Substreams when
- The consumer must process every relevant block or event in order.
- You need a historical backfill that continues as a live stream.
- Cursor recovery forms part of the production design.
Feed your database, analytics pipeline, or event processor with standard Substreams package output over gRPC.
filtered_events
Request parameters
ethereum-common@v0.4.0 · USDC Transfer
Request excerpt. See the service quickstart for authentication and a complete example.
Package and stream parameters
{
"package": "ethereum-common@v0.4.0",
"output_module": "filtered_events",
"params": "evt_addr:0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48 && evt_sig:0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef",
"start_block_num": -1,
"final_blocks_only": false
} Substreams reads blockchain data through a package of modules. The package decides which blocks are read and how they are transformed, and your consumer receives one module's output in block order over gRPC. GraphOps Substreams runs standard packages, so you can backfill historical blocks and continue with a live stream. Your consumer stores the output in your own database or processor, commits data before saving each cursor, and handles undo signals for reversible blocks.
Substreams delivers package output over gRPC, and your consumer turns it into state. These patterns cover the decisions that keep that state complete and correct.
01 start block · stop block · cursor
Load a protocol's event history into your database, then keep it current from the same consumer. Backfills and live streaming use the same stream protocol: start at the block your history needs, leave the stop block open, and the stream continues as new blocks arrive. Historical ranges can run in parallel workers, up to your organization's worker limits.
02 BlockScopedData · cursor
Every block-scoped message carries output and a cursor. Write both in one database transaction where you can, so the stored cursor always matches committed data. After a disconnect, reconnect with backoff and resume from the latest committed cursor. Replays from that point must be safe, so make writes idempotent.
BlockScopedData output to your tables.03 BlockUndoSignal · last valid block
Near the chain head, a block can be replaced. Substreams then sends a BlockUndoSignal with the last valid block and its cursor. Your consumer reverts what it wrote after that block, saves the new cursor, and continues with the replacement blocks.
BlockUndoSignal, revert rows above the last valid block.final_blocks_only, so output trails the chain head.04 .spkg · db_out · substreams sink
Reuse a package from the Substreams registry or run your own .spkg, then choose the output module your pipeline needs. Standard sinks write it to Postgres, ClickHouse, files, or Pub/Sub, or your application consumes the stream directly. The database stays yours to query, join, and retain.
.spkg, and pin its version.db_out for the SQL sink.GraphOps runs the package against chain data and streams its output. Everything after the stream, from cursors to tables, belongs to your consumer.
Package compatibility depends on the chain data model.
See supported networksx-api-key header or SUBSTREAMS_API_KEY. Route-token authentication is not supported.Each network has its own endpoint, from the live catalog.
Go from a new organization to a first response, then prepare for production traffic.
Choose your monthly credits or buy a top-up. Streams consume credits by the current billable components.
The Substreams CLI reads SUBSTREAMS_API_KEY, and the GraphOps CLI can store the key and add it, with the endpoint, to your substreams commands. Raw gRPC clients send an x-api-key header.
Pick a package and output module, then the endpoint for the chain it supports.
Run the pinned Ethereum package over six blocks to inspect USDC Transfer events.
Pin package versions, persist cursors durably, handle undo signals, and test recovery before processing a larger range.
https://<CHAIN_ID>.evm.substreams.graphops.xyz Use standard Substreams tooling with this endpoint. Its hostname starts with the decimal EVM chain ID of a supported network.
Prerequisites
Install the Substreams CLI.
Set GRAPHOPS_API_KEY to your GraphOps API key.
CLI · Substreams CLI
# The CLI sends SUBSTREAMS_API_KEY to GraphOps in the x-api-key header.
export SUBSTREAMS_API_KEY="${GRAPHOPS_API_KEY}"
# Stream USDC Transfer events (--params) for six blocks from 18000002 as JSON lines.
substreams run \
-e "https://1.evm.substreams.graphops.xyz" \
"https://spkg.io/v1/packages/ethereum-common/v0.4.0" \
"filtered_events" \
--params "filtered_events=evt_addr:0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48 && evt_sig:0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef" \
--start-block 18000002 \
--stop-block +6 \
-o jsonl
import { createHash } from "node:crypto";
import { createGrpcTransport, Http2SessionManager } from "@connectrpc/connect-node";
import {
applyParams,
createRegistry,
createRequest,
createSubstream,
streamBlocks,
unpackMapOutput,
} from "@substreams/core";
const endpoint = "https://1.evm.substreams.graphops.xyz";
const apiKey = process.env.GRAPHOPS_API_KEY;
if (!apiKey) throw new Error("Set GRAPHOPS_API_KEY");
// Check the package's SHA-256, so the program only runs the exact module code you reviewed.
const download = await fetch("https://spkg.io/v1/packages/ethereum-common/v0.4.0");
if (!download.ok) throw new Error(`Package download failed: ${download.status}`);
const bytes = new Uint8Array(await download.arrayBuffer());
const checksum = createHash("sha256").update(bytes).digest("hex");
if (checksum !== "beda75cef8a655941816a7c2be222ed590f618fe389e7c08c3ac5f0826161345") {
throw new Error("Package checksum mismatch");
}
const pkg = createSubstream(bytes);
if (!pkg.modules) throw new Error("Package has no modules");
// Keep Transfer events (evt_sig) emitted by the USDC contract (evt_addr).
applyParams(
[
"filtered_events=evt_addr:0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48 && evt_sig:0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef",
],
pkg.modules.modules,
);
// The package embeds its protobuf schemas, so the output decodes without generated code.
const registry = createRegistry(pkg);
const request = createRequest({
substreamPackage: pkg,
outputModule: "filtered_events",
startBlockNum: 18000002,
stopBlockNum: 18000008, // exclusive
// Unset starts at startBlockNum; a committed cursor resumes right after its block.
startCursor: process.env.GRAPHOPS_CURSOR || undefined,
// Production mode processes the range in parallel and sends only the output module.
productionMode: true,
// Only irreversible blocks, so the server never sends a blockUndoSignal (reorg rewind).
finalBlocksOnly: true,
});
const sessionManager = new Http2SessionManager(endpoint);
const transport = createGrpcTransport({ baseUrl: endpoint, httpVersion: "2", sessionManager });
// GraphOps authenticates every gRPC call with the x-api-key metadata header.
const headers = { "x-api-key": apiKey };
try {
for await (const response of streamBlocks(transport, request, { headers })) {
// Skip session, progress and other non-data messages.
if (response.message.case !== "blockScopedData") continue;
const { clock, cursor } = response.message.value;
const output = unpackMapOutput(response, registry);
const data = output?.toJson() as { events?: unknown[] } | undefined;
if (!data?.events) continue; // no matching transfers in this block
// Store the events and this cursor together; after a restart, pass the last stored
// cursor as startCursor to continue from the next block without gaps or duplicates.
console.log(JSON.stringify({ block: String(clock?.number), cursor, data }));
}
} finally {
sessionManager.abort(); // Close the HTTP/2 connection so the process can exit.
}
import hashlib
import json
import os
import grpc
import requests
from google.protobuf import descriptor_pool, json_format, message_factory
# Generated Substreams protocol bindings, included in the example project.
from sf.substreams.rpc.v2 import service_pb2, service_pb2_grpc
from sf.substreams.v1 import package_pb2
ENDPOINT = "https://1.evm.substreams.graphops.xyz"
PACKAGE_URL = "https://spkg.io/v1/packages/ethereum-common/v0.4.0"
PACKAGE_SHA256 = "beda75cef8a655941816a7c2be222ed590f618fe389e7c08c3ac5f0826161345"
MODULE = "filtered_events"
OUTPUT_TYPE = "sf.substreams.ethereum.v1.Events"
# Keep Transfer events (evt_sig) emitted by the USDC contract (evt_addr).
PARAMS = "evt_addr:0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48 && evt_sig:0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef"
api_key = os.environ["GRAPHOPS_API_KEY"]
# Check the package's SHA-256, so the program only runs the exact module code you reviewed.
download = requests.get(PACKAGE_URL, timeout=30)
download.raise_for_status()
if hashlib.sha256(download.content).hexdigest() != PACKAGE_SHA256:
raise RuntimeError("Package checksum mismatch")
package = package_pb2.Package.FromString(download.content)
# Set the module's params input, like `substreams run --params`.
for module in package.modules.modules:
if module.name == MODULE:
for module_input in module.inputs:
if module_input.HasField("params"):
module_input.params.value = PARAMS
# The package embeds its protobuf schemas, so the output decodes without generated code.
pool = descriptor_pool.DescriptorPool()
for proto_file in package.proto_files:
pool.Add(proto_file)
Events = message_factory.GetMessageClass(pool.FindMessageTypeByName(OUTPUT_TYPE))
# Trust gRPC's default roots, or the CA bundle in SSL_CERT_FILE when set.
roots = None
if os.environ.get("SSL_CERT_FILE"):
with open(os.environ["SSL_CERT_FILE"], "rb") as bundle:
roots = bundle.read()
credentials = grpc.ssl_channel_credentials(root_certificates=roots)
with grpc.secure_channel(ENDPOINT.removeprefix("https://"), credentials) as channel:
request = service_pb2.Request(
modules=package.modules,
output_module=MODULE,
start_block_num=18000002,
stop_block_num=18000008, # exclusive
# Empty starts at start_block_num; a committed cursor resumes right after its block.
start_cursor=os.environ.get("GRAPHOPS_CURSOR", ""),
# Production mode processes the range in parallel and sends only the output module.
production_mode=True,
# Only irreversible blocks, so the server never sends a BlockUndoSignal (reorg rewind).
final_blocks_only=True,
)
# GraphOps authenticates every gRPC call with the x-api-key metadata header.
stream = service_pb2_grpc.StreamStub(channel).Blocks(
request, metadata=[("x-api-key", api_key)]
)
for response in stream:
# Skip session, progress and other non-data messages.
if not response.HasField("block_scoped_data"):
continue
data = response.block_scoped_data
events = json_format.MessageToDict(
Events.FromString(data.output.map_output.value)
)
if "events" not in events:
continue # no matching transfers in this block
# Store the events and this cursor together; after a restart, pass the last stored
# cursor as start_cursor to continue from the next block without gaps or duplicates.
record = {"block": str(data.clock.number), "cursor": data.cursor, "data": events}
print(json.dumps(record))
package main
import (
"context"
"crypto/sha256"
"crypto/tls"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"log"
"net/http"
"os"
"strings"
rpc "github.com/streamingfast/substreams/pb/sf/substreams/rpc/v2"
pb "github.com/streamingfast/substreams/pb/sf/substreams/v1"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials"
"google.golang.org/grpc/metadata"
"google.golang.org/protobuf/encoding/protojson"
"google.golang.org/protobuf/proto"
"google.golang.org/protobuf/reflect/protodesc"
"google.golang.org/protobuf/reflect/protoreflect"
"google.golang.org/protobuf/types/descriptorpb"
"google.golang.org/protobuf/types/dynamicpb"
)
const (
endpoint = "https://1.evm.substreams.graphops.xyz"
packageURL = "https://spkg.io/v1/packages/ethereum-common/v0.4.0"
packageSHA256 = "beda75cef8a655941816a7c2be222ed590f618fe389e7c08c3ac5f0826161345"
module = "filtered_events"
outputType = "sf.substreams.ethereum.v1.Events"
// Keep Transfer events (evt_sig) emitted by the USDC contract (evt_addr).
params = "evt_addr:0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48 && evt_sig:0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef"
)
func main() {
if err := run(); err != nil {
log.Fatal(err)
}
}
func run() error {
apiKey := os.Getenv("GRAPHOPS_API_KEY")
if apiKey == "" {
return errors.New("set GRAPHOPS_API_KEY")
}
pkg, err := downloadPackage()
if err != nil {
return err
}
// Set the module's params input, like `substreams run --params`.
for _, m := range pkg.Modules.Modules {
if m.Name != module {
continue
}
for _, input := range m.Inputs {
if p := input.GetParams(); p != nil {
p.Value = params
}
}
}
// The package embeds its protobuf schemas, so the output decodes without generated code.
files, err := protodesc.NewFiles(&descriptorpb.FileDescriptorSet{File: pkg.ProtoFiles})
if err != nil {
return err
}
descriptor, err := files.FindDescriptorByName(outputType)
if err != nil {
return err
}
eventsType := descriptor.(protoreflect.MessageDescriptor)
// Trust the system roots, or the CA bundle in SSL_CERT_FILE when set (Go ignores it on macOS).
creds := credentials.NewTLS(&tls.Config{})
if caFile := os.Getenv("SSL_CERT_FILE"); caFile != "" {
if creds, err = credentials.NewClientTLSFromFile(caFile, ""); err != nil {
return err
}
}
conn, err := grpc.NewClient(strings.TrimPrefix(endpoint, "https://"), grpc.WithTransportCredentials(creds))
if err != nil {
return err
}
defer conn.Close()
// GraphOps authenticates every gRPC call with the x-api-key metadata header.
ctx := metadata.AppendToOutgoingContext(context.Background(), "x-api-key", apiKey)
stream, err := rpc.NewStreamClient(conn).Blocks(ctx, &rpc.Request{
Modules: pkg.Modules,
OutputModule: module,
StartBlockNum: 18000002,
StopBlockNum: 18000008, // exclusive
// Empty starts at StartBlockNum; a committed cursor resumes right after its block.
StartCursor: os.Getenv("GRAPHOPS_CURSOR"),
// Production mode processes the range in parallel and sends only the output module.
ProductionMode: true,
// Only irreversible blocks, so the server never sends a BlockUndoSignal (reorg rewind).
FinalBlocksOnly: true,
})
if err != nil {
return err
}
for {
response, err := stream.Recv()
if errors.Is(err, io.EOF) {
return nil // reached StopBlockNum
}
if err != nil {
return err
}
data := response.GetBlockScopedData()
if data == nil {
continue // session, progress and other non-data messages
}
output := dynamicpb.NewMessage(eventsType)
if err := proto.Unmarshal(data.GetOutput().GetMapOutput().GetValue(), output); err != nil {
return err
}
outputJSON, err := protojson.Marshal(output)
if err != nil {
return err
}
var payload map[string]any
if err := json.Unmarshal(outputJSON, &payload); err != nil {
return err
}
if payload["events"] == nil {
continue // no matching transfers in this block
}
// Store the events and this cursor together; after a restart, pass the last stored
// cursor as StartCursor to continue from the next block without gaps or duplicates.
record := map[string]any{"block": fmt.Sprint(data.Clock.Number), "cursor": data.Cursor, "data": payload}
if err := json.NewEncoder(os.Stdout).Encode(record); err != nil {
return err
}
}
}
// downloadPackage fetches the Substreams package and checks its SHA-256, so the program only
// ever runs the exact module code that was reviewed.
func downloadPackage() (*pb.Package, error) {
response, err := http.Get(packageURL)
if err != nil {
return nil, err
}
defer response.Body.Close()
if response.StatusCode != http.StatusOK {
return nil, fmt.Errorf("package download: %s", response.Status)
}
content, err := io.ReadAll(response.Body)
if err != nil {
return nil, err
}
if sum := sha256.Sum256(content); hex.EncodeToString(sum[:]) != packageSHA256 {
return nil, errors.New("package checksum mismatch")
}
pkg := &pb.Package{}
return pkg, proto.Unmarshal(content, pkg)
}
// build.rs generates the Substreams protocol types and client with Buffa and connectrpc.
mod pb {
connectrpc::include_generated!();
}
use buffa::Message;
use buffa_descriptor::{generated::descriptor::FileDescriptorSet, DescriptorPool, DynamicMessage};
use connectrpc::client::{ClientConfig, HttpClient};
use connectrpc::{rustls, Protocol};
use pb::sf::substreams::rpc::v2::{response, Request, StreamClient};
use pb::sf::substreams::v1::{module::input::Input, Package};
use ring::digest::{digest, SHA256};
use serde_json::json;
use std::{env, error::Error, sync::Arc};
const ENDPOINT: &str = "https://1.evm.substreams.graphops.xyz";
const PACKAGE_URL: &str = "https://spkg.io/v1/packages/ethereum-common/v0.4.0";
const PACKAGE_SHA256: &str = "beda75cef8a655941816a7c2be222ed590f618fe389e7c08c3ac5f0826161345";
const MODULE: &str = "filtered_events";
const OUTPUT_TYPE: &str = "sf.substreams.ethereum.v1.Events";
// Keep Transfer events (evt_sig) emitted by the USDC contract (evt_addr).
const PARAMS: &str = "evt_addr:0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48 && evt_sig:0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef";
#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
let api_key = env::var("GRAPHOPS_API_KEY")?;
// Check the package's SHA-256, so the program only runs the exact module code you reviewed.
let bytes = reqwest::get(PACKAGE_URL)
.await?
.error_for_status()?
.bytes()
.await?;
let checksum: String = digest(&SHA256, &bytes)
.as_ref()
.iter()
.map(|byte| format!("{byte:02x}"))
.collect();
if checksum != PACKAGE_SHA256 {
return Err("package checksum mismatch".into());
}
let mut package = Package::decode_from_slice(&bytes)?;
// Set the module's params input, like `substreams run --params`.
let module = package
.modules
.as_option_mut()
.ok_or("package has no modules")?
.modules
.iter_mut()
.find(|module| module.name == MODULE)
.ok_or("module not found")?;
for input in &mut module.inputs {
if let Some(Input::Params(params)) = &mut input.input {
params.value = PARAMS.into();
}
}
// The package embeds its protobuf schemas, so the output decodes without generated code.
let pool = Arc::new(DescriptorPool::new(FileDescriptorSet {
file: std::mem::take(&mut package.proto_files),
..Default::default()
})?);
let events_type = pool
.message_index(OUTPUT_TYPE)
.ok_or("output type not found")?;
// Trust the system roots, or the CA bundle in SSL_CERT_FILE when set.
let mut roots = rustls::RootCertStore::empty();
roots.add_parsable_certificates(rustls_native_certs::load_native_certs().certs);
let tls = rustls::ClientConfig::builder()
.with_root_certificates(roots)
.with_no_client_auth();
// GraphOps authenticates every gRPC call with the x-api-key metadata header.
let config = ClientConfig::new(ENDPOINT.parse()?)
.with_protocol(Protocol::Grpc)
.with_default_header("x-api-key", api_key);
let client = StreamClient::new(HttpClient::with_tls(Arc::new(tls)), config);
let mut stream = client
.blocks(Request {
modules: package.modules,
output_module: MODULE.into(),
start_block_num: 18000002,
stop_block_num: 18000008, // exclusive
// Empty starts at start_block_num; a committed cursor resumes right after its block.
start_cursor: env::var("GRAPHOPS_CURSOR").unwrap_or_default(),
// Production mode processes the range in parallel and sends only the output module.
production_mode: true,
// Only irreversible blocks, so the server never sends a BlockUndoSignal (reorg rewind).
final_blocks_only: true,
..Default::default()
})
.await?;
while let Some(message) = stream.message().await? {
// Skip session, progress and other non-data messages.
let Some(response::Message::BlockScopedData(data)) = message.to_owned_message().message
else {
continue;
};
let output = &data.output.map_output;
let events = DynamicMessage::decode(pool.clone(), events_type, &output.value)?;
let events = serde_json::to_value(&events)?;
if events.get("events").is_none() {
continue; // no matching transfers in this block
}
// Store the events and this cursor together; after a restart, pass the last stored
// cursor as start_cursor to continue from the next block without gaps or duplicates.
let record = json!({
"block": data.clock.number.to_string(),
"cursor": data.cursor,
"data": events,
});
println!("{record}");
}
Ok(())
}
Add credits and follow the quickstart, then test cursor recovery and undo handling before processing a larger range.