Hyperledger Fabric New External Peer Stuck at Block 0 — Root Cause and Fix

While adding a new external peer node into an existing Hyperledger Fabric network, I encountered an interesting bootstrap issue.

The peer successfully:

• joined the channel using genesis block (block #0)
• installed chaincode
• connected to gossip peers

But it never synchronized beyond block 0, even though the network had already reached block 210.

After extensive debugging involving gossip configuration, TLS verification, MSP validation, and delivery service behavior, the final fix turned out to be enabling:

CORE_PEER_GOSSIP_ORGLEADER=true

This article documents the entire investigation, why the issue happens, and how to properly bootstrap a new peer in this scenario.

Network Scenario

Existing network state:

• Channel already running
• Current channel height: block 210
• Existing peers:
• peer0-org2
• peer1-org2
• peer2-org2
• other org peers

New node added:

peer4-org2

The new peer:

• joined successfully with genesis block
• chaincode installed successfully
• peer process started normally

But ledger synchronization never happened.

Initial Environment Configuration

The peer was configured as:

CORE_PEER_LOCALMSPID=org2

CORE_PEER_GOSSIP_USELEADERELECTION=false
CORE_PEER_GOSSIP_ORGLEADER=false

CORE_PEER_GOSSIP_STATE_ENABLED=true
CORE_PEER_GOSSIP_DISCOVERY=true

CORE_PEER_GOSSIP_BOOTSTRAP=peer0-org2:30000

CORE_PEER_ADDRESS=peer4-org2:30000
CORE_PEER_GOSSIP_EXTERNALENDPOINT=peer4-org2:30000

CORE_PEER_DELIVERYCLIENT_ENABLED=true

The intention was:

• peer should behave as a regular follower/reference peer
• synchronization should happen from existing anchor peers
• no leader election participation required

Symptoms

The peer always stayed at block 0.

The startup logs showed:

Tried joining channel newchannel but our org(org2),
isn't among the orgs of the channel: [org1]

And the anchor peer logs showed:

Failed obtaining connection for peer4-org2:30000
reason: context deadline exceeded

Even after:

• enabling gossip discovery
• enabling state transfer
• fixing external endpoint configuration
• validating DNS reachability
• validating TLS certificates

the peer still refused to synchronize.

What Was Initially Suspected

Several possible causes were investigated:

1. Gossip Discovery Disabled

Initially:

CORE_PEER_GOSSIP_DISCOVERY=false

This was corrected to:

CORE_PEER_GOSSIP_DISCOVERY=true

2. State Transfer Disabled

Initially:

CORE_PEER_GOSSIP_STATE_ENABLED=false

This prevented block synchronization completely.

Corrected to:

CORE_PEER_GOSSIP_STATE_ENABLED=true

3. External Endpoint Mismatch

Originally:

CORE_PEER_ADDRESS=peer4-org2:30000
CORE_PEER_GOSSIP_EXTERNALENDPOINT=peer4-org2:30000

This caused inconsistent endpoint advertisement.

Corrected to:

CORE_PEER_ADDRESS=peer4-org2-dom.com:30000
CORE_PEER_GOSSIP_EXTERNALENDPOINT=peer4-org2-dom.com:30000

Connectivity Verification

Network connectivity was verified successfully.

From peer4:

nc -vz peer0-org2 30000

Result:

Connection succeeded

TLS verification also worked:

openssl s_client -connect peer0-org2:30000

This confirmed:

• DNS resolution working
• firewall/network path working
• TLS handshake working

The Actual Root Cause

The important detail was this:

The new peer joined using block 0.

At block 0:

• only org1 existed in channel configuration
• org2 MSP had not yet been added into the channel

Therefore:

• gossip membership validation could not fully initialize
• peer was not yet considered a valid channel member
• follower-mode synchronization never started

This created a bootstrap deadlock.

Why Follower Mode Failed

The peer was configured as:

CORE_PEER_GOSSIP_USELEADERELECTION=false
CORE_PEER_GOSSIP_ORGLEADER=false

This means:

• peer depends on existing org leader peers
• synchronization depends on channel gossip
• peer expects already-valid channel membership

But since the peer started from an earlier channel config epoch (before org2 org existed), gossip synchronization never activated.

As a result:

• state transfer never started
• config blocks introducing org2MSP were never received
• peer stayed permanently at block 0

The Fix That Finally Worked

The solution was to temporarily make the peer pull blocks directly from the ordering service.

Configuration changed to:

CORE_PEER_GOSSIP_USELEADERELECTION=false
CORE_PEER_GOSSIP_ORGLEADER=true

After restarting the peer:

• deliver service connected directly to orderer
• blocks started synchronizing immediately
• peer caught up from block 0 to block 210

Logs finally showed:

Committed block [1]
Committed block [2]
Committed block [3]
...

Why This Works

Setting:

CORE_PEER_GOSSIP_ORGLEADER=true

does NOT mean:

• peer becomes anchor peer
• peer participates in election
• peer becomes cluster leader

It simply means:

this peer is allowed to pull blocks directly from the orderer.

This bypasses the gossip bootstrap dependency problem.

Once the peer receives the later config blocks where org2 MSP is added, the peer becomes a fully valid channel participant.

Important Hyperledger Fabric Bootstrap Behavior

This issue appears specifically when:

1. a new peer joins using an old genesis block
2. the peer’s org was introduced only in later config blocks
3. the peer starts in follower-only mode

In such cases, follower gossip synchronization alone may not bootstrap the peer successfully.

Recommended Bootstrap Procedure

For peers joining historical channels:

Temporary Bootstrap Mode

Use:

CORE_PEER_GOSSIP_USELEADERELECTION=false
CORE_PEER_GOSSIP_ORGLEADER=true

CORE_PEER_GOSSIP_STATE_ENABLED=true
CORE_PEER_GOSSIP_DISCOVERY=true

Allow the peer to fully synchronize.

After Bootstrap Completion

Once the peer catches up completely:

peer channel getinfo -c newchannel

Verify latest height matches network.

Then optionally revert:

CORE_PEER_GOSSIP_ORGLEADER=false

and restart the peer.

At that point:

• peer already knows all channel MSPs
• gossip membership works normally
• future synchronization occurs via gossip

In many production environments, however, keeping ORGLEADER enabled is also acceptable.

Key Takeaways

1. Joining with block 0 is valid

Fabric peers must join using genesis/join block.

Later config blocks cannot be used directly for peer join.

2. Gossip synchronization depends on valid channel membership

If the org does not yet exist in the peer’s current config view, gossip bootstrap may stall.

3. ORGLEADER=true can solve historical bootstrap deadlocks

This enables direct deliver-service synchronization from orderer.

4. Connectivity alone is not enough

Even when:

• DNS works
• TLS works
• gossip endpoints work

the peer may still fail due to channel membership epoch mismatch.

Final Working Configuration

CORE_PEER_GOSSIP_USELEADERELECTION=false
CORE_PEER_GOSSIP_ORGLEADER=true

CORE_PEER_GOSSIP_STATE_ENABLED=true
CORE_PEER_GOSSIP_DISCOVERY=true

CORE_PEER_ADDRESS=peer4-org2:30000
CORE_PEER_GOSSIP_EXTERNALENDPOINT=peer4-org2:30000

Conclusion

This was an interesting Hyperledger Fabric bootstrap edge case involving:

• historical channel config evolution
• gossip membership validation
• follower-mode synchronization
• direct delivery service fallback

Hopefully this helps others facing peers stuck permanently at block 0 while joining existing Fabric channels.