IBM Support

Bookkeeper process startup fails with InvalidCookieException

Troubleshooting


Problem

Bookkeeper process startup fails with InvalidCookieException

Symptom

  • Bookkeeper service fails to restart.
  • Bookkeeper log:
[main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server org.apache.bookkeeper.bookie.BookieException$InvalidCookieException:    at org.apache.bookkeeper.bookie.LegacyCookieValidation.checkCookies(LegacyCookieValidation.java:119) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]    at org.apache.bookkeeper.server.EmbeddedServer$Builder.build(EmbeddedServer.java:408) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]    at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:277) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]    at org.apache.bookkeeper.server.Main.doMain(Main.java:216) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
at org.apache.bookkeeper.server.Main.main(Main.java:199) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]

Cause

  • Bookie Decommissioning (most common): When a bookie is explicitly decommissioned via command/API, the Auditor migrates ledger data to other bookies and deletes the corresponding cookie from ZooKeeper upon completion.
    • If the decommissioned bookie is restarted (either accidentally or intentionally), it still has:
      • Local data directories with the old cookie file intact
      • Ledger data that may now be stale or duplicated elsewhere
    • However, ZooKeeper no longer recognizes this bookie because its cookie was deleted during decommissioning. When the bookie attempts to start, it performs a cookie validation check:
      • Reads the local cookie from disk
      • Attempts to find the matching cookie in ZooKeeper
      • Fails because the cookie no longer exists in ZooKeeper
  • Auditor-triggered Auto-repair (placement policy violation): If repairedPlacementPolicyNotAdheringBookieEnabled=true is set in bookeeper.conf, the Auditor may automatically decommission bookies that violate placement policies (e.g., rack/zone rules), leading to cookie deletion as part of the process.
  • Bookie Format/Reformat Operation: Running the bookieformat command (with force) deletes the existing cookie, wipes local data, and recreates a new cookie during reinitialization.

Environment

Apache Pulsar 3.0.11

Resolving The Problem

  • Locate the Cookie file

The local cookie file is named `VERSION` and is stored in the `current` subdirectory of each data directory configured for the bookie.

grep -E "journalDirectory|ledgerDirectories|indexDirectories" conf/bk_server.conf

Example output:

journalDirectory=/data/bookkeeper_journal
ledgerDirectories=/data/bookkeeper_ledgers
indexDirectories=/data/bookkeeper_index

 

Cookie files will be located at:

Journal: `/data/bookkeeper_journal/current/VERSION`
Ledgers: `/data/bookkeeper_ledgers/current/VERSION`
Index: `/data/bookkeeper_index/current/VERSION`

 

Once location is determined, verify it's a valid cookie:

cat /data/bookkeeper_ledgers/current/VERSION

 

Expected content:

5
bookieHost: "server1.example.com:3181"
journalDir: "/data/bookkeeper_journal"
ledgerDirs: "1\t/data/bookkeeper_ledgers"
instanceId: "a1b2c3d4-e5f6-7890-abcd-ef1234567890"

 

  • Restore local cookie in the Zookeeper

Run this command from a working bookie server or any server with BookKeeper shell access:

bin/bookkeeper shell cookie_create -cf /data/bookkeeper_ledgers/current/VERSION <IP_of_problematic_bookkeeper_instance>:3181

The file /data/bookkeeper_ledgers/current/VERSION is located on the bookie that’s down.

 

  • Verify that the cookie was created
bin/bookkeeper shell cookie_get <IP_of_problematic_bookkeeper_instance>:3181

 

  • Verify bookie appears in the list
bin/bookkeeper shell listbookies -all

 

  • Start bookkeeper

On the problematic bookkeeper server, start the BookKeeper service:

systemctl start bookkeeper

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB76","label":"Data Platform"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSQWIX","label":"DataStax Luna"},"ARM Category":[{"code":"a8mgJ0000000GPlQAM","label":"Drupal Knowledge Base Article"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":""}]

Document Information

Modified date:
08 May 2026

UID

ibm17272492

Page Feedback