mirror of
https://github.com/varun-r-mallya/py-libp2p.git
synced 2025-12-31 20:36:24 +00:00
d61bca78ab
* initialise the module * added content routing * added routing module * added peer routing * added value store * added utilities functions * added main kademlia file * fixed create_key_from_binary function * example to test kademlia dht * added protocol ID and enhanced logging for peer store size in provider and consumer nodes * refactor: specify stream type in handle_stream method and add peer in routing table * removed content routing * added default value of count for finding closest peers * added functions to find close peers * refactor: remove content routing and enhance peer discovery * added put value function * added get value function * fix: improve logging and handle key encoding in get_value method * refactor: remove ContentRouting import from __init__.py * refactor: improved basic kademlia example * added protobuf files * replaced json with protobuf * refactor: enhance peer discovery and routing logic in KadDHT * refactor: enhance Kademlia routing table to use PeerInfo objects and improve peer management * refactor: enhance peer addition logic to utilize PeerInfo objects in routing table * feat: implement content provider functionality in Kademlia DHT * refactor: update value store to use datetime for validity management * refactor: update RoutingTable initialization to include host reference * refactor: enhance KBucket and RoutingTable for improved peer management and functionality * refactor: streamline peer discovery and value storage methods in KadDHT * refactor: update KadDHT and related classes for async peer management and enhanced value storage * refactor: enhance ProviderStore initialization and improve peer routing integration * test: add tests for Kademlia DHT functionality * fix linting issues * pydocstyle issues fixed * CICD pipeline issues solved * fix: update docstring format for find_peer method * refactor: improve logging and remove unused code in DHT implementation * refactor: clean up logging and remove unused imports in DHT and test files * Refactor logging setup and improve DHT stream handling with varint length prefixes * Update bootstrap peer handling in basic_dht example and refactor peer routing to accept string addresses * Enhance peer querying in Kademlia DHT by implementing parallel queries using Trio. * Enhance peer querying by adding deduplication checks * Refactor DHT implementation to use varint for length prefixes and enhance logging for better traceability * Add base58 encoding for value storage and enhance logging in basic_dht example * Refactor Kademlia DHT to support server/client modes * Added unit tests * Refactor documentation to fixsome warning * Add unit tests and remove outdated tests * Fixed precommit errora * Refactor error handling test to raise StringParseError for invalid bootstrap addresses * Add libp2p.kad_dht to the list of subpackages in documentation * Fix expiration and republish checks to use inclusive comparison * Add __init__.py file to libp2p.kad_dht.pb package * Refactor get value and put value to run in parallel with query timeout * Refactor provider message handling to use parallel processing with timeout * Add methods for provider store in KadDHT class * Refactor KadDHT and ProviderStore methods to improve type hints and enhance parallel processing * Add documentation for libp2p.kad_dht.pb module. * Update documentation for libp2p.kad_dht package to include subpackages and correct formatting * Fix formatting in documentation for libp2p.kad_dht package by correcting the subpackage reference * Fix header formatting in libp2p.kad_dht.pb documentation * Change log level from info to debug for various logging statements. * fix CICD issues (post revamp) * fixed value store unit test * Refactored kademlia example * Refactor Kademlia example: enhance logging, improve bootstrap node connection, and streamline server address handling * removed bootstrap module * Refactor Kademlia DHT example and core modules: enhance logging, remove unused code, and improve peer handling * Added docs of kad dht example * Update server address log file path to use the script's directory * Refactor: Introduce DHTMode enum for clearer mode management * moved xor_distance function to utils.py * Enhance logging in ValueStore and KadDHT: include decoded value in debug logs and update parameter description for validity * Add handling for closest peers in GET_VALUE response when value is not found * Handled failure scenario for PUT_VALUE * Remove kademlia demo from project scripts and contributing documentation * spelling and logging --------- Co-authored-by: pacrob <5199899+pacrob@users.noreply.github.com>
125 lines
6.0 KiB
ReStructuredText
125 lines
6.0 KiB
ReStructuredText
Kademlia DHT Demo
|
|
=================
|
|
|
|
This example demonstrates a Kademlia Distributed Hash Table (DHT) implementation with both value storage/retrieval and content provider advertisement/discovery functionality.
|
|
|
|
.. code-block:: console
|
|
|
|
$ python -m pip install libp2p
|
|
Collecting libp2p
|
|
...
|
|
Successfully installed libp2p-x.x.x
|
|
$ cd examples/kademlia
|
|
$ python kademlia.py --mode server
|
|
2025-06-13 19:51:25,424 - kademlia-example - INFO - Running in server mode on port 0
|
|
2025-06-13 19:51:25,426 - kademlia-example - INFO - Connected to bootstrap nodes: []
|
|
2025-06-13 19:51:25,426 - kademlia-example - INFO - To connect to this node, use: --bootstrap /ip4/127.0.0.1/tcp/28910/p2p/16Uiu2HAm7EsNv5vvjPAehGAVfChjYjD63ZHyWogQRdzntSbAg9ef
|
|
2025-06-13 19:51:25,426 - kademlia-example - INFO - Saved server address to log: /ip4/127.0.0.1/tcp/28910/p2p/16Uiu2HAm7EsNv5vvjPAehGAVfChjYjD63ZHyWogQRdzntSbAg9ef
|
|
2025-06-13 19:51:25,427 - kademlia-example - INFO - DHT service started in SERVER mode
|
|
2025-06-13 19:51:25,427 - kademlia-example - INFO - Stored value 'Hello message from Sumanjeet' with key: FVDjasarSFDoLPMdgnp1dHSbW2ZAfN8NU2zNbCQeczgP
|
|
2025-06-13 19:51:25,427 - kademlia-example - INFO - Successfully advertised as server for content: 361f2ed1183bca491b8aec11f0b9e5c06724759b0f7480ae7fb4894901993bc8
|
|
|
|
|
|
Copy the line that starts with ``--bootstrap``, open a new terminal in the same folder and run the client:
|
|
|
|
.. code-block:: console
|
|
|
|
$ python kademlia.py --mode client --bootstrap /ip4/127.0.0.1/tcp/28910/p2p/16Uiu2HAm7EsNv5vvjPAehGAVfChjYjD63ZHyWogQRdzntSbAg9ef
|
|
2025-06-13 19:51:37,022 - kademlia-example - INFO - Running in client mode on port 0
|
|
2025-06-13 19:51:37,026 - kademlia-example - INFO - Connected to bootstrap nodes: [<libp2p.peer.id.ID (16Uiu2HAm7EsNv5vvjPAehGAVfChjYjD63ZHyWogQRdzntSbAg9ef)>]
|
|
2025-06-13 19:51:37,027 - kademlia-example - INFO - DHT service started in CLIENT mode
|
|
2025-06-13 19:51:37,027 - kademlia-example - INFO - Looking up key: FVDjasarSFDoLPMdgnp1dHSbW2ZAfN8NU2zNbCQeczgP
|
|
2025-06-13 19:51:37,031 - kademlia-example - INFO - Retrieved value: Hello message from Sumanjeet
|
|
2025-06-13 19:51:37,031 - kademlia-example - INFO - Looking for servers of content: 361f2ed1183bca491b8aec11f0b9e5c06724759b0f7480ae7fb4894901993bc8
|
|
2025-06-13 19:51:37,035 - kademlia-example - INFO - Found 1 servers for content: ['16Uiu2HAm7EsNv5vvjPAehGAVfChjYjD63ZHyWogQRdzntSbAg9ef']
|
|
|
|
Alternatively, if you run the server first, the client can automatically extract the bootstrap address from the server log file:
|
|
|
|
.. code-block:: console
|
|
|
|
$ python kademlia.py --mode client
|
|
2025-06-13 19:51:37,022 - kademlia-example - INFO - Running in client mode on port 0
|
|
2025-06-13 19:51:37,026 - kademlia-example - INFO - Connected to bootstrap nodes: [<libp2p.peer.id.ID (16Uiu2HAm7EsNv5vvjPAehGAVfChjYjD63ZHyWogQRdzntSbAg9ef)>]
|
|
2025-06-13 19:51:37,027 - kademlia-example - INFO - DHT service started in CLIENT mode
|
|
2025-06-13 19:51:37,027 - kademlia-example - INFO - Looking up key: FVDjasarSFDoLPMdgnp1dHSbW2ZAfN8NU2zNbCQeczgP
|
|
2025-06-13 19:51:37,031 - kademlia-example - INFO - Retrieved value: Hello message from Sumanjeet
|
|
2025-06-13 19:51:37,031 - kademlia-example - INFO - Looking for servers of content: 361f2ed1183bca491b8aec11f0b9e5c06724759b0f7480ae7fb4894901993bc8
|
|
2025-06-13 19:51:37,035 - kademlia-example - INFO - Found 1 servers for content: ['16Uiu2HAm7EsNv5vvjPAehGAVfChjYjD63ZHyWogQRdzntSbAg9ef']
|
|
|
|
The demo showcases key DHT operations:
|
|
|
|
- **Value Storage & Retrieval**: The server stores a value, and the client retrieves it
|
|
- **Content Provider Discovery**: The server advertises content, and the client finds providers
|
|
- **Peer Discovery**: Automatic bootstrap and peer routing using the Kademlia algorithm
|
|
- **Network Resilience**: Distributed storage across multiple nodes (when available)
|
|
|
|
Command Line Options
|
|
--------------------
|
|
|
|
The Kademlia demo supports several command line options for customization:
|
|
|
|
.. code-block:: console
|
|
|
|
$ python kademlia.py --help
|
|
usage: kademlia.py [-h] [--mode MODE] [--port PORT] [--bootstrap [BOOTSTRAP ...]] [--verbose]
|
|
|
|
Kademlia DHT example with content server functionality
|
|
|
|
options:
|
|
-h, --help show this help message and exit
|
|
--mode MODE Run as a server or client node (default: server)
|
|
--port PORT Port to listen on (0 for random) (default: 0)
|
|
--bootstrap [BOOTSTRAP ...]
|
|
Multiaddrs of bootstrap nodes. Provide a space-separated list of addresses.
|
|
This is required for client mode.
|
|
--verbose Enable verbose logging
|
|
|
|
**Examples:**
|
|
|
|
Start server on a specific port:
|
|
|
|
.. code-block:: console
|
|
|
|
$ python kademlia.py --mode server --port 8000
|
|
|
|
Start client with verbose logging:
|
|
|
|
.. code-block:: console
|
|
|
|
$ python kademlia.py --mode client --verbose
|
|
|
|
Connect to multiple bootstrap nodes:
|
|
|
|
.. code-block:: console
|
|
|
|
$ python kademlia.py --mode client --bootstrap /ip4/127.0.0.1/tcp/8000/p2p/... /ip4/127.0.0.1/tcp/8001/p2p/...
|
|
|
|
How It Works
|
|
------------
|
|
|
|
The Kademlia DHT implementation demonstrates several key concepts:
|
|
|
|
**Server Mode:**
|
|
- Stores key-value pairs in the distributed hash table
|
|
- Advertises itself as a content provider for specific content
|
|
- Handles incoming DHT requests from other nodes
|
|
- Maintains routing table with known peers
|
|
|
|
**Client Mode:**
|
|
- Connects to bootstrap nodes to join the network
|
|
- Retrieves values by their keys from the DHT
|
|
- Discovers content providers for specific content
|
|
- Performs network lookups using the Kademlia algorithm
|
|
|
|
**Key Components:**
|
|
- **Routing Table**: Organizes peers in k-buckets based on XOR distance
|
|
- **Value Store**: Manages key-value storage with TTL (time-to-live)
|
|
- **Provider Store**: Tracks which peers provide specific content
|
|
- **Peer Routing**: Implements iterative lookups to find closest peers
|
|
|
|
The full source code for this example is below:
|
|
|
|
.. literalinclude:: ../examples/kademlia/kademlia.py
|
|
:language: python
|
|
:linenos:
|