d61bca78ab
* initialise the module * added content routing * added routing module * added peer routing * added value store * added utilities functions * added main kademlia file * fixed create_key_from_binary function * example to test kademlia dht * added protocol ID and enhanced logging for peer store size in provider and consumer nodes * refactor: specify stream type in handle_stream method and add peer in routing table * removed content routing * added default value of count for finding closest peers * added functions to find close peers * refactor: remove content routing and enhance peer discovery * added put value function * added get value function * fix: improve logging and handle key encoding in get_value method * refactor: remove ContentRouting import from __init__.py * refactor: improved basic kademlia example * added protobuf files * replaced json with protobuf * refactor: enhance peer discovery and routing logic in KadDHT * refactor: enhance Kademlia routing table to use PeerInfo objects and improve peer management * refactor: enhance peer addition logic to utilize PeerInfo objects in routing table * feat: implement content provider functionality in Kademlia DHT * refactor: update value store to use datetime for validity management * refactor: update RoutingTable initialization to include host reference * refactor: enhance KBucket and RoutingTable for improved peer management and functionality * refactor: streamline peer discovery and value storage methods in KadDHT * refactor: update KadDHT and related classes for async peer management and enhanced value storage * refactor: enhance ProviderStore initialization and improve peer routing integration * test: add tests for Kademlia DHT functionality * fix linting issues * pydocstyle issues fixed * CICD pipeline issues solved * fix: update docstring format for find_peer method * refactor: improve logging and remove unused code in DHT implementation * refactor: clean up logging and remove unused imports in DHT and test files * Refactor logging setup and improve DHT stream handling with varint length prefixes * Update bootstrap peer handling in basic_dht example and refactor peer routing to accept string addresses * Enhance peer querying in Kademlia DHT by implementing parallel queries using Trio. * Enhance peer querying by adding deduplication checks * Refactor DHT implementation to use varint for length prefixes and enhance logging for better traceability * Add base58 encoding for value storage and enhance logging in basic_dht example * Refactor Kademlia DHT to support server/client modes * Added unit tests * Refactor documentation to fixsome warning * Add unit tests and remove outdated tests * Fixed precommit errora * Refactor error handling test to raise StringParseError for invalid bootstrap addresses * Add libp2p.kad_dht to the list of subpackages in documentation * Fix expiration and republish checks to use inclusive comparison * Add __init__.py file to libp2p.kad_dht.pb package * Refactor get value and put value to run in parallel with query timeout * Refactor provider message handling to use parallel processing with timeout * Add methods for provider store in KadDHT class * Refactor KadDHT and ProviderStore methods to improve type hints and enhance parallel processing * Add documentation for libp2p.kad_dht.pb module. * Update documentation for libp2p.kad_dht package to include subpackages and correct formatting * Fix formatting in documentation for libp2p.kad_dht package by correcting the subpackage reference * Fix header formatting in libp2p.kad_dht.pb documentation * Change log level from info to debug for various logging statements. * fix CICD issues (post revamp) * fixed value store unit test * Refactored kademlia example * Refactor Kademlia example: enhance logging, improve bootstrap node connection, and streamline server address handling * removed bootstrap module * Refactor Kademlia DHT example and core modules: enhance logging, remove unused code, and improve peer handling * Added docs of kad dht example * Update server address log file path to use the script's directory * Refactor: Introduce DHTMode enum for clearer mode management * moved xor_distance function to utils.py * Enhance logging in ValueStore and KadDHT: include decoded value in debug logs and update parameter description for validity * Add handling for closest peers in GET_VALUE response when value is not found * Handled failure scenario for PUT_VALUE * Remove kademlia demo from project scripts and contributing documentation * spelling and logging --------- Co-authored-by: pacrob <5199899+pacrob@users.noreply.github.com>
py-libp2p
The Python implementation of the libp2p networking stack.
⚠️ Warning: py-libp2p is an experimental and work-in-progress repo under development. We do not yet recommend using py-libp2p in production environments.
Read more in the documentation on ReadTheDocs. View the release notes.
Maintainers
Currently maintained by @pacrob, @seetadev and @dhuseby, looking for assistance!
Feature Breakdown
py-libp2p aims for conformity with the standard libp2p modules. Below is a breakdown of the modules we have developed, are developing, and may develop in the future.
Legend: ✅: Done 🛠️: In Progress/Usable 🌱 Prototype/Unstable ❌: Missing
Transports
| Transport | Status | Source |
|---|---|---|
libp2p-tcp |
✅ | source |
libp2p-quic |
🌱 | |
libp2p-websocket |
❌ | |
libp2p-webrtc-browser-to-server |
❌ | |
libp2p-webrtc-private-to-private |
❌ |
NAT Traversal
| NAT Traversal | Status |
|---|---|
libp2p-circuit-relay-v2 |
❌ |
libp2p-autonat |
❌ |
libp2p-hole-punching |
❌ |
Secure Communication
| Secure Communication | Status | Source |
|---|---|---|
libp2p-noise |
🌱 | source |
libp2p-tls |
❌ |
Discovery
| Discovery | Status |
|---|---|
bootstrap |
❌ |
random-walk |
❌ |
mdns-discovery |
❌ |
rendezvous |
❌ |
Peer Routing
| Peer Routing | Status |
|---|---|
libp2p-kad-dht |
❌ |
Publish/Subscribe
| Publish/Subscribe | Status | Source |
|---|---|---|
libp2p-floodsub |
✅ | source |
libp2p-gossipsub |
✅ | source |
Stream Muxers
| Stream Muxers | Status | Status |
|---|---|---|
libp2p-yamux |
🌱 | |
libp2p-mplex |
🛠️ | source |
Storage
| Storage | Status |
|---|---|
libp2p-record |
❌ |
General Purpose Utilities & Datatypes
| Utility/Datatype | Status | Source |
|---|---|---|
libp2p-ping |
✅ | source |
libp2p-peer |
✅ | source |
libp2p-identify |
✅ | source |
Explanation of Basic Two Node Communication
Core Concepts
(non-normative, useful for team notes, not a reference)
Several components of the libp2p stack take part when establishing a connection between two nodes:
- Host: a node in the libp2p network.
- Connection: the layer 3 connection between two nodes in a libp2p network.
- Transport: the component that creates a Connection, e.g. TCP, UDP, QUIC, etc.
- Streams: an abstraction on top of a Connection representing parallel conversations about different matters, each of which is identified by a protocol ID. Multiple streams are layered on top of a Connection via the Multiplexer.
- Multiplexer: a component that is responsible for wrapping messages sent on a stream with an envelope that identifies the stream they pertain to, normally via an ID. The multiplexer on the other unwraps the message and routes it internally based on the stream identification.
- Secure channel: optionally establishes a secure, encrypted, and authenticated channel over the Connection.
- Upgrader: a component that takes a raw layer 3 connection returned by the Transport, and performs the security and multiplexing negotiation to set up a secure, multiplexed channel on top of which Streams can be opened.
Communication between two hosts X and Y
(non-normative, useful for team notes, not a reference)
Initiate the connection: A host is simply a node in the libp2p network that is able to communicate with other nodes in the network. In order for X and Y to communicate with one another, one of the hosts must initiate the connection. Let's say that X is going to initiate the connection. X will first open a connection to Y. This connection is where all of the actual communication will take place.
Communication over one connection with multiple protocols: X and Y can communicate over the same connection using different protocols and the multiplexer will appropriately route messages for a given protocol to a particular handler function for that protocol, which allows for each host to handle different protocols with separate functions. Furthermore, we can use multiple streams for a given protocol that allow for the same protocol and same underlying connection to be used for communication about separate topics between nodes X and Y.
Why use multiple streams?: The purpose of using the same connection for multiple streams to communicate over is to avoid the overhead of having multiple connections between X and Y. In order for X and Y to differentiate between messages on different streams and different protocols, a multiplexer is used to encode the messages when a message will be sent and decode a message when a message is received. The multiplexer encodes the message by adding a header to the beginning of any message to be sent that contains the stream id (along with some other info). Then, the message is sent across the raw connection and the receiving host will use its multiplexer to decode the message, i.e. determine which stream id the message should be routed to.